Tweet
can any one tell me what should be in this file and what it does and where it goes c panel shows ip's have been looking for it
-
-
The robots.txt file is a file that search index spiders read before crawling your site to index the content. You can tell the spiders to stay away from your site, or certain files or directories. So they don't get included in search engines.
Personally I do not believe in this really because there are enough search engines that ignore the robots.txt file.
You place it in the public_html/ root folder. It is the first file on the web site, which applies to all files/folders below it.
-
Is it possible to have a robot.txt capability to tell bots where to crawl the site and for SEO in the cloud platform? I know there were tickets in when I searched from 2016.
If there is,...
Comment