Robots.txt for VB 5
Collapse
X
-
Tags: None
-
Not something we supply, you will need to create your own. -
The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.
Translations provided by Google.
Wayne Luke
The Rabid Badger - a vBulletin Cloud demonstration site.
vBulletin 5 APIComment
-
Alsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.👍 1Comment
-
I attach a file robots.txt to which you haveComment
-
I want the file that you haveComment
-
Here is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!Comment
-
It is just a blank file with only one entry anyone attached....
Attached FilesComment
-
Here is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!Comment
-
Comment
-
Here is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Comment
-
i have exclude many (bad) bots - not with robot.txt - i do this over the .htaccess
-->> http://badbots.vps.tips/
-
Related Topics
Collapse
-
by panoramiaDear Sirs,
I am trying to submit vBulletin 5 xml sitemap to Google. I am a bit confused with the instructions in vBulletin backend. It says that it will automatically submit sitemap to Google...2 Photos-
Channel: Support Issues & Questions
-
-
by dtwsolutionsMy client's VB forum is running the latest version, but we can not get the XML sitemap to be loaded in Google. It says "Sitemap index has errors" then has:
This required tag is...-
Channel: Support Issues & Questions
-
-
by HIZIRBEYafter the creating my website it looks like that. what do I need to do to fix this ?1 Photo
-
Channel: vBulletin 5 Installs & Upgrades
-
Comment