Hi,
I thought it would be a good idea to create a thread showing what the ideal VBulletin robots.txt file should look like.
At the moment, this file is nowhere near perfect, but it will be after a bit of input from you guys.
Rhys
I thought it would be a good idea to create a thread showing what the ideal VBulletin robots.txt file should look like.
Code:
Sitemap: Sitemap URL here User-agent: BoardTracker Disallow: / User-agent: Gigabot Disallow: / User-agent: Twiceler Disallow: / User-agent: Slurp Crawl-delay: 2 User-agent: msnbot Crawl-delay: 2 User-agent: * Disallow: /editpost.php Disallow: /gamercard.php Disallow: /inlinemod.php Disallow: /member.php Disallow: /memberlist.php Disallow: /newreply.php Disallow: /newthread.php Disallow: /payments.php Disallow: /printthread.php Disallow: /private.php Disallow: /profile.php Disallow: /report.php Disallow: /search.php Disallow: /sendmessage.php Disallow: /showpost.php Disallow: /usercp.php Disallow: /usernote.php User-agent: Mediapartners-Google Allow: /member.php Allow: /private.php Allow: /usercp.php
At the moment, this file is nowhere near perfect, but it will be after a bit of input from you guys.
Rhys
Comment