[Forum] What is robots.txt best practices in VB 4.0 ?

[cheefoo]

Hi,

May I know what is the best practices to place inside robots.txt to suit VB 4.0 ?
Are these enough for the new features such as CMS and Blog ? Please kindly advice.

Disallow: /forums/admincp/
Disallow: /forums/announcement.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/image.php
Disallow: /forums/images/
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/attachment.php
Disallow: /forums/payments.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/subscriptions.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php

[Loco.M]

This is the list I used for 3.8

http://forum.vbulletinsetup.com/f7/vbulletin-seo-optimization-tips-2.html

it should work for 4.0 as well

[cheefoo]

Hi dude,

Thanks! It's good info out there! How about blogs ? The crawling for blogs seems to take ages.

Happy New Year !!

[Loco.M]

what kind of blog?
the vbulletin blog, wordpress or some other script?

[cheefoo]

Hi,

It's VB 4.0 blogs. I am using GSiteCrawler, and it's taking a long time to crawl over similiar links like
www.mydomainxxxx.com/forums/blog.php/?do=list&m=11&y=1990...