Robot.txt (what to include)

[MRGTB]

Can some experts here please post some paths that should be included in your "robot.txt" file on a clean vBulletin install (no add-ons installed).

Which should exclude robots from having access to certain files and directorys, that will NOT have a negative effect on you being indexed. If anything, it will help you get better indexed, because search bots will not be wasting there time looking at directorys and files that are off no importance to them what-so-ever. I don't want to add paths that I "THINK" should exlude bots, I'd rather be 100% sure there right too add. So I'm asking the experts here what should be included in the robot.txt file.

For example: Should the "admincp" , "modcp" , "install" and "images" directories be added to exclude bots from looking at files in them?

[RedTyger]

Mine:
User-agent: *
Disallow: /attachment.php
Disallow: /image.php
Disallow: /editpost.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /sendmessage.php
Disallow: /subscription.php
Disallow: /showpost.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /admin/
Disallow: /mod/
Disallow: /images/
User-agent: Googlebot-Image
Disallow: /

Most of those robots should not be finding anyway, but there are some like showpost I've denied to deal with duplicate content.

[Milado]

User-agent: *
# Disallow vBulletin Directories
Disallow:/forum/archive/
Disallow:/forum/clientscript/
Disallow:/forum/cpstyles/
Disallow:/forum/customavatars/
Disallow:/forum/customsignatures/
Disallow:/forum/images/
Disallow:/forum/includes/
# Disallow vBulletin Files
Disallow:/forum/ajax.php
Disallow:/forum/announcement.php
Disallow:/forum/attachment.php
Disallow:/forum/calendar.php
Disallow:/forum/clear.gif
Disallow:/forum/cron.php
Disallow:/forum/editpost.php
Disallow:/forum/global.php
Disallow:/forum/image.php
Disallow:/forum/inlinemod.php
Disallow:/forum/joinrequests.php
Disallow:/forum/login.php
Disallow:/forum/member.php
Disallow:/forum/memberlist.php
Disallow:/forum/misc.php
Disallow:/forum/moderator.php
Disallow:/forum/newattachment.php
Disallow:/forum/newreply.php
Disallow:/forum/newthread.php
Disallow:/forum/online.php
Disallow:/forum/payment_gateway.php
Disallow:/forum/payments.php
Disallow:/forum/poll.php
Disallow:/forum/postings.php
Disallow:/forum/printthread.php
Disallow:/forum/private.php
Disallow:/forum/profile.php
Disallow:/forum/register.php
Disallow:/forum/report.php
Disallow:/forum/reputation.php
Disallow:/forum/search.php
Disallow:/forum/sendmessage.php
Disallow:/forum/showgroups.php
Disallow:/forum/showpost.php
Disallow:/forum/subscription.php
Disallow:/forum/threadrate.php
Disallow:/forum/usercp.php
Disallow:/forum/usernote.php

I disallow the archive pages.

[Milado]

I allow index, showthread and forumdisplay.

[RedTyger]

I allow index, showthread and forumdisplay.

Hah, wish I'd thought to do it that way. Might want to allow the FAQ and any other text-heavy pages as well. Announcements?

[MRGTB]

Should you include these:

cron.php
online.php
image.php
ajax.php
login.php
inlinemod.php

I've noticed in awats these files get hit quite a lot

Why does nobody so far not also disallow the install directory.

PS: Milado

There should be a SPACE between the dots ":" and "/"

Yours looks like this below:
Disallow:/forum/online.php

Should look like this:
Disallow: /forum/online.php

[---MAD---]

After you get a solid list, can you post your list for future referances ?

[MRGTB]

It's hard to get a solid list, because you always find that some will include directory's and files, that other won't.

I would have much preferred to have seen a staff member here like Wayne, go through the files and post a list of all directory's and files that should be included in a robot text after a clean install of vBulletin.

But not include the archive and printthread pages though for exclusion. Just leave that for people who choose to add them as an extra to the list for better seo.

It's just impossible, because each person does it differently, as you can see above. And you just don't know who's right and who's wrong.

[Milado]

Hah, wish I'd thought to do it that way. Might want to allow the FAQ and any other text-heavy pages as well. Announcements?

Actually, FAQs are not related to my site content.

Announcement include laws & some advises for members.

It's up to you what you want crawlers to index

[Milado]

PS: Milado

There should be a SPACE between the dots ":" and "/"

Yours looks like this below:
Disallow:/forum/online.php

Should look like this:
Disallow: /forum/online.php

It works for me, and it's OK with google webmasters.
Is there any reference for your correction?

Thanks in advance.

[Milado]

For the install directory, I never leave it exist, I delete it immediately after having the upgrading process done.

[MRGTB]

It works for me, and it's OK with google webmasters.
Is there any reference for your correction?

Thanks in advance.

Well here is what I've put together so far, I have not added the "Archive Pages" because I have Google Adsense banners advertised in them, but I have added the Printthread pages for exclusion. Otherwise I would have added both!

There are one or two files that I've not added which I'm not too sure about, I've not added the "includes" directory because again I'm not sure if bots need to access some files there to use other files not listed in my robot.txt like the search.php file. I've not added the "search.php" because I allow Guests too search the site to get content indexed better (I've seen bots using the search in the past). Also I don't allow Guests to view Members Profiles, so that was added as well as I don't want members profiles indexed in Google etc.

Also note: I have vBulletin installed in the ROOT itself, after Wayne said it's better to do that for indexing (not as many sub directorys then for bots to browse through), I don't use a "forum folder" if you wondering about my paths below not listing a forum directory in them

User-agent: Slurp
Crawl-delay: 10
 
User-agent: *
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /install/
Disallow: /modcp/
Disallow: /signaturepics/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /clear.gif
Disallow: /cron.php
[COLOR=red]Disallow: /google_adsense_script.htm (extra file I use for adsense)[/COLOR]
Disallow: /image.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

PS: I delete the install.php file, but not the install directory, as your only told after you install vBulletin to remove that file only, and not the whole install directory itself.