robots.txt

[Lissa]

Google and Inktomi spiders were indexing my forum pages and using lots of bandwidth, so I included a robots.txt file in my root directory:

User-agent: *
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/editpost.php
Disallow: /forum/member.php
Disallow: /forum/member2.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/private2.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/admincp/
Disallow: /forum/modcp/
Disallow: /forum/images/
Disallow: /forum/sendmessage.php
Disallow: /forum/register.php
Disallow: /forum/subscription.php

I've noticed over the past few days that I'm still seeing Inktomi spiders, but I haven't seen even ONE Google spider (I used to have tons of them every day).

Did I do something wrong in my robots.txt file? I just want to disallow stuff that doesn't matter to search engines, not the content.

[Scott MacVicar]

They cache the robots.txt so it might take a while for them to realise that you dont want any of the traffic.

[Reverend]

Why have you disallowed register.php

Having your register page indexed could be beneficial.IMHO.

[Lissa]

Actually, I just took a list from someone else on this forum, I wasn't really sure what to allow and what not to allow.


I just didn't want them spidering what they didn't need to, I still want the search engines to spider the forum.

[TeenForums]

Ive never used these robot.txt files before, do i jsut creat a file in my root directory called robots.txt and paste that code in???

I must seem like a dumbass!!!

[cmeinck]

I've always been afraid to modify the robots.txt in fear that I would either send away the robot or disallow a wrong directory.

[TeenForums]

Well ive made it now and there are like 3 bots online at once compared to the 1 google bot i usually had, odd!

[Triple Ox]

(Currently running patched 3.0.6)

I just did a search on this forum for "bad google indexing" and ended up at this thread.

I've just run a site:mysite.co.uk check on Google. It turns out that i have 45,000 pages indexed, which some might say is good, but a lot of the search results didn't even include a description. Looking like they're penalised.

The majority of the the indexed pages are newreply.php, showmessage.php and showthread.php. Would it be worth adding these to a robots.txt file? Surely they're worthless from a seach engine point of view.

What does the vBulletin team suggest?

[noppid]

Let the bot decide what to index IMO. A listing is a listing. Put internal link backs to your site on those pages or in the footer like the footer navbar links so the bots at least get a link to follow back in.

[Triple Ox]

Good point, noppid.

[Macgyver1978]

Call me captain overkill, but if I don't want search engines to find my site, couldn't I just disallow everything? I just did, we'll find out.. I'm pushing 50 gigs this month, 98% are .php files, lol..

[gregsbaby64]

I put a robots.txt in the forum directory as well as the root a few days ago and now Google is hunting or trying to view the admin and other members profiles. Can they see anything if I have it closed to the public?

The little stop sign is there which means: Viewing 'No Permission' Message but why do they still come around? Unless you are registered AND validated then added by the admin (me) you cannot see anything. I have tested it and tested it. Google and Ask Jeeves were on earlier today.

Thanks!

[Steve Machol]

The search engines will only be able to view and index what you allow Guests to view.

[gregsbaby64]

Ok, thanks Steve. Just kinda freaked me out a little when they were viewing members and it showed names.