Google and Inktomi spiders were indexing my forum pages and using lots of bandwidth, so I included a robots.txt file in my root directory:
User-agent: *
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/editpost.php
Disallow: /forum/member.php
Disallow: /forum/member2.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/private2.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/admincp/
Disallow: /forum/modcp/
Disallow: /forum/images/
Disallow: /forum/sendmessage.php
Disallow: /forum/register.php
Disallow: /forum/subscription.php
I've noticed over the past few days that I'm still seeing Inktomi spiders, but I haven't seen even ONE Google spider (I used to have tons of them every day).
Did I do something wrong in my robots.txt file? I just want to disallow stuff that doesn't matter to search engines, not the content.
User-agent: *
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/editpost.php
Disallow: /forum/member.php
Disallow: /forum/member2.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/private2.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/admincp/
Disallow: /forum/modcp/
Disallow: /forum/images/
Disallow: /forum/sendmessage.php
Disallow: /forum/register.php
Disallow: /forum/subscription.php
I've noticed over the past few days that I'm still seeing Inktomi spiders, but I haven't seen even ONE Google spider (I used to have tons of them every day).
Did I do something wrong in my robots.txt file? I just want to disallow stuff that doesn't matter to search engines, not the content.
Comment