I wouldn't block image.php, or faq.php, or announcement.php. But thats me.
Perfect VBulletin 4 robots.txt
Collapse
X
-
-
This thread is a really good idea OP, thanks.
It would be great if vB had a built in robots.txt generator in the settings menu, it would be very straightforward to code and could use forum variables to automatically generate lists with the correct ./path/to/files - then the user would just need to copy paste the generated text.
Given that member profiles are headed in more of a social-media direction, i'm not sure I'd want to block the search engines from that - but it probably depends on the site. If people have filled out a profile with reasonable content then definately don't block it as its just adding to your backlinks.👍 1Comment
-
tags.php
I would block tags.php ..if you have many many tags..i think it disrupts the search engine listings.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Visit India's technology discussion forum at TechCentral.inComment
-
I would not recommend using this list. For a start the robots.txt file is exactly that 'a text file', anyone can read. You have told everyone where your admincp and modcp directories are located. I, for one rename mine - so no one knows the path to either of them. I certainly would not broadcast where my admincp or modcp directories are, some people even password protect these directories just to be that bit safer.
My robots.txt
Code:User-agent: * Disallow: /admincp/ Disallow: /arcade/ Disallow: /archive/ Disallow: /cgi-bin/ Disallow: /clientscript/ Disallow: /cpstyles/ Disallow: /customavatars/ Disallow: /customgroupicons/ Disallow: /customprofilepics/ Disallow: /images/ Disallow: /includes/ Disallow: /modcp/ Disallow: /packages/ Disallow: /signaturepics/ Disallow: /vb/ Disallow: /ajax.php Disallow: /album.php Disallow: /announcement.php Disallow: /asset.php Disallow: /assetmanage.php Disallow: /attachment.php Disallow: /attachment_inlinemod.php Disallow: /blog_ajax.php Disallow: /blog_external.php Disallow: /blog_inlinemod.php Disallow: /blog_report.php Disallow: /blog_search.php Disallow: /blog_subscription.php Disallow: /blog_usercp.php Disallow: /calendar.php Disallow: /clear.gif Disallow: /converse.php Disallow: /cron.php Disallow: /editor.php Disallow: /editpost.php Disallow: /external.php Disallow: /faq.php Disallow: /global.php Disallow: /group_inlinemod.php Disallow: /groupsubscription.php Disallow: /image.php Disallow: /infraction.php Disallow: /inlinemod.php Disallow: /joinrequests.php Disallow: /list.php Disallow: /login.php Disallow: /member.php Disallow: /memberlist.php Disallow: /misc.php Disallow: /moderation.php Disallow: /moderator.php Disallow: /newattachment.php Disallow: /newreply.php Disallow: /newthread.php Disallow: /online.php Disallow: /payment_gateway.php Disallow: /payments.php Disallow: /picture.php Disallow: /picture_inlinemod.php Disallow: /picturecomment.php Disallow: /posthistory.php Disallow: /postings.php Disallow: /printthread.php Disallow: /private.php Disallow: /profile.php Disallow: /register.php Disallow: /report.php Disallow: /reputation.php Disallow: /search.php Disallow: /sendmessage.php Disallow: /sendtofriend.php Disallow: /showgroups.php Disallow: /showpost.php Disallow: /subscription.php Disallow: /threadrate.php Disallow: /threadtag.php Disallow: /usercp.php Disallow: /usernote.php Disallow: /visitormessage.php Disallow: /widget.php
Comment
-
Comment
-
I have /forum/search.php in my robots.txt file and Googld adsense tells me it cant access the file during crawls. I guess it works but I also noticed that 95% of the time when my forum is using the search.php file(like looking for new posts, etc) no relevant ads will be shown, only PSAs or irrelevant ads. Wondering if blocking the google crawler hurts relevant ad display during search.vBulletin 3.8.2Comment
-
There isn't one, develop your own based on your needs.Jake Crause
www.homebuiltairplanes.comComment
-
Saw your website in your sig line. I just sold a Pitts S1-SS project to fund my current biz. I'll check out your site today.
BartComment
-
It covers every aspect for a successful VBulletin forum. It has all the necessary VB4 files disallowed and combats against high bandwidth bots.
Building accurate robots.txt files is my profession, so I thought I would give something back to the VBulletin community for all the help I've received in the past.
Here is mine if you want to compare, enjoy!Comment
-
Comment
-
any one know how to restrict yahoo paid spider?
everytime we set a PPC they send a spider to every term at once 1000+ in one go!!Comment
Related Topics
Collapse
-
by Maik StorckHello,
I think it would be very nice if a standard robots.txt file could be included with vbulletin.
In this file, the basic rules should be included to prohibit all non-essential links...-
Channel: vBulletin 5 Connect Feedback
-
Comment