[Forum] Mod Rewrite Friendly URL's & robots.txt

[too_cool_3]

I just made my own robots.txt file from a combination of Wayne Luke (link) from vbulletin.com, Brandon Sheley (link) from vbulletinsetup.com, and Brian Cummiskey(link) from vbseo.com

I'm using vB 4.1.7 installed in my root with Automatic Sitemap Generation, my Sitemap File Path stated in AdminCP is './store_sitemap' however, when I click to view my sitemap URL in AdminCP it shows me this link: http://tbmotoring.com/xmlsitemap.php

1.) Is http://tbmotoring.com/xmlsitemap.php the correct sitemap link to put in my robots.txt file? And do I need to add '.gz' to the end (http://tbmotoring.com/xmlsitemap.php.gz)?

2.) I have Mod Rewrite Friendly URL's enabled, does that mean I need to remove the all the '.php' extensions in my robots.txt? (ie. instead of Disallow: /ajax.php should I put Disallow: /ajax)

User-agent: ia_archiver
Allow: /
 
User-agent: Baiduspider
Disallow: /
 
User-agent: BoardTracker
Disallow: /
 
User-agent: Gigabot
Disallow: /
 
User-agent: Twiceler
Disallow: /
 
User-agent: Slurp
Crawl-delay: 2
 
User-agent: msnbot
Crawl-delay: 2
 
User-agent: *
Disallow: *.js
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /cron.php
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /editpost.php
Disallow: /global.php
Disallow: /includes/
Disallow: /inlinemod.php
Disallow: /install/
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /post.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
 
Sitemap: http://tbmotoring.com/xmlsitemap.php 

Feel free to give me your feedback on my robots.txt file - if you think anything needs to be added/removed/modified ?

Thank you,
-Marc

[Wayne Luke]

1) You shouldn't like Robots.txt anywhere. Just put it in your root directory.

2) Shouldn't matter. The Friendly URLs only redirect things you want to be indexed anyway.

[too_cool_3]

Pardon me Wayne, I don't understand what you mean by your response for 1)

[TheNewOne]

sitemap does not have php at the end put what u see

[Wayne Luke]

Pardon me Wayne, I don't understand what you mean by your response for 1)

Robots.txt has nothing to do with any kind of sitemap.

Robots.txt is not linked from anywhere in vBulletin. It is just a file that bots will request automatically. Just put the file in your root directory and forget about it.

I don't know what the response above me is about because the xmlsitemap.php file does indeed have a php extension.

[too_cool_3]

To clarify, Wayne, I read in other threads that you should put a link to your sitemap in your robots.txt file, is that proper procedure and is this: http://tbmotoring.com/xmlsitemap.php the proper link to put in my robots.txt?

Previously, what I meant by removing .php extension is that if I am using mod rewrite friendly url's do I leave .php at the end of each path in robots.txt ( ie. instead of Disallow: /ajax.php should I put Disallow: /ajax)?

Thanks for your patience.

[Wayne Luke]

Don't see anything about a sitemap here: http://www.robotstxt.org/

Leave the .php.

[too_cool_3]

Thank you for the helpful link, I just finished reading the FAQ at http://www.robotstxt.org lot of good information. The only reason I put my sitemap in the robots.txt file is from seeing it done by Brian Cummiskey over at vbseo in this thread: http://www.vbseo.com/f2/tutorial-how...70/#post253465

Thanks again Wayne for all the info!