robots.txt does not work

[joesdomain]

I created and added the robots.txt listing all files/directories to ignore. Well, I still get the msnbots/ yahoo slurp etc still visiting my site. I also banned their ip address and they still seem to say they are creating event. In whos online, i do see a stop sign icon.

My website is a regional one and i do not want people worldwide registering

Joe

[David Grove]

See www.robotstxt.org, and make sure you implemented your robots.txt file correctly. I know the Googlebot only requests robots.txt once every 24 hours, so you have to let that time pass before it will obey the rules in robots.txt. Also, try validating your robots.txt file.

[Wayne Luke]

Robots.txt is not a vBulletin feature. It is an optional protocol that bots can choose to follow. We cannot make it work for you. I do know it needs to be in the root directory of your site, not the forum directory. A Webmaster related site like SitePoint.com could probably provide more assistance in their forums.

If you see a stop sign, then they are "seeing" a no permission screen.

[joesdomain]

the robots.txt validator states its correct.

when i ftp into my site, the following directories are listed:

.cp
.session_data
.tmp
logs
public_html

my VBulletin is installed in the public_html directory as is the robots.txt
Is this not correct?

Joe

[David Grove]

As Wayne states, this has nothing to do with vBulletin, per se. What is your URL?

[joesdomain]

http://www.northernbuysell.com

[David Grove]

Looks like you're good to go. Remember not all robots obey it, and others only read the file periodically.

[joesdomain]

ok thanks!

Joe