Google Bots Vs. Yahoo Bots

[dkon26]

What is the deal with google bots and yahoo bots?

Let me explain. Three days ago I noticed a google bot on my board. It has not gone away since but it has brought a friend. Now I have two google bots that seem to love the calendar and registration page.

This morning, yahoo has started with not one, not two, not three but FOUR yahoo bots that keep coming and going and then come at all the same time. I now have 6 bots that will not go away. Google never goes away and then yahoo will send 4 at a time.

So my question is, is there any trick to get them to stop coming in droves? Is it normal to have so many on at the same time?

P.S. I never submitted the site to them. They shouldnt know about it yet!

[Wayne Luke]

That is nothing. Wait until you get 300-400 of them at a time.

Bots are actually a lot like viral processes. As they hit links they want to follow, the spawn additional copies of themselves to check it out. As time goes on, you get more and more copies on your site building a web of links around the site.

You need to look into robots.txt specifications to control and maintain them. Both Google and Yahoo also have recomendations on their sites about keeping the bots in check.

[dkon26]

Thanks Wayne, I guess I never paid close enough attention to them before. I'll look into the .txt and see what google and yahoo have to say. Not that its a big deal but I imagine it takes away resources having them look at the same pages a million times a day. Im only 3 days into this so I can imagine what it'll be like in a month.

[---MAD---]

That is nothing. Wait until you get 300-400 of them at a time.

Bots are actually a lot like viral processes. As they hit links they want to follow, the spawn additional copies of themselves to check it out. As time goes on, you get more and more copies on your site building a web of links around the site.

You need to look into robots.txt specifications to control and maintain them. Both Google and Yahoo also have recomendations on their sites about keeping the bots in check.

How exactly do they work? As in how do the run around knowing which sites / links to go to and keeping them up etc?

[dkon26]

How exactly do they work? As in how do the run around knowing which sites / links to go to and keeping them up etc?

Their crazy. I have had almost 10 in the last hour all with different IP addresses. They seem to come around the same time. Maybe they want to talk to each other on the boards? They keep looking at the same things and then one will go through the board. Makes no sense to me. Artificial intelligence is great!

[Wayne Luke]

How exactly do they work? As in how do the run around knowing which sites / links to go to and keeping them up etc?

Read and enjoy:

http://www-128.ibm.com/developerworks/linux/library/l-spider/?ca=dgr-lnxw01WebSpiderLinux

I am sure that Googlebot and Yahoo! Slurp are more advanced than the ones listed in the article but it gives a basic overview.

[Shazz™]

yahoo bots seem to lag more =\

[jeffinj]

Wayne, do you have a ready made robot.txt for forums? I think they are slowing my board down. If I restrict them, would it harm my ranking?

[Joe Gronlund]

I have had 189 today so far..

[MRGTB]

maybe we should create a vbulletin bot, to kick yahoo slurp off our sites when he comes. kind of like a forum bodyguard

[dkon26]

Its just sad having more bots than actual people. =(

[MRGTB]

One thing that I don't like about Yahoo bots is, they comes in droves and steal tons of bandwidth. But when you look at your Stats to see which search engine hits come from. I don't know about you. But very little comes from Yahoo, even though they steal most of the bandwidth above all other bots, except Google maybe (but then it's a very close second). And I'm getting a ratio of like 40 Google to 1 Yahoo hit

[dkon26]

One thing that I don't like about Yahoo bots is, they comes in droves and steal tons of bandwidth. But when you look at your Stats to see which search engine hits come from. I don't know about you. But very little comes from Yahoo, even though they steal most of the bandwidth above all other bots, except Google maybe (but then it's a very close second). And I'm getting a ratio of like 40 Google to 1 Yahoo hit

I had 9 yahoo bots on before with 2 google bots. 7 of the yahoo bots were in the same place and then 2 of them were actually crawling. Again, this is only day 3 the site has been running but I think I figured out where they came from. I think they followed my sig here because google and yahoo both have some of my posts here listed. Im sure a lot of other engines will soon follow. At least it looks like people are there so new people will sign up! Not sure how long i'll be happy about them though. One of the google bots has been trying to sign up since it loves the registration page.

When it gets into the hundreds like wayne said, I'll use those ways to keep em check.

[MRGTB]

Yeah you'll get a lot trying to sign up.

[MRGTB]

What you better doing now is creating a robot.txt file to stop them browsing certain files and directory's now. Like the CGI-BIN folder and register.php file etc. This will help send them to the right places you want them on your site to get things indexed better and faster, so there not wasting there time looking at files that can't be indexed and are of no use to them. Plus it will cut down on bandwidth also.

Basically you control where they should look and where they shouldn't