And a right pain it is too. Last night I had 35 guests online and five members. Strange I thought and did a quick who's online. Well the 35 guests were inktomisearch spiders see image below. Seems it ignores the robots.txt file too. My members quote it being on my site for over 5 hours, and I just noted that these forums are also being hit right now by the sod.
The inktomisearch spider thread
Collapse
X
-
I think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.Comment
-
She (?) indexes your site for the inktomi based search engines (http://www.inktomi.com/customers/). Inktomi was purchased by Yahoo! earlyer this year, I (along with others), expect yahoo to move from google to Ink by the end of this year. You really do not want to block this spider from your sites!
My personal spider is this one:Comment
-
Originally posted by JakemanI think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.HTML Code:<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Comment
-
Search Engine Spiders...
Whats been everyones experience with them?? I have had the Inktomi spider on my side non stop for the last 2 days, it just doesnt give up. When ever a new thread is made its like its starts over again and goes through them all. Very wierd, so far the highest number of spiders has been 14 at one time. Just curious what everyone else has seen on their boards...Comment
-
-
It was on my site for two weeks, and my forums don't allow guests to access them. It must like "no permission" pages. It's a retarded spider.
Comment
widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
Comment