More Spiders / Indexers / Archives for vB3 - list

[Floris]

Latest info:

http://www.vbulletin.com/forum/showp...5&postcount=12

[Freddie Bingham]

It doesn't work off what the ip address is, or what the ip address resolves to but rather the USER_AGENT string that the spider's client sends to the server.

[Floris]

That is why I said 'without useragents, we can't do much'. You have an idea why it sometimes doesn't provide the useragents? I noticed it sometimes doesn't even list my own useragent. And sometimes it does.

[Freddie Bingham]

That is a bug in beta 3/4 that has been fixed.

[reefland]

String:
msnbot

Display:
MSN

I was actually surprised to see this since MSN Search is powered by another service but it was on one of my forums about 5 days ago.

Regards,
Scott Z.

[Floris]

String:
msnbot

Display:
MSN

I was actually surprised to see this since MSN Search is powered by another service but it was on one of my forums about 5 days ago.

Regards,
Scott Z.

Added to the first post Thank you !

[bromyaur]

string
Crawl your own stuff

display

grub

info on this bot if anyone cares, my 1'st time seeing it

http://grub.org/

and

string

Board Reader

Display

Board Reader

and info on them if no one has heard of them

http://www.boardreader.com/aboutus.asp

[reefland]

Here's a link with pleny of bots and their user-agents.
http://www.robotstxt.org/wc/active/html/contact.html

[Stadler]

Here's a link with pleny of bots and their user-agents.
http://www.robotstxt.org/wc/active/html/contact.html

I've cleaned up the list with a small script now.

Now you have a list, that only contains

{BOTNAME} Agent: {USERAGENT}

in each line.

Maybe, I'll clean it up some more, so I could run a script that creates the 2 lists we need for the fields in our options ...

Or perhaps another one volunteers?

Important note: The file attached here is a very old version and not usable in the vBulletin-config. See Post #12 instead.

[Floris]

Thank you for that stadler

[Stadler]

OK, I'm done with the cleanup. (See Attachment)

List for AdminCP follows in the next post.

I've removed version numbers from the Agents, stuff like 'Spider', 'Robot' and so on from the name for example and I've removed spiders, where no user agent was specified.

Then I've merged it with the existing lists in the first post.

Anyway, this list may still contain some problems, like dupe agents (or agents, that match two or more spiders), typos among other, so I need feedback to keep it as bugless and complete as possible.

Important note: The file attached here is a very old version and not usable in the vBulletin-config. See Post #12 instead.

[Stadler]

What is a spider:
http://www.vbulletin.com/docs/html/what_is_spider

Spider Identification Strings:
http://www.ragnarokonline.de/spiderlist/spiderident.txt

Spider Identification Description:
http://www.ragnarokonline.de/spiderlist/spiderdesc.txt

Spiderlist XML-File (vB 3.5.0 and up):
http://www.ragnarokonline.de/spiderlist/spiderlist.xml

More Files and Infos:
http://www.ragnarokonline.de/spiderlist/ (Mainly not related to vBulletin alone)

Some notes about the files:
The XML file is supported in vB 3.5.0. To use the list posted here, see http://www.vbulletin.com/docs/html/spider_xml_file

I'm maintaining the list using a MySQL table (same layout as in the SQL-file) through phpMyAdmin. All other files are created by calling a script that generates them.

[Stadler]

OK, I've added WiseNut now (forgot that) and fixed Inktomi.

[Stadler]

Changes:
Description: Googlebot -> Google

And I've switched the lists somehow before (corrected that)

[Princeton]

Ok, I'll ask...
What's the point in 'enabling' spiders on who's online?
While I'm at it ... what's the point in enabling "What's going On?" to non-members?