Fast-Webcrawler??

[MarkB]

Does anyone know what Fast-Webcrawler is? It's showing up as a spider on my web logs, and in a day and a half has made (according to the logs) over 23,000 hits on my site -- could this be why my bandwidth is going through the roof?

It's not something I want, obviously, so any tips on stopping spiders in their tracks?

[filburt1]

Check out how to add a robots.txt to your root dir to stop bots.

[MarkB]

I've just added one, thanks But I understand webcrawler ignores robots.txt files? Or at least that's what I'm understanding after reading some usenet posts...

Here's hoping it's blocked now. Cheers!

[andrewpfeifer]

http://fast.no/support/

That may help you, also, if the previous thing didn't.

[MarkB]

I have put in the robots.txt file as well as the NOINDEX meta tag, which has slowed it a little - but it's still made almost 10,000 hits since yesterday

[Thomas P]

Hi,
maybe checking the netblock of the IP (e.g. at www.all-nettools.com or similar tools) the bot come from and contacting that ISP helps.
good luck anyway,
-Tom

[filburt1]

You can also try asking your host to block requests from that source. Better yet, add a .htaccess file that bans that IP.

[MarkB]

Originally posted by filburt1
You can also try asking your host to block requests from that source. Better yet, add a .htaccess file that bans that IP.

What format would the .htaccess file take?

[Ian]

Originally posted by MarkB


What format would the .htaccess file take?

Something like this:

<Limit GET>
order allow,deny
deny from 123.456.789.0
deny from 123.45.67
allow from all
</Limit>

[MarkB]

Thanks - did that, and access to my forums came up with a 500 Error

[filburt1]

500 is Internal Server Error You didn't deny yourself acccess, did you?

[neocivitas]

Originally posted by Ian

<Limit GET>
order allow,deny
deny from 123.456.789.0
deny from 123.45.67
allow from all
</Limit>

What IPs should you have listed to deny?

[MarkB]

Originally posted by filburt1
500 is Internal Server Error You didn't deny yourself acccess, did you?

hehehe - no

This is the log entry for FAST-WebCrawler:

66.77.73.69 - - [28/Apr/2002:20:23:56 -0400] "GET /forum/showthread.php?goto=lastpost&threadid=21745 HTTP/1.0" 302 0 "-" "FAST-WebCrawler/3.5 (atw-crawler at fast dot no; [url]http://fast.no/support.php?c=faqs/crawler[/url])"

So I put 66.77.73 in the htaccess file.

I also have robots.txt with the correct deny info as per their FAQ...

[MarkB]

This is the htaccess file:

<Limit GET>
order allow,deny
deny from 66.77.73
allow from all
</Limit>

[MarkB]

Problem solved! I had a clashing <directory> entry in my httpd.conf file - fixed that, and the bot hasn't hit my site for the last 30 minutes (and it was there 24/7 seemingly!).

Ahh... The Days Of Our Server continues