Controlling bad bots through .htaccess

[eJM]

I haven't been hit as hard as some of the members here have been, but it is keeping me busy. One of the methods I have always used to prevent malicious visitors is by denying them access via the .htaccess file. But today I had a spammer post several messages as a guest (they all go to moderation) and it turns out the IP address they used was already in my deny list in the .htaccess file.

Is that possible? Could I have my format wrong? Here is my current deny list as I have it in my .htaccess file, which is located in my domain root:


<Files 403.php>
order deny,allow
deny from 150m.com
deny from 1mml.biz
deny from buyultramnow.com
deny from ce287.vdswin.com
deny from corbina.ru
deny from cr.net.cable.rogers.com
deny from cheap-24h.com
deny from dearcomputer.nl
deny from fillbest.com
deny from forum.18sex18.info
deny from infostore.org
deny from internetserviceteam.com
deny from keyweb.de
deny from server2.yourdomain9.com
deny from spycamgirl.info
deny from spyfu.com
deny from voyeurez.tv
deny from 65.39.72.142
deny from 67.212.189.144/29
deny from 68.57.62.139
deny from 79.143.176.0/22
deny from 79.143.177
deny from 82.128.35.72
deny from 85.12.25.49
deny from 87.118.70.221
deny from 89.149.227.78
deny from 93.80.0
deny from 93.81.0
deny from 121.156.65.229
deny from 194.165.42
deny from 209.63.57
deny from 213.155.0.0/19
deny from 213.155.5.160 - 213.155.5.167
allow from all
</Files>

My research has show that this is an acceptable format. But I don't know why the spammer got through. Is there a way for spammers to now ignore the .htaccess file?

Any insight would be appreciated.

Thanks,

Jim McClain

[Floris]

Why limit it to 403.php?
http://corz.org/serv/tricks/htaccess.php

[eJM]

Why limit it to 403.php?
http://corz.org/serv/tricks/htaccess.php

Nice link, thanks. However, I feel my 403 is a pretty good page. www.thefloorpro.com/403.php. But I may have to re-think that strategy for the bad bots. I wish there was a directive that would cause the bot owner to catch a bad case of the clap, but I don't think the technology is there yet. ;)

Apparently my deny IP routine isn't working for some reason. I tested it by putting my own IP address in the file. I never saw the 403 and it didn't have any affect on my access. I sent a support ticket to my host to see what's up.

Thanks again for the useful link - and that's a really attractive website too.

Jim

[SNN]

The spammer got through as that is a invalid format.
What Floris meant when limiting it to 403.php was that you're only blocking them from 403.php
In order to block them from everything and use 403 forbidden as 403.php:


ErrorDocument 403 403.php
deny from 150m.com
deny from 1mml.biz
deny from buyultramnow.com
deny from ce287.vdswin.com
deny from corbina.ru
deny from cr.net.cable.rogers.com
deny from cheap-24h.com
deny from dearcomputer.nl
deny from fillbest.com
deny from forum.18sex18.info
deny from infostore.org
deny from internetserviceteam.com
deny from keyweb.de
deny from server2.yourdomain9.com
deny from spycamgirl.info
deny from spyfu.com
deny from voyeurez.tv
deny from 65.39.72.142
deny from 67.212.189.144/29
deny from 68.57.62.139
deny from 79.143.176.0/22
deny from 79.143.177
deny from 82.128.35.72
deny from 85.12.25.49
deny from 87.118.70.221
deny from 89.149.227.78
deny from 93.80.0
deny from 93.81.0
deny from 121.156.65.229
deny from 194.165.42
deny from 209.63.57
deny from 213.155.0.0/19
deny from 213.155.5.160 - 213.155.5.167
I would suggest without the "allow from all" just incase something goes wrong in apache - which it does sometimes for me, don't know about your server and the fact that it's already allowing from anyone but the specified deny's.

[eJM]

Apparently the problem was in the format, but not in the way you suggested. I had to change the IP deny section to this:


order allow,deny
allow from all
deny from 150m.com
deny from 1mml.biz
deny from buyultramnow.com
deny from ce287.vdswin.com
deny from corbina.ru
deny from cr.net.cable.rogers.com
deny from cheap-24h.com
deny from dearcomputer.nl
deny from fillbest.com
deny from forum.18sex18.info
deny from infostore.org
deny from internetserviceteam.com
deny from keyweb.de
deny from server2.yourdomain9.com
deny from spycamgirl.info
deny from spyfu.com
deny from voyeurez.tv
deny from 65.39.72.142
deny from 67.212.189.144/29
deny from 68.57.62.139
deny from 79.143.176.0/22
deny from 79.143.177
deny from 82.128.35.72
deny from 85.12.25.49
deny from 87.118.70.221
deny from 89.149.227.78
deny from 93.80.0
deny from 93.81.0
deny from 121.156.65.229
deny from 194.165.42
deny from 209.63.57
deny from 213.155.0.0/19
deny from 213.155.5.160 - 213.155.5.167

I removed the <files></files> part because the purpose of adding an IP deny is to keep them from being able to use your site - including any custom error pages. Apparently, my server was going into a loop when I had the <files> directive and put my own IP address in the deny list. I never got a 403. I tried putting Google in there - <files google.com></files> - but I got a 500 internal server error. I'd like to be able to send bad bots off to a porn site or someplace - anyplace but my server (I have a VPS).

So, it's working now, even though it doesn't give a custom destination. At least I know they can't use the site.

Jim
PS: I do have several custom error pages defined near the top of my .htaccess file - for 403, 404 and 500

[MRGTB]

Why don't you just use a "deny from all", by dropping a .htaccess in directories you don't want anybody to access. Like directorys that have 777 permissions. Rather than creating a long list of IP's

<Files *>
Order Allow,Deny
Deny from All
</Files>

[eJM]

Why don't you just use a "deny from all", by dropping a .htaccess in directories you don't want anybody to access. Like directorys that have 777 permissions. Rather than creating a long list of IP's

<Files *>
Order Allow,Deny
Deny from All
</Files>

Perhaps it wasn't clear. The intention of the denied IP's in my root's .htaccess file is to ban bad bots from the whole website. The bad bots I am concerned about are the ones that register or post messages in those few forums I allow guest posting, or bots that seem to be there to eat up bandwidth and falsely increase the number of visitors (I had something like this coming from 1&1.com's servers and they were responsible for the 499 visitors I had one 30 minute period in April). I don't want them accessing any part of the website, not just the forums.

I am familiar with using .htaccess files in other directories, which I do to prevent all other people and bots from accessing those directories I feel need it. However, it is much better to use a modified version of your directive:

<Files *>
Order Allow,Deny
Allow from my.own.ip.address
Deny from All
</Files>

R'gards,

Jim