[Forum] Googlebots

[ozzy47]

How can I stop googlebots from trying to log into or look at my ftp?

They keep getting blocked by my server, here is a couple of the block notices I get:

66.249.72.78 # lfd: 66.249.72.78 (US/United States/crawl-66-249-72-78.googlebot.com), 1 distributed ftpd attacks on account [anonymous] in the last 300 secs - Sun Oct 2 11:52:00 2011
66.249.68.169 # lfd: (ftpd) Failed FTP login from 66.249.68.169 (US/United States/crawl-66-249-68-169.googlebot.com): 1 in the last 300 secs - Tue Oct 4 06:29:51 2011

[Wayne Luke]

You need to create a robots.txt file that can control search engine bots. Do a search here or at www.vbulletin.org for robots.txt for examples.

[ozzy47]

I have searched and see no way of blocking the ftp, as the ftp is not a file in the forum root, or part of the site index.

[Wayne Luke]

Googlebots won't be getting into your FTP. They are only interested in HTTP.

[ozzy47]

If that were the case how come I keep getting the csf blocks on them saying they are trying to access it?

Here is a full list From Sept 21st to now.

66.249.71.226 # lfd: 66.249.71.226 (US/United States/crawl-66-249-71-226.googlebot.com), 1 distributed ftpd attacks on account [anonymous] in the last 300 secs - Wed Sep 21 21:53:14 2011
66.249.71.86 # lfd: 66.249.71.86 (US/United States/crawl-66-249-71-86.googlebot.com), 1 distributed ftpd attacks on account [anonymous] in the last 300 secs - Sun Sep 25 02:57:42 2011
66.249.72.113 # lfd: (ftpd) Failed FTP login from 66.249.72.113 (US/United States/crawl-66-249-72-113.googlebot.com): 1 in the last 300 secs - Sun Sep 25 12:52:47 2011
66.249.67.228 # lfd: (ftpd) Failed FTP login from 66.249.67.228 (US/United States/crawl-66-249-67-228.googlebot.com): 1 in the last 300 secs - Tue Sep 27 12:13:52 2011
66.249.71.206 # lfd: (ftpd) Failed FTP login from 66.249.71.206 (US/United States/crawl-66-249-71-206.googlebot.com): 1 in the last 300 secs - Thu Sep 29 01:17:54 2011
66.249.71.0/24 # lfd: (NETBLOCK) 66.249.71.0/24 has had more than 4 blocks in the last 86400 secs - Thu Sep 29 03:59:35 2011
66.249.71.80 # lfd: (ftpd) Failed FTP login from 66.249.71.80 (US/United States/crawl-66-249-71-80.googlebot.com): 1 in the last 300 secs - Thu Sep 29 03:59:36 2011
66.249.72.138 # lfd: (ftpd) Failed FTP login from 66.249.72.138 (US/United States/crawl-66-249-72-138.googlebot.com): 1 in the last 300 secs - Sun Oct 2 09:21:07 2011
66.249.67.73 # lfd: (ftpd) Failed FTP login from 66.249.67.73 (US/United States/crawl-66-249-67-73.googlebot.com): 1 in the last 300 secs - Mon Oct 3 09:51:10 2011
66.249.68.169 # lfd: (ftpd) Failed FTP login from 66.249.68.169 (US/United States/crawl-66-249-68-169.googlebot.com): 1 in the last 300 secs - Mon Oct 3 08:58:09 2011
66.249.72.0/24 # lfd: (NETBLOCK) 66.249.72.0/24 has had more than 4 blocks in the last 86400 secs - Sun Oct 2 11:52:00 2011
66.249.72.78 # lfd: 66.249.72.78 (US/United States/crawl-66-249-72-78.googlebot.com), 1 distributed ftpd attacks on account [anonymous] in the last 300 secs - Sun Oct 2 11:52:00 2011
66.249.68.169 # lfd: (ftpd) Failed FTP login from 66.249.68.169 (US/United States/crawl-66-249-68-169.googlebot.com): 1 in the last 300 secs - Tue Oct 4 06:29:51 2011

[Wayne Luke]

Yeah.. Most likely someone posing as Googlebot. You'll need to contact your hosting company for support with FTP as it isn't related to vBulletin.

[ozzy47]

I don't believe it is someone posing as googlebot as the ip addy's resolve back to google, I guess I will have to some more research on the issue.

Thanks for your time Wayne!

[mainframe]

I'm having the exact same problem and my firewall is setup to instantly ban on wrong login so have to remove them from my banlist all the time

Time: Thu Dec 15 16:42:04 2011 +0100
IP: 66.249.71.217 (US/United States/crawl-66-249-71-217.googlebot.com)
Failures: 1 (ftpd)
Interval: 300 seconds
Blocked: Permanent Block

Log entries:

Dec 15 16:42:03 server2 pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [anonymous]

Time: Thu Dec 15 16:41:49 2011 +0100
IP: 66.249.71.152 (US/United States/crawl-66-249-71-152.googlebot.com)
Failures: 1 (ftpd)
Interval: 300 seconds
Blocked: Permanent Block

Log entries:

Dec 15 16:41:49 server2 pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [anonymous]

[Wayne Luke]

Well googlebot won't be trying to log into your FTP unless you have a link to it somewhere on a webpage. FTP access is not relevant to vBulletin support. You should contact your hosting company about switching to a secure form of FTP.

[Rik Brown]

We RARELY use ftp on our site so we disable it when we are not using it which plugs that possible security hole.

So consider disabling ftp when you are not using it if that is a concern for you.

-- Rik

[JackieChun]

I would really prefer not to disable ftp. I have no problem with googlebot attempting to crawl whatever it wants, but once csf bans googlebot, it can't crawl my actual forum, which is a problem.