Spambots

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Acestumacher
    New Member
    • Aug 2010
    • 9
    • 4.0.x

    [Forum] Spambots

    Hi Guys, should I be concerned by this spambot?,

    Google Spider 10:41 PM Modifying Private Message Folders 66.249.71.144



    Thanks Ace
  • setishock
    Senior Member
    • Jun 2005
    • 1334
    • 4.2.x

    #2
    It's a google bot not a spam bot. They crawl your site to index it on Google. If you see a red icon next to the IP of the search bot it means they don't have permission to do whatever it says they're trying to do.
    They are harmless. However if you want to tighten up what they can and can't look at, check your robot.txt file. If you don't have one you can Google the term and get some tuts that will guide you.
    ...

    Comment

    • Acestumacher
      New Member
      • Aug 2010
      • 9
      • 4.0.x

      #3
      Ah Thank you.

      Comment

      • Videx
        Senior Member
        • Jan 2007
        • 848

        #4
        The basic content of a robots.txt for vb:
        User-agent: *
        Disallow: /forum/printthread.php
        Disallow: /forum/subscription.php
        Disallow: /forum/payments.php
        Disallow: /forum/profile.php
        Disallow: /forum/faq.php
        Disallow: /forum/calendar.php
        Disallow: /forum/private.php
        Disallow: /forum/sendmessage.php
        Disallow: /forum/sendmessage.php?do=
        Disallow: /forum/showgroups.php
        Disallow: /forum/reputation.php
        Disallow: /forum/report.php
        Disallow: /forum/threadrate.php
        Disallow: /forum/postings.php
        Disallow: /forum/newthread.php
        Disallow: /forum/search.php
        Disallow: /forum/newreply.php
        Disallow: /forum/register.php
        Disallow: /forum/login.php
        Disallow: /forum/faq.php
        Disallow: /forum/image.php
        Disallow: /forum/cron.php
        Disallow: /forum/joinrequests.php
        Disallow: /forum/usercp.php

        User-agent: Slurp
        Crawl-delay: 1500

        User-agent: *
        Disallow: /admin/
        Disallow: /contrib/
        Disallow: /doc/
        Disallow: /lib/
        Disallow: /modules/
        Disallow: /plugins/
        Disallow: /scripts/
        Disallow: /tmp/

        Comment

        • djbaxter
          Senior Member
          • Aug 2006
          • 1418
          • 4.2.5

          #5
          The robots.txt will stop Google and other well-behaved bots from indexing that content but it won't necessarily stop them from crawling the content looking for other links to follow.

          Basically, it's a non-issue that requires no intervention. Spiders will attempt to follow links - ALL links. Some ofn those are deadends with "you don't have permission..." messages. That's as it should be. vBulletin working as designed.
          Psychlinks Web Services Affordable Web Design & Site Management
          Specializing in Small Businesses and vBulletin/Xenforo Forums

          Comment

          • sebbo
            Member
            • May 2006
            • 58

            #6
            How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?

            Comment

            • Andy
              Senior Member
              • Jan 2002
              • 5886
              • 4.1.x

              #7
              Originally posted by sebbo
              How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?
              Yes.

              Comment

              • beishe8
                Senior Member
                • Oct 2005
                • 6782
                • 4.2.X

                #8
                Originally posted by sebbo
                How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?
                Are not they "Baidu" ?


                vB5 is unequivocally the best forum software, but not yet...

                Comment

                • Videx
                  Senior Member
                  • Jan 2007
                  • 848

                  #9
                  I gotta disagree that 160 bots all from the same outfit at the same time is 'normal'. This sounds like one of those mysterious 'guest attacks' we've seen intermittently. But when it happens to us they're all from different sites, so I guess not. Still, any one site really only needs one spider to crawl one site.

                  Right now our site is showing only two Google Adsense spiders, but 12 Bing spiders. One Yandex and one Baidu.

                  Comment

                  • sebbo
                    Member
                    • May 2006
                    • 58

                    #10
                    What is your session time out? I change it to 60 and now i have 21 spiders ( 3 x Google AdSense Spider and 18 x Google Spider). Those spiders have 20 different IP addresses. My normal session time out is 3600. With that i have 71 AdSense Spiders and 66 Google Spiders. Those come from 70 different IP´s.

                    It is a little bit embarrassing that i have many guest but almost all of those are spiders.

                    Comment

                    • Videx
                      Senior Member
                      • Jan 2007
                      • 848

                      #11
                      Me? Our timeout has been 900 (15 minutes) for ages. I can't imagine lowering it to 1 minute - nobody could post!?

                      Comment

                      • Simon Lloyd
                        Senior Member
                        • Apr 2008
                        • 610
                        • 3.7.x

                        #12
                        This will help you control them and bring down the load...etc on your server http://www.vbulletin.org/forum/showthread.php?t=268208
                        Kind regards,
                        Simon
                        Microsoft Office Discussion

                        Comment

                        widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                        Working...