The inktomisearch spider thread

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • centris
    Senior Member
    • Jun 2001
    • 314
    • 3.8.x

    The inktomisearch spider thread

    And a right pain it is too. Last night I had 35 guests online and five members. Strange I thought and did a quick who's online. Well the 35 guests were inktomisearch spiders see image below. Seems it ignores the robots.txt file too. My members quote it being on my site for over 5 hours, and I just noted that these forums are also being hit right now by the sod.
    Attached Files
  • IDN
    Senior Member
    • Apr 2002
    • 4030
    • 3.5.x

    #2
    Not new, just annoying
    Running vB since 4-14-2002

    Comment

    • Jake Bunce
      Senior Member
      • Dec 2000
      • 46598
      • 3.6.x

      #3
      I think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.

      Comment

      • IDN
        Senior Member
        • Apr 2002
        • 4030
        • 3.5.x

        #4
        What does he do anyways?
        Running vB since 4-14-2002

        Comment

        • Joe
          Senior Member
          • May 2000
          • 2435

          #5
          She (?) indexes your site for the inktomi based search engines (http://www.inktomi.com/customers/). Inktomi was purchased by Yahoo! earlyer this year, I (along with others), expect yahoo to move from google to Ink by the end of this year. You really do not want to block this spider from your sites!

          My personal spider is this one:
          Attached Files
          Bike Forums.net

          Comment

          • DWZ
            Senior Member
            • Jan 2002
            • 985
            • 2.2.9

            #6
            Originally posted by Jakeman
            I think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.
            Yes:
            HTML Code:
            <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
            But if it's ignoring robots.txt, it will probably ignore the meta tag too.

            Comment

            • Floris
              Senior Member
              • Dec 2001
              • 37767

              #7
              From all the people that complain, does anybody actually know how much traffic the spider is pulling ?

              Comment

              • IDN
                Senior Member
                • Apr 2002
                • 4030
                • 3.5.x

                #8
                Or how much taffic it is costing us?
                Running vB since 4-14-2002

                Comment

                • Radon3k
                  Senior Member
                  • Nov 2001
                  • 1889

                  #9
                  Originally posted by IDN
                  Or how much taffic it is costing us?
                  Well you have to weigh in how much bandwidth you have, how much you use (on average per month), and how much you're willing to set aside for that type of thing.

                  Comment

                  • IDN
                    Senior Member
                    • Apr 2002
                    • 4030
                    • 3.5.x

                    #10
                    Sorry, meant, traffic.

                    bandwidth is the site of the pipe, traffic is how much goes through the pipe.

                    if it visites every page on the site with mutiple spiders, must be costing us alot
                    Running vB since 4-14-2002

                    Comment

                    • Chris M
                      Senior Member
                      • Apr 2002
                      • 1387
                      • 3.6.x

                      #11
                      My site is still to be hit by them lol...

                      When they come, I'm not worried

                      Satan

                      Comment

                      • mymilkexpired
                        Member
                        • Mar 2003
                        • 35

                        #12
                        Search Engine Spiders...

                        Whats been everyones experience with them?? I have had the Inktomi spider on my side non stop for the last 2 days, it just doesnt give up. When ever a new thread is made its like its starts over again and goes through them all. Very wierd, so far the highest number of spiders has been 14 at one time. Just curious what everyone else has seen on their boards...

                        Comment

                        • mymilkexpired
                          Member
                          • Mar 2003
                          • 35

                          #13
                          Its been on my site for 2 days now (Inktomi) i think this spider crawls really slow :/

                          Comment

                          • Jake Bunce
                            Senior Member
                            • Dec 2000
                            • 46598
                            • 3.6.x

                            #14
                            It was on my site for two weeks, and my forums don't allow guests to access them. It must like "no permission" pages. It's a retarded spider.

                            Comment

                            • poolking
                              Senior Member
                              • Aug 2002
                              • 2061

                              #15
                              I've had 45 google spiders on at once one day and I have loads of inkitomis come on, but they have been quiet over past couple of days.

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...