Yahoo robot tx file?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • JillK
    Senior Member
    • Mar 2006
    • 135

    Yahoo robot tx file?

    Hi,
    Last night I put up a post saying my board was slow. I later discovered there are 250 yahoo bots going thru the site at once. I read other post saying there is a code you insert into your public htm? Can someone please tell me how to set up the robot TX file?

    Thanks
    Jill
    Last edited by JillK; Tue 17 Jul '07, 3:46am. Reason: typo
  • JillK
    Senior Member
    • Mar 2006
    • 135

    #2
    More to question?

    Hi,
    I wanted to add im finding that many of these bots are inktomisearch.com .. Who are they and can I just block them?

    Thanks again for help I really need it..

    Jill

    Comment

    • JillK
      Senior Member
      • Mar 2006
      • 135

      #3
      Another question to my first question?

      Hi,
      Since I did not get a reply I went to Google and did a search. I found out how to set up a robot.tx I did so and then I uploaded it to the root of my site.

      My question now is at my site I have my website and I have my forum. My forum is in a different folder but from what I can see on my web tracker both my site and the board are getting bots so should I upload an robot.tx to both folders as right now the tx file is not in the folder where the forum is its just in the root of where both sites are?

      If I do upload one to my forum folder is there a particular place in that folder where the tx file should go?

      Thanks
      Jill

      Comment

      • JillK
        Senior Member
        • Mar 2006
        • 135

        #4
        I'm feeling un loved

        In 2 days and 2 post no replies..



        Jill

        Comment

        • Indy
          Senior Member
          • Jun 2002
          • 134
          • 3.6.x

          #5
          An Example

          User-agent: *
          Disallow: /forums/
          Disallow: /cgi-bin/
          Disallow: /photopost/


          I have my robots.txt file placed in the root dir. For every dir that I don't want spiders to enter I place a Disallow: statement. As you can see from the above example, those three dir's won't be spider'd but if I had a dir called /images it would be vulnerable.

          That's basically how I've been doing it for a few years now and I don't have any problems with spiders, bots or whatever you call them!
          Version: vB 3.6.4
          Website: RimfireCentral.com
          RFC Server: (2) Dual Core Intel 5130 Woodcrest Processor 1333 FSB | 4GB RAM | (2) 146GB 10K RPM SCSI/SAS Hard Drives RAID 1 | PHP 4.4.4 | Apache 1.3.37 | MySQL 4.1.22 | XCache 1.2.0 with Optimizer = Off | APC APC-3.0.12p2 (temporarily disabled)
          Host: ThePlanet

          Comment

          • JillK
            Senior Member
            • Mar 2006
            • 135

            #6
            Originally posted by Indy
            User-agent: *
            Disallow: /forums/
            Disallow: /cgi-bin/
            Disallow: /photopost/


            I have my robots.tx file placed in the root dir. For every dir that I don't want spiders to enter I place a Disallow: statement. As you can see from the above example, those three dir's won't be spider'd but if I had a dir called /images it would be vulnerable.

            That's basically how I've been doing it for a few years now and I don't have any problems with spiders, bots or whatever you call them!
            Hi Indy
            Thank you for your reply. I truly appreciated it. When I did a search on Google it said so that you dont entirely stop search engines from visting your site to set up the robot.tx file like what I will put below. If this is wrong please let me know.

            I know where the root file is as far as my website goes but not sure if I should just upload the TX file in the main forum folder on the site or in a specific folder within that folder. That is the problem Im really getting hit hard with search bots and its slowing my board down to a dead crawl so im trying very hard to get this fixed.

            Here is what I have for my robot.tx file.

            User-agent: *
            Disallow: /admincp/
            Disallow: /announcement.php
            Disallow: /calendar.php
            Disallow: /cron.php
            Disallow: /editpost.php
            Disallow: /faq.php
            Disallow: /joinrequests.php
            Disallow: /login.php
            Disallow: /member.php
            Disallow: /misc.php
            Disallow: /modcp/
            Disallow: /moderator.php
            Disallow: /newreply.php
            Disallow: /newthread.php
            Disallow: /online.php
            Disallow: /printthread.php
            Disallow: /private.php
            Disallow: /profile.php
            Disallow: /register.php
            Disallow: /search.php
            Disallow: /sendmessage.php
            Disallow: /showgroups.php
            Disallow: /showpost.php
            Disallow: /subscription.php
            Disallow: /subscriptions.php
            Disallow: /threadrate.php
            Disallow: /usercp.php

            Will that help?

            Thanks again
            Jill

            Comment

            • Indy
              Senior Member
              • Jun 2002
              • 134
              • 3.6.x

              #7
              I would simply use the command line: Disallow: /forums/ (Change "forums" to the name of your forum directory). Put the file in your root dir and that should be it. I have never had a problem with bots in the last several years by doing it this way.
              Version: vB 3.6.4
              Website: RimfireCentral.com
              RFC Server: (2) Dual Core Intel 5130 Woodcrest Processor 1333 FSB | 4GB RAM | (2) 146GB 10K RPM SCSI/SAS Hard Drives RAID 1 | PHP 4.4.4 | Apache 1.3.37 | MySQL 4.1.22 | XCache 1.2.0 with Optimizer = Off | APC APC-3.0.12p2 (temporarily disabled)
              Host: ThePlanet

              Comment

              • Jose Amaral Rego
                Senior Member
                • Feb 2005
                • 11058
                • 1.1.x

                #8
                /cgi-bin/ - You should add .htaccess to prevent hacking.

                Comment

                • JillK
                  Senior Member
                  • Mar 2006
                  • 135

                  #9
                  Thanks..

                  Hi Indy and Jose.
                  Thanks so much for your replies. If Ipower ever gets my site back up I will add the files.

                  Thanks again

                  Jill

                  Comment

                  widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                  Working...