BoardReader.com

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Gary King
    Senior Member
    • Feb 2002
    • 860

    #46
    I'm sure they'll remove it once they see this post

    Comment

    • spurdon
      Member
      • Jul 2001
      • 41

      #47
      What is your site. I will take care of it for you. Sorry about that!!!

      Comment

      • IDN
        Senior Member
        • Apr 2002
        • 4030
        • 3.5.x

        #48
        Sent you a PM with the URL.
        Running vB since 4-14-2002

        Comment

        • spurdon
          Member
          • Jul 2001
          • 41

          #49
          It has been updated and should take effect during our next build (seven to tens days or so).


          Best,

          Scott

          Comment

          • IDN
            Senior Member
            • Apr 2002
            • 4030
            • 3.5.x

            #50
            Thanks. I'll see you then.

            Just curious what your bot name is? Do you follow robots.txt?
            Running vB since 4-14-2002

            Comment

            • Christine
              Senior Member
              • Aug 2001
              • 592
              • 3.0.6

              #51
              Originally posted by spurdon
              We do respect all robots.txt and also emails from users and or webmasters when they don't want to be crawled....
              Hi Scott,

              I have never had any problems with search sites as my robots.txt is set to block all. Seeing this thread, I ran my site through yours and found several hundred cached threads.

              While I appreciate your service (and applaud your efforts!), I do not wish for my robots.txt file to be overriden on the site.

              I will gladly send you an email for exclusion, but as my robots.txt was *supposed* to do this, can you also provide the bot-id for me and others who may wish to set controls for it?

              Thanks!

              L'chaim

              Comment

              • spurdon
                Member
                • Jul 2001
                • 41

                #52
                Christine,


                Please send me an email for your site and I will make sure this problem gets corrected. Sorry about that. I'm leaving out of town for the next week so I will not have computer access for a while but I will make sure I handle that before I leave.

                Thanks,
                Scott

                Comment

                • Christine
                  Senior Member
                  • Aug 2001
                  • 592
                  • 3.0.6

                  #53
                  Will do Scott. Thanks, and have a great trip!

                  L'chaim

                  Comment

                  • spurdon
                    Member
                    • Jul 2001
                    • 41

                    #54
                    I still have not gotten your site. Please email me.

                    Comment

                    • Christine
                      Senior Member
                      • Aug 2001
                      • 592
                      • 3.0.6

                      #55
                      I just sent it. Sorry -- was going through the new posts since last night. All 8 pages of them. Heh.

                      L'chaim

                      Comment

                      • akiy
                        Senior Member
                        • Apr 2000
                        • 157

                        #56
                        BoardReader just came to my site and then proceeded to try to grab twenty showthread.php pages in three seconds, causing the load to spike up to unreasonable heights. Suffice it to say that I have blocked their very unfriendly spider from going through my site.
                        AikiWeb Aikido Information

                        Comment

                        • spurdon
                          Member
                          • Jul 2001
                          • 41

                          #57
                          AKIY,

                          This was a very strange issue that I am very SORRY about. I just checked your site (www.aikiweb.com) in my admin and saw that for some reason the crawler got caught and crawled back 1000 days. I then got an alert telling me of this and I have since corrected the problem. Your crawl back days should have been no more than 1 or 2 days for your size site.

                          I'm ok with you not wanting us to index and post your site on Boardreader.com, but just know we are very aware of both sides (yours and mine) bandwidth use. It is expensive for both of us that we do our best to limit use at ALL TIMES.

                          I ask you to try this correction that I just made and tell us what you think. I have currently left your site inactive so if you would like to review our bot with this 1 day crawl back correction that would be GREAT just let me know. Users have to understand that always during a first crawl there is going to be a little spike but Akit is right we should not have used that much during the first/second/or even third pass of his site.

                          If nothing else I'm very sorry for this, but please know it was a mistake and nothing else. If you would like to review our bot like I said please let me know and I will activate your site again for you to review our bot.

                          Thanks for the statements. Have a great day!

                          Scott

                          Comment

                          • jilly
                            Senior Member
                            • Apr 2001
                            • 1235
                            • 3.7.x

                            #58
                            I feel it is stealing because they are overwriting my robots.txt file, which disallows all spiders.

                            My site is set to private, all search engine spiders are blocked and only logged in, registered users are supposed to be accessing the site. How is boardreader.com getting around this? Why, if robots are blocked, and guest/unlogged in users can't see content, are they able to take it and display it on their site? And why, when I have contacted them twice before, and had my site content removed twice before, is my site content now BACK up visible on their site again? How can I permanently block boardreader.com from accessing my site and stealing my content?

                            Comment

                            • spurdon
                              Member
                              • Jul 2001
                              • 41

                              #59
                              Jilly,

                              What is your website. I don't believe we are crawling your site unless you submitted it for crawling. Please advise and I can double check.

                              best,

                              scott

                              Comment

                              • spurdon
                                Member
                                • Jul 2001
                                • 41

                                #60
                                Jilly,

                                I just dug up your email from over a year ago, which had your two sites (one site with a redirect), and we are not crawling it and have not crawled it since 7/2/2007. I did a search on BoardReader and could not find any messages. This data is gone from boardreader from what I can see in my DB. Please advise if your seeing it on boardreader and if so please send me that link so I can deal with it, but please know we removed some time ago.

                                Thanks,

                                Scott

                                Comment

                                Related Topics

                                Collapse

                                • alex2220
                                  seo issues
                                  by alex2220
                                  Hi all,
                                  Running this cloud version for a few months nou make me thinks that there is something slow in seo
                                  Good traffic, good rank for the age but still shows in page 4,5,6 on google searches...
                                  Sat 4 Mar '17, 2:02am
                                • dnnglobal
                                  Sitemap Not Working
                                  by dnnglobal
                                  Hello,

                                  I noticed I possibly don't have a sitemap. I'm wondering how I can create one so my site is easily readable and crawlable by Google. Do i need to manually create a sitemap? My search...
                                  Fri 28 Sep '18, 4:06am
                                Working...