I'm sure they'll remove it once they see this post
BoardReader.com
Collapse
This topic is closed.
X
X
-
Originally posted by spurdonWe do respect all robots.txt and also emails from users and or webmasters when they don't want to be crawled....
I have never had any problems with search sites as my robots.txt is set to block all. Seeing this thread, I ran my site through yours and found several hundred cached threads.
While I appreciate your service (and applaud your efforts!), I do not wish for my robots.txt file to be overriden on the site.
I will gladly send you an email for exclusion, but as my robots.txt was *supposed* to do this, can you also provide the bot-id for me and others who may wish to set controls for it?
Thanks!
L'chaimComment
-
Christine,
Please send me an email for your site and I will make sure this problem gets corrected. Sorry about that. I'm leaving out of town for the next week so I will not have computer access for a while but I will make sure I handle that before I leave.
Thanks,
ScottComment
-
BoardReader just came to my site and then proceeded to try to grab twenty showthread.php pages in three seconds, causing the load to spike up to unreasonable heights. Suffice it to say that I have blocked their very unfriendly spider from going through my site.Comment
-
AKIY,
This was a very strange issue that I am very SORRY about. I just checked your site (www.aikiweb.com) in my admin and saw that for some reason the crawler got caught and crawled back 1000 days. I then got an alert telling me of this and I have since corrected the problem. Your crawl back days should have been no more than 1 or 2 days for your size site.
I'm ok with you not wanting us to index and post your site on Boardreader.com, but just know we are very aware of both sides (yours and mine) bandwidth use. It is expensive for both of us that we do our best to limit use at ALL TIMES.
I ask you to try this correction that I just made and tell us what you think. I have currently left your site inactive so if you would like to review our bot with this 1 day crawl back correction that would be GREAT just let me know. Users have to understand that always during a first crawl there is going to be a little spike but Akit is right we should not have used that much during the first/second/or even third pass of his site.
If nothing else I'm very sorry for this, but please know it was a mistake and nothing else. If you would like to review our bot like I said please let me know and I will activate your site again for you to review our bot.
Thanks for the statements. Have a great day!
ScottComment
-
I feel it is stealing because they are overwriting my robots.txt file, which disallows all spiders.
My site is set to private, all search engine spiders are blocked and only logged in, registered users are supposed to be accessing the site. How is boardreader.com getting around this? Why, if robots are blocked, and guest/unlogged in users can't see content, are they able to take it and display it on their site? And why, when I have contacted them twice before, and had my site content removed twice before, is my site content now BACK up visible on their site again? How can I permanently block boardreader.com from accessing my site and stealing my content?Comment
-
Jilly,
I just dug up your email from over a year ago, which had your two sites (one site with a redirect), and we are not crawling it and have not crawled it since 7/2/2007. I did a search on BoardReader and could not find any messages. This data is gone from boardreader from what I can see in my DB. Please advise if your seeing it on boardreader and if so please send me that link so I can deal with it, but please know we removed some time ago.
Thanks,
ScottComment
Related Topics
Collapse
-
by alex2220Hi all,
Running this cloud version for a few months nou make me thinks that there is something slow in seo
Good traffic, good rank for the age but still shows in page 4,5,6 on google searches...-
Channel: vB Cloud Support & Troubleshooting.
Sat 4 Mar '17, 2:02am -
-
by dnnglobalHello,
I noticed I possibly don't have a sitemap. I'm wondering how I can create one so my site is easily readable and crawlable by Google. Do i need to manually create a sitemap? My search...-
Channel: Support Issues & Questions
Fri 28 Sep '18, 4:06am -
Comment