Boardtracker.com

[David Bott]

Hey Guys...

Heads up just in case. (Unless I am late to this.)

I just found a site called http://www.Boardtracker.com where it goes out and takes our content to make it searchable on their site. It goes deeper then by linking to items like user profiles and also the runs a search query to find the posts by that user. It does this all as a USER on the site but returns the results to their site.

This was brought to my attention by some large queries happening. I was NOT HAPPY to find this site is making $$$ off of our content. (They use a ton of Google Ads with the content found.)

This is a case of them user our servers for results. (unlike Google which indexes it all.) Seeing our content is now on their site, it is also indexed by Google thus also sending them traffic based on our content.

Look for a user named BoardTracker and ban the user ID if you do not want this happening. I find this totally underhanded. (was not even asked.)

Kyle...PbNation.com is also in it as well as most of the sites listed in this thread.

[RedWingFan]

Good catch. According to their site, this should get rid of their tracking:

User-agent: BoardTracker
Disallow: /

Going to install it now...our site is listed there also.

Edit: found the user BoardTracker among our members. I've disabled it for now, until their spider can remove us from their index.

Interesting though--it joined up around the end of November, and it's possible we may have started seeing some of our slowdowns around that time.

[David Bott]

Upon further review of what they are doing...We will be turing this over to legal firm to look into seeing they have been using the content of our sites to make money. We will be demanding for all material related to our sites to be removed and to not visit the site again. (Not just site content, but our web services to get the data with each call.) They MAKE AN ACCOUNT to get the information. This goes far beyond the spider for SOMEONE has to make the account.

You should not have to add in a directive to stop it. That more or less says, "We did not add it in not to so it is ok to take our data." Their is also nothing to say that they HAVE TO HONOR the directive. (This is true for any spider.)

[TonyPBN]

The user's ip on our BoardTracker member came back as 62.90.210.65 (mail.mocl.co.il).

The domain is also registered to a Tel Aviv address. http://pidgintech.com/

Might be a tad more complicated going across borders.

[Reeve of Shinra]

fyi -- There's a big discussion about boardtracker over at theadminzone.com -- the owner posted some replies there too.

[TonyPBN]

Erwin, glad to hear you made it! Must have been a long 5 days?


As for the boardtracker discussion, the thread is here: http://www.theadminzone.com/forums/s...ad.php?t=13611

Thanks Reeve of Shinra for the info.

[kmike]

Another forum search engine to watch: http://www.boardreader.com
User agent: BoardReader
It has a nasty habit of firing a dozen of requests in a second when it goes round for forum update.

Erwin: congratulations, I hope you'll get everything up and running soon!

[David Bott]

Hi...

This kills me. (Sorry I was late to know about BoardTracker.) But they CLEARLY ARE pulling content not open to non-registered users. We do not allow for member information to be viewed unless registered. Thus being registered it can pull information.

Also we have Club areas. If they then choose to join the club, they then can index the private forums.

He also said..."We notify all boards BEFORE adding to boardtracker so they can choose to be spidered, fully excluded, or modify their robots.txt accordingly to control access."

This is total BS...I was not contacted for AVS or TiVo Community sites.

All in all, we have requested all contact to be removed and we will be bring this up with Google as what BoardTracker is doing with our content on Googles site.

Sorry all, now back on topic.

Erwin...Congrats. OUCH on the down time. Still not looking forword to using it based on some of the posts here.

[BoardTracker]

Clearly there are some issues and misunderstandings about what BoardTracker is and how it works and I hope this post will clarify those..

#1 - We DO NOT use your servers resources for search, BoardTracker is like any other search engine and we have our own index of the data and only use our resources to provide search and all our other services. Further, since BoardTracker is designed specifically for boards/forums our spider is extremely efficient and uses the absolute minimal resources to do its job, it does not wander around aimlessly spidering every page on your site over and over.

#2 - We are robots.txt compliant and as soon as you (David Bott) updated your robots.txt file with a BoardTracker exclusion, ALL data from your boards was removed. We even contacted you directly to discuss the matter since we noticed from your posts here that you have misunderstood some aspects of how BoardTracker works. I hope this post will address your concerns and the reasons you decided to block BoardTracker in your robots.txt will no longer be an issue for you but either way its entirely your choice and the relevant robots.txt directives will always be respected immediately (as soon as our spider next visits of course) on all boards.

#3 - BoardTracker, like any search engine, aims to also make money from the services we provide. We make money by providing a valuable service to our users and that service is sending users to forums to read threads that match their search and alerts terms. We send fresh, highly targeted traffic and new members to boards that we index so everybody benefits - our users, your users, your board as a whole and BoardTracker itself. Doesn't Google make money from the many thousands of pages it has indexed from your site? Naturally it does, as does Yahoo and all the rest. They all have advertising and various other ways to generate income however it is never a one way street, sites of all kinds benefit from being indexed by search engines, search engines benefit from the data they index. It is the nature of the web and generally it works well.

#4 - As for notifying boards in advance - this is not BS. But first of all - has any other search engine EVER notified you or anyone in advance before spidering your site? I have never heard of such a thing happening. HOWEVER, as you can see from the thread on TAZ, there were concerns raised by some board owners and we listened and we decided (in september) to start notifying in advance. BoardTracker is in BETA and in constant development and we welcome feedback so that we can improve the service for board owners and for our users. Your boards were indexed *before* we made this decision (19-Jun-05 for AVForums, 19-Aug-05 for Tivo) which is why you were not notified in advance. Also as you know we use an account on boards which clearly identifies us so you can easily check out BoardTracker, in addition our spider also identifies itself in the useragent string and includes a url to our spider info page so again you can easily see our activity and modify your robots.txt if you desire.

#5 - BoardTracker does not 'steal' content from boards, we do not display whole threads, we do not display replies, we do not show a cached version of your threads. BoardTracker works like any search engine in that it generates a small snippet of a thread which links directly back to the original thread on the board itself. You cannot read threads on boardtracker, you have to visit the board to view it.

#6 - BoardTracker does not index private forums. What BoardTracker does when indexing a new board is it reads the list of forums VISIBLE TO GUESTS and it is only these forums that will be indexed. We never intentionally index private forums (unless requested by the board owner) but if for some reason a forum was visible to guests on the forum list yet the threads within it were only visible to logged in users then in theory those threads could be indexed - however this is very rare and in the unlikely event of such a forum being indexed we provide tools for the board owner to exclude such a forum either by using special robots.txt directives to block a specific forum within a board, or by contacting us to request a removal. PLUS when we notify boards in advance we send them a link to a board owners control panel which shows exactly which forums will be indexed so they can take any required action in advance. This control panel provides various data about the board and tools for categorizing forums etc.

#7 - All data from a board that is indexed on BoardTracker links back to the board, always, every time. This includes thread poster/member information, you can't find member profile data on BoardTracker besides the regular and public username, homepage, post counts, etc. - for anything else there is simply a link to the profile on the board itself which of course is then subject to the boards own access policies. The data indexed by boardtracker is the same as you will find in all search engines like Google, we simply display it in a way that better suits forum search.

#8 - There are various technical reasons why we use an account on boards including the ability to set the GMT timezone to facilitate syncing times on messages from boards all over the world, to clearly identify BoardTracker to board owners so they know what is going on, to ensure our spiders are issued a single user session rather than multiple guest sessions each time which helps maximize efficiency when spidering threads, to use boards referral systems (such as with vb) so board owners can see that we refer new members, plus we often use the account (like now) to respond to questions on the boards about BoardTracker. Also on certain boards where required we can use the account to change the default language to english or change the date format or various other things that may be needed depending on how they are configured.

I hope I have covered all concerns but as always we are available to discuss any concerns and welcome all constructive feedback. We are in the process of building a portal on boardtracker for board owners which will provide links to various boardtracker tools, resources and FAQ's which will help to clarify all the issues raised here and others and hopefully improve overall communication between us allowing us all to work together more closely to provide the best service to all our users and respective communities.

[TonyPBN]

Ah great, here we go.

If anyone (like myself) was not aware of the boardtracker fiasco over at theadminzone I suggest reading the thread I linked to there on the previous page before propelling this thread into another debate about boardtracker.

[David Bott]

Ah great, here we go.

If anyone (like myself) was not aware of the boardtracker fiasco over at theadminzone I suggest reading the thread I linked to there on the previous page before propelling this thread into another debate about boardtracker.

Your right. Should not be in this thread. I mentioned it for large sites like ours stand to loose so much more.

We will let out legal firm handle it at this point and we may even look into a class action suite. While they removed the contet from coming up, it still can be searched on their site and found with ads showing even though the results do not get displayed (Multi pages of items found.) Thus they still can use the content for Google indexing.

[John]

Split from http://www.vbulletin.com/forum/showt...111239&page=45

[Wayne Luke]

I have always been leery of Boardtracker and they have never satisfied the question of why they need to register a user (and violate registration rules on forums in many cases) to spider forums, at least in my opinion. For that reason, whenever I start a new project I ban the userid and prohibit anyone from their IP addresses from registering.

[David Bott]

Yes, we ended up the a letter from our legal firm going to them. As mentioned, they have removed the content from view, but NOT from the database. If you search for AVSForum it will find pages of content, but not show it. Yet the search works and ads are then shown.

They also used our trademark logo giving the look, in our opinion, that we endorse the site. This is way over the top for a "search engine". While other sites may not have a trademark, we do.

They have been advised that our content is not to be stored in their systems with all the content removed. This means, anything taken from our site should be FULLY REMOVED and not be searchable. (Again, even if it does not display the posts...the ads as still shown on result pages and pages.)

Such a shame sites need to go to this extent, but what is being done is not right in any way shape or form.

[David Bott]

On #1...The site uses the user login to get user profile information and the last current threads. This is done on the forums server to get the data. Thus not stored by you and not available unless a person is a registered member of our site.

On #2...We added in the directive for one reason...To statifiy the courts if we bring it to that end. We should not have to set a directive for we should have been asked. Again, an account was made to reads the site contents which goes far beyond a search engine.

On #3...You use the content of other sites to get your placement in Google to this gain traffic. This is NOT what a search engine like Google does. When a search is made in Google and your site is shown with our content, that is purely wrong.

On #4...It was YOUR statement that your contacted the sites. This was not the case with two of the sites I operate. I did not see the date mentioned when you choose to out this new rule into affect. I am sorry you choose not to contact sites you started to take content from prior to thinking it may be good to check first. FYI...you would have been told no. How may others of the 22,000+ forums do not know?

On #5 & 7..See #3.

On #8...You use the account to pull other users account profile information. This is done on the forum at the time of the request thus using the account to gain access to inforamtion that is not open to a user that is not a logged in member. At the same time it uses the search to find posts by the user. Again, far beyond what a search engine does.

From our user agreement...

"All rights reserved. By posting on this forum you grant AVS Forum, Inc., and its successors and assigns a nonexclusive irrevocable right to re-use your posting in any manner it or they see fit without notice or compensation to you. No material contained in this site may be republished or reposted without express written consent of the AV Science, Inc., except that message board users retain the right to republish or repost their own work."

and

"9. NO RESALE OF SERVICE You agree not to reproduce, duplicate, copy, sell, resell or exploit for any commercial purposes, any portion of the Service, use of the Service, or access to the Service."


and


"AVS FORUM, and the AVS FORUM logo, trademarks and service marks, and other AVS FORUM logos and product and service names are trademarks of AVS FORUM (the "AVS FORUM Marks"). Without AVS FORUM's prior permission, you agree not to display or use in any manner, the AVS FORUM Marks."


All of which you would have had to agree to when you made the accoount for your spider.


All in all you have been advised on what you need to do with the content you have taken from our site. We will be checking by doing seaches on past threads you had already harvested. Nothing should be found at all and we are not taking about just not displaying it. If it is searchable, you sill have our data stored.

Thank you.