Can search engines spider all pages if guests cant view?

[John Diver]

Hey,
I want to change one of my forums so spiders can index all pages but guests have to register to view the pages.



Is this done by default if guests have to register to read posts?

[Floris]

No.

A bot basicly sees what a guest can see. A bot is basicly a guest (without the session hash).

If you have set guests to ' Can view forum: NO '.
Then a spider bot can not view it either.

[John Diver]

Pitty...this is a great feature.

I just converted one of my sites from invision to vBulletin - This worked great to keep visitors and get more registered members with IPB.

Is there any way to put spiders into their own usergroup and have guests in another so spiders can still show posts in SE results?

[treasureman]

I would like to know this also, having just converted from Invision to VBulletin.

With Invision you could assign the search engine spiders to any user group therefore giving them access to spider the forums even though guests could not read it.

Anyone?

[Steve Machol]

There is currently no built-in function to do this. This requires modifying the code. We cannot officially support code modifications or forums running modified code, however you can try searching or asking for help with this over at www.vbulletin.org.

[Floris]

I would like to know this also, having just converted from Invision to VBulletin.

With Invision you could assign the search engine spiders to any user group therefore giving them access to spider the forums even though guests could not read it.

Anyone?

I consider this to be a security/privacy issue. With FireFox I can set my useragent to say Googlebot, giving me access to these usergroups/forums.

[treasureman]

Since the information would be spidered and available in Google anyway, it wouldn't matter. It isn't like the information is sensitive or private, if it was I wouldn't want it to be spidered.

I am just looking for a way to have Google spider the forums, but not allow guests to view the post content.

Invision Board has this feature and when I used it I got double the registrations than when my board was open for guests to read. Double the registrations means more money for me as registered users tend to come back more than non-registered users and getting people to come back is what it is all about. The more they come back the more likely they are to buy my other (paying) services.

If someone wanted to go to the trouble to set their browser to useragent googlebot, more power to them, again, the information isn't sensitive or private. I think that 99.9% of my visitors wouldn't know how to do that, or go to the trouble to do it anyway. It would be easier to just register.

That is about the only feature I miss from Invisionboard. VBulletin is far and away better than Invisionboard over all and I am glad I converted after using Invision for over 2 years and I would never go back to Invision.

But I do miss that feature because it made for more registrations. The feature allowed you to assign the search engines to have the same access rights as whatever group you decided to assign them to. I used that feature for 2 years as did thousands of other Invision users and never had a problem with it, and all my forums were spidered and listed in Google and other search engines.

[John Diver]

I agree with treasureman 110%...

I have lost registrations because of this...I have an IPB forum I was considering changing to another vb license but this is the only thing holding me back, its a large forum so I dont want to lose anything.

[treasureman]

I agree with treasureman 110%...

I have lost registrations because of this...I have an IPB forum I was considering changing to another vb license but this is the only thing holding me back, its a large forum so I dont want to lose anything.

I think I am going to make an official request for this feature.

Every time I mention it in the forums the Vbulletin people are negative toward it, but it is one feature I really miss from Invision. I see it requested over and over again, and I think VB needs to implement it if people want it. The users could then decide whether or not to use it, just like any other feature.

It would be an easy feature to incorporate, just assign the useragent of the search engines to whatever group you wanted.

I ran my Invision board for over 2 years with this feature enabled where I assigned the search engines to the registered user group, but had the guest group to where they couldn't read the messages. I did very well in the search engines, and attracted double the registrations than before I had it set up that way.

[Marco van Herwaarden]

It would also be against the TOS of any major search engine, and it would be considered cloaking. If they catch you with it, your site will be removed from their listings.

[treasureman]

It would also be against the TOS of any major search engine, and it would be considered cloaking. If they catch you with it, your site will be removed from their listings.

Again, I used this feature for over 2 years, as do thousands of Invision board users, with no problems!

I really love Vbulletin, but Invision has you beat on this one feature.

Why not implement it and let the user decide whether or not to use it?

[John Diver]

It would also be against the TOS of any major search engine, and it would be considered cloaking. If they catch you with it, your site will be removed from their listings.

I really doubt any site would get banned for it....I don't know of any IPB board banned and there probably hundreds of thousands of IPB forums using the feature, its one of the best selling points of IPB to me..

[Marco van Herwaarden]

I did a Google search for "site banned from google", below are some result i found on teh first page. Some maybe not exactly the same as what you want, but it will help showing the general picture:
- http://blog.outer-court.com/archive/2006-02-04-n60.html

- http://www.bigoakinc.com/seo-article...-by-google.php

2. Cloaking (A Big Google No-No)

Straight from Google's website: "The term "cloaking" is used to describe a website that returns altered web pages to search engines crawling the site. In other words, the web server is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they'll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking to distort their search rankings."
If your website or web pages are set up to display different information for a search engine spider versus a real person, then you are cloaking. Cloaking delivers one version of a page to an Internet user and a different version to a search engine. The cloaked page is packed with keyword and terms that the site wants to be highly rank for so, in essence, they are cheating. There are good reasons for cloaking as well, such as targeted advertising, but if you are trying to manipulate your rankings you should put an end to this immediately.

If you do a search you will find many sites get banned for cloaking. Google (for example) not only uses the known spider, but also sometimes sent a single bot that is not recognisble as a spider to urls listed. If the result is different (not 1 time, but many times) it will be assumed you are cloaking.

PS A lot on this topic can be found at http://forums.searchenginewatch.com/

[treasureman]

Well it seems that Vbulletin has it's mind made up about this. All I am asking is for you to take another look at it and not be closed minded about it just because you decided in the past that it was not something you wanted to do.

Look at this statement you quoted from Google: "Cloaking delivers one version of a page to an Internet user and a different version to a search engine. The cloaked page is packed with keyword and terms that the site wants to be highly rank for so, in essence, they are cheating."

That is NOT what I want! We are not showing different content, or a page packed with keywords and terms just to rank highly.

We are showing the SAME content, just that the search engines can get deeper into the site. The content is the SAME. When someone registers they will see the SAME content. The unregistered user will see the SAME content, just that they are limited as to how far into the forum they can get.

Cloaking is when someone has two different home pages, one for search engines, and one for users. We have the SAME home page for both! It is not misleading when the users register they will be able to read exactly what the search engines read.

If Google banned you for changing content on a site then when they sent their different spiders out as you described and you had rearranged or made chagnes on your site then you would instantly be banned.

Again, I HAVE used this feature for years with problems!

[Marco van Herwaarden]

First of all what i post is not the opinion of Jelsoft or the developers. It seems (from what i read) that they don't believe in adding features that could open potential security holes or get the users of vBulletin penalised by Search Engines. They have also stated that if individual board owners want this functionality, it would be relative simple to add it by the means of plugins, and they have showed you where you can ask support in developing such a plugin. They just don't believe that this should be standard functionality to all their customers.

Now back to your reply:

That is NOT what I want! We are not showing different content, or a page packed with keywords and terms just to rank highly.

You ARE showing different content. The text posted in posts on your forum is your content, not the framework of the forums, but the actual post.

By showing different thread/post text (ie. no permission error, or all the text) you are showing different content to an Internet user (this is anybody browsing your site, member or not) and to the search spider.

That you have used this for years without problems, only means that you have been lucky in my opinion. But ofcourse you can have your own view on this.