Search engine spider question

[CCWBumper]

How can I (or can I) restrict the search engine spiders from going into the "Viewing User Profile" area and other areas that they have no business being in? I want them to spider the site, of course, but jeez, they get into everything....

[Steve Machol]

Set 'Can view member info' to 'No' for the Unregistered usergroup.

[CCWBumper]

Thanks for the quick reply, Steve

[CCWBumper]

Set 'Can view member info' to 'No' for the Unregistered usergroup.

I just edited that group and that particular setting was already set to "No". Is there anything else that could be allowing it? Could this be tied, in some way, to the directory permissions?

[silkroad]

Yes, you can add or modify your robots.txt file.

See ours for an example:

https://www.unix.com/robots.txt

Neo

[CCWBumper]

Yes, you can add or modify your robots.txt file.

See ours for an example:

www.unix.com/robots.txt

Neo

Thanks alot, Neo, that had not crossed my mind. I may have created the problem myself since I put in a wide open robots.txt file a couple of weeks ago.

[silkroad]

It would be good if Jelsoft would reorganize the directory structure to put files that should not be, as a general rule, spidered by search bots, in a distinct directory.

Then, for example, we could change the robots.txt file to read:

Disallow: /no-bots

or something like that.....

The current vB structure requires a large robots.txt file, and there is a better way!