Can bots an spriders still roam my forums if they are set to private

[volcom5799]

Can bots an spriders still roam my forums/posts even if they are set to private and only able to be accessed by registered users????

[Tolitz]

As far as I know, spiders and bots have the same permissions as a guest to your forum... so they can only access parts of your forum that a guest can access.

If you check your "who's online" page, you'll see that some spiders are met with the "stop" icon when they try to access parts of your forum that are off-limits to guests, meaning they can only see either a login page or an "access forbidden" message, depending on your forum settings on permissions...

- T

[volcom5799]

yeah i get a lot of those stop signs on my view online members page. i am going to try it without my forums set to private and see what happens over a few days.

[eJM]

Why don't you just set up a robots.txt file for your domain root and prevent the spiders from trying to access pages you don't want them to? Part of mine looks like this:

User-agent: *
Disallow: /403.html
Disallow: /404.html
Disallow: /errordocument.html
Disallow: /showpic.php
Disallow: /community/a-floor-pro-comedy-tour/
Disallow: /community/a-floor-pro-network-support/
Disallow: /community/test-forum/
Disallow: /community/chiefcp/
Disallow: /community/announcement.php
Disallow: /community/attachment.php
Disallow: /community/calendar.php?do=add
Disallow: /community/cron.php
Disallow: /community/editpost.php
Disallow: /community/joinrequests.php
Disallow: /community/login.php
Disallow: /community/memberlist.php?
Disallow: /community/misc.php
Disallow: /community/staffcp/
Disallow: /community/moderator.php
Disallow: /community/newreply.php
Disallow: /community/newthread.php
Disallow: /community/online.php
Disallow: /community/payments.php
Disallow: /community/poll.php
Disallow: /community/postings.php
Disallow: /community/printthread.php
Disallow: /community/private.php
Disallow: /community/profile.php
Disallow: /community/register.php
Disallow: /community/report.php
Disallow: /community/reputation.php
Disallow: /community/sendtofriend.php
Disallow: /community/search.php
Disallow: /community/sendmessage.php
Disallow: /community/showgroups.php
Disallow: /community/showpost.php
Disallow: /community/subscription.php
Disallow: /community/subscriptions.php
Disallow: /community/threadrate.php
Disallow: /community/usercp.php

I also have entries specific to Google and Google Image because they will adhere to directives that other bots won't.

Jim

[ComputerVitals]

I was updating my Adsence profile and seen a spot there, you can set it up with a user and pass (I think thats how it is) so it can acccess the content.

But I dont know if the bots that bring in ads are the same as the ones that spider the site.

[dethfire]

boardreader scans site by logging in

[MrNase]

Do they register manually?

[eJM]

But I dont know if the bots that bring in ads are the same as the ones that spider the site.

No, they are not. Google will punish you if you use any methods to get them to spider content that isn't available to unregistered visitors.

Jim