Folks, I need one solution to my problem: a robot.txt document with the necessary information on it to keep google, yahoo, and other spiders from crawling my site. I don't know what exactly to put in the .txt document, and that's where you come in. I am running RC3, and need a document that will stop spiders from accessing my site altogether. I have John's arcade installed, so I have an arcade.php file included with the rest of the other files.
Any help is appreciated!! I have google adsense. Should I get rid of it to stop google spiders, or can I still generate money while blocking spiders?
Please show me a document to use, and where to put it. Thanks.
Any help is appreciated!! I have google adsense. Should I get rid of it to stop google spiders, or can I still generate money while blocking spiders?
Please show me a document to use, and where to put it. Thanks.
Comment