How To Get Rid of Duplicate Content - A Solution

[kevinmanphp]

Well... now that the googlebot accepts wildcards in robots.txt... I have put together some disallows that will hopefully solve all of vB's duplicate content penalizations (supplemental index). One or two dupes are no big deal.. but if google gets all the links on your forums like mine... you can get up to 8+ dupes and that is NO GOOD - all go to sup index.

Someone please check this robots.txt entry and let me know if this is safe to use. I am not too hip on the robots exclusions syntax and would appreciate a friendly "looks good to me" for my lines below.

NOTE: I WANT to have showthread.php indexed... just without the params listed below:

User-agent: Googlebot
Disallow: /*goto=
Disallow: /*mode=
Disallow: /*showthread.php?p=
Disallow: /*&pp=
Disallow: /*postcount=
Disallow: /*daysprune=
Disallow: /*&sort=

Any flaws with this? Thanks!

[unitedpunjab]

Just curious,what will showthread.php do without parameters ?

[kevinmanphp]

showthread.php is the file that serves up all the posts in a thread. The problem is that showthread.php accepts a lot of different parameters that can display the same content just in different ways.

[kmike]

Ok I know I'm late to the party...
Not sure how big a flaw it is for you, but with that robots.txt you'll lose the value of 3rd party links matching those lines. And there could be quite a lot of them, people tend to bookmark and re-use some really weird URLs.