[RELEASE] vbSpiderFriend - Search Engine Friendliness

[Overgrow]

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[Overgrow]

...

VERSION 1.0

NEW VERSION, May 21 9:38 am PST

Upgrading: Unzip and copy your variables from the top of the old index.php to the new index.php.


FIXES:

v1.1b release

-That forwarding scheme added in 1.0b is considered 'cloaking' by Google so it has been turned off by default. If you wish to enable it, make $refresh=1 in the top options.

-Added a new link at the top saying that this is the text-only version, click for the real thing. Idea by robertusss.

v1.0b release:

-Added a forwarding scheme. If the archive is spidered they will see the search-friendly version of the thread. If a user clicks onto that page from a search engine, they will be automatically forwarded to the real showthread.php. This is done with the REFERER tag.

-Added the top link to the footer as well for more keyword density

-Fixed all minor bugs

v0.1a:

-Made the private forums actually hide themselves

-Made the query string parser more flexible to work on any install location

-Added $privateForums variable so they will not be shown

-Added header("Status: 200 OK"); for the really picky engines

[Overgrow]

View an online example:

http://www.overgrow.com/edge/archive

[eva2000]

woah nicely laid out output too

[Overgrow]

I know you must be piling on the sarcasm.. cracking on my UI like that! Look how flexible it is-- you can change the spacing to however many nbsp's you want! hehe

This is never meant to be viewed by a user except when they click through the search engine to the plain-text thread. Then the point is to get them to click on one of the top links to bring them into your real forum system. Before I launch this on my site I am definitely going to "pretty it up" but I figured I'd release the raw code and let everyone else have at it as well.

p.s. my site is damn slow today, I'm sure when installed on a real system it won't be so slow as that example.

[eva2000]

strange i just installed this on my vb 2 rc2 forum and it loads with all forums but no threads are shown when i click on any of the forum links/dates ?

i'd post the url but my private forums are revealed as well

is this meant to be set to mysql

var $CONN = "";

[Overgrow]

Does it tell you "No posts, please go back" ?

Does it give you a totally blank screen?

or does it at least show what forum you are in?

This change is just for eva to troubleshoot... FIND

echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

	$query = "SELECT title,threadid,lastpost FROM thread 
WHERE lastpost > '$ts1' AND lastpost < '$ts2' 
AND forumid='$forumID' ORDER BY dateline ASC";

change to

echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

	$query = "SELECT title,threadid,lastpost FROM thread 
WHERE lastpost > '$ts1' AND lastpost < '$ts2' 
AND forumid='$forumID' ORDER BY dateline ASC";

echo $query;

Then when it spits the query out, run it in phpmyadmin and see if it is a valid query.

[eva2000]

it outputs the 'No posts, please go back' message

[eva2000]

i ran this in phpmyadmin and it got mysql query error

SELECT title,threadid,lastpost FROM thread WHERE lastpost > '1136102400' AND lastpost < '1136620800' AND forumid='2000' ORDER BY dateline ASC

must be the forumid ? i only have 94 forums

[Overgrow]

It's just parsing the query string incorrectly, using the year instead of the forum. If you're having trouble like eva, change this:

            $forumID=$urlArray[3];
	$year=$urlArray[4];
	$month=$urlArray[5];
	$week=$urlArray[6];
	$threadID=$urlArray[7];

to this

            $a=0;

	while($urlArray[$a] != "archive") {
		$a++;
	}

	$forumID=$urlArray[($a+1)];
	$year=$urlArray[($a+2)];
	$month=$urlArray[($a+3)];
	$week=$urlArray[($a+4)];
	$threadID=$urlArray[($a+5)];

[eva2000]

great works perfectly now

one last thing.. how do i prevent private forums from being listed/outputted and indexed ?

thanks much appreciated

[Chris Schreiber]

Excellent and easy to install hack, thanks

I even liked the URL submission tool you linked to!

[Overgrow]

There ya go.. that is an important addition, eh?

There's a new zip file up top yonder.. the index.php has a new variable that holds your private forumid's.

[Brian]

One suggestion would be to show a suffix (.php , .shtml etc) so that is dynamic so the search engines know to spider it slower than if it was html. If they assume its html they might take on a tone at once as doing that to normal html files should be fine but this is actually doing all the db calls etc.

Just a suggestion but this is very nice!!

-Brian

[Streicher]

I have tested your hack and my private forums are not hidden.

And also some thread are not found, by the index.php

When i click on some forumlinks the forumpage is simply reloaded.