[RELEASE] vbSpiderFriend - Search Engine Friendliness

[Overgrow]

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit (http://128.121.225.20/submit/index.html) /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[Overgrow]

...

VERSION 1.0

NEW VERSION, May 21 9:38 am PST

Upgrading: Unzip and copy your variables from the top of the old index.php to the new index.php.


FIXES:

v1.1b release

-That forwarding scheme added in 1.0b is considered 'cloaking' by Google so it has been turned off by default. If you wish to enable it, make $refresh=1 in the top options.

-Added a new link at the top saying that this is the text-only version, click for the real thing. Idea by robertusss.

v1.0b release:

-Added a forwarding scheme. If the archive is spidered they will see the search-friendly version of the thread. If a user clicks onto that page from a search engine, they will be automatically forwarded to the real showthread.php. This is done with the REFERER tag.

-Added the top link to the footer as well for more keyword density

-Fixed all minor bugs

v0.1a:

-Made the private forums actually hide themselves

-Made the query string parser more flexible to work on any install location

-Added $privateForums variable so they will not be shown

-Added header("Status: 200 OK"); for the really picky engines

[Overgrow]

View an online example:

http://www.overgrow.com/edge/archive

[eva2000]

woah nicely laid out output too http://vbulletin.com/forum/images/icons/icon14.gif

[Overgrow]

I know you must be piling on the sarcasm.. cracking on my UI like that! :) Look how flexible it is-- you can change the spacing to however many nbsp's you want! hehe

This is never meant to be viewed by a user except when they click through the search engine to the plain-text thread. Then the point is to get them to click on one of the top links to bring them into your real forum system. Before I launch this on my site I am definitely going to "pretty it up" but I figured I'd release the raw code and let everyone else have at it as well.

p.s. my site is damn slow today, I'm sure when installed on a real system it won't be so slow as that example.

[eva2000]

strange i just installed this on my vb 2 rc2 forum and it loads with all forums but no threads are shown when i click on any of the forum links/dates ?

i'd post the url but my private forums are revealed as well

is this meant to be set to mysql

var $CONN = "";

[Overgrow]

Does it tell you "No posts, please go back" ?

Does it give you a totally blank screen?

or does it at least show what forum you are in?

This change is just for eva to troubleshoot... FIND


echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

$query = "SELECT title,threadid,lastpost FROM thread
WHERE lastpost > '$ts1' AND lastpost < '$ts2'
AND forumid='$forumID' ORDER BY dateline ASC";


change to


echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

$query = "SELECT title,threadid,lastpost FROM thread
WHERE lastpost > '$ts1' AND lastpost < '$ts2'
AND forumid='$forumID' ORDER BY dateline ASC";

echo $query;


Then when it spits the query out, run it in phpmyadmin and see if it is a valid query.

[eva2000]

it outputs the 'No posts, please go back' message

[eva2000]

i ran this in phpmyadmin and it got mysql query error


SELECT title,threadid,lastpost FROM thread WHERE lastpost > '1136102400' AND lastpost < '1136620800' AND forumid='2000' ORDER BY dateline ASC

must be the forumid ? i only have 94 forums

[Overgrow]

It's just parsing the query string incorrectly, using the year instead of the forum. If you're having trouble like eva, change this:


$forumID=$urlArray[3];
$year=$urlArray[4];
$month=$urlArray[5];
$week=$urlArray[6];
$threadID=$urlArray[7];


to this


$a=0;

while($urlArray[$a] != "archive") {
$a++;
}

$forumID=$urlArray[($a+1)];
$year=$urlArray[($a+2)];
$month=$urlArray[($a+3)];
$week=$urlArray[($a+4)];
$threadID=$urlArray[($a+5)];

[eva2000]

great works perfectly now

one last thing.. how do i prevent private forums from being listed/outputted and indexed ? :D

thanks much appreciated :)

[Chris Schreiber]

Excellent and easy to install hack, thanks :)

I even liked the URL submission tool you linked to!

[Overgrow]

There ya go.. that is an important addition, eh? :)

There's a new zip file up top yonder.. the index.php has a new variable that holds your private forumid's.

[Brian]

One suggestion would be to show a suffix (.php , .shtml etc) so that is dynamic so the search engines know to spider it slower than if it was html. If they assume its html they might take on a tone at once as doing that to normal html files should be fine but this is actually doing all the db calls etc.

Just a suggestion but this is very nice!!

-Brian

[Streicher]

I have tested your hack and my private forums are not hidden.

And also some thread are not found, by the index.php

When i click on some forumlinks the forumpage is simply reloaded.

[Overgrow]

Thanks guys,

Streicher, you need to specify which forums are private at the top of index.php. Look for

$privateForums="|17|18|";

and put whatever forums in there that should be hidden. Enclose them with |pipes|. Anyone else having these same problems with reloading, etc?


Brian, my main target here is google. It does fine on other parts of my site where I do these type of query strings. If you do find that some engines are hesitant, let me know or make the change and tell us what you did.

[Stephan Whelan]

Overgrow,

I'm interested in the hack but can't seem to get it to work.

I've followed your instructions but am getting an Error 500 Internal Server Error when I try and run the script.

I'm running on a Cobalt RAQ4i so I don't know if there is a problem with the .htaccess file and the way the RAQ handles it.

Any ideas?

[NickyNet]

hi..

great hack.. :)

but not working for me..

http://www.nicky.net/foren/archiv/ :(

[Overgrow]

Nicky:

Here is your error

"The requested URL /foren/archiv/11 was not found on this server.
Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request."

That bottom line is the key.. it says your .htaccess is not pointing to the proper file. Edit the .htaccess and play around with it until you get it right. Sometimes the link has to be ../relative sometimes you can make it absolute /foren/archiv/index.php


Stephan can you give me a link? I use a Raq4i as well and it works for me. I did have to change the access.conf and put Allow Override ALL in there so htaccess could control the errordocument.

[Stephan Whelan]

Originally posted by Overgrow
Stephan can you give me a link? I use a Raq4i as well and it works for me. I did have to change the access.conf and put Allow Override ALL in there so htaccess could control the errordocument.

http://forums.deeperblue.net/archive/index.php

[Streicher]

Originally posted by Overgrow
Thanks guys,

Streicher, you need to specify which forums are private at the top of index.php. Look for

$privateForums="|17|18|";

and put whatever forums in there that should be hidden. Enclose them with |pipes|. Anyone else having these same problems with reloading, etc?

I know that and i have done it already. But it does not work.

[eva2000]

Originally posted by Streicher


I know that and i have done it already. But it does not work. same here some private forums disappeared but some didn't :(

[KeithMcL]

Just finished installing this hack but am having some problems.

I'm getting the error:

"Fatal error: Call to unsupported or undefined function htmlheader() in /home/keith/webdevforums-www/archive/index.php on line 76"

where line 70-78 reads:
header("Status: 200 OK");

//echo "<br>&nbsp;<br>$forumID - $year - $month - $week - $threadID";

if (empty($forumID)) {
htmlHeader();
forumList();
exit;

You can see the page at http://www.webdevforums.com/archive/

[Overgrow]

Sorry... please forgive my sloppy coding. I'm running a fever and worshipping the porcelain god. You can either replace the forumList function with that below, or download the new zip:


function forumList() {

global $db,$baseURL,$privateForums;

echo "Archives<br>";

$privateForums=preg_replace("/^\|/","",$privateForums);
$privateForums=preg_replace("/\|$/","",$privateForums);

$pfs = explode("|",$privateForums);
$whereclause="";
$wherecounter=0;


while($pf=array_shift($pfs)) {
if(ereg("[0-9]",$pf)) {

if($wherecounter==0) {
$whereclause= " WHERE forumid!='$pf'";
$wherecounter++;
} else {
$whereclause .= " AND forumid!='$pf'";
}
}
}



$query = "SELECT title,forumid FROM forum$whereclause ORDER BY forumid ASC";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$forum ) = each ($data)) {

echo spacer(1)."<a href=\"$baseURL/$forum[forumid]\">$forum[title]</a><br>";
}
}

}

[Overgrow]

No clue, coolKeith... Why is your server not capitalizing the function name? See in the code it is htmlHeader() and your server says htmlheader()... Look down in the very bottom of the index.php and you will see the function htmlHeader()... it is there and working. All I can say is try and download it again. If that still doesn't work, change the function name to lower case and see if that helps.

[Streicher]

Thanks Overgrow. It works fine.

And i have find out that only forums with id <10 (1-9) makes a reload of index.php, when clicking on the link.

[Peteruk]

Originally posted by Overgrow

Anyone else having these same problems with reloading, etc?
.

I have sorted it now it was a problem within .htaccess

[Peteruk]

Just one problem (I hope :) ) it all loads up ok no errors or anything but when you click on a forum it loads up but where it should look like below

May
week1
week2
week3
week4

It just displays a page like below

Archives
Competition
2001




vbSpiderFriend by ~shabang~ of Overgrow

any idea what the problem is no doubt something I've done :)

[Peteruk]

Here I go again I have sorted the above and the problem was the date is in MM/DD/YYYY I thought I saw DD/MM/YYYY :rolleyes: :p

[Overgrow]

Sorry.. typical N.American-centric time code eh! But that's the way that the php function strtotime is written...

I'm glad people have this working. Now let's hear the success stories in a few months when spiders actually list our posts.

[eiko]

Having read this thread, I'm getting the same error:

"Fatal error: Call to unsupported or undefined function htmlheader() in /home/user/domain-www/forums3/archive/index.php on line 77"

I downloaded the latest zip as well. Any thoughts?

Also, the question was asked but I didn't see the answer?
In class.mysql.php, at the top, what is "conn" supposed to be set to?

Thanks

[Peteruk]

lol overgrow :)

Come to daddy you cybersurfin beauties

Top class site by the way love it, the design is top draw, and many thanks for this hack much appreciated.

[Overgrow]

Thanks peteruk :p

To the htmlheader() folks, are you by chance using PHP3? The reason I ask is because I define my functions last, but PHP3 needs them defined first. If so, try this version... I bet it will give an array_pop error or something since that's not a PHP3 function but that will tell us what's wrong...

ps. conn should be left blank

[eiko]

tes to php3

the error:
"Parse error: parse error in /home/user/domain-www/forums3/archive/index.php on line 284"

Thanks for the effort!

[JenniferS]

the above error can be fixed by replacing "=" with "echo"

unfortunately, i don't know what to do about:

Fatal error: Call to unsupported or undefined function array_shift() in /home/kdpublis/kdpublish-www/ssoda/archive/index.php on line 247


such a cool hack! think i'll change my host so i can have php4 already...

[Dontom]

Cool hack,
hasselfree installation
:D
Tom

[veedee]

Heh !

great hack !

Looks like Google have caught on and are ready for a bit of mass spidering :D:D:D

see pic !

[supernut]

would be nice if there is away to add adverts to the top and bottom of the created pages

[Overgrow]

Simple!

After this:

$pagetitle="Grow Marijuana @ Overgrow";


Insert this:

$header=""; // OPTIONAL header to insert at the top of all pages
$footer=""; // OPTIONAL footer to insert at the bottom of all pages



After this:

} else {
showThread();
}


Insert this:

echo $footer;



and one more

After this:

<BODY bgcolor="#ffffff">


Insert this:

<?=$header?>



These changes have been reflected in this file if you want to download it and change the other variables again ----> next post

[Overgrow]

p.s. Funny google gif! I thought you were going to show us how your board was somehow listed already. :rolleyes:

[veedee]

Originally posted by Overgrow
p.s. Funny google gif! I thought you were going to show us how your board was somehow listed already. :rolleyes:

No mate, i have too much free and time i like to have a laugh :)

I will install this hack when i

a) learn to do it
b) stop pissing about

good luck in your future hacks :)

[Limey]

this is an incredible hack...thank you shabang!

[Overgrow]

I know the formatting looks really plain but it is really simple to customize. At the bottom of the script, look for

function HTMLHeader() {

and in there you will see normal HTML. Edit it with colors, a style sheet for fonts, etc, however you want. If you look through the script you will find loops that print out normal HTML if you want to change the background color of rows, etc...

It's intended to create simple doorway pages that will rank very high on search engines-- hence the way it dynamically pulls post title into the metas (one reason phpbuilder's pages rank so high). The pages have very little text before the meat of the content for high keyword prominence.

Be sure to add your main site keywords into the options at the top for better ranking overall. Somehow I expect some of my forum posts to rank higher than my normal site because of the search-engine-friendliness of the pages and the number of times keywords are repeated in long threads.

[GMTalk]

Overgrow. You asked if anyone was having the reload error.

I like the concept of this hack and have installed it on my server. I get the list of the forums. The private forums are hidden and all. When I click one of the listings I get the listing page over again.

Any suggestions?

http://discussions.gmforums.com/archive

TIA for any help

John

[GMTalk]

Got it. Nevermind my last post. I figured it out and it is working very well.

[eiko]

MAN! I'd LOVE to make this hack work :(

Fatal error: Call to unsupported or undefined function htmlheader() in /home/name/dir/forums3/archive/index.php on line 80

anybody had any luck with this?

[Overgrow]

Two options:

1) I continue back-porting this hack to PHP3.

2) You get your host to upgrade to PHP4.

In the ideal world, #2 is the correct choice :D It will help with any other hacks you try and install plus everything, including your VB, will run faster.

[eiko]

Problem is is a dedicated server... it's all up to me, which is unfortunate, big time newbie when it comes to installing new software, packages and the like.

I don't hold much hope for the NOS to do it for me. Patience I guess.

[Limey]

Use this and skip the frontpage extensions instructions.
http://www.linuxnewbie.org/nhf/intel/webserving/a_m_f1.html

[eiko]

Thanks for the link! Good stuff.
I'm wondering about RedHat and "packages" ... will have to look into it, but it seems that it's a matter of uploading a php "package" and executing it. Just begining to read up.

Thanks for the input.

[Gilby]

Originally posted by Overgrow
This is never meant to be viewed by a user except when they click through the search engine to the plain-text thread.

How about doing a redirect for users? So when they click on the link from a search engine like google, the script will redirect them to the vbulletin page that has that thread instead of showing them the page that isn't formatted as much.

[Overgrow]

Oooh yea ya know that thought had crossed my mind.. then I quickly smoked it out of there.

I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?

Help me out on how the spiders will handle this and I will get it implemented quickly.. I think it's a great idea.

Good point also is that it won't matter when you fix that-- the search engines will have your info and if you put a redirect in, all further links in will use it.

[eiko]

Most spiders ignore pages with redirects.

[Gilby]

Originally posted by Overgrow
I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?

I was thinking of a redirect from the script side, it will check out the referrer, and if it is from a link external to the site (such as from a search engine), then it will execute the php code to redirect it, so it'd be:
header("Location:/forums/showthread.php?threadid=1234");

[Overgrow]

Ok... it's easy to detect if the Referer is from your host. Getting the header(location:) code to work is being a bugger! I've used this many times before.. I know the error that happens if you sent previous headers, but using the header(location:) here just fails! So I used both metas and jscript refresh.

Try this index.php and let me know if it works for you before I make it an overall change in the main zip. You won't notice any difference if you follow a link down the tree like a spider. IF you copy and paste out that final thread URL, go to a new site, then paste that URL back in then you should be forwarded to the real thread...

ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal

[Gilby]

Originally posted by Overgrow
ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal

Didn't work. I got:
Warning: Cannot add header information - headers already sent by (output started at
/home/sites/site3/web/forums/archive/index.php:88) in /home/sites/site3/web/forums/archive/index.php on line 316

Also, when I got that error, it redirected me to the real thread, which it's not supposed to. Make sure that when the referrer is blank, that it doesn't redirect. It should only redirect when the referrer is defined and is from an external link.

[Overgrow]

This is the if statement


if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) {


It is checking that the referer is greater than 0 length. This does work for me :( Can you troubleshoot it a bit on your end?

[Gilby]

Originally posted by Overgrow
This is the if statement


if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) {


It is checking that the referer is greater than 0 length. This does work for me :( Can you troubleshoot it a bit on your end?

Looking at that line, I see the problem. strstr() is case sensitive, and I defined my $homeURL to be a different case than what I accessed it from, so you should change that to use stristr() instead.

[Overgrow]

Thanks! I changed to stristr, good point. As soon as you let me know that this works I will make the changes in the main download-zip. That is a new index.zip up there a few posts back (with stristr).

[steven]

This is one awesome hack, the only problem that I am having is that the spider is accessing forums that I have specified it not to. Is there a quick fix for this?

Below is my index.php file


<?

/*

vbSpiderFriend v0.1a by ~shabang~

** Free License: YOU MUST LEAVE THE FOOTER INTACT **

*/

$privateForums="|4|5||6||7||8||9||33|"; // Hidden forumids, enclosed by | pipes
$firstPost="07/04/2000"; // MM/DD/YYYY of your forum's first post
$spacer="&nbsp;&nbsp;"; // The characters or spaces to use as one indent
$forumURL="/v2"; // Base URL of your forum

$homeURL="http://www.baddealings.com"; // The link URL for the top of the page
$homeLink="Baddealings.com - Online Source For Consumer Complaints"; // The link text for the top of the page

$keywords="complaints, complain, consumer complaints, consumer, fraud, scams, consumer fraud, grievances, auctions, electronics, feedback, shopping, buyers/sellers, forums, community, ripped off, chat"; // SET META INFORMATION HERE
$description="Baddealings.com provides an arena for members to complain about a product or service. Search our online database for the lastest information on consumer complaints and scams."; // The script will add to these fields
$pagetitle="Baddealings.com - Online Source For Consumer Complaints"; // with the info from the thread

include("class.mysql.php");



// NOTHING MORE TO EDIT BELOW /////////////////////////////////////////////////////////////




$baseURL=$forumURL."/archive";
$dateSplit = split("/",$firstPost);
$firstMonth = preg_replace("/^0/","",$dateSplit[0]);
$firstYear = $dateSplit[2];
$currentYear=date("Y",time());
$currentMonth=preg_replace("/^0/","",date("m",time()));


// setup the DB connection

$db = new MySQL;

if (!$db->init()) {
echo "no DB connection<br>";
exit;
}



// parse query string

if (ereg("/archive/[0-9]",getenv('REQUEST_URI'))) {

$urlArray=explode("/",getenv('REQUEST_URI')); //split the URL path
$a=0;

while($urlArray[$a] != "archive") {
$a++;
}

$forumID=$urlArray[($a+1)];
$year=$urlArray[($a+2)];
$month=$urlArray[($a+3)];
$week=$urlArray[($a+4)];
$threadID=$urlArray[($a+5)];

if (eregi("[a-z]",$forumID) or $forumID < 1 or strstr($privateForums,"|".$forumID."|"))
$forumID=1;
}

header("Status: 200 OK");

//echo "<br>&nbsp;<br>$forumID - $year - $month - $week - $threadID";

if (empty($forumID)) {
htmlHeader();
forumList();
exit;
} else if (empty($year) or empty($month)) {
htmlHeader();
weekList();
} else if (empty($threadID)) {
htmlHeader();
threadList();
} else {
showThread();
}

echo "<center><font size=1><br>&nbsp;<br>vbSpiderFriend by ~shabang~ of <a href=\"http://www.overgrow.com/\">Overgrow</a></font></center></body></html>";


function showThread() {

global $db,$baseURL,$forumID,$threadID,$homeURL,$homeLink ,$forumURL,$keywords,$description,$pagetitle;

$query = "SELECT thread.title as ttitle,forum.title as ftitle FROM thread LEFT JOIN forum ON thread.forumid=forum.forumid WHERE thread.threadid='$threadID'";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$forum ) = each ($data)) {

$forumline = "Forum: <a href=\"$forumURL/forumdisplay.php?forumid=$forumID\">$forum[ftitle]</a><br>";
$threadline = "Thread: <a href=\"$forumURL/showthread.php?threadid=$threadID\"><b>$forum[ttitle]</a></b><br>";
$threadtitle = $forum[ttitle];
}
} else {
echo $query;
}

$ks = str_replace(" ",",",$threadtitle);

$keywords.=",$ks";
$description.=": $threadtitle";
$pagetitle.=" : $threadtitle";
htmlHeader();

echo "<center><a href=\"$homeURL\"><b>$homeLink</b></a><br>&nbsp;</center><br>";

echo spacer(1).$forumline;
echo spacer(2).$threadline;

$data=$db->select("SELECT post.dateline as dateline,post.postid as postid,post.pagetext as pagetext,post.username as fakename,
post.title as title,post.userid as userid, user.userid as userid,user.username as username,
user.usertitle as usertitle,user.posts as posts
FROM post
LEFT JOIN user ON (user.userid = post.userid)
WHERE post.threadid=$threadID AND visible=1
ORDER BY dateline");

if(!empty($data)) {

echo "<table border=0 cellpadding=10>";

while ( list ( $key,$posts ) = each ($data)) {

$username=$posts[username];

if(empty($username))
$username=$posts[fakename];

echo "<tr><td valign=top>$username</td><td>".nl2br($posts[pagetext])."</td></tr>";
}

echo "</table>";
}

}



function threadList(){

global $db,$baseURL,$forumID,$year,$month,$week;

echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

$query = "SELECT title FROM forum WHERE forumid='$forumID'";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$forum ) = each ($data)) {

echo spacer(1)."<a href=\"$baseURL/$forumID\">$forum[title]</a><br>";
}
}

switch($week) {

case("1"):
$fw = "01";
$lw = "07";
break;
case("2"):
$fw = "08";
$lw = "14";
break;
case("3"):
$fw = "15";
$lw = "21";
break;
case("4"):
$fw = "22";
$lw = "31";
break;

default:
$fw = "01";
$lw = "07";

}

$date1 = "$month/$fw/$year";
$date2 = "$month/$lw/$year";

$ts1 = strtotime("$date1");
$ts2 = strtotime("$date2");

echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

$query = "SELECT title,threadid,lastpost FROM thread WHERE lastpost > '$ts1' AND lastpost < '$ts2' AND forumid='$forumID' ORDER BY dateline ASC";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$thread ) = each ($data)) {

echo spacer(3)."<a href=\"$baseURL/$forumID/$year/$month/$week/$thread[threadid]\">$thread[title]</a><br>";
//echo spacer(3)."<a href=\"http://www.overgrow.com/edge/showthread.php?threadid=$thread[threadid]\">$thread[title]</a><br>";
}
} else {
echo spacer(3)."No posts, please <a href=\"".getenv(HTTP_REFERER)."\">go back</a>.";
}

}

function spacer($n) {

global $spacer;

while($n!=0){
$return.=$spacer;
$n--;
}
return $return;
}

function removeZero($n) {

preg_replace ("/./","",$n);
return $n;
}


function weekList() {

global $db,$baseURL,$forumID,$firstYear,$currentYear,$cur rentMonth,$firstMonth;

$months = array("blank","January","February","March","April","May","June","July","August","September","October","November","December");

echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

$query = "SELECT title FROM forum WHERE forumid='$forumID'";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$forum ) = each ($data)) {

echo spacer(1)."$forum[title]<br>";
}
}

for($y=$firstYear;$y<=$currentYear;$y++) {

echo spacer(2)."$y<br>&nbsp;<br>";

if($y==$firstYear)
$fm=$firstMonth;
else
$fm=1;

if($y==$currentYear)
$lm=($currentMonth + 1);
else
$lm=13;


for($m=$fm;$m<$lm;$m++){

echo spacer(3).$months[$m]."<br>";

for($w=1;$w<5;$w++) {

if (strlen($m) < 2)
$mo="0".$m;
else
$mo=$m;

echo spacer(4)."<a href=\"$baseURL/$forumID/$y/$mo/$w\">week $w</a><br>";
}

echo "&nbsp;<br>";
}
}

}


function forumList() {

global $db,$baseURL,$privateForums;

echo "Archives<br>";

$privateForums=preg_replace("/^\|/","",$privateForums);
$privateForums=preg_replace("/\|$/","",$privateForums);

$pfs = explode("|",$privateForums);
$whereclause="";
$wherecounter=0;


while($pf=array_shift($pfs)) {
if(ereg("[0-9]",$pf)) {

if($wherecounter==0) {
$whereclause= " WHERE forumid!='$pf'";
$wherecounter++;
} else {
$whereclause .= " AND forumid!='$pf'";
}
}
}



$query = "SELECT title,forumid FROM forum$whereclause ORDER BY forumid ASC";

$data = $db->select($query);

if(!empty($data)) {

while ( list ( $key,$forum ) = each ($data)) {

echo spacer(1)."<a href=\"$baseURL/$forum[forumid]\">$forum[title]</a><br>";
}
}
}


function htmlHeader() {

global $keywords,$description,$pagetitle;

?>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE> <?=$pagetitle?> </TITLE>
<META NAME="Generator" CONTENT="EditPlus">
<META NAME="Author" CONTENT="">
<META NAME="Keywords" CONTENT="<?=$keywords?>">
<META NAME="Description" CONTENT="<?=$description?>">

<style type="text/css">

BODY {font: 10pt verdana,arial,sans-serif;}
TD {font: 10pt verdana,arial,sans-serif;}

</style>

</HEAD>

<BODY bgcolor="#ffffff">

<?
}

?>

Thanks Steven

[steven]

It appears as though the spider is only not indexing the first 2 forums that are specified for exclusion, but if there are more than 2 forums specified in the index.php file, then it still spiders the forums that I don't want to be included.

[steven]

Solved my problems, on last question. Does anybody know how often that the vbspiderfriend spiders you site? So let's say I got 100 posts on my site today and get 200 tommorrow, when does the vbspiderfriend spider those new posts?

Thanks
Steven

[Overgrow]

I have a new version that fixes all of the bugs with redirection etc, I'll try and get it uploaded in the morning. It won't require any major changes if you update, just copy over your existing variables.

Glad you got your problem sorted, maybe post the solution too so others that run into it know how to fix it as well.

re: how often does it spider?

vbSpiderFriend is not a spider itself, it is simply a new way to view your forum-- a way that is more friendly for a spider to navigate and archive. The data you see from vbSF is totally fresh and updated all the time since it reads it dynamically from the database.

The question is-- how often do you re-submit your site to the search engines? If you submit it today with 100 posts, it will always have those same 100 posts on a search engine until you re-submit. At that point, the engine spider can crawl all of your new posts. Some spiders automatically return after a certain time, some do not.

[mrlister]

on mine when i click on one of the forums it says this page cannot be found. that's when it goes to something like /archive/14 why does it do this?

[Overgrow]

Your .htaccess ErrorDocument is not pointed correctly. You MAY need to edit your access.conf to allow override all for .htaccess files.

You probably just have to edit the .htaccess and mess around with the file pointer. It needs absolute directory names most likely.

[Overgrow]

EVERYONE: Please download the latest version from the first page of this thread. Thanks!

[mrlister]

i'm using microsoft iss. how would i do it under there? if i select the file as a error thing it shows the source code and nothing else when you click on one of them.

[Overgrow]

Sorry :( No clue on how to do the URL trick under IIS.

[Gilby]

Originally posted by mrlister
i'm using microsoft iss. how would i do it under there? if i select the file as a error thing it shows the source code and nothing else when you click on one of them.

From http://www.microsoft.com/technet/iis/steps.asp

Custom Error Messages

In Apache, you provide custom error messages by editing the Error Document and referring to it by using the command:
ErrorDocument 404 http://www.domain.com/404.html

To customize error messages in IIS 5.0, in the IIS snap-in open Properties for the Web site. On the Custom Errors tab, you'll see the location of the error message files. From here, you can map custom error messages to a file or to a URL on the local server.

[Cyrus]

hi

I would really like to get this thing working but Icant for some reason :(

here is the link to my arschive files

http://66.78.8.167/~thehood/archive/index.php

i keep getting 404 errors

do I need to make archives of the forums or sumthing ?? and if so, how do i make these.

thanx all !!!

[Overgrow]

OK here is the important line on your error screen:

>>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.

That means your .htaccess ErrorDocument is not setup correctly. Try different paths-- absolute not relative normally.

[Cyrus]

hi overgrow

i went to my space and i cant seem to see a .htaccess file in it.
i tried uploading it again, but it doesnt seem to want to uplaoad into there.

any hints ?

[mrlister]

ok i set the 404 error to be at /archive/index.php and now instead of it saying 404 error not found it pretty much just reloads. any ideas?

[Cyrus]

I am still having the same proble, and cannot see a way to resolve it yet.

the addy to it is this :-

http://66.78.8.167/~thehood/archive/index.php

I put this in my .htaccess file which I uploaded to the forum directory:-

ErrorDocument 404 /archive/index.php

I do not see what I am doing wrong :confused:

also, wat is the var $CONN = ""; in index.php ??

Can someone please help me !!!!!!!! ?

Thank you

[Streicher]

The Version Number in the ZIP-File is still 0.1a. I have compare the file with the previous downloaded version and find no fixes.

When i click on a forumlink with an ID <10 the page reloads only.

[Overgrow]

EVERYONE: Please download the newest version on page 1 and make sure it's titled "vbSpiderfriend_v1"... I had some trouble getting the latest version to upload, it seems to be there now. The top of the file will say v1.0b if you have the correct one.

Streicher: Please give a link to the archive-- no one else has said anything about a problem like this with forums < 10.

Cyrus/Lister: $CONN should be left blank.

/archive/index.php is not the full path to the file. It will be something like:

/~thehood/archive/index.php

or

/home/cyrus/archive/index.php

or

/home/wwwdocs/HTML/~cyrus/archive/index.php

etc, etc..

[Cyrus]

hi

i did everything u said, even updated to the latest hack version

but it still refuses to work :(

my htaccess file has thsis in it now

/home/thehood/archive/index.php

which is the FULL path to it.

It doesnt work still 4 me.

it goes to some file like "2" in archive ... but theres nothing there, how is it supposed to open that ???

[Streicher]

@Overgrow:

here is the Link: http://www.studenten-city.de/forum/archiv/

If have updated to Version 1.

[Cyrus]

hmm .....


I think this is related to my problem :-

This uses ErrorDocument and query string parsing to get the variables needed

if it is, can anyone help me with this plsssssssssss ?????????

:)

[Gilby]

Originally posted by Cyrus
http://66.78.8.167/~thehood/archive/index.php

I put this in my .htaccess file which I uploaded to the forum directory:-

ErrorDocument 404 /archive/index.php

I do not see what I am doing wrong :confused:


You need it as
ErrorDocument 404 /~thehood/archive/index.php

[Cyrus]

hi

thanx Gilby, its finalyl been done.

one problem tho, when i go to see posts, it says no new posts although I KNOw there should be.

any hints ?

[robertusss]

On each page you should add a link saying:

this is the text-only lowtech version of this thread.
Click HERE to see this thread with all graphics, options, features and links.

[Overgrow]

After reading that the forwarding-mechanism I'm using would be considered 'cloaking' by Google, I have now uploaded a new version which initially starts with the auto-refresh turned off. If you wish to have it turned on, simply enable $refresh=1;

Also incorporated robertussss' idea since we're no longer auto-refreshing.


Streicher: Sorry :( I have no idea on your problem-- you are the only one who has reported this and I cannot duplicate it.

[Cyrus]

wat about me overgrow ??

what shall "I" do :(

[Overgrow]

Hi Cyrus,

The first thing that I will ask you to do is to restore the footer as it was with my name, copyright, and link to Overgrow. I ask nothing from you when I give out my code except that you leave the footer/links intact-- it says so at the top of the file:

** Free License: YOU MUST LEAVE THE FOOTER INTACT **

Restore that and then we can work see about the problem. Thanks.

[Cyrus]

lol

oops, sorry. I didnt read that !

itll be restored in 5 mins

sorry

:(

[eva2000]

Originally posted by Overgrow
After reading that the forwarding-mechanism I'm using would be considered 'cloaking' by Google, I have now uploaded a new version which initially starts with the auto-refresh turned off. If you wish to have it turned on, simply enable $refresh=1;

Also incorporated robertussss' idea since we're no longer auto-refreshing.


Streicher: Sorry :( I have no idea on your problem-- you are the only one who has reported this and I cannot duplicate it. well i only have the original released you made installed... so what's with the cloaking ?

[Overgrow]

The cloaking applies to 1.0b so if you have the original you can leave it as is. Cloaking refers to giving the spider a different page than you give the user-- no matter how well intentioned you are-- it is a bannable offense on google.

It was a nice idea for a feature but should not have been included as the default, so now it is an option.

I would upgrade to the latest simply because it provides a better link at the top, but there's no real need.

[Overgrow]

Cyrus I'm still looking into your problem, I will probably have a piece of code for you to paste in tomorrow.

[Cyrus]

excellent, thanx

ill be waiting for it.

[Overgrow]

Cyrus, OK here is your problem.. your VB has the month/day reversed from standard American time :rolleyes: hehe...

Find:

$month=$urlArray[($a+3)];
$week=$urlArray[($a+4)];


Replace with:

$week=$urlArray[($a+3)];
$month=$urlArray[($a+4)];


That's the easy fix. It won't have the dates right on the top of some of the pages but it won't affect the spider.

[robertusss]

(removed a stupid question here)

[Cyrus]

hi overgrow,

i made that fix, but i get the same problem :(

what shallI do now :confused:

[Overgrow]

Uhh Cyrus? It does work sometimes...


http://www.t-hood.com/archive/2/2001/02/3

http://www.t-hood.com/archive/2/2001/04/2

http://www.t-hood.com/archive/9/2001/03/3


Sorry, I don't know what to tell you. The reason you have problems is because your date format is not the one the program was intended to work with.. since it reads the date from the query, it is touchy about how it's formatted.

Here is a great chance to learn PHP and figure out how to fix it. Otherwise I would need FTP access to your site so I can try new scripts, there is no other way for me to troubleshoot it.. (besides switching my vb over to the other format but mine is a bit busy to be doing that with)

[Overgrow]

robert, I just looked over the class and it seems like it should work. I'm sorry I wrote it using my personal mysql class file instead of just including VB's.. but at the time I didn't have a copy of 2.0 handy and I didn't want to write to 1.x in case it had changed.

Having said all that, I don't see a problem.. I would re-download the latest version, and type in the info again.. Otherwise it would not be difficult to remove my class stuff and just use VB's if you wanted to edit the script.

[Cyrus]

Hi Overgrow,

I know wat u mean.

But seeing the dates arent correct, can I revert the dates to the what they should be ?? Will it work fine then ?? I do not mind doing that, its fine with me :)

[Overgrow]

No guarantee, but it's a good bet!

[krohn]

http://www.forumoc.com/archive/

not working here either...

.htaccess looks like this
ErrorDocument 404 /home/web/forumoc.com/archive/index.php

[robertusss]

@krohn try:
ErrorDocument 404 /archive/index.php

here is the man-page for apache:

http://httpd.apache.org/docs/mod/core.html#errordocument


But on my site it doesn't work either... maybe there is a switch somewhere in httpd.conf to enable customized error-pages globally...

[jojo85]

Nice hack!!!
Congrats
I love it!

[etones]

Overgrow ... Ya just a god damm GENIUS!

Your hacks are just toooo good man, keep em up..

and is that you with the afro...? If so.. damm your a kewl succer :D

[chrispadfield]

just installed in less than 5 minutes, amazing hack thanks so much

[Overgrow]

Welcome guys... I don't sport the fro no mo... (hey man slap mah fro)


....did anyone get their archive in this last Google update? I was too slow for this one, but I have seen the googlebot back a few hundred times since then so I think he's starting to take the bait.

[Overgrow]

OK! If everyone has left their footer intact (you did read the big licensing agreement at the top, right?) --- then we officially have our FIRST VBULLETIN SPIDERED INTO GOOGLE!

Congrats Streicher of Studenten-city.de!

The one guy that had some of the worst trouble installing this piece-of-hack-- he's the guy that manages to get in Google first! Only a few pages so far but now the spider knows where you live :)

http://www.google.com/search?q=vbspiderfriend

[chrispadfield]

someone's referral stats are going to get pushed up by this in google ;) but you deserve it and that 5 times over.

my footer is of course still there :)

www.ascifi.com/forums/archive/

[chrispadfield]

http://www.alltheweb.com/cgi-bin/search?type=all&query=vbspiderfriend

[jamez]

I can get it to work, but not on a subdomain. Is there away to make it work with subdomains?

[robertusss]

sorry, but I do not get it to work. is there a hhtpd.conf setting I am missing?

I have the .htacces-file in place in /archiv, but i still get the 404-message that is defined in the .htaccess in htdoc-root - doesn't the .htaccess in /archive-directory overwrite the root one?

[Overgrow]

jamez: sorry no clue on subdomains, you need more of a linux expert than me.

robert! didn't know you were still having trouble.. I believe the setting you are looking for will be in access.conf (possibly httpd.conf)... it is called "Allow Override" for .htaccess on a directory level. It is fairly well documented inside of the file. You need to Allow Override All or at least the part about redirects.

hope this helps... I'm sure someone with more apache knowledge can offer better advice.

[robertusss]

Yeah, I found that on apache.org and in httpd.conf and I enabled it. Now my customized error document defined via a .htaccess file in / works fine, but not in /archiv.

anyone here that can help me?!

[Jesse69]

Hi - Im very interested at the script - but cant get at the first Page. - please tell me where to get it ....

Jesse

http://161.58.84.213/forum/showthread.php?s=&threadid=15628
produces errors

[Jesse69]

Ok , seems someone fixed that prob- no I got the Hack , but :
Fatal error: Call to unsupported or undefined function htmlheader() in /homepages/2/d21873614/htdocs/forum/archive/index.php3 on line 69

any hints ?

Jesse

[Overgrow]

Hi Jesse,

You need to upgrade from PHP3 to PHP4. The script will work and everything will be faster and more pleasant.

og

[chrispadfield]

google is being incredibly slow to index us... um..

[Overgrow]

The google update has already happened for last month and the databases appear stabilized across www-www2-www3. Check next month around this time and I'm sure we'll have more VBSFs in there.

[Jesse69]

@ Overgrow

php4 ? No chance for this year at my provider ... :-(

looks like the searchengines will not found us anymore ...

Jesse

[Overgrow]

IMPORTANT MESSAGE FOR SPIDERFRIENDLY FORUMS:

This little piece of Google info came from a great interview about Google's Page Rank and Term Vectors (http://www.webposition.com/mp-current.htm#SEVEN).

...while Google might spider your site, it won't be added to the database until someone else links to it.

I was guilty of this myself with the vbSpiderFriend here. I didn't want this link to be visible to my users-- or they might start demanding a bloat-free text only version of my forum! So I manually submitted my Archive to Google hoping it would be added to the database.

According to that article (and interview with a head Goog) if a website is an island-- with no links leading into it from anywhere-- it will never be added to the database. So! Lesson learned: you must put a link to your archive somewhere on your site that Google normally indexes each month. Also, your Page Rank will of course be higher since it was found and not submitted.

[eva2000]

Originally posted by Overgrow
IMPORTANT MESSAGE FOR SPIDERFRIENDLY FORUMS:

This little piece of Google info came from a great interview about Google's Page Rank and Term Vectors (http://www.webposition.com/mp-current.htm#SEVEN).

...while Google might spider your site, it won't be added to the database until someone else links to it.

I was guilty of this myself with the vbSpiderFriend here. I didn't want this link to be visible to my users-- or they might start demanding a bloat-free text only version of my forum! So I manually submitted my Archive to Google hoping it would be added to the database.

According to that article (and interview with a head Goog) if a website is an island-- with no links leading into it from anywhere-- it will never be added to the database. So! Lesson learned: you must put a link to your archive somewhere on your site that Google normally indexes each month. Also, your Page Rank will of course be higher since it was found and not submitted. hence i linked the my url when i installed your hack to the bottom of my forum pages - i have already been indexed too :)

[Brian]

How could you make this only display the months/weeks when their are posts made.. IE for not popular forums some areas dont get posted often and I would rather the search engine not get the "No posts, please go back.:" phrase so often.

[Overgrow]

Doing this would make the display time unbearable I believe. Maybe not if your forum DB is really small.. but if you want to remove the dead weeks you would have to query each week as it is displayed in the list..... adding many many queries to the week-list.

Good idea, easy to implement, but mind-boggingly slow if it's done.

[Brian]

gotcha make sence

[Brian]

Here is an odd question. If I have more forums private then public it would be easier for me to on the top just list the ones I want displayed. What would I need to do to change it so it only displays the ones I want. I know I could simply put all the ones I dont want on top but I have a ton of the ones I dont want displayed and just a few of the ones I do want displayed.

[Overgrow]

(This is untested) Make your changes to the top variable and...

Find:

if (eregi("[a-z]",$forumID) or $forumID < 1 or
strstr($privateForums,"|".$forumID."|"))


Replace with:

if (eregi("[a-z]",$forumID) or $forumID < 1 or
!strstr($privateForums,"|".$forumID."|"))



Find:

while($pf=array_shift($pfs)) {
if(ereg("[0-9]",$pf)) {

if($wherecounter==0) {
$whereclause= " WHERE forumid!='$pf'";
$wherecounter++;
} else {
$whereclause .= " AND forumid!='$pf'";
}
}
}


Replace with:


while($pf=array_shift($pfs)) {
if(ereg("[0-9]",$pf)) {

if($wherecounter==0) {
$whereclause= " WHERE forumid='$pf'";
$wherecounter++;
} else {
$whereclause .= " AND forumid='$pf'";
}
}
}

[julius]

Overgrow, great hack!

I don't know why, but some threads are not displayed.

If I put a link to the archive, at the bottom of myhomepage.com (not forum index) of the same color of the background, so it is invisible, it would be fine to google?

[GMYachtsman]

This looks great !!

I'd love to get the search engines' notice.

Yesterday I just purchased VBulletin. I am hosted on a server that supports .php, .php3 .phtml, and .php4 (HostRocket). When I went to download from VBulletin, I was faced with this surprise:

Click here to download vBulletin version 2.0.1 with .PHP file extensions (default).

Click here to download vBulletin version 2.0.1 with .PHP3 file extensions.


Well, I chose .PHP3 file extensions, not knowing the differences.

Will your hack work on that version? Should I change to .PHP instead?

[jarvis]

Does anyone have this running under IIS? I have set the custom error 404 to point to my /archive/index.php but all I get from the links generated by the hack is:

a) the unparsed php code of /archive/index.php

or

b) 404 file not found (only if I let IIS do the 404 default redirects)

Ideas?

Thanks!!

[etones]

i's change to the .PHP version, it will be better for you in the future.

[jarvis]

i's change to the .PHP version
PHP version of what? My homepage? My question pertains more to getting this running under IIS which is handling vB fine, but many of the hacks are intended for unix-based systems. I'm just curious if anyone has my configuration?

Thanks!

[GMYachtsman]

Thanks, etones. I replaced the php3 version with php. In addition to installing the php files in place of the php3 file, I had to edit my database to replace all the "php3" references with "php". The results were very good!! Just about all my changes were preserved.

Then I went on to install VBSpiderfriend. It is working fine now, but I had the following problems:

1. I had not realized that VB adds a /~databasename folder into the file path. I was trying to make the path work that was in my own directory (not what actually shows up on the URL itself).

2. I had named my archive folder "webarchive" instead of "archive". Before I fixed (1) above, while my first page list of forums showed up fine, when I clicked through, I got a standard 404 file not found error. After I fixed (1) above I got a file not found error right on the front page (this was a different error template...one I had modified for my own site's use). What I learned: the archive folder must have that exact name.

3. Every time I uploaded it, my .htaccess file would disappear without a trace. But I guessed it was there, since it kept overwriting itself and now it is working (though still invisible).

Next for the customizing, the outside links, better Meta tags, etc (the easy parts).

Thanks, Outgrow, for a beauty.

[Bane]

I have tried every variation I can think of in my .htaccess file and cant seem to get this to work.

Either it will go to http://www.influx9.com/archive/3
or http://www.influx9.com/archive/index.php and just relist

I have tried a lot of variations on the .htaccess file.. even just the http://www.influx9.com/archive/index.php (which will make it reload) any ideas what Im doing wrong? My FTP client puts the file at /home/influx9/www/archive/index.php

[GMYachtsman]

Bane, read my last post above; then open your Vbulletin board and look at the address in the URL box.

http://www.influx9.com/index.php


Then go to your VBulletin Forums Admin Control panel and look at the URL there:

I don't know what it is, but it will be different.


I think that difference may have the clue you need.

I am not familiar with the code you use to change the web pages (i.e., php? action=forums), but it is really cool. Is that or is your directory structure (archive not being in the forums folder) making a problem? Too early for me to tell, though others probably could. But before wondering any more do what I just had to above.

I am far less expert than the others around here, but I hope that this may help.

[chilliboy]

Have a look at this - it may be the 'perfect' alround solution, using a samll script in a customised 404 error page:

http://vbulletin.com/forum/showthread.php?s=&threadid=21723

Here's the key stuff if you can't be bothered to read all:

that 404 is not complex, I think that's the best content management system...a lot of hosts let you do a custom 404, but most of us will have their own server/clients server for their files I think?

now the script I posted here has nothing to do with the 404 thing, the 404 is just a few lines, you just put

header("Status: 200 OK"); before any html, and then you do:

$url = explode("/",$REQUEST_URl);
$page=$url[1];

then you check if that content exists

$content = mysql_query("SELECT content FROM content WHERE page=$page");
if (!$content) header ("HTTP/1.0 404 Not Found"); (wich will override the status: 200 OK)

and then you just put the content there!! I didn't test this, but I'm planning to use this if nobody else knows a better content management system, but I believe this is the way it's done. There are peeps using this, so it does work, just not sure all my code is correct ;D

[Overgrow]

Congrats to Eva of Animeboards who now has his vBulletin spidered by Google! He has a good page rank evidently since Google went very deep on only it's first pass. Animeboards now has 11,700 new pages listed in Google with the spiderFriend!

Also to Deadbodies.org who got their spiderFriend noticed, but without much page rank, the bot didn't travel down the links yet. Hopefully next pass...

julius-- do not use invisible links.. if Google notices this, your site will be banned. Better to use a real link, let a few users see it if they must, but keep your position in Google.

Also another note in case you missed it... do not ORPHAN your archive. It MUST be listed as a link from a page that Google already knows (ie, your home page). If you have an orphan archive and submit that link directly to Google, but the bot never finds it's way there itself, your page will never be listed in the search results. Google hates orphans...


edit: changed Eva's gender to more correctly reflect the actual person :)

[chilliboy]

OverGrow - I haven't really had time to check exactly how you area getting this hack to work but I guess you are doing something with the '404' method I posted two posts up. I did some checking on sitepoint and found some threads by you on how you use this method for most of your site.

Do you think it would be possible to develop your hack further so that these 'perfect' URLs are not only used by search engines but users as well? ie this sort of link is used throughout your vB as standard, and not just an extra trick for getting search engine listed.

It would be really cool if you could use the "<!-- breadcrumb, nav links -->" as the URL eg this post would have a URL like:

vBulletin_Community_Forum/Customising_vBulletin/vBulletin_Code_Hacks/Releases_Version 2.x/vbSpiderFriend_ -_ Search Engine Friendliness/

These would then also be available as for use as dynamic meta keywords.

Cheers

[Overgrow]

oh man dynamic meta keywords inserted as the page name........... BRILLIANT! Google will eat it up...

I dunno when I might get around to implementing this, but I love the idea. Thanks a million, I'll be sure to mention your name if I ever publish the hack.

I do use the 404 trick for the rest of my site including user links... now when people post a link to an article that they've bookmarked, its:

http://www.overgrow.com/article/2/3

(article 2, page 3).. same thing for the FAQ...

I haven't done it with vB yet since it would require a huge overhaul for little benefit (in my eyes). OTOH, Wayne Luke has been modifying the SitePoint forums to do just that. I'm sure a search here or there will find the W.Luke threads on vB urls.

[Overgrow]

ps. after thinking it through a bit, I'm not sure if we can do the vB message title as the final part of the URL... without having to query to match the title back with the threadid when the user requests it. I was actually more excited about using it for my articles and FAQ. It would be easy to build a lookup table for those without having to run a huge query like you would for the vB.

So I don't want to sound to excited about implementing your idea with the spiderFriend.. not sure if it's possible or feasible.. but it triggered a thought of how to do it with the rest of the site.

[julius]

I've disabled html in Vb posts.
But, if someone put some html or javascript tag, with the spyderfriend they will run.
Maybie to prevent this it's better to censor in vB some dangerous words like "javascript"?

I found some threads are not in the list. Any idea?

[ldydvr]

Just wondering ...

Is the zip in the second post the latest updated version of the hack?

I noticed the date read 05-21-2001 and just wanted to make sure before starting.

=-)

[dwh]

Originally posted by Overgrow
Some spiders automatically return after a certain time, some do not.

!! That's the first time I heard this! Which spiders (or major ones anyway) crawl only once?

Nice hack btw.

[JackG]

Will this work on Windows NT ?

Has anyone tired?

[Godfather1]

I have the same problem like Streicher when i click on a forumlink with an ID <10 the page reloads only.
you have an idea why?


cu

[robertusss]

soon after I included this hack on my site, google kicked me out completely... :(
Hope it doesn't have to do anything with this hack...

[chilliboy]

Hi Overgrow,

These are just some general questions as regard to this hack.

As google and a few others can spider ? and & URLs what other advantages does this hack have other than all the dynamic meta tags?

What I'm getting at is - if I can do all the dynamic meta tags etc straight into my vB then removing them from the equation why would this hack improve the ability for search engines to spider.

As far as I can see the only difference would be the / in the URL's as opposed to ? and &'s. Are there any other advantages? Are they easier to spider because 'less clutter' from CSS, images etc?

Just wondering as I'm not that enlightened on exactly how search engines (specifically google) work, and what are all the benifits of this script over just inserting dynamic meta tags straight into your actual forum pages.

Cheers

[eva2000]

Originally posted by Overgrow
Congrats to Eva of Animeboards who now has his vBulletin spidered by Google! He has a good page rank evidently since Google went very deep on only it's first pass. Animeboards now has 11,700 new pages listed in Google with the spiderFriend!

edit: changed Eva's gender to more correctly reflect the actual person :) thanks overgrow, my pagerank boost was from linking to my main site eva2000.com which has a very very high rank :)

unfortunately your hack doesn't seem to show threads anymore? :(

any updates?

[eva2000]

never mind i found my problem, each forum upgrade i start with a new database and perserve the previous version database as a copy but forgot to change the config file for your hack to point to the new db :rolleyes:

[robertusss]

eva2000: did you submit your page to google again, or did google visit you automatically?
If you submitted, did you submit www.yourdomain.com or did you submit www.yourdomain.com/forum/archive/ ?

[eva2000]

Originally posted by robertusss
eva2000: did you submit your page to google again, or did google visit you automatically?
If you submitted, did you submit www.yourdomain.com or did you submit www.yourdomain.com/forum/archive/ ? i submited my main domain page along with a particular archive thread i really want to see indexed and just wait :)

[dwh]

Are you sure you folks aren't putting out 404 error pages?

http://www.phpbuilder.net/annotate/message.php3?id=1000365

[JohnL]

Did anyone ever figure out why some people had a problem where any of the links reloads the index.php page?

Here is an example http://www.reefcentral.com/vbulletin/archive/index.php

I'd love to get this hack to work.

[ForzaGrifo]

What are the major search engines other than Google that this hack would work on?

[MacUser]

The forums with id=1,2,3,4,5,6,7,8,9 reloads the index.php.

http://www.macuser.de/forum/archive/index.php

Some people have the same problem .
Any idea ?

MacUser

[ForzaGrifo]

Originally posted by Overgrow
I am tired of my 200,000 posts not being listed in Google.

I just read in clickz that Google can now crawl dynamically generated pages. Is that true??

the article is here:
http://www.clickz.com/search/opt/article.php/874131

[Alun]

I just read in clickz that Google can now crawl dynamically generated pages. Is that true?? = Yup.

Search Google for "Thorikos" and fifth link down is my forum. I think I've had a few referrals from posts to do with oikoi too.

[MacUser]

Originally posted by MacUser
The forums with id=1,2,3,4,5,6,7,8,9 reloads the index.php.

http://www.macuser.de/forum/archive/index.php

Some people have the same problem .
Any idea ?

MacUser

Someone Please help. :confused:

[JohnL]

Are you using an IIS server also? I am and can not get the links to work properly either. I would really love to get this working.

[Raz Meister]

Small Bug with displaying private forums:

If you have a private forum with id 5 and try acessing page - /archive/5 it won't show the threads BUT if you add a zero to the front and try - /archive/05 it WILL lists the private forum.

To fix find: while($urlArray[$a] != "archive") {
$a++;
}

$forumID=$urlArray[($a+1)];
add below:$forumID = $forumID + 0;

[CoolaShacka]

Thanks, Overgrow.
Realy nice work.

I have a litle Problem. Could you please tell be what's the Bug?

******************************
link deleted
******************************

Thank you in advance

[Raz Meister]

Have you set the .htaccess file?

[CoolaShacka]

Yes.
my .htaccess is ErrorDocument 404 /home/csbvu/public_html/archive/index.php I am hosted by VO

[JohnL]

Is there any way to get this to work on a W2K server with IIS?

I did the redirect for the 404 error but it just reloads the index page. Here is an example http://www.reefcentral.com/vbulletin/archive/index.php

[Raz Meister]

Originally posted by CoolaShacka
Yes.
my .htaccess is ErrorDocument 404 /home/csbvu/public_html/archive/index.php I am hosted by VO

The redirect should be an absolute web address not a unix path!

Replace with:
ErrorDocument 404 /archive/index.php

[Raz Meister]

Originally posted by JohnL
Is there any way to get this to work on a W2K server with IIS?

I did the redirect for the 404 error but it just reloads the index page. Here is an example http://www.reefcentral.com/vbulletin/archive/index.php

This is probably occuring because the PHP file isn't able to get the original URL that was requested.

Create a new index.php and add <? phpinfo(); ?> in it. Now try to cause a 404 and look at the enviromental variables that hold the requested URL. Use that enviromental variable inside the original index.php file.

[JohnL]

THANK YOU...THANK YOU...THANK YOU :)

OK all you IIS guys. Thanks to Razzie...

Do a search for REQUEST_URI and replace it with QUERY_STRING and you are good to go!

Originally posted by Razzie
This is probably occuring because the PHP file isn't able to get the original URL that was requested.

Thanks again Razzie :)

[JohnL]

Originally posted by Razzie
Small Bug with displaying private forums:

Excellent catch! Razzie, you da man, grandma!

[CoolaShacka]

Originally posted by Razzie


The redirect should be an absolute web address not a unix path!

Replace with:
ErrorDocument 404 /archive/index.php
Thank you Man. :D
All works fine now. :)

[Pk]

Originally posted by CoolaShacka

Thank you Man. :D
All works fine now. :)


My god, so many pages, anyway is this hack done now?
If so could someone just post it on the hack release forum? :]

[OmniSlash31]

I just started my forum, but when i go to for exaple: vbb/archive/2 it goes to my error404 page:confused:

[JohnL]

Originally posted by OmniSlash31
I just started my forum, but when i go to for exaple: vbb/archive/2 it goes to my error404 page:confused:

You need to redirect 404 errors to the /archive/index.php file as has been discussed in this thread.

[OmniSlash31]

Ok, i have a folder called "vbb" on my server for the forum.
So, .htaccess should be
ErrorDocument 404 /vbb/archive/index.php, right?

[CoolaShacka]

No.
.htaccess should be
ErrorDocument 404 /archive/index.php, but you have to put the .htaccess in your vbb folder

[Chen]

OmniSlash31 please enter your license information in your profile. Thanks.