[howto] Make your forum search-engine spiderable

[filburt1]

Mods, I'm posting this here because it doesn't really seem to fit in any of the other forums. I'm hoping it can be moved to vB FAQ.

I got this idea from fastforward's hack at vB.org. This does not modify your actual PHP files though. The only downside of this hack is that WOL will just show a bunch of Unknown Location links when users are viewing a forum or viewing a thread. All other aspects of WOL will still function.

Also different to fastfoward's release is that (1) this one works (I never got his working successfully), and (2) broken images are a nonissue as this doesn't emulate directories, just a ton of flat files.

Requirements: Apache with mod_rewrite enabled (if your phpinfo page contains the text "mod_rewrite" then it's installed)
Time required: 20 minutes, plus backup time

If you mess up you can grandly break your forums! Do a database backup first, and if it doesn't work, revert the backup.

1. Turn your forums off in the admin CP
2. Back up your database (seriously)
3. Download the attached .htaccess file. Please don't quote its contents; I don't want unlicensed users to be able to see its contents.
4. Upload it to your forums' root directory. If you already have a .htaccess file there, just append this one's contents to the existing one.
5. In the templates forumhome_forumbit_level1_nopost, forumhome_forumbit_level1_post, forumhome_forumbit_level2_nopost, forumhome_forumbit_level2_post, forumdisplay_forumbit_level1_nopost, forumdisplay_forumbit_level1_post, forumdisplay_forumbit_level2_nopost, and forumdisplay_forumbit_level2_post (eight in total), replace:

<a href="forumdisplay.php?s=$session[sessionhash]&forumid=$forum[forumid]">

With:

<a href="$bburl/forumid$forum[forumid]session$flathash.html">

7. In forumdisplaybit, replace:

<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]">

with:

<a href="$bburl/threadid$thread[threadid]session$flathash.html">

8. In your phpinclude template, add the following:

$useflathashes = 0; // Change to 1 to turn on
if ($useflathashes)
{
    $flathash = $session[sessionhash];
} 

This disables sessionhashes in the flat URLs. This will cause problems for a few people; namely those behind a proxy, but Google won't index your site unless sessionhashes are gone.
9. Reopen your forums in the admin CP. If all goes well, links like showthread.php?s=sessionhash&threadid=threadid will be equivalent to threadidthreaidsessionsessionhash.html. Note that these .html files don't actually exist! They are effectively aliases to the proper pages.
10. Submit your forums to Google and wait for Mr. Bot to come-a-knocking!

[Erwin]

Re: [howto] Make your forum search-engine spiderable

I actually run an extended version of this that encompasses my whole site which involves hacking most of my PHP files. I have removed sessionhash altogether though. My site appears to work fine without it. I'll PM you the URL to my site for you to have a look.

Edited: Thanks Filburt for answering my question.

[filburt1]

Replied.

WOOHOO!

Mr. Googlebot is back and this time going into the individual forums!

[SkuZZy]

Nice hack or "template modification" filburt... very nice indeed! I will see about installing it on my forums later on today

[filburt1]

Updated to include a way to turn off sessionhashes for the flat URLs

[Thomas P]

Hi filburt,

that sounds really good, since I tried to get fastforwards hack running, too - no chance.

I have a notice from my hoster, who says I need RewriteCond(itions) - without that conditions the RewriteEngine won't work with their servers.
So... do you have the appropriate RewriteCond for your Rewrites, so that I can use this?
Thanks,
-Tom

[filburt1]

It took me two hours to learn regexs and how to use mod_rewrite, so I can't say that I know how to use RewriteCond yet

[Thomas P]

Okay, fair enough
Will see, if I can get it working...
thx,
-Tom

[nuno]

Re: [howto] Make your forum search-engine spiderable

Originally posted by filburt1
The only downside of this hack is that WOL will just show a bunch of Unknown Location links when users are viewing a forum or viewing a thread.

Originally posted by fastforward at vB.org
easypeasy
In online.php : Change the 'default' block of the CASE statement on or around line 626 to look like this:

default:
    if (preg_match ("/t([0-9]+)\.html/", $filename, $match)) {
       $userinfo[threadid] = $match[1];
       $userinfo[activity] = 'showthread';
       $threadids .= ",$userinfo[threadid]";
    } elseif (preg_match ("/f([0-9]+)/", $filename, $match)) {
       $forumid = $match[1];
       $forumids .= ",$forumid";
       $userinfo[activity] = 'forumdisplay';
       $userinfo[forumid] = $forumid;
    } elseif (preg_match ("/robots\.txt/", $filename)) {
       $userinfo[activity] = 'spider';
    } else {
      $userinfo[activity] = 'unknown';
    }
  } 

[filburt1]

Ah, but that would be hacking

[nuno]

Oops! Busted!

[BigCheeze]

Originally posted by filburt1
Updated to include a way to turn off sessionhashes for the flat URLs

By doing so, will this allow users behind Proxy servers to visit the site properly?

I really want to install this update, but I'm worried that at least a good % of my possible users will get lock out becasue they come from behind a proxy server (i.e. bussiness users)

Thanks!!!!!!!!!!!!!!

[filburt1]

You'll have to turn on $useflathashes to 1 but it's iffy as to whether bots will crawl the site.

[nocturnix]

Maybe the VB Team should get clever and include this option in vB3, or even better, figure out a way to make it crawlable without causing the drawbacks! ::drools:: would be so sweet.

[SkuZZy]

Originally posted by nocturnix
Maybe the VB Team should get clever and include this option in vB3, or even better, figure out a way to make it crawlable without causing the drawbacks! ::drools:: would be so sweet.

There is a way to make it crawlable without "drawbacks". It's called mod_Rewrite. But the vb developers don't want to use it cause it doesn't work on all servers (only apache). I've been considering releasing a complete spiderable hack (or template mod) for vbulletin v2, but since vb3 is almost here, i've decided to wait. Once vb3 comes out i'll see about doing it and releasing it