Archiving

[bira]

Hi,

I did my best to look at other threads before I start a new one, and hopefully I'm not posting a duplicate one here.

I am really interested in an option to archive threads. I DON'T want to prune or delete old threads - they are priceless, and we all mourn to this day the threads that were lost during the UBB prune-every-60-days era.

Currently I have threads running up to March 2000. At some point - probably after, say, March 2001, I will want to archive year old threads - make them into static HTML pages that can be read normally, but cannot be replied to.

Is there anything similar to this included in vB 2.0? Or are there any recommendations on what to do with this?

Thanks,

Bira

[Wayne Luke]

This is a great idea.

[KeithMcL]

Wayne, I noticed on your forums that you have a locked archive forum.

Is this similar to what's being mentioned here? Are you just mass moving old threads to this forum?

I also think that the archiving option is a very good idea.

[Wayne Luke]

Originally posted by KeithMcL
Wayne, I noticed on your forums that you have a locked archive forum.

Is this similar to what's being mentioned here? Are you just mass moving old threads to this forum?

I also think that the archiving option is a very good idea.

It was just a mass move of threads out of the General Chat forum and placed in the forum which was then locked. Only way vBulletin handles it.

I also do not want to prune messages. The postid column supports numbers of over 4 billion. I want to see how close I can get... (yeah it will take awhile).

[Sharg]

Yes great an needed idea we discussed some time ago with not much interest :(

Old threads---> statics would be really excellent.

Benj

[Martin]

yikes...

the conversion process, depending on how many you wanted to convert at a time, could bring even the hardiest of servers to its knees...

[Sharg]

Well if you do incremental 100 posts jump it might be doable....

Benj

[bira]

Martin, I'd rather shut down the BB for a few minutes every few months and do it in 100 posts cycle than prune old thread :(

[Martin]

I think it's a great idea, it's just how to go about implementing it without crashing a server.

I guess it really depends on how many posts you have before you archive and how it's set up to search and find the posts of the criteria you want to archive.

[bira]

Look, I don't know enough about programing/php/MySQL, but this is what I have in mind (and maybe there's a far better way):

1) do a select query for all threadid's of threads that the last post was >x time ago (bb-owner submitted).

2) run a while loop for the above query result, that does:

for each threadid, do:

// read thread and write it to a static html page, named $threadid.html
$file = fopen("./showthread.php?threadid=$threadid", "r+");
$source = fread($file);
$grab = eregi("(*)", $source, $newfile);
$newfilename="$threadid.html";
fopen($newfilename, w+);
fputs($newfilename, $newfile[0]);
fclolse($newfilename);

// delete threadid etc' from database


Or whatever the correct syntax may be.


There's bound to be an easier way to do this, I can imagine. Which is why I am bringing this up in the suggestions forum, and not trying to write a hack myself :)

Also, there are two drawbacks that need to be addressed:

1) How do you include the thread titles in the forum listing, or at least have a listing of them somewhere;
(you can add to the above process another fopen with a+, whereby it appends to the bottom of the file a line with the thread title, linking it to the static html, date last posted and forum name, thus making this page - say, archive.html - the index for all archived threads. However, with hundreds and hundreds of threads, it can be huge!)

2) how do you make these threads searchable along with the rest of the active forums.

Like I said, if I had all the answers I'd have written it myself. Since I don't have enough knowledge to even attempt to answer my own questions, I submit this to the developers of this BB.

Thanks,

Bira





[Edited by bira on 12-08-2000 at 09:46 PM]

[Kier]

There is a way to do this, with a little code-fiddling.

Open showthread.php and go to the end of the document and find this: eval("echo dovars(\"".gettemplate("$usetemplate")."\");");
And replace it with this:
//eval("echo dovars(\"".gettemplate("$usetemplate")."\");");

eval("\$html = dovars(\"".gettemplate("$usetemplate")."\");");

$filename = "vBthread_$threadid.html";
$fp = fopen($filename,"w");
fwrite($fp, $html);
fclose($fp);
Now save this as savethread.php and you're all set. When you want to archive a thread, simply look at the URL for the current showthread.php and replace it with savethread.php - so http://forums.mydomain.com/showthread.php?threadid=101 would become http://forums.mydomain.com/savethread.php?threadid=101

This will cause PHP to write the resulting page to a file called vBthread_101.html (in this case), in the root of the forums folder.

Obviously this is some way from a full solution, but it shows that it can be done...

I'll have another look at it soon and see if I can get a fully operation version, which will save a selection of threads in one operation, and automatically index the saved pages etc.

[bira]

This is encouraging, indeed. I am hoping that, nonetheless, John is reading this and will take it further to be included in the vB2.0 release. However, if he doesn't, I will be interested in a hack very much.

[Richard Tafoya]

I need this as well.

Would it be an option to create a secondary database or a partition that's not queried except for an advanced search option?

I'm thinking along the lines of Deja.com, where search queries default to recent but can also include past and all.

Would board performance be enhanced by splitting out an archive database that is only accessible in this way?

Perhaps this way, threads with a last post older than xx days could be mass-moved into the archive with a maintenance routine, but still be accessible and not locked for response. If someone responds, the thread reverts back to the current database.



[Edited by Richard Tafoya on 12-12-2000 at 10:56 PM]

[bira]

I like the dejanews idea. Hope John is reading this

[SonnetCelestial]

I've been hoping for this all along and I wonder why no one else understood me or supported me back then (Maybe I just didn't express it correctly) but I really hope someone can do this for the next version!

BTW I'm sure john is watching most of the threads! ;)

[SonnetCelestial]

Also I would like to have an option that someone else can store in zips the threads and so forth! :) This is such a wonderful idea!

[Sharg]

Im' for it 100%

HTMLized all thread, but with a non querried database that would allow us to:

1) Search in old threads
2) REbuild html from this database.

Well, maybe not for 2.0 (we should NOT delay this version anymore) but as a future add one.

Benj

[Wayne Luke]

The newer versions of MySQL support compressed read-only tables. You could accomplish most of the archiving features using this format.

Of course the archiving process would be a little more difficult as you would have to copy the read-only table to a writable table, add new archived elements, drop the previous table and rename the new one. However I think this would have minimal impact on overall system resources because you are not going to archive constantly, at most on a daily basis but probably monthly.

This archiving script could be running as a CRON job to make it automatic.

Now what are the perceived benefits of writing all of these threads to HTML files? I can only see negatives in increased diskspace usage and creating a more difficult search process.

[Richard Tafoya]

If a board has a million posts and 800,000 of them are in a read-only archive, wouldn't the board perform more quickly?

I'm assuming that the archive would only be queried in advanced searches. The 800,000 old posts wouldn't factor into the data shown on the main pages showing total threads and posts.

[bira]

I'm bringing this thread back to life in the hope that this will make its way into the 2.1 developing stages :)

[Hooper]

I noticed how old the thread was and wondered how it made its way back up here. Bira was the answer. Yes, would be a superior function and when I finally find a way to integrate my news, articles etc..to the vBforum threads, this function would would be fantastic for permanent archiving. This has to be the most excellent idea I have heard of. Outputting to .html would be great.

Thanks Bira for coming up with this idea.

[Meli]

Has there been any word from the developers as to whether they are even considering something like this?

I have been running a board for about a year and a half (the past few months as vB) that, for all intents and purposes, is running down, due to increased competition from other boards on a similar topic.

Thus, I've been considering switching my board to a new topic, completely unrelated to the old one. But I would still like to have all those old posts available for browsing/searching as well, since there's some great material there.

I suppose I could try to do the co-branding thing, but these two topics are /completely/ unrelated to each other (one is a hobby/fan-type thing, the other is professional/business-related), and I wouldn't want visitors to my 'professional' site to see the other one at all.

I also suppose I could buy a second vB license and pay my hosting service for a second MySQL database (keeping the two completely separate), but I'm not made of $$$. :)

Anyway, this is long, rambling, and confusing, so I'll stop now. Thanks for reading this far. ;)

[brookelyn]

Has anyone come up with an auto-archiving method yet?
My board has tons of threads and I believe it has caused it to get slower and slower.... it's ridiculously slow now. I really don't want to delete any posts, but I'm going to have to unless I can archive them somehow

[Hooper]

I'm not sure if anyone has taken any serious thought to this thread unfortunately. Bira, have you made any progress with accomplishing this in any way?

I wish I could write this hack myself. Would be super. ;)

[cobradude]

This is becoming an important issue for me too. Currently my database is 280mb, and after convincing, my host will allow to 300mb, but that's it, and my site is growing. Obviously I need to do something quickly, but what? Remove all the older, yet very valuable posts? That hurts!

I need a way to have a second database for archived data that my members could still search, but were not part of the every day queries. That would help my host, my site speed up, and help me.

Can someone from Jelsoft respond to this?

[Omero]

Any news here? With a 3 gigs DB, this is a serious issue for me...

[Wayne Luke]

At this time, I do not know of any initiative to develop this feature. However if you need to look into this functionality, there is a section in the MySQL documentation on handling replication and creating backup databases.

[Omero]

Originally posted by wluke
At this time, I do not know of any initiative to develop this feature. However if you need to look into this functionality, there is a section in the MySQL documentation on handling replication and creating backup databases.

What do you mean exactly? Im no mysql expert ;)

[Wayne Luke]

Neither am I... However you can read this section of the MySQL manual to learn more: http://www.mysql.com/documentation/mysql/bychapter/manual_MySQL_Database_Administration.html#Replicat ion

It isn't light reading though. I would actually recommend that everyone read chapter 4 completely.

[werehere]

Wluke,

I use the fake archiving method like you do (ie: mass moved threads to a forum that no one could reply to), but it still allows for searching. Now I could go around that and drop the search index, then rebuild it skipping the threadid=?'s correct? Would that be a quick option for some people to restrict the data from being searched? :)

[SonnetCelestial]

wluke I thought you liked the feature in the beginning?

But the reason admins such as myself would want HTML versions is because it can be stored on any site regardless of database or cgi-bin and people could even download them to their hard drive if I were to delete or prune the forums.

In a way archiving to zipped HTML would take the burden off my hands and into those who cared about those threads.

It's really useful for sites that want to keep some threads or for forums that have become extinct to keep their threads alive.

In addition I run a roleplay forum so threads aren't merely screaming delete me when a story or roleplay is completed.