Large UBB 6.72 Import

[Levelbest]

Issue:

We are importing from a very large UBB to vbulletin. The operation was moving along fairly well, though slow. We got up to importing forum 10... and it appears the number of posts it imports at on time has changed causing the process to occasionally time out. We had left it at the default, whick was a smaller number, so I am not sure what changed during the import.

Suggestions on how to address this are appreciated. Right now, I upped the CGI timeout to a crazy number, but it would be nice to get it processing smaller amounts of posts again.

I started the import process yesterday around 1 - it's now 24 hours later and because of time outs (that happened while I slept), we are way behind schedule.

Also, is it wise to keep the boards closed till the import process completes?

[Jerry]

We are importing from a very large UBB to vbulletin. The operation was moving along fairly well, though slow.

How many users and posts and what is slow ? I've done a 1.4 million post import in 2 hours from classic.

We got up to importing forum 10... and it appears the number of posts it imports at on time has changed causing the process to occasionally time out.

Do you actually get the "this script has hit the time limit" PHP error ?

We had left it at the default, whick was a smaller number, so I am not sure what changed during the import.

Well unless you edit the source nothing should change.

Suggestions on how to address this are appreciated. Right now, I upped the CGI timeout to a crazy number, but it would be nice to get it processing smaller amounts of posts again.

Up the PHP memory limit as well, for ubb.classic imports I usual set the mem limit to 256M.

I started the import process yesterday around 1 - it's now 24 hours later and because of time outs (that happened while I slept), we are way behind schedule.

Run impex stand alone, i.e. take it out of the forum/ folder and run it in the web root and just set up the ImpExConfig.php completely.

Also, is it wise to keep the boards closed till the import process completes?

Never ever close a board with out having done successful imports first, ever.

My first statment here :

http://www.vbulletin.com/forum/showthread.php?t=116398

Practice the import from a copy of the site.

[Levelbest]

How many users and posts and what is slow ? I've done a 1.4 million post import in 2 hours from classic.

20,000 users, 35 forums, and how may posts exactly I am not sure

Do you actually get the "this script has hit the time limit" PHP error ?

What I was getting was a CGI timeout. So I increased the timeout from 300 secs to 10,000. Things are running again at least.

Well unless you edit the source nothing should change.

we didn't do that.

Up the PHP memory limit as well, for ubb.classic imports I usual set the mem limit to 256M.

in the php ini? right now it's at 16M

Run impex stand alone, i.e. take it out of the forum/ folder and run it in the web root and just set up the ImpExConfig.php completely.

not sure I understand this... if it's back running presently (albeit slow), will this speed things up?

Never ever close a board with out having done successful imports first, ever.

My first statment here :

http://www.vbulletin.com/forum/showthread.php?t=116398

Practice the import from a copy of the site.

Not blaming anyone but myself here... but I didn't want to run our current board with posts adding to it and running an import at the same time. That was my choice.

Thanks for your assistance...

JOHN

[Jerry]

Righty :

in the php ini? right now it's at 16M

Yup, ramp that right up, impex eats memory and the ubb.classic importer is the worst due to the amount of files it has to deal with. if you can go for 256 or thereabouts.

not sure I understand this... if it's back running presently (albeit slow), will this speed things up?

A little yea as there is no vBulletin over head, also it doesn't time out in the admincp so you can leave it for a while.

So, I'd say, do the import on a copy of the site. Open up your classic site. Up the memory as much as you can (pending on how much the server has) and run impex standalone..

[Levelbest]

OK, I indeed opened it up to 256 and stuff is importing and not timing out now.

If I do decide to do it on a copy and open the boards... what is the best/easiest way to flush out the existing imported data / users etc from an import gone awry?

Thanks jerry for your help.

[Jerry]

At the top of every impex page there is a link to :

Database cleanup & restart

Follow that then choose :

Delete Session and all imported data

To delete the import session and all imported data for a clean retry, click here.

[Levelbest]

Thanks... should have been obvious... but sometimes when you are so new to something, things just arent!

[Jerry]

Thanks... should have been obvious... but sometimes when you are so new to something, things just arent!

For sure

Best thing to do is take a copy of your source data and set up a testing and planning area so you can get used to the import, what has to be done afterwards, time it etc.

Then you can plan the migration and have a list of after import tasks (update counters, rebuild search index, etc) so you can say to your users "We are moving and its going to take X hours, brb" because you've tested it a few times and know how long its takes.

Or even do the test and have some mods/admins go over the importer board looking for cleanup, by nature of ubb.classic source and if the board has ever been updated its going to need cleaning up.

[TXXJ]

this probably a stupid question, but when you're importing a UBB forum, where do you need to put the files to copy/import? like what directory structure? I'm currently working on converting from UBB 6.7.1 on a different host. I've read the How-to thread, and the Impex manual (a couple of times), but for the life of me I haven't found any info on where to put the UBB files that are going to be imported, other than I need to have them on the same machine.

[Jerry]

this probably a stupid question, but when you're importing a UBB forum, where do you need to put the files to copy/import? like what directory structure? I'm currently working on converting from UBB 6.7.1 on a different host. I've read the How-to thread, and the Impex manual (a couple of times), but for the life of me I haven't found any info on where to put the UBB files that are going to be imported, other than I need to have them on the same machine.

You just need them on the same server and with their permissions set so the web server can read them. As long as they are in the same layout as they are when in a working ubb.classic set up, impex will do the rest.

[TXXJ]

Thanks for the quick reply, once I actually started the import process, I saw that you specify the directories in Impex.

Another hopefully quick question... I've just finished all the import processes finally, and rebuilt the threads, forums, search index, and did the update user names function, and so far everything looks ok, but... all the threads are locked? and stickied threads aren't sticky anymore

[Jerry]

This SQL will open all threads :

UPDATE thread SET open=1;

The stickies you will have to re stick as that value isn't currently imported.