Importing 400,000 posts - taking forever

[Verbose]

What is the recommended batch size for importing 400,000 posts? I've tried doing 25,000 at a time and it is just too sluggish and I'd rather not have to sit here and click "next" a million times.

A good feature would be to automatically refresh to the next page, and a way to turn off the data dump to your browser window.

[Verbose]

also... I purposely closed my browser to try to killl the posts import module because it said there was some mysql error where it couldn't reach the server for a minute. So i closed my browser window to redo the import - well, when i logged back in the import was still actively running.

Is there a kill switch somewhere or do you just have to let it run it's course for 30+ minutes?

[Jerry]

What is the recommended batch size for importing 400,000 posts? I've tried doing 25,000 at a time and it is just too sluggish and I'd rather not have to sit here and click "next" a million times.

A good feature would be to automatically refresh to the next page, and a way to turn off the data dump to your browser window.

In ImpEx config :



define('shortoutput', true); // Depending on your source system, only some Tier 1 supported at the moment.
define('step_through', false);


As for speed, that depends mainly on the PHP memory limit and the power of the database, 25,000 seems fine, I would expect that to import in about 15 mins depending on the over all power of the server.

[Verbose]

In ImpEx config :



define('shortoutput', true); // Depending on your source system, only some Tier 1 supported at the moment.
define('step_through', false);
As for speed, that depends mainly on the PHP memory limit and the power of the database, 25,000 seems fine, I would expect that to import in about 15 mins depending on the over all power of the server.

Hi Jerry, thanks for your response :)

I set the batch import size to 10,000 posts at a time - this took roughly 3 hours to import 400,000 posts. We've spoken to tech support and have our serving settings pretty high with fairly powerful hardware, so I'm not sure what else to try.

I am doing the live import today, and will try it with the output turned off. FYI, our source board is UBB.threads 6.5.2 - I think that is Tier 1, so hopefully this feature is supported?

[Jerry]

I've just updated the 007.php file in the ubb_threads system so it has the short output, if you download and over write that file it will have it now.