Fatal error while importing

[coloradok5]

I got this error while test importing from UBBT 6.4:

Fatal error: Allowed memory size of 52428800 bytes exhausted (tried to allocate 112 bytes) in /home/bla/bla/bla.com/blaforums/impex/ImpExData.php(209) : runtime-created function on line 1

Fatal error: Allowed memory size of 52428800 bytes exhausted (tried to allocate 0 bytes) in Unknown on line 0



Any ideas?

[Jerry]

Lower the amount of what ever it is you are doing per page.

[coloradok5]

(posts) OK. the default was 2,000 I will lower that to 500, do I have to start completly over again? Nothing happens when I click on the Import/Export link.

[Jerry]

You will have to re-start that module yes.

If it isn't responding the session might be corrupt. Does it display the links at the top of the page to "database clean" ?

[coloradok5]

Does it display the links at the top of the page to "database clean" ?
Yep, those links are there.

[Jerry]

Click on the Database clean up and restart, and clear all the imported data and session and start again, set it to 500 per page for starters.

[coloradok5]

Will do, thanks.

[Gary Fischman]

I get the same error - even when its set to 500. Is this due to a system level variable?

[Jerry]

Its the memory limit being hit when creating Lambda functions.

The best way around it for the moment is to edit ImpExData.php

line 182 : function is_valid()

make the function return true so the error checking function isn't created, so :



function is_valid()
{
return true;
// If any of the madatory values are null return false.
.....
.......


This is something I am working on at the moment as the memory managment is an issue.

[Gary Fischman]

Actually, I found another way around it. I edited the php.ini file to increase the memory limit. :)

[Jerry]

Actually, I found another way around it. I edited the php.ini file to increase the memory limit. :)

Well that is the best way, though I am looking at how to use a more efficent way of error checking the data as lambda does eat memory.

[Jerry]

This is why we advise testing and not taking down the live site, and yes I know it can be a pain on a board that big.

Did you edit ImpExData.php or up the memory limit ?

I've also just taken out some referencing in the post import module that should help with using less memory, that should help out as well. I've updated the download in the members area.

[coloradok5]

This is why we advise testing and not taking down the live site, and yes I know it can be a pain on a board that big.

Did you edit ImpExData.php or up the memory limit ?

I've also just taken out some referencing in the post import module that should help with using less memory, that should help out as well. I've updated the download in the members area.

I was importing to a test forum, but I thought I have to keep my current forum closed so the source db does not get accessed during the Import, is that correct?
The problem I see with the import is that it takes about 20-30 sec to access my source database and only about 4 sec to import 500 posts, it was importing about 60,000 posts per hour, a long time to get 1 million posts moved over.
Does this sound normal?

[Jerry]

I was importing to a test forum, but I thought I have to keep my current forum closed so the source db does not get accessed during the Import, is that correct?
The problem I see with the import is that it takes about 20-30 sec to access my source database and only about 4 sec to import 500 posts, it was importing about 60,000 posts per hour, a long time to get 1 million posts moved over.
Does this sound normal?

20-30 to access the source is a bit mad, though I've tried to tune the ubb.t importer a bit to help with that.

Nope, you can leave the source board open, ImpEx will only ever read from it.

[coloradok5]

20-30 to access the source is a bit mad, though I've tried to tune the ubb.t importer a bit to help with that.

Nope, you can leave the source board open, ImpEx will only ever read from it.

OK, I could have sworn that it was a bad thing to leave the source forum open because people would be making posts etc.?

So what files do I need to reupload? Also should I continue to use 500 posts at a time to import?

[coloradok5]

Well after 27 hours I got almost half of my million posts imported when I got hit with the memory limit. :mad:
By the way I was not joking about the 27 hours, and all my users are pissed at me for taking it down that long, now I have to find a way to tell them I have to do it again for even longer. :eek:

[Jerry]

OK, I could have sworn that it was a bad thing to leave the source forum open because people would be making posts etc.?

So what files do I need to reupload? Also should I continue to use 500 posts at a time to import?

If you can up your php memory limit to 128 Meg or so it would be best, importing eats memory. Then you should be able to do 5000 or so a page.

If you can do that and reupload systems/ubb_threads/007.php

If you want to sacrifice the parent id's for the posts, i.e. loose any threading information you can speed it up a little bit as well that way.

People can post to a source board, impex doesn't mind as it reads in chunks from 0-n , it will hit n at some point though at some point it will select the last chunk of posts, unless people are posting faster than it can import, which is unlightly.

[coloradok5]

If you can up your php memory limit to 128 Meg or so it would be best
OK, how do I do this?

[Jerry]

its an php setting, its in the php.ini file.

http://www.vbulletin.com/forum/showpost.php?p=501777&postcount=4

[coloradok5]

its an php setting, its in the php.ini file.

http://www.vbulletin.com/forum/showpost.php?p=501777&postcount=4

If I am unable to gain access to the php.ini file will the corrections you made to iimpex be enough to work?

[Jerry]

If I am unable to gain access to the php.ini file will the corrections you made to iimpex be enough to work?

I have no idea, without and completly identical serve to yours to test on.

This is why we advise testing.

[coloradok5]

Welp, I added what you have in post number 9 and so far I just passed 500,000 and looking good so far (knocks on wood). It has timed out but I simply refresh and click on the inport link and it continues where it left off, looks like not having the error message is the ticket, I will let you know how it turns out.

[Jerry]

Using the latest downloaded impex ?

[coloradok5]

Using the latest downloaded impex ?
Yep, and just got done eating some Turkey and it is up to 844,000 so far. :D

[coloradok5]

OK, it worked, got all 1,078,930 replies imported. :D Looks like polls are taking forever though, I hope I don't have to many.

[Jerry]

Big board, lots of refrencing going on looking up post id's etc so its all realtive :)

[coloradok5]

OK, it worked, got all 1,078,930 replies imported. :D Looks like polls are taking forever though, I hope I don't have to many.
Just an FYI, the polls imported to incorrect posts, when I import again for real I will not bother with them at all as they are not really important to my users but just letting you know. :)

[Jerry]

Just an FYI, the polls imported to incorrect posts, when I import again for real I will not bother with them at all as they are not really important to my users but just letting you know. :)

Thanks, I've submitted the bug :

http://www.vbulletin.com/forum/bugs.php?do=view&bugid=3514

[meanshark]

Click on the Database clean up and restart, and clear all the imported data and session and start again, set it to 500 per page for starters.

So, the Database clean up and restart link ONLY cleans out the session of the inporter. NOT the actual database right?

[Jerry]

So, the Database clean up and restart link ONLY cleans out the session of the inporter. NOT the actual database right?

No, that link takes you to a page where you have some options as to exactly what you want to remove from the database.

[meanshark]

I am inporting the posts and got the memory error, and now it just flashes a loop of the same page with no links, other than the two at the top.

[Jerry]

The the session has crashed, run this SQL :




DELETE FROM datastore WHERE title='ImpExSession'



You will need a table pre-fix on datastore if you have one.

Idealy, hit the stop button and click the click at the top of the impex page.