Import freezing at same point

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • baker011
    Member
    • Dec 2004
    • 43
    • 4.2.X

    Import freezing at same point

    Clean import on a newly installed testing forum.
    Importing from 3.6.4 to 3.6.4
    Error happens on Threads module

    I am trying to set up a test version of a large forum I manage. My ultimate aim is to import the forum and then test merging another vBulletin forum with it. However, I cant even import the original form yet as it is constantly freeing at importing threads. It gets to 230,000 - 232,000 and it just hangs until it brings up a "The page cannot be displayed" error and the importer is not useable after. It locks up everytime I try to use it once it breaks at this stage.

    Any ideas?
  • Jerry
    Senior Member
    • Dec 2002
    • 9137
    • 1.1.x

    #2
    Can you just take a copy of the first database, then use that to test importing the 2nd forum into ?
    I wrote ImpEx.

    Blog | Me

    Comment

    • baker011
      Member
      • Dec 2004
      • 43
      • 4.2.X

      #3
      Sorry, the way we have done is is we have backed up the live database on our live server, restored it on our dev server and then trying to import this restored version. It connects fine and imports all the users, forums, usergroups, just when it hits this point in the threads does it cause issues.

      The databases are on the same server, and its a very low traffic server as its just a dev server.

      Comment

      • Jerry
        Senior Member
        • Dec 2002
        • 9137
        • 1.1.x

        #4
        Anything in the PHP error logs ?
        I wrote ImpEx.

        Blog | Me

        Comment

        • baker011
          Member
          • Dec 2004
          • 43
          • 4.2.X

          #5
          No, absolutely no error logs. It just refuses to go past that stage then after Impex is completely locked up and refuses to load. I did all the suggested script changes too for large forums as this is a 4,000,000+ post forum.

          Comment

          • Jerry
            Senior Member
            • Dec 2002
            • 9137
            • 1.1.x

            #6
            No error logs for PHP ? Eeek, that’s going to make it tricky !!

            On something that size, its probably memory or the database. PHP is going to use a huge amount of memory holding all the threadids.

            Have you turned off the PHP memory limit all together ?
            I wrote ImpEx.

            Blog | Me

            Comment

            • baker011
              Member
              • Dec 2004
              • 43
              • 4.2.X

              #7
              If I remember rightly our Server guy upped it to 64mb. When it first acted up it was set to 8mb but it has failed at the same point from the very first try, so I am not so sure its down to the memory limit. I have got the server guy trying another import today with this database and I have requested he search for any error reports, but each time he has checked it has come back with none. Could you confirm where these error logs would be just to make sure we are checking the correct place.

              Thanks

              Comment

              • Jerry
                Senior Member
                • Dec 2002
                • 9137
                • 1.1.x

                #8
                Depends on the OS version and lay out, the sysadmin that built it should know exactly where the logs are, I can't think of any good reason to turn logs off, they are created for very good reasons.

                Check the php.ini to see what and where, see if this is set :

                Code:
                error_log =
                I wrote ImpEx.

                Blog | Me

                Comment

                • baker011
                  Member
                  • Dec 2004
                  • 43
                  • 4.2.X

                  #9
                  Sorry I was getting mixed up with the db logs that you choose to enable for the Impex. I will get our server admin to look through the PHP logs.

                  Comment

                  • baker011
                    Member
                    • Dec 2004
                    • 43
                    • 4.2.X

                    #10
                    Ok, we have tried another import, upped the thread limit to 5000 a time and it stalled on 225,000-230,000 (almost exactly the same point it froze at when doing 2,000 a time. Then the PHP logs were checked and there were no reported errors. So back at square one

                    Comment

                    • Jerry
                      Senior Member
                      • Dec 2002
                      • 9137
                      • 1.1.x

                      #11
                      Originally posted by baker011
                      Ok, we have tried another import, upped the thread limit to 5000 a time and it stalled on 225,000-230,000 (almost exactly the same point it froze at when doing 2,000 a time. Then the PHP logs were checked and there were no reported errors. So back at square one
                      True, though we have eliminated one avenue of investigation

                      Do you have any forums with a large amount of redirects ?

                      On line 107 of impex/systems/vb_36/006.php add this after the brace :

                      PHP Code:
                      echo "<br>Importing - {$thread_id}"
                      That will show you which one its working on, so which one it dies on.
                      I wrote ImpEx.

                      Blog | Me

                      Comment

                      • baker011
                        Member
                        • Dec 2004
                        • 43
                        • 4.2.X

                        #12
                        Pretty normal amount of redirects to be honest, I will have that line of code added and run another test import.

                        Thanks for the help so far, it really is appreciated!

                        Comment

                        • baker011
                          Member
                          • Dec 2004
                          • 43
                          • 4.2.X

                          #13
                          Ok we ran the import with that update to the script, it stopped at this thread:



                          I dont see a lot wrong with it, I have asked our tech guy to run it once again to see if its consistantly that thread.

                          Comment

                          • baker011
                            Member
                            • Dec 2004
                            • 43
                            • 4.2.X

                            #14
                            I just got this email from our Server guy after the latest set of test imports:


                            After restarting it, it looks to be trying to run the same section of thread inserts over and it gets to this one...

                            131 Query INSERT INTO thread
                            (
                            forumid, title, importforumid,
                            importthreadid, firstpostid, lastpost,
                            pollid, open, replycount,
                            postusername, postuserid, lastposter,
                            dateline, views, iconid,
                            notes, visible, sticky,
                            votenum, votetotal, attach, similar,
                            hiddencount, deletedcount
                            )
                            VALUES
                            (
                            '154',
                            'RCTCC Survivor',
                            '421',
                            '557336',
                            '8302002',
                            '1168495448',
                            '0',
                            '1',
                            '11',
                            'aoshi',
                            '71233',
                            'Bitter Jeweler',
                            '1168493355',
                            '66',
                            '0',
                            '',
                            '1',
                            '0',
                            '0',
                            '0',
                            '0',
                            '',
                            '0',
                            ''
                            )
                            ... Then starts running a bunch of inserts...
                            070111 1:14:45 129 Query SELECT threadid FROM thread where importthreadid = 77172
                            131 Query SELECT threadid FROM thread where importthreadid = 77265
                            070111 1:14:46 129 Query SELECT threadid FROM thread where importthreadid = 77231
                            131 Query SELECT threadid FROM thread where importthreadid = 77971
                            070111 1:14:47 129 Query SELECT threadid FROM thread where importthreadid = 77265
                            131 Query SELECT threadid FROM thread where importthreadid = 79124
                            070111 1:14:48 129 Query SELECT threadid FROM thread where importthreadid = 77971
                            070111 1:14:49 131 Query SELECT threadid FROM thread where importthreadid = 80142
                            070111 1:14:50 129 Query SELECT threadid FROM thread where importthreadid = 79124
                            131 Query SELECT threadid FROM thread where importthreadid = 80465
                            070111 1:14:51 129 Query SELECT threadid FROM thread where importthreadid = 80142
                            131 Query SELECT threadid FROM thread where importthreadid = 82199
                            070111 1:14:52 129 Query SELECT threadid FROM thread where importthreadid = 80465
                            131 Query SELECT threadid FROM thread where importthreadid = 82445
                            070111 1:14:53 129 Query SELECT threadid FROM thread where importthreadid = 82199
                            131 Query SELECT threadid FROM thread where importthreadid = 83025
                            070111 1:14:54 131 Query SELECT threadid FROM thread where importthreadid = 83066
                            129 Query SELECT threadid FROM thread where importthreadid = 82445
                            070111 1:14:55 131 Query SELECT threadid FROM thread where importthreadid = 83798
                            129 Query SELECT threadid FROM thread where importthreadid = 83025
                            070111 1:14:56 131 Query SELECT threadid FROM thread where importthreadid = 83879
                            129 Query SELECT threadid FROM thread where importthreadid = 83066
                            070111 1:14:57 131 Query SELECT threadid FROM thread where importthreadid = 84956
                            129 Query SELECT threadid FROM thread where importthreadid = 83798
                            070111 1:14:58 131 Query SELECT threadid FROM thread where importthreadid = 85512
                            129 Query SELECT threadid FROM thread where importthreadid = 83879
                            070111 1:14:59 131 Query SELECT threadid FROM thread where importthreadid = 86339
                            070111 1:15:00 129 Query SELECT threadid FROM thread where importthreadid = 84956
                            131 Query SELECT threadid FROM thread where importthreadid = 86532
                            070111 1:15:01 129 Query SELECT threadid FROM thread where importthreadid = 85512
                            070111 1:15:02 131 Query SELECT threadid FROM thread where importthreadid = 87237
                            129 Query SELECT threadid FROM thread where importthreadid = 86339
                            070111 1:15:03 131 Query SELECT threadid FROM thread where importthreadid = 87296
                            129 Query SELECT threadid FROM thread where importthreadid = 86532
                            070111 1:15:04 131 Query SELECT threadid FROM thread where importthreadid = 88375
                            129 Query SELECT threadid FROM thread where importthreadid = 87237
                            070111 1:15:05 131 Query SELECT threadid FROM thread where importthreadid = 88421
                            129 Query SELECT threadid FROM thread where importthreadid = 87296
                            070111 1:15:06 131 Query SELECT threadid FROM thread where importthreadid = 89624
                            129 Query SELECT threadid FROM thread where importthreadid = 88375
                            070111 1:15:07 131 Query SELECT threadid FROM thread where importthreadid = 89721
                            129 Query SELECT threadid FROM thread where importthreadid = 88421
                            070111 1:15:08 131 Query SELECT threadid FROM thread where importthreadid = 89838
                            129 Query SELECT threadid FROM thread where importthreadid = 89624
                            070111 1:15:09 131 Query SELECT threadid FROM thread where importthreadid = 89894
                            129 Query SELECT threadid FROM thread where importthreadid = 89721
                            070111 1:15:10 131 Query SELECT threadid FROM thread where importthreadid = 89945
                            070111 1:15:11 129 Query SELECT threadid FROM thread where importthreadid = 89838
                            131 Query SELECT threadid FROM thread where importthreadid = 91687
                            070111 1:15:12 129 Query SELECT threadid FROM thread where importthreadid = 89894
                            131 Query SELECT threadid FROM thread where importthreadid = 91822
                            070111 1:15:13 129 Query SELECT threadid FROM thread where importthreadid = 89945
                            070111 1:15:14 131 Query SELECT threadid FROM thread where importthreadid = 93187
                            129 Query SELECT threadid FROM thread where importthreadid = 91687
                            131 Query SELECT threadid FROM thread where importthreadid = 94411
                            070111 1:15:15 129 Query SELECT threadid FROM thread where importthreadid = 91822
                            131 Query SELECT threadid FROM thread where importthreadid = 95533
                            070111 1:15:16 129 Query SELECT threadid FROM thread where importthreadid = 93187
                            131 Query SELECT threadid FROM thread where importthreadid = 95551
                            070111 1:15:17 129 Query SELECT threadid FROM thread where importthreadid = 94411
                            070111 1:15:18 131 Query SELECT threadid FROM thread where importthreadid = 96532
                            129 Query SELECT threadid FROM thread where importthreadid = 95533
                            070111 1:15:19 131 Query SELECT threadid FROM thread where importthreadid = 96946
                            129 Query SELECT threadid FROM thread where importthreadid = 95551
                            070111 1:15:20 131 Query SELECT threadid FROM thread where importthreadid = 96958
                            129 Query SELECT threadid FROM thread where importthreadid = 96532
                            070111 1:15:21 131 Query SELECT threadid FROM thread where importthreadid = 100549
                            129 Query SELECT threadid FROM thread where importthreadid = 96946
                            070111 1:15:22 131 Query SELECT threadid FROM thread where importthreadid = 102470
                            129 Query SELECT threadid FROM thread where importthreadid = 96958
                            070111 1:15:23 131 Query SELECT threadid FROM thread where importthreadid = 103453
                            070111 1:15:24 129 Query SELECT threadid FROM thread where importthreadid = 100549
                            131 Query SELECT threadid FROM thread where importthreadid = 103595
                            070111 1:15:25 129 Query SELECT threadid FROM thread where importthreadid = 102470
                            ...that take a long time (each one takes a second or so). It times out while running these selects. All in all, it seems like the SQL isn't very optimized. If we could just have it spit out the SQL we could run it from the command line and not have to worry about timeouts. I've rerun it several times and it does the exact same thing every time. I don't know enough about the importer or vBulletin to debug this.

                            Comment

                            • Jerry
                              Senior Member
                              • Dec 2002
                              • 9137
                              • 1.1.x

                              #15
                              And that is during the post module ?

                              That could be updating the poll id's for threads, just to check that comment out line 173 of impex/systems/vb_35/006.php.

                              Also add an index to thread.importthreadid if it isn't already there, after the 001 module, but before the thread module.
                              I wrote ImpEx.

                              Blog | Me

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...