Impex Import super slow

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • croeder
    New Member
    • Nov 2012
    • 4

    Impex Import super slow

    I am importing from a custom MSSQL based forum using the fusetalk template for reference. The data is importing, but at an extremely slow rate. It currently processes 10 users/min. I have 316,000 users and 13,000,000 posts to import. At this rate it will take me 2 months to import. I tweaked the php memory limit to 1024M and MySQL buffers but it is still slow. The load on the servers is less than .1% so I have resources to spare. Any Help/Guidance is appreciated.
  • Trevor Hannant
    vBulletin Support
    • Aug 2002
    • 24325
    • 5.7.X

    #2
    What's your server spec?
    Vote for:

    - Admin Settable Paid Subscription Reminder Timeframe (vB6)
    - Add Admin ability to auto-subscribe users to specific channel(s) (vB6)

    Comment

    • croeder
      New Member
      • Nov 2012
      • 4

      #3
      Web Server (Nginx /php-fpm):

      OS: Red Hat Enterprise Linux Server release 6.3 (Santiago)
      CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz x2
      Memory: 4 GB RAM
      NIC: VMware VMXNET3 Ethernet Controller


      MySQL Server (5.5.22)

      OS: Ubuntu 12.04 LTS
      CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz x2
      Memory: 4 GB RAM
      NIC: VMware VMXNET3 Ethernet Controller

      Comment

      • croeder
        New Member
        • Nov 2012
        • 4

        #4
        Is there any documentation on the minimum data that is required for importing to VBulletin? I want to try and break down the process into manageable chunks as the import process seems to trip up on itself, and I get no debug info even though the flag is set to true. I end up with the same users getting imported more than once, the first time they are correct, the second their usernames end up with "Imported" pre-pended to them.

        Comment

        • Trevor Hannant
          vBulletin Support
          • Aug 2002
          • 24325
          • 5.7.X

          #5
          Before you run the import again, you should clear the imported data.

          When you say it trips up, where exactly is it stopping? Is it saying it's failed or is it simply stopping mid-process?
          Vote for:

          - Admin Settable Paid Subscription Reminder Timeframe (vB6)
          - Add Admin ability to auto-subscribe users to specific channel(s) (vB6)

          Comment

          • croeder
            New Member
            • Nov 2012
            • 4

            #6
            The problem is the process never seems to stop. It cycles through all the users in the origin db and then appears to start cycling through them again. It like it gets stuck in an infinite loop. In the vBulletin db I have the correct username (I.e. Croeder) at first and then I end up with another username of Imported_Croeder after it starts cycling again. I know it is doing something strange as I only have 400,000 users and it starts counting higher than that number.

            Comment

            • jpsandiego
              New Member
              • Mar 2013
              • 7
              • 4.2.X

              #7
              I seem to be encountering something similar with Impex. I'm importing from phpBB v3 to vBulletin 4.2. I have 3.5 million user records and it seems that Impex is actually getting through them at a decent rate but then bombs out. The numbers in the web interface stop incrementing but the process never completes. Apache continues to run a high load with no traffic indefinitely until I can go into impex and delete the session.

              About to start my 3rd attempt on the users import after tweaking down the max memory in php.ini (it was set to more than the machine has available). Hopefully this time it can go through.

              Support - how can I get logging of what Impex is doing if this next run doesn't work?

              Comment

              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
              Working...