vBulletin 4.x upgrade process for large vBulletin forums - eva2000's method

Collapse
X
Collapse
 

  • vBulletin 4.x upgrade process for large vBulletin forums - eva2000's method

    Basic re-post of my vB 4.x upgrade process which I have been using for years. Basically, you upgrade on a copy of your live database so as to leave live database untouched in case you mess up the upgrade.

    eva2000's upgrade process

    My vB 3.8.x to vB 4.1.x upgrade process which I personally use myself as well as for private paying clients is as follows and assumes:
    1. you have ssh2 telnet access
    2. your vB 3.x forums are only using MySQL default MyISAM/Memory tables with NO InnoDB tables
    3. you have enough free disk space to hold all the backups approximately 3x to 6x times the size of your forum database/files/attachments in free disk space.
    4. public root directory = /home/username/public_html
    5. live database name = livedbname
    6. new databasename name = newdbname



    Step 1. Tune php and mysql settings specifically for backup, restore and upgrade process. Tune mysql to ensure enough memory is allocated to buffers to speed things up, increase timeout values for mysql to ensure no mysql timeouts and for php. Ensure mysql server has InnoDB support disabled otherwise you’ll have to deal with InnoDB tuning parameters before you start as vB4 automatically converts 6 tables to InnoDB if mysql server has InnoDB support enabled. Read the reasons of how and why you’d want InnoDB support disabled in my blog post at http://vbtechsupport.com/675/ and if you do decide to use InnoDB storage engine for some database tables, I’d also look at upgrading your MySQL version to a better performing MySQL version such as MySQL 5.5.12, Percona 5.5.10-20.2 or my recommendation MariaDB 5.2.x series.

    Step 2. Backup live forums

    Code:
    cd /home/username/
    mysqldump --opt -Q -u mysqlusername -p [b]livedbname[/b] > /home/username/vb3backup.sql
    cp -a public_html public_html_oldvb38
    if you have attachments outside of public_html i.e. /home/username/attachments

    Code:
    cd /home/username
    cp -a attachments attachments_oldvb38
    Step 3. Create new empty database called newdbname and import backup vb3backup.sql into newdbname

    mysql -u mysqlusernae -p newdbname < /home/username/vb3backup.sql

    You could compress these vBulletin forum sql, attachment and public_html backups, but for large vBulletin forums it justs adds additional time to the upgrade task and if you have to revert to one backup it adds more time in uncompressing the vBulletin backups. So I usually don’t compress these backups until after the vBulletin 3.x to 4.x upgrade process is completed fully.

    Step 4. Get new vB version files uploaded to /home/username/public_html. Two ways of doing this - I prefer 2nd method if I am on the clock and don't have much time.

    Method #1
    • Empty out /home/username/public_html and upload new vB version files from zip folder /upload


    Method #2
    • On local computer, extract vB new version zip file and in /upload rename config.php.new to config.php and populate all the settings pointing to step #3's newdbname database which now has a copy of your live database imported into newdbname.
    • Also add to vB new version config.php following 2 lines to disable plugins and mail for upgrade process
      PHP Code:
      define("DISABLE_HOOKS"true);
      define('DISABLE_MAIL'1); 
    • If you use vB datastore caching in config.php either APC, Xcache, File or Memcached datastore caching, disable it temporarily for the upgrade by commenting it out with double forward slashes.
    • Then zip up /upload folder i.e. upload.zip and upload that to /home/username/ via FTP
    • Then ssh2 telnet into server as same username as your vB forum site account (this is so extracted files retain the correct file owner/group permissions).
    • Once in ssh2 go to /home/username directory and extract upload.zip you'll have a folder at /home/username/upload now which contains all the new vB version's php files etc

      Code:
      unzip upload
    • Still in ssh2 telnet, copy all files in /home/username/upload and overwrite all files in /home/username/public_html

      Code:
      cd [b]/home/username/upload[/b]
      unalias cp
      cp -Rfp * [b]/home/username/public_html[/b]


    Now you should have overwritten all files in /home/username/public_html with new vB version files

    At this stage, if you are using vBSEO addon and have MaxCDN or any other CDN hosted vB php/js/css files served externally, please flush and purge the CDN caches from your MaxCDN or CDN provider control panels to ensure you're upgrading using the uploaded new files and not cached CDN files.

    Step 5. vB 3.6.x to vB 3.8.x upgrades.
    • Run upgrade to 3.7.x and then repeat Step 2 and backup 3.7.x forums.
    • Run upgrade from 3.7.x to 3.8.x and then repeat Step 2 and backup 3.8.x forums.
    • Note: if you are upgrading using vB 4.0.9+ and higher vB php files, you can from the start using PHP CLI command line upgrade method outlined in Step 7. Just make sure to drop the fulltext indices mentioned in Step 6, BEFORE running PHP CLI command line upgrade.


    Step 6. vB 3.8.x to vB 4.0.9 upgrades.
    • Similar to large imports mentioned here http://www.vbulletin.com/docs/html/med_large_import, drop the post table's fulltext index on title BEFORE running vB 3.8.x to vB 4.1.x stage as vB 4.x no longer uses it. Run query on newdbname database (copy of your live forums). In ssh2 telnet type following to enter mysql command line:

      Code:
      mysql -u mysqlusername -p [b]newdbname[/b]
    • Then run query to drop fulltext indices in post, thread, groupmessage and socialgroup tables. If you get error saying the key doesn't exist you can ignore it and move on.

      Code:
      ALTER TABLE `post` DROP INDEX `title`;
      ALTER TABLE `thread` DROP INDEX `title`;
      ALTER TABLE `groupmessage` DROP INDEX `gm_ft`;
      ALTER TABLE `socialgroup` DROP INDEX `name`
    • Now begin upgrade process from vB 3.8.x to 4.0.9
    • Note: if you are upgrading using vB 4.0.9+ and higher vB php files, you can from the start using PHP CLI command line upgrade method outlined in Step 7. Just make sure to drop the fulltext indices mentioned in Step 6, BEFORE running PHP CLI command line upgrade.


    Step 7. vB 4.0.9 to vB 4.1.x latest - utilise the new command line upgrade process oulined at http://www.vbulletin.com/docs/html/upgrade_cli
    • ssh2 telnet into server as same username as vB site account and run from /home/username/public_html/install directory this command
      Code:
      php -f upgrade.php


    Step 8. vB 4.x upgrade tasks and backup
    • Once upgrade complete, I like to make another 2 backups by repeat Step 2 and backup 4.1.x forums BEFORE vB 4 upgrade tasks are done and another backup AFTER vB4 upgrade tasks are done.
    • vB4 upgrade tasks are those notice prompts when you first log into vB4.x admincp, you'll be prompted to do the following: rebuild search index, adding keywords to threads and rebuilding thread information.


    Rebuilding Search Index

    Rebuilding search index can be a time consuming task, so vBulletin included a way of doing this rebuild via SSH2 telnet via PHP command line. In the downloaded vB zip file folder called do_not_upload there’s a file called searchindex.php. You upload searchindex.php to the root directory of your vBulletin forum installation. In this example it would be uploaded to /home/username/public_html/searchindex.php. You then log in via SSH2 telnet as the same user as the forum domain/site user and run the following two commands:

    Code:
    cd /home/username/public_html
    php searchindex.php
    You’ll be prompted to enter the path to your vBulletin installation in this case = /home/username/public_html.

    Rebuilding Threads

    Rebuilding threads can also be time consuming for large vB forums. vBulletin developer, Michael Anders unofficially posted a PHP command line script for such task as well called threadcli.php (download link for registered vB members only as it isn’t included in official vB4 zip downloads). Works pretty much the same way as searchindex.php but asks for starting thread id and number of threads to update per pass.

    Run it from includes folder:

    Code:
    cd /home/username/public_html/includes
    php threadcli.php
    If threadcli.php doesn’t work for you and you get access denied message, edit threadcli.php and add right after
    PHP Code:
    <?php
    define
    ('VB_ENTRY','1');
    or you can try Paul K’s cli.php script instead which has more options.

    Code:
    cd /home/username/public_html
    php cli.php
    Code:
    php cli.php
    > Checking vB installation
    > Installation seems to be fine, continuing
    #################################################
    > Please select an option
    1) Quit
    2) Update User Titles and Ranks
    3) Rebuild Thread Information
    4) Rebuild Forum Information
    5) Add Add Missing Thread Keywords
    6) Update User Names
    7) Update Post Counts
    8) Fix Broken User Profiles
    9) Rebuild Post Cache
    10) Empty Signature Cache
    11) Delete Duplicate Threads
    Step 9. Revert your php and mysql settings back to everyday 24/7 values and remove from you config.php the lines

    PHP Code:
    define("DISABLE_HOOKS"true);
    define('DISABLE_MAIL'1); 
    Then re-enable vB datastore caching in config.php if you had disabled it for the upgrade.

    Step 10. Update YUI version – by either updating to vB 4.1.3 PL2 or higher or you can edit includes/class_core.php and update the version of YUI used from v2.7.0 to v2.9.0. Then after vB 4.x is upgraded, in AdminCP -> Options -> Server Settings and Optimization Options -> Use Remote YUI = select Google

    from

    Code:
    define('YUI_VERSION', '2.7.0'); // define the YUI version we bundle
    to

    Code:
    define('YUI_VERSION', '2.9.0'); // define the YUI version we bundle
    So basically, you do the upgrade on a copy of your live database in database named newdbname – leaving live database untouched in case you have to revert to a backup if upgrade messes up. You’ll end up with a few backups at different stages so it’s easier to revert to a particular stage rather than completely start over from scratch if upgrade messes up somewhere between.

    You’ll have vB 3.6.x backup, vB 3.7.x backup, vB 3.8.x backup and vB 4.1.x BEFORE upgrade tasks done and AFTER upgrade tasks done. If your database is very large then disk space could be an issue, so you could be selective as to how many backups you do, I’d at least have 3 backups, original live database backup, vB 3.8.x backup and vB 4.1.x backup.

    Largest vB forum I have tried this process on was aproximately 38GB size. If you have problems or vBulletin errors during upgrade, best to check out vBulletin 4 Installations and Upgrades support forums.

    Once you have upgraded to vBulletin 4.x, you can check out Shawn/Digitalpoint.com’s vBulletin 4 Optimization Guide.

    I've also posted this as an CMS Article in How to Install/Upgrade section and you can discuss the article here.


    Update: June 9th, 2012
    One year anniversary from original blog post. Added youtube video for steps 1 to 4 outlined above as an example of upgrading from vB 4.2.0 to 4.2.0 PL1. Creating this video was a bit awkward as for years now, I normally use multiple monitors for different apps and screens so having to condense this video to single monitor recording was a bit of a challenge.

    Video best viewed in HD 720P mode full screen.



    Updated Notes:
    If upgrading vBulletin 3.x with vBulletin 3.x API installed to vBulletin 4.2.x, you may get an error at vBulletin 4.1.10 Alpha 1 part of the upgrade script as follows:

    Code:
    Upgrading to 4.1.10 Alpha 1
    ----------------------------------
    Step 1 - Updating setting table
    Step 2 - Updating settinggroup table
    Step 3 - Database error in vBulletin 4.1.9:
     
    Invalid SQL:
    DELETE FROM navigation WHERE productid = 'vbapi';
     
    MySQL Error : Table 'databasename.navigation' doesn't exist
    Error Number : 1146
    Request Date : 
    Error Date : 
    Script : http://
    Referrer :
    IP Address :
    Username :
    Classname : vB_Database
    MySQL Version : 5.1.61-log
    This bug is fixed in upcoming vBulletin 4.2.1

    • sdi_employee
      #11
      sdi_employee commented
      Editing a comment
      Don't know if anyone will see this here or not, but I've followed this process and have everything up to running the search index script completing in about 2.5 hours, which is pretty reasonable.

      That being said, the search index script is taking *days* to finish, and I can't find the thread script to run anywhere so I am just running that in the browser and it also takes a very long time. At this point, the only solution I can find without taking my site down for days is to let my users know that they are going to lose posts for the upgrade period, leave the site up, and upgrade a backup. Replacing the running site with the backup and losing the posts for those couple of days that the post-upgrade tasks were running is definitely not an ideal solution, but it's all I have right now.

      For context - I'm running a site with a few thousand active users at any one time, and 20 million posts currently in the database. (Also, it seems like the database size itself balloons quite a bit after the upgrade to 10Gb from around 3-4 before the upgrade, and that has me a bit worried from a backup prospective, but I don't think there's anything to be done about that.)

    • George L
      #12
      George L commented
      Editing a comment
      Originally posted by sdi_employee;bt11808
      Don't know if anyone will see this here or not, but I've followed this process and have everything up to running the search index script completing in about 2.5 hours, which is pretty reasonable.

      That being said, the search index script is taking *days* to finish, and I can't find the thread script to run anywhere so I am just running that in the browser and it also takes a very long time. At this point, the only solution I can find without taking my site down for days is to let my users know that they are going to lose posts for the upgrade period, leave the site up, and upgrade a backup. Replacing the running site with the backup and losing the posts for those couple of days that the post-upgrade tasks were running is definitely not an ideal solution, but it's all I have right now.

      For context - I'm running a site with a few thousand active users at any one time, and 20 million posts currently in the database. (Also, it seems like the database size itself balloons quite a bit after the upgrade to 10Gb from around 3-4 before the upgrade, and that has me a bit worried from a backup prospective, but I don't think there's anything to be done about that.)
      really depends on many factors

      1. server hardware
      2. server configuration/optimal settings for best upgrade performance
      3. size of database and number of rows

    • IggyP
      #13
      IggyP commented
      Editing a comment
      i have read some conflicting infos....is the cli upgrade indeed only for after 4.x???

      i am confused by the dated blog but some good informations none the less thank you
    Posting comments is disabled.

Related Topics

Collapse

Working...