Tweet
Last Friday we upgraded a rather large forum from vBulletin 3 to vBulletin 4 and I must say I am less than impressed. This forum has 11.5 million posts, 180,000+ members, and averages 120-150 page requests per second. This is on a dedicated server, some server specifics on hardware and packages:
Ubuntu 12.04
16 cores at 3.0 ghz
32 gigs of RAM
mysql 5.5
lighttpd 1.4.28
fast-cgi
memcache
x-sendfile
imagemagick
The first problem is load, and its a big problem. As mentioned above, the upgrade was performed last Friday morning, and load since then has been through the roof. Consider this screen shot of Observium stats. Everything to the left of Friday morning is vBulletin 3 with all kinds of plugins loaded. Everything to the right of Friday morning is vBulletin 4 with 0 plugins loaded. Other than the vBulletin upgrade no changes to server configuration were made.
As you can see, its a HUGE jump in CPU use. Now some of those spikes on Saturday evening was the rebuilding of some of the post counts and stuff like that. The rest of it is straight vBulletin CPU use. We had been using Sphinx for fulltext searches on vBulletin 3 and I left searches disabled on vBulletin 4 as I tried some of the newer Sphinx packages which are not working out that great. On Tuesday I switched over to the default vBulletin search and rebuilt those incices which accounts for the big jump on Tues/Weds. These are 5 minute averages.
On Friday March 1st you can actually see where I shut everything down so I could take a closer look at things. This is because loads were up over 30 on a 16 core machine and the site was running like crap. Over this weekend, loads were actually spiking up into the 50s so I took everything down for a while and started digging. Part of the problem ended up being a cronjob in cleanup2.php that calls class_dm_attachment.php in order to clean up orphaned attachments. It started throwing this in the error log on March 1st.
fast-cgi was killing the script because it was calling for more memory than is allocated to a single process. Normally this isnt a big deal, whatever process it is just cannot run. But the vBulletin cronjob doesnt know why its being killed, it just knows that it is supposed to run, so it spawns it back up again. I literally have 85 megs of that error over 3 days because it was trying to run every single second of every single day. Every time it got killed it just respawns, and whats worse it stacks on top of itself. By today it was trying to allocate 16 gigs of RAM to this process.
Now I solved this problem by changing max_memory in php.ini from 128 megs to 256 megs, killing all the processes and starting the server back up. Thats causing me to actually use some swap but I can live with that for now. Once that process was able to actually run a couple of times my loads have dropped down below 16. As I am writing this they are averaging 12-14.
So my first couple of questions are:
1. What would be causing vb4 to use so much more CPU than vb3, and is there anything I can do to reduce that? As stated I am now at a load of 12-14 but historically over the last 3 years we rarely went over 10 on this same server. I can document that if needed.
2. Why is vb4 needing more RAM to process orphaned files? And more importantly, when that process gets killed by fast-cgi why is it trying to respawn like this? Is the RAM use related to the fact that albums are now considered attachments? If not then this is not making sense to me as we have the same attachment size caps that we have always had, and the same fast-cgi and php settings.
The second real problem I have to solve is actual album pictures not working. My understanding of this (which may be wrong) is that with vb4 albumpics are considered to be attachments, and in vb3 they were not. We have always had albumpics and stuff like that stored in the file system instead of in the database and for several reasons did not move them back into the database before the upgrade. Its actually pretty unreasonable to require that in the update process to begin with. When you already have a database that is 35 gigs the last thing you want to do is import 50 gigs of album pics and another 120 gigs of file attachments into it, uprade, and then export them back out into the file system. The partition that contains the database wouldnt have handled that anyways.
So what would be the best way to get my albums working again
Ubuntu 12.04
16 cores at 3.0 ghz
32 gigs of RAM
mysql 5.5
lighttpd 1.4.28
fast-cgi
memcache
x-sendfile
imagemagick
The first problem is load, and its a big problem. As mentioned above, the upgrade was performed last Friday morning, and load since then has been through the roof. Consider this screen shot of Observium stats. Everything to the left of Friday morning is vBulletin 3 with all kinds of plugins loaded. Everything to the right of Friday morning is vBulletin 4 with 0 plugins loaded. Other than the vBulletin upgrade no changes to server configuration were made.
As you can see, its a HUGE jump in CPU use. Now some of those spikes on Saturday evening was the rebuilding of some of the post counts and stuff like that. The rest of it is straight vBulletin CPU use. We had been using Sphinx for fulltext searches on vBulletin 3 and I left searches disabled on vBulletin 4 as I tried some of the newer Sphinx packages which are not working out that great. On Tuesday I switched over to the default vBulletin search and rebuilt those incices which accounts for the big jump on Tues/Weds. These are 5 minute averages.
On Friday March 1st you can actually see where I shut everything down so I could take a closer look at things. This is because loads were up over 30 on a 16 core machine and the site was running like crap. Over this weekend, loads were actually spiking up into the 50s so I took everything down for a while and started digging. Part of the problem ended up being a cronjob in cleanup2.php that calls class_dm_attachment.php in order to clean up orphaned attachments. It started throwing this in the error log on March 1st.
[01-Mar-2013 08:20:08 UTC] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 34 bytes) in /data/www/forums/includes/class_dm_attachment.php on line 393
[03-Mar-2013 10:34:25 UTC] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 17179869184 bytes) in /data/www/forums/includes/class_dm_attachment.php on line 393
So my first couple of questions are:
1. What would be causing vb4 to use so much more CPU than vb3, and is there anything I can do to reduce that? As stated I am now at a load of 12-14 but historically over the last 3 years we rarely went over 10 on this same server. I can document that if needed.
2. Why is vb4 needing more RAM to process orphaned files? And more importantly, when that process gets killed by fast-cgi why is it trying to respawn like this? Is the RAM use related to the fact that albums are now considered attachments? If not then this is not making sense to me as we have the same attachment size caps that we have always had, and the same fast-cgi and php settings.
The second real problem I have to solve is actual album pictures not working. My understanding of this (which may be wrong) is that with vb4 albumpics are considered to be attachments, and in vb3 they were not. We have always had albumpics and stuff like that stored in the file system instead of in the database and for several reasons did not move them back into the database before the upgrade. Its actually pretty unreasonable to require that in the update process to begin with. When you already have a database that is 35 gigs the last thing you want to do is import 50 gigs of album pics and another 120 gigs of file attachments into it, uprade, and then export them back out into the file system. The partition that contains the database wouldnt have handled that anyways.
So what would be the best way to get my albums working again
