Issues with segmentation faults and vBulletin 4.2.0 PL3

[jpsandiego]

Our installation of vBulletin is generating dozens of segmentation faults per minute which look to be causing 'connection reset' errors for users:
[Wed Oct 09 11:22:48 2013] [notice] child pid 23034 exit signal Segmentation fault (11)
[Wed Oct 09 11:22:52 2013] [notice] child pid 23656 exit signal Segmentation fault (11)
[Wed Oct 09 11:22:54 2013] [notice] child pid 23246 exit signal Segmentation fault (11)
[Wed Oct 09 11:22:57 2013] [notice] child pid 23643 exit signal Segmentation fault (11)

Not seeing very much information on what to do to isolate these issues and gain stability...

The only change recently to the environment was to change our MySQL VM cluster - downgrading RAM (24G down to 8G) and CPU (4 down to 2). Since that change we have restarted Apache and continue to see these issues.

One interesting issue happened earlier today when one of the Memcache (3 total) nodes failed and we saw the huge increase in the number of these segmentation faults. I disabled the failed memcache node in the config and all got back to the normal rate of faults, but it left me wondering if there is an issue with the vBulletin -> PHP -> Memcache config. This is the cache config from the server

$config['Cache']['class'][0] = 'vB_Cache_Memcached'; //regular cache
$config['Cache']['class'][1] = 'vB_Cache_Memory'; //fastest cache
$config['Cache']['class'][2] = 'vB_Cache_Db'; //largest cache and longest life.

Would enabling this within the config help shed some light - " $config['Misc']['debuglogging'] = true;" ?

Any ideas on how to approach this issue?


Apache 2.2.14 / PHP Version 5.3.10 / MySQL client 5.1.67 / Memcache client 3.0.4 (all Ubuntu packages)

[jpsandiego]

It turned out that this issue was being caused by one of our (three) Memcache nodes going offline temporarily. They're all VMs and another machine was configured with the same MAC as one of the Memcache nodes which caused connectivity to the 'real node' to sometimes be interrupted unexpectedly. It was so brief that our monitoring didn't even notice the issue. It took some manual investigation into each node (to look for any package manage changes that could explain why things were unstable) for someone to notice how their SSH session got dropped randomly which led to the duplicate MAC discovery.

Appears the deficiently is in the Apache+PHP+memcache module not reporting the issue and instead blowing up the Apache thread that was running - hence the segmentation fault.

Lovely problem, eh? =)

[Wayne Luke]

Are you using vBulletin 5 and posted in the wrong forum? The cache config you listed above does no occur in vBulletin 4.X.

You should use a single Memcache server and point all instances of your vBulletin files using to that server.