Slow Posting response when posting to SOME threads

[flynnibus]

Our forum is 3.6.8 on linux with a pretty generic apache, php5, and mysql4 setup.

Recently we've noticed very long delays when posting - but the rest of the site is fast and its not consistent. Only certain threads really seem to suffer from it - and these are NOT huge threads (maybe 30-40 posts).

Doing some searching here, we changed some settings related to this.
- we disabled similar thread searching
- we switched to fulltext searching from vb default
- we enabled mail queuing
- we expanded the server ram to 1gig


When the problem is visible, it may take upwards of 30 seconds for the forum to complete a posting.

The problem is isolated to posting - the rest of the forum is snappy and responsive. The forum has been running smoothly for 2+ years and its not that large (about 1k active users.. about 200 posts a day peak). Our post table is about 48megs.

Any pointers where to go next? I wouldn't expect at this level we'd need extra caching products, or wild optimization. I see that eva's thread gets quite a number of optimization requests and wanted to know if there is other SOP methods that we should be exploring first, or that we could be killing ourselves over.

The only plugins we have I don't think are related
- template mod for unread posts in welcome box
- move for inactive users (done as part of the hourly cron)
- template mod for the display IP page
- multiple login detector (uses cookies on login stuff.. not posting)
- opt-out for new posts per forum
- passivevid - which when displaying posts changes youtube/etc links but the actual post itself is not modified

None of the above should affect saving posts from my interpretation. Pointers on where to go next?

[Steve Machol]

Try removing or disabling your plugins and template modifications to see if that helps.

Another possibility is that the slowdown is caused by the server when trying to send out the email notifications for those threads.

[flynnibus]

Try removing or disabling your plugins and template modifications to see if that helps.

I listed them out to help ID that the plugins I do have would not be related to posting.

Another possibility is that the slowdown is caused by the server when trying to send out the email notifications for those threads.

The server is never slowing down response wise - its not going under load at all (CPU wise as I'm monitoring) - its only the vb page that is slow to respond. And these threads I can see the delay in do not have many subscribers. I can see this delay in threads with as few as 10-20 posters. Surely the code can't try to send email to every subscriber and wait for success before committing the post.

We've watched the sql server - it spits out results effortlessly. We've watched the CPU load - it never gets above about 40%. The only thing peaky about it all is the httpd's process load.

I've even seen the slowdown here on vb.com. The initial post that created this thread took about 15 seconds to commit. Yet the rest of the site is quite snappy. But the slow-down isn't on every post - which is what is making it hard to ID

[Lynne]

Have you repaired/optimized the thread or post tables lately?

[flynnibus]

Have you repaired/optimized the thread or post tables lately?

No - why would I and how? They aren't reporting errors and the tables are indexed on their own.

[Lynne]

When I was on a single server and things would slow down, optimizing the tables seemed to always help out. You can do it through the admin cp under Maintainance > Repair/Optimize tables. I don't know how big your site is, but if you've never done it before, I would start by just doing a couple of tables at a time.

[flynnibus]

It's clear its not a general server slow-down because I can be navigating the forums just fine in another window while waiting for the post to submit.

Even after doubling our RAM to 1 gig and making all the changes - while the server only has 8 people on it.. some threads.. the one this morning which only had 58 posts in it.. still took upwards of 30 seconds to submit a reply to.

There has got to be a way to trace what the server is doing during this time? Is there any debugging to enable in the vbulletin code that can write out what steps its doing and which are taking so long to narrow this down?

[Lynne]

It's clear its not a general server slow-down because I can be navigating the forums just fine in another window while waiting for the post to submit.

Even after doubling our RAM to 1 gig and making all the changes - while the server only has 8 people on it.. some threads.. the one this morning which only had 58 posts in it.. still took upwards of 30 seconds to submit a reply to.

There has got to be a way to trace what the server is doing during this time? Is there any debugging to enable in the vbulletin code that can write out what steps its doing and which are taking so long to narrow this down?

If you turn on debug mode, then you may see the queries on the page and how much time it is taking per query. I think there may be a mod over a vb.org to turn on debug just for your IP or your userid (unless you don't mind everyone seeing it).

Just curious, but did you try an optimize/repair on your post/thread tables or decide to not try that?

[flynnibus]

Just curious, but did you try an optimize/repair on your post/thread tables or decide to not try that?

I did not because based on its description, it does not seem to apply. I have not deleted lots of data from the table and its relatively small.

sql does a good job of maintaining its data - there should not be a reason to run this command blindly.

[flynnibus]

I have watched this more closely and ID'd the following behavior

- this can happen on new threads as well as existing

The real interesting one is this.. and probably why people see so many double postings... The post is posted almost immediately to the forum, yet the user who submitted the post is still sitting for a long time with the wait screen. I can sit in another tab in the browser and see the post in the forum almost instantly. My other tab is perfectly fine speed wise to the same site, and same forum.. even the same thread. This is not the server getting bogged down overall nor is it network based.

So user hits post... sees waiting screen. Post is on the forums almost immediately... user gets tired of waiting or browser times out. They refresh their page and hit post again - Double post results.

This is obviously something going during the post cycle that is taking way too long - yet we can't get any help from Jelsoft here but 'disable plugins' when we've already shown the plugins in use should not be relative to the issue. Guess we'll have to open a ticket if we want to get a real answer towards helping ID the problem.

[peterska2]

To troubleshoot this, please reupload all the original vB non-image files (except install.php). Make sure you upload these in ASCII format and overwrite the ones on the server. Also be sure to upload the admincp files to whichever directory you have set in your config.php file. Then run 'Suspect File Versions' in Diagnostics to make sure you have all the original files for your version:

Admin CP -> Maintenance -> Diagnostics -> Suspect File Versions

[Note: In some cases you may also need to remove any of the listed .xml files in the includes/xml directory.]

Next, disable all plugins.

Note: To temporarily disable the plugin system, edit config.php and add this line right under <?php

define('DISABLE_HOOKS', true);

Then if you still have this problem, create a new style and choose no parent style. This will force it to use the default templates. Finally empty your browser cache, close all browser windows then try again. Make sure you change to the new style and view your forums with it. Now check again.

Do you still have the same problem?

[flynnibus]

Sounds like the 'reboot' answer from most support lines

All these things require taking the site offline so you can understand why admins are not happy with the SOP answer rather then actually being given diagnostic tools or debugging suggestions.

The CYA approach of Jelsoft is getting tiresome rather then actually LOOKING at the issue at hand before suggesting people take their site down to revert everything. So what if it still happens, then what? I still don't have any methods to diagnose or work with it. So I stay up late at night, take my site down, and still don't get anything done.

[flynnibus]

I am currently cloning the current site into another directory on the same server so that testing can be done without further disrupting the main site. By using the same server, I should have the same httpd, php, and sql configs to mimic the test case as closely as possible.

[Steve Machol]

The CYA approach of Jelsoft is getting tiresome rather then actually LOOKING at the issue at hand before suggesting people take their site down to revert everything. So what if it still happens, then what? I still don't have any methods to diagnose or work with it. So I stay up late at night, take my site down, and still don't get anything done.

When an issue cannot be duplicated with the default vB code, then this is standard procedure to troubleshoot the problem. It has nothing to do with CYA as you so nicely put it.

If you do not like this and want to keep you modifications, then you can try asking for help at www.vbulletin.org

[flynnibus]

When an issue cannot be duplicated with the default vB code, then this is standard procedure to troubleshoot the problem. It has nothing to do with CYA as you so nicely put it.

Then why not tell someone how to debug it rather then say 'reboot'

As for it not being duplicated with the default vb code - how do you know? You haven't isolated the conditions that cause it to fail - can you claim it doesn't exist?

Rather then look at any problem, your answer is always 'default it'. I understand the burden of dealing with 3rd party code - but there is a difference between SUPPORTING third party code and denying support for your product when it exists. Usually you work to isolate the two - but your standard answer is 'default it'. When your server misbehaves - Microsoft doesn't demand you remove all installed programs before they even take your call.... which seems to be the Jelsoft Support SOP.

Your support policy of 'default or we don't even talk' is what is tiresome. You expect me to default the entire site BEFORE telling me how to even diagnose the situation. So when I sit around at 3am, disable everything, and the problem still happens - then what?

Instead we are forced to work around the blackbox of vbull because changing the entire production site isn't practical and duplicating the site doesn't give a real equal either, but that is all we are left with.

We are currently using the debug mode in a parallel site to try to dig into the php operation when the slow down is observed.

See... a GOOD support organization would start by helping ID what the posting operation goes through so you can start to monitor which part of the process is the one getting stalled. Instead your lack of support forces someone to treat the entire process one one big black box.. from the time the user hits submit.. to when the page is redisplayed to the poster, rather then looking at the elements of the post being submitted, to the web server.. to the vbull code processing the post, the database writes, any post-processing that happens after a post is submitted, to having the thread page redrawn for the user. We've isolated that its not the db speed, we know its not network between the user and server, we know other pages are being served full speed during the 'stall', and we know the post has been committed to the post table at least at full speed because we know other viewers see it immediately.

A GOOD support organization would work to ID/describe what happens during that period for the user so that the proper tools can be used to diagnose those steps. Once the problem area is ID'd, then you go and identify if its an 1st or 3rd party induced problem.

Instead we get the POOR service which puts the support organization's needs ahead of the practicality of the world the customers live in.

In our TAC we have to deal with all kinds of 3rd party problems we can't control, but you work to ID the root cause - rather then demand the customer put own product in a total vacuum before helping them understand where and how the dependencies work together. Not believe that the world is a perfectly clean white sheet of paper and demand everyone else think the same way.