Really slow post times

[dzeanah]

Well, everything was running pretty well, except the packages in the linux distro I was running were a little quirky and nonstandard, and I wanted to install a RAID controller anyway, so we reinstalled.

Now we're looking at a situation where everything's working fine, except a certain percentage of new posts take forever to post. Like minutes. Even on test forums with just a few total posts...

Anyway, here's the info:

1. your server specs, such as mysql and php version
Redhat 8.0, 2x PIII 933MHz, 3 SCSI-3 9g drives set up as a hardware RAID-5 array, 1.5 G RAM. Newest MySql (4.0.9) with PHP 4.3.0.
2. if possible how mysql was compiled/installed
Unknown. Can find out if it really matters.
3. your top stats 6:20pm up 1 day, 16:15, 4 users, load average: 1.60, 2.14, 2.13
173 processes: 169 sleeping, 1 running, 3 zombie, 0 stopped
CPU0 states: 8.4% user, 2.1% system, 0.0% nice, 88.3% idle
CPU1 states: 17.2% user, 1.4% system, 0.0% nice, 80.2% idle
Mem: 1547576K av, 1445656K used, 101920K free, 0K shrd, 120448K buff
Swap: 2040244K av, 0K used, 2040244K free 1121872K cached


Yeah -- don't know what's up with the zombies. Bad sendmail BAD!

4. your mysql configuration variables located at /etc/my.cnf
# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-innodb
skip-external-locking
set-variable = max_connections=650
set-variable = key_buffer=64M
set-variable = myisam_sort_buffer_size=64M
set-variable = join_buffer=2M
set-variable = record_buffer=2M
set-variable = sort_buffer=4M
set-variable = table_cache=1024
set-variable = thread_cache_size=512
set-variable = wait_timeout=7200
set-variable = connect_timeout=10
set-variable = query_cache_size=16M
[mysql.server]
user=mysql
basedir=/var/lib
[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192
[mysqldump]
set-variable = max_allowed_packet=128M
[myisamchk]
set-variable = key_buffer=128M
set-variable = sort_buffer=128M
set-variable = read_buffer=16M
set-variable = write_buffer=16M

5. your mysql extended-status outputYou have new mail in /var/spool/mail/derek
[root@internal admin]# mysqladmin extended-status
+--------------------------+-----------+
| Variable_name | Value |
+--------------------------+-----------+
| Aborted_clients | 8 |
| Aborted_connects | 0 |
| Bytes_received | 30577326 |
| Bytes_sent | 413443729 |
| Com_admin_commands | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 10400 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 677 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 1069 |
| Com_insert_select | 66 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 351 |
| Com_replace_select | 5 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_select | 59980 |
| Com_set_option | 0 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 0 |
| Com_show_databases | 0 |
| Com_show_fields | 0 |
| Com_show_grants | 0 |
| Com_show_keys | 0 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 21 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 2 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 0 |
| Com_show_variables | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 0 |
| Com_update | 16248 |
| Connections | 7693 |
| Created_tmp_disk_tables | 18 |
| Created_tmp_tables | 4685 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 1658 |
| Handler_read_first | 4036 |
| Handler_read_key | 815395 |
| Handler_read_next | 4747128 |
| Handler_read_prev | 93610 |
| Handler_read_rnd | 443752 |
| Handler_read_rnd_next | 4007321 |
| Handler_rollback | 0 |
| Handler_update | 15513 |
| Handler_write | 368277 |
| Key_blocks_used | 5534 |
| Key_read_requests | 2082249 |
| Key_reads | 5486 |
| Key_write_requests | 15101 |
| Key_writes | 12626 |
| Max_used_connections | 20 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 83 |
| Open_files | 148 |
| Open_streams | 0 |
| Opened_tables | 89 |
| Questions | 181856 |
| Qcache_queries_in_cache | 1704 |
| Qcache_inserts | 59968 |
| Qcache_hits | 85362 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 12 |
| Qcache_free_memory | 13980208 |
| Qcache_free_blocks | 786 |
| Qcache_total_blocks | 4254 |
| Rpl_status | NULL |
| Select_full_join | 24 |
| Select_full_range_join | 0 |
| Select_range | 9753 |
| Select_range_check | 0 |
| Select_scan | 5477 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 0 |
| Sort_merge_passes | 0 |
| Sort_range | 7628 |
| Sort_rows | 451874 |
| Sort_scan | 5116 |
| Table_locks_immediate | 111434 |
| Table_locks_waited | 215 |
| Threads_cached | 13 |
| Threads_created | 21 |
| Threads_connected | 8 |
| Threads_running | 1 |
| Uptime | 4690 |
+--------------------------+-----------+
[root@internal admin]#

6. oh and is your vB the only thing on the server? or other scripts? sites?
A few more very low-bandwidth sites are hosted here. Just flat files and images.

7. how many average and max concurrent users on your vB forum ?
Probably 275-325 concurrent (70-110 connections at any given moment at the firewall) users, and max of 457.

8. create a file named phpinfo.php and place this code in it and post the url/link to it from your web site
Link here (http://www.thehighroad.org/phpinfo.php).

Probably unimportant, but httpd settings here:Timeout 300
Keepalive On
MaxKeepAliveRequests 100
KeepAliveTimeout 10
MinspareServers 5
MaxSpareServers 30
Startservers 50
MaxClients 250

Any thoughts?

[dzeanah]

I should point out that the problem is inconsistent -- one post might go perfectly, and the next time out for 3 minutes.

A post that looks like it's going to time out ends up going through, even if the user backs up and resubmits (making it 2 posts), and resubmits (3 posts), etc.

Mods are busy. ;)

[wajones]

Originally posted by dzeanah
I should point out that the problem is inconsistent -- one post might go perfectly, and the next time out for 3 minutes.

A post that looks like it's going to time out ends up going through, even if the user backs up and resubmits (making it 2 posts), and resubmits (3 posts), etc.

Mods are busy. ;)

I'm having the same problems with 2.2.9 ?

[dzeanah]

Originally posted by wajones
I'm having the same problems with 2.2.9 ? 2.2.9 here too.

Note that I didn't have this problem before the upgrade, running older binaries.

More info: originally I let the Redhat install do its thing, and ended up with Apache 2 and MySQL 3.23.52 and some older version of PHP -- same issue. We hoped the upgrade/downgrade process would help, but nope...

[dzeanah]

Turns out the problem was either with sendmail, or with the way apache chose to interact with sendmail.

We ended up with a couple hundred zombie sendmail processes, which struck me as a lot. Each of the zombies had apache as owner. Apparently something was screwing up when people were posting to a thread with e-mail notification turned on, which caused a delay, which somehow caused sendmail to stop responding, and so on.

I turned off e-mail functions on the board and the problem seemed to go away. *poof*

So, now we're running procmail and everything seems to be going ok. <crossing fingers>

[eva2000]

Originally posted by dzeanah
Turns out the problem was either with sendmail, or with the way apache chose to interact with sendmail.

We ended up with a couple hundred zombie sendmail processes, which struck me as a lot. Each of the zombies had apache as owner. Apparently something was screwing up when people were posting to a thread with e-mail notification turned on, which caused a delay, which somehow caused sendmail to stop responding, and so on.

I turned off e-mail functions on the board and the problem seemed to go away. *poof*

So, now we're running procmail and everything seems to be going ok. <crossing fingers> hmmm alot of still unproven versions you have there.. rh8, php 4.3 and mysql 4.x

but from what you described it's definitely sendmail,

have you looked at your sendmail logs ? do you require members on your forum to verify their email addresses ? some could be dead and bouncing back to your server

[dzeanah]

Originally posted by eva2000
hmmm alot of still unproven versions you have there.. rh8, php 4.3 and mysql 4.x

but from what you described it's definitely sendmail,

have you looked at your sendmail logs ? do you require members on your forum to verify their email addresses ? some could be dead and bouncing back to your server Well, when the older, more stable versions weren't doing the job anyway... :D

Looks like there might be some advantages to MySql 4's ability to cache queries. You can read this better than I can -- here:| Handler_read_next | 51504818 |
| Handler_read_prev | 841193 |
| Handler_read_rnd | 3974061 |
| Handler_read_rnd_next | 41449677 |
| Handler_rollback | 0 |
| Handler_update | 148454 |
| Handler_write | 2941162 |
| Key_blocks_used | 14036 |
| Key_read_requests | 20020897 |
| Key_reads | 13379 |
| Key_write_requests | 167505 |
| Key_writes | 134315 |
| Max_used_connections | 40 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 120 |
| Open_files | 190 |
| Open_streams | 0 |
| Opened_tables | 126 |
| Questions | 1709871 |
| Qcache_queries_in_cache | 4262 |
| Qcache_inserts | 568192 |
| Qcache_hits | 793931 |
| Qcache_lowmem_prunes | 4336 |
| Qcache_not_cached | 71 |
| Qcache_free_memory | 10733712 |
| Qcache_free_blocks | 2276 |
| Qcache_total_blocks | 10885 |
| Rpl_status | NULL |
| Select_full_join | 373 |
| Select_full_range_join | 0 |
| Select_range | 87540 |
| Select_range_check | 0 |
| Select_scan | 51946 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 0 |
| Sort_merge_passes | 0 |
| Sort_range | 70614 |
| Sort_rows | 4133267 |
| Sort_scan | 48547 |
| Table_locks_immediate | 1056109 |
| Table_locks_waited | 1373 |
| Threads_cached | 39 |
| Threads_created | 41 |
| Threads_connected | 2 |
| Threads_running | 1 |
| Uptime | 66140 |
+--------------------------+------------+
[root@internal root]# uptime
11:23am up 2 days, 9:18, 2 users, load average: 0.15, 0.29, 0.32
[root@internal root]#


Looks like a number of queries are being drawn directly from the cache, instead of hitting the disk. Here's another look at the board: Server Load Averages0.70, 0.43, 0.37 282 users online (188 members & 94 guests).That's the highest I've seen it all morning; it has been much higher (see original post for a "top" while under the load of 314 simultaneous users). I'm inclined to think it's because we're less bound to the disk -- most people browse more than they post, after all.

Can't comment on the newer PHP version though -- my co-admin made that call.

[dzeanah]

Whoops -- didn't respond to Sendmail questions...

I just peeked in the log, and saw a lot of delays in there. Looks like Sendmail might have been doing a good job of slowing down spam, but at the cost of response time for my own machine. I'm not willing to commit to that being all of it though -- no way should 200 zombie sendmail processes have accumulated in 2 hours. No way.

Of course, it could be entirely my fault -- I'm a bright guy, but Sendmail isn't something I have the time or desire to master. :o

Anyway, a switch to Procmail was the fix in my case. For anyone browsing this thread looking for a fix to their own problem, try disabling e-mail functions and see if it helps.

[waddy]

Thanks so much for the info, i have the exact same problem, the site and forums are fast, as soon as someone tries to post a new thread or reply = Very slow

I have been pulling my hair out on this one. Looks like i may have the same problem you desctribe, and yes it only started happening after the upgrade to 2.2.9

Thanks.