VPS server optimization request

[funkmiester]

I'm experiencing sluggish performance, for example when clicking on forum & thread links, I'm seeing 4-5 second delay before I get a response back and shown either the forum/thread listing or the thread itself. Other than this, things seem to be ok. It was quicker earlier in the forums history, am seeing it get sluggish like this recently though.

1. Virtual (VPS) server. Approx. 15 additional sites sites share this VPS

2. Intel dual xeons with 2.8Ghz to 3.6Ghz CPU
12GB of RAM, multiple SCSI drives set in a RAID 10

My VPS is configured as:

Centos 4
Equal share CPU
2GB Burst RAM
512MB RAM guaranteed
20GB Disk Space

Apache/1.3.34 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.11 FrontPage/5.0.2.2635 mod_ssl/2.8.25 OpenSSL/0.9.7e

PHP version: PHP/4.3.11
MySQL version: 4.1.13

Processor InformationProcessor #1 Vendor: AuthenticAMD
Processor #1 Name: Dual Core AMD Opteron(tm) Processor 265
Processor #1 speed: 896.380 MHz
Processor #1 cache size: 1024 KB
Processor #2 Vendor: AuthenticAMD
Processor #2 Name: Dual Core AMD Opteron(tm) Processor 265
Processor #2 speed: 896.380 MHz
Processor #2 cache size: 1024 KB
Processor #3 Vendor: AuthenticAMD
Processor #3 Name: Dual Core AMD Opteron(tm) Processor 265
Processor #3 speed: 896.380 MHz
Processor #3 cache size: 1024 KB
Processor #4 Vendor: AuthenticAMD
Processor #4 Name: Dual Core AMD Opteron(tm) Processor 265
Processor #4 speed: 896.380 MHz
Processor #4 cache size: 1024 KB

System Information
Linux 2.6.8-022stab078.9-enterprise #1 SMP Thu Jun 8 12:38:51 MSD 2006 i686 athlon i386 GNU/Linux

Current Memory Usage total used free shared buffers cached
Mem: 8239196 8180332 58864 0 109680 2801408
-/+ buffers/cache: 5269244 2969952
Swap: 8289532 3833216 4456316
Total: 16528728 12013548 4515180
Current Disk UsageFilesystem Size Used Avail Use% Mounted on
/dev/vzfs 19G 15G 4.3G 78% /
vzfs 308M 92K 308M 1% /tmp

3. No innodb type databases/tables

4. Unknown

5.
top - 13:56:08 up 7 days, 21:00, 1 user, load average: 0.08, 0.06, 0.07
Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6% us, 0.1% sy, 0.0% ni, 99.1% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8239196k total, 8156716k used, 82480k free, 120760k buffers
Swap: 8289532k total, 3833416k used, 4456116k free, 2805432k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1962 nobody 16 0 49752 24m 42m S 0.7 0.3 0:55.98 httpd
27702 nobody 15 0 48716 22m 42m S 0.7 0.3 0:30.67 httpd
9888 nobody 15 0 45128 5568 42m S 0.7 0.1 0:00.41 httpd
11755 root 16 0 44740 4372 42m S 0.3 0.1 0:18.75 httpd
5862 nobody 15 0 50704 25m 42m S 0.3 0.3 0:47.38 httpd
25989 nobody 15 0 48792 20m 42m S 0.3 0.3 0:33.14 httpd
19933 ghpg 15 0 1820 944 1620 R 0.3 0.0 0:00.17 top
1 root 15 0 1476 488 1320 S 0.0 0.0 0:02.18 init
11367 root 15 0 1464 536 1300 S 0.0 0.0 0:07.79 syslogd
11387 named 25 0 67200 13m 4780 S 0.0 0.2 0:00.00 named
11403 root 16 0 3656 1316 3440 S 0.0 0.0 0:12.16 sshd
11417 root 16 0 2016 720 1688 S 0.0 0.0 0:00.61 xinetd
11450 root 15 0 7228 2472 3044 S 0.0 0.0 0:18.48 chkservd
11507 mailnull 16 0 6764 1644 6304 S 0.0 0.0 0:01.56 exim
11513 mailnull 15 0 6732 1540 6304 S 0.0 0.0 0:00.02 exim
11520 root 16 0 3404 1652 2792 S 0.0 0.0 0:03.00 antirelayd
11605 root 15 0 28096 9424 4132 S 0.0 0.1 0:04.14 spamd
11664 root 16 0 5860 1452 5432 S 0.0 0.0 0:00.89 pure-ftpd
11669 root 16 0 3556 748 3216 S 0.0 0.0 0:00.01 pure-authd
11695 root 15 0 1516 628 1360 S 0.0 0.0 0:01.01 crond
11707 root 17 0 2056 900 1876 S 0.0 0.0 0:00.00 mysqld_safe
11744 mysql 15 0 100m 40m 5348 S 0.0 0.5 0:07.09 mysqld
11747 mysql 16 0 100m 40m 5348 S 0.0 0.5 0:00.57 mysqld
11748 mysql 16 0 100m 40m 5348 S 0.0 0.5 0:31.74 mysqld
11776 mysql 16 0 100m 40m 5348 S 0.0 0.5 1:42.54 mysqld
11958 mailnull 15 0 7472 3120 4472 S 0.0 0.0 0:02.15 eximstats
12034 root 34 19 13176 8372 4804 S 0.0 0.1 0:57.85 cpanellogd
12040 mysql 16 0 100m 40m 5348 S 0.0 0.5 1:41.44 mysqld
12061 root 16 0 9940 3148 7568 S 0.0 0.0 0:02.52 cppop
12127 mailman 17 0 7916 2652 4688 S 0.0 0.0 0:00.02 mailmanctl
12140 root 18 0 3972 816 3628 S 0.0 0.0 0:00.00 saslauthd

6.
[mysqld]
skip-innodb
max_connections = 350
key_buffer = 16M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 10
thread_concurrency = 4
query_cache_limit = 1M
query_cache_size = 64M
query_cache_type = 1
query_prealloc_size = 16384
query_alloc_block_size = 16384

[mysqld_safe]
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M


7.
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_length_for_sort_data | 1024 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| myisam_data_pointer_size | 4 |
| myisam_max_extra_sort_file_size | 2147483648 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 67108864 |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | OFF |
| open_files_limit | 3960 |
| pid_file | /var/lib/mysql/hq.harrisonhq.com.pid |
| port | 3306 |
| preload_buffer_size | 32768 |
| protocol_version | 10 |
| query_alloc_block_size | 16384 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 67108864 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 16384 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 1044480 |
| read_only | OFF |
| read_rnd_buffer_size | 262144 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| server_id | 0 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_net_timeout | 3600 |
| slave_transaction_retries | 0 |
| slow_launch_time | 2 |
| socket | /var/lib/mysql/mysql.sock |
| sort_buffer_size | 2097144 |
| sql_mode | |
| storage_engine | MyISAM |
| sql_notes | OFF |
| sql_warnings | OFF |
| sync_binlog | 0 |
| sync_replication | 0 |
| sync_replication_slave_id | 0 |
| sync_replication_timeout | 0 |
| sync_frm | ON |
| system_time_zone | EST |
| table_cache | 1800 |
| table_type | MyISAM |
| thread_cache_size | 384 |
| thread_stack | 126976 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| tmp_table_size | 67108864 |
| tmpdir | |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| version | 4.1.13-standard |
| version_comment | MySQL Community Edition - Standard (GPL) |
| version_compile_machine | i686 |
| version_compile_os | pc-linux-gnu |
| wait_timeout | 20 |
+---------------------------------+------------------------------------------+
+----------------------------+------------+
| Variable_name | Value |
+----------------------------+------------+
| Aborted_clients | 1376 |
| Aborted_connects | 0 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 0 |
| Bytes_received | 231803029 |
| Bytes_sent | 2857083894 |
| Com_admin_commands | 0 |
| Com_alter_db | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 75994 |
| Com_change_master | 0 |
| Com_check | 131 |
| Com_checksum | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_dealloc_sql | 0 |
| Com_delete | 10047 |
| Com_delete_multi | 0 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_drop_user | 0 |
| Com_execute_sql | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 27047 |
| Com_insert_select | 84 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 3166 |
| Com_optimize | 131 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 0 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 19414 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 209933 |
| Com_set_option | 4387 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_charsets | 0 |
| Com_show_collations | 0 |
| Com_show_column_types | 0 |
| Com_show_create_db | 0 |
| Com_show_create_table | 1072 |
| Com_show_databases | 0 |
| Com_show_errors | 0 |
| Com_show_fields | 1072 |
| Com_show_grants | 0 |
| Com_show_innodb_status | 0 |
| Com_show_keys | 0 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_privileges | 0 |
| Com_show_processlist | 2279 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 7 |
| Com_show_storage_engines | 0 |
| Com_show_tables | 1112 |
| Com_show_variables | 115 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_prepare | 0 |
| Com_stmt_execute | 0 |
| Com_stmt_send_long_data | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_close | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 3166 |
| Com_update | 303215 |
| Com_update_multi | 347 |
| Connections | 82958 |
| Created_tmp_disk_tables | 5750 |
| Created_tmp_files | 50 |
| Created_tmp_tables | 17951 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 18552 |
| Handler_discover | 0 |
| Handler_read_first | 29967 |
| Handler_read_key | 17229094 |
| Handler_read_next | 6057190 |
| Handler_read_prev | 128464 |
| Handler_read_rnd | 1543808 |
| Handler_read_rnd_next | 37189104 |
| Handler_rollback | 0 |
| Handler_update | 3431973 |
| Handler_write | 1779690 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 10225 |
| Key_blocks_used | 9943 |
| Key_read_requests | 31341230 |
| Key_reads | 32625 |
| Key_write_requests | 397230 |
| Key_writes | 41520 |
| Max_used_connections | 11 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 705 |
| Open_streams | 0 |
| Open_tables | 393 |
| Opened_tables | 574 |
| Qcache_free_blocks | 2666 |
| Qcache_free_memory | 46900144 |
| Qcache_hits | 445692 |
| Qcache_inserts | 202050 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 7883 |
| Qcache_queries_in_cache | 6638 |
| Qcache_total_blocks | 16534 |
| Questions | 1190715 |
| Rpl_status | NULL |
| Select_full_join | 51 |
| Select_full_range_join | 50 |
| Select_range | 34518 |
| Select_range_check | 0 |
| Select_scan | 72010 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slave_retried_transactions | 0 |
| Slow_launch_threads | 0 |
| Slow_queries | 66 |
| Sort_merge_passes | 25 |
| Sort_range | 37214 |
| Sort_rows | 3069060 |
| Sort_scan | 28193 |
| Table_locks_immediate | 785258 |
| Table_locks_waited | 111 |
| Threads_cached | 10 |
| Threads_connected | 1 |
| Threads_created | 11 |
| Threads_running | 1 |
| Uptime | 681505 |
+----------------------------+------------+
Uptime: 681506 Threads: 2 Questions: 1190725 Slow queries: 66 Opens: 574 Flush tables: 1 Open tables: 393 Queries per second avg: 1.747
mysqladmin Ver 8.41 Distrib 4.1.13, for pc-linux-gnu on i686
Copyright (C) 2000 MySQL AB & MySQL Finland AB & TCX DataKonsult AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license
Server version 4.1.13-standard
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/lib/mysql/mysql.sock
Uptime: 7 days 21 hours 18 min 26 sec
Threads: 2 Questions: 1190725 Slow queries: 66 Opens: 574 Flush tables: 1 Open tables: 393 Queries per second avg: 1.747

8. 2 vB installations - 1 with it's own database and 1 with both vB and PhotoPost Pro sharing a single database

9. 15 max concurrent users on each of the forums. Session timeout is 900 seconds

10. http://www.ghpg.net/phpinfo.php

11.
KeepAlive = On
MaxKeepAliveRequests = 100
KeepAliveTimeout = 5
MinSpareServers = 5
MaxSpareServers = 10
StartServers = 5
MaxClients = 150
AND Maxrequestsperchild value = 500

12. vB 3.6.4

13. None

14.
uname -a
Linux 2.6.8-022stab078.9-enterprise #1 SMP Thu Jun 8 12:38:51 MSD 2006 i686 athlon i386 GNU/Linux

ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited


cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 265
stepping : 2
cpu MHz : 896.380
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni
bogomips : 3522.56
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 265
stepping : 2
cpu MHz : 896.380
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni
bogomips : 3579.90
processor : 2
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 265
stepping : 2
cpu MHz : 896.380
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni
bogomips : 3579.90
processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 265
stepping : 2
cpu MHz : 896.380
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni
bogomips : 3579.90

[jason|xoxide]

That server is swapping heavily so your provider is probably oversubscribing the box quite a bit. They're also lying about that 12GB of RAM as that box only has 8GB.

In any case, ask them to update to the newest OpenVZ kernel (they've long since moved to a 2.6.9 based on the RHEL4 kernel) as that one is pretty old.

Also please post the output of 'cat /proc/user_beancounters', that will let us know if you are reaching your VPS limits.

[funkmiester]

Thanks Jason - how do the mysql and apache settings look?

cat /proc/user_beancounters


Version: 2.5
uid resource held maxheld barrier limit failcnt
2390: kmemsize 10021053 10038750 37925683 45510819 0
lockedpages 0 0 14744 16384 0
privvmpages 114414 114419 471860 524288 0
shmpages 9051 9051 524288 524288 0
dummy 0 0 0 0 0
numproc 68 68 500 500 0
physpages 50000 50001 0 2147483647 0
vmguarpages 0 0 131072 2147483647 0
oomguarpages 71625 71626 131072 2147483647 0
numtcpsock 33 33 500 500 0
numflock 2 2 800 880 0
numpty 1 1 255 255 0
numsiginfo 0 0 2048 2048 0
tcpsndbuf 46788 57928 21463040 41943040 0
tcprcvbuf 0 692 21463040 41943040 0
othersockbuf 37004 37004 6012928 16252928 0
dgramrcvbuf 0 692 943720 1048576 0
numothersock 27 27 500 500 0
dcachesize 1162924 1165794 33554432 34537472 0
numfile 3005 3006 52428 52428 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 531 531 15360 15360 0


If I'm close to reaching my limits - the next bump up available from my provider is:

Centos 4 (same)
Equal share CPU (same)
2.5GB Burst RAM (1/2 gig more)
768MB RAM guaranteed (256MB more)
25GB Disk Space (extra 5GB disk space)

Is that sufficient to last a while longer with demonstratable performance improvements?

[jason|xoxide]

Thanks Jason - how do the mysql and apache settings look?

cat /proc/user_beancounters

Well, you haven't exceeded any resource limits (when you do, the last column will increment) so that's good.

As to the httpd.conf settings, you don't seem to have that many concurrent users so you should probably try dialing those back to conserve memory. Something like the following would work (it's what I use on my small forums).

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 3

StartServers 4
MinSpareServers 3
MaxSpareServers 5
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 512

Try it with keepalives on and off and use whichever seems more responsive. On sites that are short on RAM, "off" generally works better.

You should consider upgrading your PHP to a newer version as that one is pretty old. If possible, move either to 4.4.4 or to 5.1.6 (or 5.2.0 if you are feeling ambitious). Keep in mind though that any PHP upgrade is going to mean that you will need to reinstall eAccelerator and the Zend Optimizer.

On that, your version of eAccelerator is pretty old. Even if you don't upgrade PHP you should still upgrade to the newest version of eA (which is 0.9.5). You should also remove Zend Optimizer if you don't actually have any software that requires it.

As to your my.cnf, it doesn't really matter much when you're only getting 1.747 queries per second. Nothing you change here is going to make an explosive difference. In any case, eva2000 is better at that than I am so I'll leave it to him (my expertise on MySQL is pretty much related to what works for me).

[funkmiester]

OK thanks, I'll wait to hear from eva2000.

[eva2000]

yeah your queries/sec is low so my.cnf looks okay but PHP and MySQL are severely out dated

PHP 4.4.4 and MySQL 4.1.22 would be more ideal

Happy New Year :)

[funkmiester]

Ok, I've upgraded Apache, PHP, eAccelerator and MySQL - so my configuration files look ok and are still valid with these updates to the core applications?

If so, with just the upgrades in place should I be seeing instant performance improvements or will it take a few days as things are cached by eAccelerator? (If I'm understanding how it works correctly)! Right at this moment, I don't see any dramatic improvements.

Thanks.

[jason|xoxide]

I just visited your site and it seems fine. If you are seeing performance problems at certain points of the day it's probably not related to your site. In all likelihood, the reason your site is slow is because the server is oversubscribed.

[funkmiester]

Ok, thanks for your help Jason and eva2000, much appreciated.

Will have a word with my hosting company if things don't improve :)