Server Optimization Request - Large Forum [Archive]

Colosus

Tue 5th Feb '08, 2:17pm

Hello,

Our forum is relatively large and running on a single dedicated server. We have limited funds so buying another server is out of the question. I am hoping we can find some optimizations that we can make so that our site will run more smoothly. We've disabled a lot of the performance hits built into vB already and switched search over to sphinx that gets indexed nightly. But we still have some massive queries that kill our database during the day. Any help would be appreciated. The answers to the questions are listed below.

1. Dedicated server.

2. Server Stats
CPU Dual AMD Operton 280 (dual core, 4 cores total)
RAM 4gb RAM
HD 2x 65GB SCSI RAID 1
OS CentOS 4.5
Apache 2.0.59
PHP 5.2.3
MySQL 4.1.22

3. vB 3.6.4

4. No innodb

5. No extra config options.

6. Top Stats

top - 12:02:52 up 119 days, 13:48, 2 users, load average: 3.15, 3.98, 3.30
Tasks: 133 total, 1 running, 132 sleeping, 0 stopped, 0 zombie
Cpu0 : 62.2% us, 6.0% sy, 0.0% ni, 31.3% id, 0.5% wa, 0.0% hi, 0.0% si
Cpu1 : 12.4% us, 2.0% sy, 0.0% ni, 83.6% id, 2.0% wa, 0.0% hi, 0.0% si
Cpu2 : 28.4% us, 2.5% sy, 0.0% ni, 64.7% id, 4.5% wa, 0.0% hi, 0.0% si
Cpu3 : 24.4% us, 2.0% sy, 0.0% ni, 73.6% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 4146500k total, 3709024k used, 437476k free, 435436k buffers
Swap: 2040244k total, 95512k used, 1944732k free, 2001960k cached

7. mySQL Config

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-locking
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
max_connections = 500
wait_timeout = 120

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 1024M
sort_buffer_size = 1024M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

8. mySQL Stats

+---------------------------------+------------------------------------------+
| Variable_name | Value |
+---------------------------------+------------------------------------------+
| back_log | 50 |
| basedir | / |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| concurrent_insert | ON |
| connect_timeout | 5 |
| datadir | /var/lib/mysql/ |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| expire_logs_days | 0 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
| group_concat_max_len | 1024 |
| have_archive | NO |
| have_bdb | NO |
| have_blackhole_engine | NO |
| have_compress | YES |
| have_crypt | YES |
| have_csv | NO |
| have_example_engine | NO |
| have_geometry | YES |
| have_innodb | YES |
| have_isam | NO |
| have_merge_engine | YES |
| have_ndbcluster | NO |
| have_openssl | NO |
| have_query_cache | YES |
| have_raid | NO |
| have_rtree_keys | YES |
| have_symlink | YES |
| init_connect | |
| init_file | |
| init_slave | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 8388608 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_fast_shutdown | ON |
| innodb_file_io_threads | 4 |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_open_files | 300 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 402653184 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| language | /usr/share/mysql/english/ |
| large_files_support | ON |
| lc_time_names | en_US |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | OFF |
| log_error | |
| log_slave_updates | OFF |
| log_slow_queries | ON |
| log_update | OFF |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 1047552 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 1073741824 |
| max_connect_errors | 10 |
| max_connections | 500 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| myisam_data_pointer_size | 4 |
| myisam_max_extra_sort_file_size | 2147483648 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 67108864 |
| myisam_stats_method | nulls_unequal |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | OFF |
| open_files_limit | 4096 |
| pid_file | /var/lib/mysql/www.tribalwar.com.pid |
| port | 3306 |
| preload_buffer_size | 32768 |
| prepared_stmt_count | 0 |
| protocol_version | 10 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 33554432 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 2093056 |
| read_only | OFF |
| read_rnd_buffer_size | 8384512 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| server_id | 1 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_net_timeout | 3600 |
| slave_transaction_retries | 0 |
| slow_launch_time | 2 |
| socket | /var/lib/mysql/mysql.sock |
| sort_buffer_size | 2097144 |
| sql_mode | |
| sql_notes | ON |
| sql_warnings | ON |
| storage_engine | MyISAM |
| sync_binlog | 0 |
| sync_frm | ON |
| sync_replication | 0 |
| sync_replication_slave_id | 0 |
| sync_replication_timeout | 0 |
| system_time_zone | CST |
| table_cache | 512 |
| table_type | MyISAM |
| thread_cache_size | 8 |
| thread_stack | 196608 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| tmp_table_size | 33554432 |
| tmpdir | |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| version | 4.1.22-standard-log |
| version_comment | MySQL Community Edition - Standard (GPL) |
| version_compile_machine | i686 |
| version_compile_os | pc-linux-gnu |
| wait_timeout | 120 |
+---------------------------------+------------------------------------------+
+----------------------------+------------+
| Variable_name | Value |
+----------------------------+------------+
| Aborted_clients | 1058 |
| Aborted_connects | 330 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 0 |
| Bytes_received | 1631614924 |
| Bytes_sent | 1819380579 |
| Com_admin_commands | 0 |
| Com_alter_db | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 18058 |
| Com_change_db | 1371189 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_checksum | 0 |
| Com_commit | 17989 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_dealloc_sql | 0 |
| Com_delete | 318838 |
| Com_delete_multi | 0 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_drop_user | 0 |
| Com_execute_sql | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 526699 |
| Com_insert_select | 0 |
| Com_kill | 1 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 1 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 0 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 45562 |
| Com_replace_select | 10 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 12319941 |
| Com_set_option | 331 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 5 |
| Com_show_charsets | 83 |
| Com_show_collations | 83 |
| Com_show_column_types | 0 |
| Com_show_create_db | 6 |
| Com_show_create_table | 5 |
| Com_show_databases | 36 |
| Com_show_errors | 0 |
| Com_show_fields | 496 |
| Com_show_grants | 5 |
| Com_show_innodb_status | 0 |
| Com_show_keys | 4542 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_ndb_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_privileges | 0 |
| Com_show_processlist | 1697 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 12069 |
| Com_show_storage_engines | 0 |
| Com_show_tables | 419 |
| Com_show_variables | 211 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_close | 0 |
| Com_stmt_execute | 0 |
| Com_stmt_prepare | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_send_long_data | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 0 |
| Com_update | 2513241 |
| Com_update_multi | 0 |
| Connections | 1372471 |
| Created_tmp_disk_tables | 184930 |
| Created_tmp_files | 51840 |
| Created_tmp_tables | 267683 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 18058 |
| Handler_delete | 207882 |
| Handler_discover | 0 |
| Handler_read_first | 2073056 |
| Handler_read_key | 337756118 |
| Handler_read_next | 2318777882 |
| Handler_read_prev | 1016568838 |
| Handler_read_rnd | 345180111 |
| Handler_read_rnd_next | 3091295677 |
| Handler_rollback | 11030 |
| Handler_update | 2868137 |
| Handler_write | 501762512 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 24807 |
| Key_blocks_used | 324093 |
| Key_read_requests | 1792786968 |
| Key_reads | 640836 |
| Key_write_requests | 979965 |
| Key_writes | 779328 |
| Max_used_connections | 421 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 573 |
| Open_streams | 0 |
| Open_tables | 512 |
| Opened_tables | 57466 |
| Qcache_free_blocks | 2390 |
| Qcache_free_memory | 18294504 |
| Qcache_hits | 10133647 |
| Qcache_inserts | 12094284 |
| Qcache_lowmem_prunes | 154025 |
| Qcache_not_cached | 225369 |
| Qcache_queries_in_cache | 6977 |
| Qcache_total_blocks | 16629 |
| Questions | 29435491 |
| Rpl_status | NULL |
| Select_full_join | 2983 |
| Select_full_range_join | 0 |
| Select_range | 1228086 |
| Select_range_check | 0 |
| Select_scan | 1113072 |
| Slave_open_temp_tables | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 2474 |
| Sort_merge_passes | 118597 |
| Sort_range | 1312741 |
| Sort_rows | 2550311286 |
| Sort_scan | 247873 |
| Table_locks_immediate | 23467026 |
| Table_locks_waited | 276885 |
| Threads_cached | 0 |
| Threads_connected | 42 |
| Threads_created | 57104 |
| Threads_running | 41 |
| Uptime | 395800 |
+----------------------------+------------+
Uptime: 395800 Threads: 42 Questions: 29435492 Slow queries: 2474 Opens: 57466 Flush tables: 1 Open tables: 512 Queries per second avg: 74.370
mysqladmin Ver 8.41 Distrib 4.1.22, for pc-linux-gnu on i686
Copyright (C) 2000 MySQL AB & MySQL Finland AB & TCX DataKonsult AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Server version 4.1.22-standard-log
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/lib/mysql/mysql.sock
Uptime: 4 days 13 hours 56 min 40 sec

Threads: 35 Questions: 29435570 Slow queries: 2474 Opens: 57466 Flush tables: 1 Open tables: 512 Queries per second avg: 74.370
+---------+--------------+-----------+--------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+--------------+-----------+--------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+
| 1331620 | root | localhost | tw_twforums | Sleep | 3 | | |
| 1372429 | tw_tribalwar | localhost | tw_twforums | Query | 0 | update | INSERT INTO post
(showsignature, allowsmilie, username, userid, title, pagetext, iconid, visible, |
| 1372432 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372434 | tw_tribalwar | localhost | tw_twforums | Query | 7 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372436 | tw_tribalwar | localhost | tw_twforums | Query | 7 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372437 | tw_tribalwar | localhost | tw_tribalwar | Sleep | 0 | | |
| 1372439 | tw_tribalwar | localhost | tw_twforums | Query | 6 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372440 | tw_tribalwar | localhost | tw_twforums | Query | 6 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372441 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372442 | tw_tribalwar | localhost | tw_twforums | Query | 6 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372443 | tw_tribalwar | localhost | tw_twforums | Query | 6 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372444 | tw_tribalwar | localhost | tw_twforums | Query | 5 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372445 | tw_tribalwar | localhost | tw_twforums | Query | 5 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372446 | tw_tribalwar | localhost | tw_twforums | Query | 5 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372447 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372448 | tw_tribalwar | localhost | tw_twforums | Query | 4 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372449 | tw_tribalwar | localhost | tw_twforums | Query | 4 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372450 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372451 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372452 | tw_tribalwar | localhost | tw_twforums | Query | 3 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372453 | tw_tribalwar | localhost | tw_twforums | Query | 3 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372454 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372455 | tw_tribalwar | localhost | tw_twforums | Query | 2 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372456 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372457 | tw_tribalwar | localhost | tw_twforums | Query | 0 | | SELECT
thread.threadid, thread.title AS threadtitle, thread.forumid, pollid, open, replycount, |
| 1372458 | tw_tribalwar | localhost | tw_twforums | Query | 2 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372460 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372461 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372462 | tw_tribalwar | localhost | tw_twforums | Query | 1 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372463 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372464 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372465 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
| 1372468 | tw_tribalwar | localhost | tw_twforums | Query | 1 | Sorting result | SELECT post.postid, post.pagetext, IFNULL( user.username , post.username ) AS username, dateline
F |
| 1372469 | root | localhost | | Query | 0 | | show processlist |
| 1372470 | tw_tribalwar | localhost | tw_twforums | Sleep | 0 | | |
+---------+--------------+-----------+--------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+

MySQL 4.1.22-standard-l uptime 4 13:58:44 Tue Feb 5 12:06:55 2008

__ Key __________________________________________________ _______________
Buffer used 316.60M of 384.00M %Used: 82.45
Current 359.88M %Usage: 93.72
Write hit 20.47%
Read hit 99.96%

__ Questions __________________________________________________ _________
Total 29.45M 74.4/s
DMS 15.73M 39.7/s %Total: 53.42
QC Hits 10.14M 25.6/s 34.43
Com_ 1.43M 3.6/s 4.85
COM_QUIT 1.37M 3.5/s 4.66
+Unknown 778.81k 2.0/s 2.64
Slow (10) 2.47k 0.0/s 0.01 %DMS: 0.02 Log: ON
DMS 15.73M 39.7/s 53.42
SELECT 12.33M 31.1/s 41.85 78.35
UPDATE 2.51M 6.4/s 8.54 15.98
INSERT 527.00k 1.3/s 1.79 3.35
DELETE 318.97k 0.8/s 1.08 2.03
REPLACE 45.58k 0.1/s 0.15 0.29
Com_ 1.43M 3.6/s 4.85
change_db 1.37M 3.5/s 4.66
begin 18.08k 0.0/s 0.06
commit 18.01k 0.0/s 0.06

__ SELECT and Sort __________________________________________________ ___
Scan 1.11M 2.8/s %SELECT: 9.03
Range 1.23M 3.1/s 9.97
Full join 2.99k 0.0/s 0.02
Range check 0 0/s 0.00
Full rng join 0 0/s 0.00
Sort scan 247.91k 0.6/s
Sort range 1.31M 3.3/s
Sort mrg pass 118.62k 0.3/s

__ Query Cache __________________________________________________ _______
Memory usage 12.78M of 32.00M %Used: 39.92
Block Fragmnt 19.52%
Hits 10.14M 25.6/s
Inserts 12.10M 30.6/s
Insrt:Prune 78.57:1 30.2/s
Hit:Insert 0.84:1

__ Table Locks __________________________________________________ _______
Waited 277.13k 0.7/s %Total: 1.17
Immediate 23.48M 59.3/s

__ Tables __________________________________________________ ____________
Open 512 of 512 %Cache: 100.00
Opened 57.47k 0.1/s

__ Connections __________________________________________________ _______
Max used 421 of 500 %Max: 84.20
Total 1.37M 3.5/s

__ Created Temp __________________________________________________ ______
Disk table 184.93k 0.5/s
Table 267.74k 0.7/s Size: 32.0M
File 51.85k 0.1/s

__ Threads __________________________________________________ ___________
Running 2 of 6
Cached 3 of 8 %Hit: 95.84
Created 57.14k 0.1/s
Slow 0 0/s

__ Aborted __________________________________________________ ___________
Clients 1.06k 0.0/s
Connects 330 0.0/s

__ Bytes __________________________________________________ _____________
Sent 1.95G 4.9k/s
Received 1.63G 4.1k/s

9. We have a website, but the forums are the main traffic by about 85%. Main website uses different database on same server.

10. 300-500 users. Session timeout 600

11. http://www.tribalwar.com/staff/colosus/info.php

12. Worker Module

KeepAlive On
MaxKeepAliveRequests 2000
KeepAliveTimeout 2

StartServers 10
MaxClients 1024
MinSpareThreads 50
MaxSpareThreads 300
ThreadsPerChild 25
MaxRequestsPerChild 20000

13. We have Apache logging turned off. No files over limit.

14. Linux info

root@www [/home/colosus]# uname -a
Linux www.tribalwar.com 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:30:58 EDT 2007 i686 athlon i386 GNU/Linux
root@www [/home/colosus]# ulimit -aH
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 14335
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@www [/home/colosus]# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2411.002
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4825.52

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2411.002
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4821.62

processor : 2
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2411.002
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4821.66

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2411.002
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4821.62

Colosus

Wed 6th Feb '08, 1:15pm

I switched from eAccellerator to xcache and added memcached instead of the file-based datastore right after I posted this. It helped a bit, but we are still seeing performance issues during peak times.

I really think it has to do with the fact that our post table is 5gb, but we really don't want to prune our old threads as there's a lot of history there. Still looking for any advice you have.

kmike

Thu 7th Feb '08, 7:36am

It's probably not what you wanted to hear, but your forum has outgrown this server, plain and simple. It is still possible to hold longer with the current setup, but it requires a tedious task of scouring for all possible bits of performance everywhere.

What will help the most is an addition of another 4GB of RAM, so the database will (hopefully) fit into RAM completely - it will reduce database-induced I/O load to almost nil, eliminating the locking problems.
Then you can bump the key buffer size to the sum of all used indexes to let MySQL manage them most effectively, reducing IO and CPU load at the same time.

Before going for the RAM upgrade though, run "vmstat 1" for a few minutes during a peak load time and check the "wa" column, it's the percentage of time the systems waits for I/O. If you consistently see values over 20-30%, your system is indeed I/O limited and adding enough RAM for the database to fit in it will help tremendously.

Regardless of that, your MaxClients and Min/MaxSpareThreads values are insanely high, especially that you're using worker MPM. I think you'll be better with the traditional prefork module.
Anyway, go with 256 for MaxClients and 16/32 for Min/MaxSpareThreads, it will free some RAM to spare. Also KeepAlive should be off, no need to keep those Apache processes lingering there waiting for more requests.

Of course it will be much better to drop Apache completely and switch to PHP/FastCGI and lighttpd or nginx as the webserver.

Your my.cnf looks almost good, but you can shave off some memory by decreasing various buffer sizes. Also thread and table cache values are is too low.

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-locking
key_buffer = 512M
max_allowed_packet = 16M
table_cache = 1024
join_buffer_size = 1M
sort_buffer_size = 1M
read_buffer_size = 1M
# leave at default, or no more than 1M
# read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 64M
thread_cache_size = 32
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
max_connections = 500
wait_timeout = 120
max_heap_table_size = 64M
max_allowed_packet = 64M

vBulletin tweaks: switch to "CSS stylesheets as files" ASAP, and I see you have vbSEO but no fancy URLs, any reason you have it enabled? I guess it just sits there sucking the CPU right now.

Don't expect a sudden performance gain from these tweaks though, they are just bandaids until you take the plunge and upgrade (or better, split) the system.

Good luck.

I manage the forum very similar to yours in terms of number of messages, much more members and activity though.

eva2000

Fri 8th Feb '08, 8:37am

Yeah using Apache prefork worker instead of worker module might help somewhat or dump Apache and move to lighttpd or nginx or litespeed web serving software ?. But you are the cross roads of needing a 2nd server i.e. dedicated web + dedicated mysql server.

Try the following in this exact order. You can ignore any of the suggestions that you have already done.

1. Upgrade MySQL server to 4.1.22
2. Upgrade PHP to 5.2.5 if you're on 5.x or PHP 4.4.8 if you're on PHP 4.x after upgrading MySQL (ensure you install as ISAPI and NOT CGI)
3. Edit mysql server's /etc/my.cnf or c:\my.ini for windows and place the following mysql server settings in /etc/my.cnf and restart mysql server afterwards. Make sure to restart mysql server everytime you make changes to your my.cnf for the changes to take effect.

If mysql doesn't restart properly after my.cnf changes and you're on VPS server, make sure skip-innodb entry is removed or commented out from below my.cnf

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
safe-show-database
old_passwords
back_log = 75
skip-innodb
max_connections = 650
key_buffer = 400M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 4000
thread_cache_size = 512
wait_timeout = 15
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
net_buffer_length = 16384
max_connect_errors = 10
thread_concurrency = 8
read_rnd_buffer_size = 786432
bulk_insert_buffer_size = 8M
query_cache_limit = 4M
query_cache_size = 64M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
default-storage-engine = MyISAM

[mysqld_safe]
nice = -10
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 256M
sort_buffer = 256M
read_buffer = 64M
write_buffer = 64M

[mysqlhotcopy]
interactive-timeout

If you get mysql server gone away error messages, then keep increasing wait_timeout value in my.cnf by 60 second increments, then restart mysql after my.cnf changes for it to take effect. Test for a few days and see if you get less or eliminate that error message. If it still occurs, then keep repeating the 60 second increment until the message goes away. Each vB forum and server will have different optimal wait_timeout values depending on your vB forum traffic patterns and server hardware specifications.

4. Remove Eaccelerator or APC Cache if they're installed (check phpinfo.php url of yours to see) and instead install Xcache v1.2.1 final release or Xcache v1.2.2 http://xcache.lighttpd.net/wiki/Release-1.2.2 which seems to be a bit faster than APC Cache - http://www.vbulletin.com/forum/showthread.php?t=213267. Xcache site http://xcache.lighttpd.net/, documentation http://trac.lighttpd.net/trac/wiki/Docs and forums http://forum.lighttpd.net/forum/4

5. Upgrade to vB 3.0.17 http://www.vbulletin.com/forum/showthread.php?t=209720 if you're on vB 3.0.xx or upgrade to vB 3.5.8 http://www.vbulletin.com/forum/showthread.php?t=221903 if you're on vB 3.5.x. Upgrade to vB 3.6.8 PL2 http://www.vbulletin.com/forum/showthread.php?t=247739 if you're on vB 3.6.x

6. If you just upgraded to vB 3.5.x/3.6.x try to disable these 4 options:

Admin CP -> vBulletin Options -> Forums Home Page Options -> Display Logged in Users?

Admin CP -> vBulletin Options -> Forum Display Options (forumdisplay) -> Show Users Browsing Forums

Admin CP -> vBulletin Options -> Thread Display Options -> Show Users Browsing Thread

Admin CP -> vBulletin Options -> Message Searching Options -> Automatic Similar Thread search

7. Check phpinfo.php url to see if you have mod_gzip (Apache 1.3.x) or mod_deflate (Apache 2.x) loaded/installed (on phpinfo.php url page in browser you can do CTRL+F to bring up find prompt and type in mod_gzip or mod_deflate to quickly see). If you have either mod_gzip or mod_deflate loaded, then ensure vB admincp gzip compression is disabled since it's the same thing as mod_gzip or mod_deflate and double compression will just increase cpu loads. If you don't have mod_gzip or mod_deflate installed, then DO NOT set vB admincp gzip compression to a level higher than 1. Higher than 1, will only increase cpu loads unncessarily.

8. Edit httpd.conf values to following and restart apache

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 1

StartServers 20
MaxClients 256
MinSpareThreads 20
MaxSpareThreads 25
ThreadsPerChild 10
MaxRequestsPerChild 1000

Colosus

Fri 8th Feb '08, 1:57pm

After tweaking memcached and xcache, we are down to 0.69 - 1.2 load average throughout the day. Occasionally load average will spike to 2 or 3 for a few seconds, but drop immediately.

There are still more tweaks we plan to do this weekend such as re-indexing the post table and thread table and we're looking at splitting the post table and possibly adding in lighttpd for static content.

I know there's a lot of talk on these forums about just throwing hardware at issues, but I do feel like there's a lot of tweaks and improvements that can be made to increase performance without costing a lot of money. The only hardware upgrade I think we need is adding a bit more RAM.

Thanks for the advice though.