PDA

View Full Version : [Server Optimisation] Very High Load Problem - Over 50 load on peak hours


pulsorock
Thu 5th Jul '07, 8:19pm
I'm just setup a new server and the specs are pretty good, but some reason the load is being too high. So, hopefully is some configuration problems that can be optimized. Here are the details:

1. Dedicated Server

2. Server Details:

- HP ProLiant DL380 G4 Dual Xeon 3.6Ghz 6GB RAM Server with 3x 36.4GB Hot Plug SCSI Hard Drive
- Processor - 2x Intel Xeon 3.6 GHz/800MHz - 1MB L2
- Memory - 6 GB PC2-3200R 400MHz DDR2 with Advanced ECC and online spare memory capabilities
- Hard Drive - 3x original COMPAQ HP 36.4GB 10K Hot Pluggable SCSI Hard Drive
- Storage Controller - Ultra320 Smart Array 6i Controller (integrated on system board) with 128MB Battery Backed Write Cache...

CentOS Linux 5.0
Apache/2.2.3 (CentOS)
PHP Version 5.1.6
MySQL 5.0.22

3. vB 3.6.7

4. No innodb tables/db

5. MySQL was installed at setup of CentOS 5.0

6. Top Stats as of now (been higher):
top - 18:34:03 up 6 days, 18:26, 1 user, load average: 15.58, 30.34, 22.30
Tasks: 144 total, 1 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 31.8%us, 1.7%sy, 0.0%ni, 66.2%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3632116k total, 2590008k used, 1042108k free, 271248k buffers
Swap: 2031608k total, 0k used, 2031608k free, 1636788k cached7. MySQL my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid8. MySQL Stats
+---------------------------------+--------------------------------------------------------+
| Variable_name | Value |
+---------------------------------+--------------------------------------------------------+
| auto_increment_increment | 1 |
| auto_increment_offset | 1 |
| automatic_sp_privileges | ON |
| back_log | 50 |
| basedir | /usr/ |
| bdb_cache_size | 8388600 |
| bdb_home | /var/lib/mysql/ |
| bdb_log_buffer_size | 32768 |
| bdb_logdir | |
| bdb_max_lock | 10000 |
| bdb_shared_data | OFF |
| bdb_tmpdir | /tmp/ |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 5 |
| datadir | /var/lib/mysql/ |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| div_precision_increment | 4 |
| engine_condition_pushdown | OFF |
| expire_logs_days | 0 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
| group_concat_max_len | 1024 |
| have_archive | NO |
| have_bdb | YES |
| have_blackhole_engine | NO |
| have_compress | YES |
| have_crypt | YES |
| have_csv | NO |
| have_example_engine | NO |
| have_federated_engine | NO |
| have_geometry | YES |
| have_innodb | YES |
| have_isam | NO |
| have_ndbcluster | NO |
| have_openssl | DISABLED |
| have_query_cache | YES |
| have_raid | NO |
| have_rtree_keys | YES |
| have_symlink | YES |
| init_connect | |
| init_file | |
| init_slave | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 8388608 |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_io_threads | 4 |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_open_files | 300 |
| innodb_support_xa | ON |
| innodb_sync_spin_loops | 20 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| innodb_thread_sleep_delay | 10000 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 8388600 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| language | /usr/share/mysql/english/ |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_error | |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 1048576 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 1073741824 |
| max_connect_errors | 10 |
| max_connections | 100 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| multi_range_count | 256 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 8388608 |
| myisam_stats_method | nulls_unequal |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | ON |
| open_files_limit | 1024 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| pid_file | /var/run/mysqld/mysqld.pid |
| prepared_stmt_count | 0 |
| port | 3306 |
| preload_buffer_size | 32768 |
| protocol_version | 10 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 131072 |
| read_only | OFF |
| read_rnd_buffer_size | 262144 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| server_id | 0 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_compressed_protocol | OFF |
| slave_load_tmpdir | /tmp/ |
| slave_net_timeout | 3600 |
| slave_skip_errors | OFF |
| slave_transaction_retries | 10 |
| slow_launch_time | 2 |
| socket | /var/lib/mysql/mysql.sock |
| sort_buffer_size | 2097144 |
| sql_mode | |
| sql_notes | ON |
| sql_warnings | ON |
| storage_engine | MyISAM |
| sync_binlog | 0 |
| sync_frm | ON |
| system_time_zone | AST |
| table_cache | 64 |
| table_lock_wait_timeout | 50 |
| table_type | MyISAM |
| thread_cache_size | 0 |
| thread_stack | 196608 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| timed_mutexes | OFF |
| tmp_table_size | 33554432 |
| tmpdir | |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| updatable_views_with_limit | YES |
| version | 5.0.22 |
| version_bdb | Sleepycat Software: Berkeley DB 4.1.24: (May 25, 2006) |
| version_comment | Source distribution |
| version_compile_machine | i686 |
| version_compile_os | redhat-linux-gnu |
| wait_timeout | 28800 |
+---------------------------------+--------------------------------------------------------+
+-----------------------------------+------------+
| Variable_name | Value |
+-----------------------------------+------------+
| Aborted_clients | 5384 |
| Aborted_connects | 122 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 0 |
| Bytes_received | 754184230 |
| Bytes_sent | 3014396152 |
| Com_admin_commands | 66355 |
| Com_alter_db | 0 |
| Com_alter_table | 1 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 203799 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_checksum | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_dealloc_sql | 0 |
| Com_delete | 50199 |
| Com_delete_multi | 0 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_drop_user | 0 |
| Com_execute_sql | 0 |
| Com_flush | 0 |
| Com_grant | 2 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 56216 |
| Com_insert_select | 0 |
| Com_kill | 0 |
| Com_load | 1 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 0 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 3562 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 3234926 |
| Com_set_option | 761425 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 12 |
| Com_show_charsets | 87 |
| Com_show_collations | 87 |
| Com_show_column_types | 0 |
| Com_show_create_db | 2 |
| Com_show_create_table | 15 |
| Com_show_databases | 83 |
| Com_show_errors | 0 |
| Com_show_fields | 88 |
| Com_show_grants | 39 |
| Com_show_innodb_status | 0 |
| Com_show_keys | 60 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_ndb_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_privileges | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 5 |
| Com_show_storage_engines | 1 |
| Com_show_tables | 192 |
| Com_show_triggers | 0 |
| Com_show_variables | 179 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_close | 0 |
| Com_stmt_execute | 0 |
| Com_stmt_fetch | 0 |
| Com_stmt_prepare | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_send_long_data | 0 |
| Com_truncate | 194 |
| Com_unlock_tables | 0 |
| Com_update | 105421 |
| Com_update_multi | 0 |
| Com_xa_commit | 0 |
| Com_xa_end | 0 |
| Com_xa_prepare | 0 |
| Com_xa_recover | 0 |
| Com_xa_rollback | 0 |
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 215335 |
| Created_tmp_disk_tables | 136101 |
| Created_tmp_files | 5 |
| Created_tmp_tables | 310484 |
| Delayed_errors | 1 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 356 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 7410 |
| Handler_discover | 0 |
| Handler_prepare | 0 |
| Handler_read_first | 163721 |
| Handler_read_key | 32652392 |
| Handler_read_next | 42851611 |
| Handler_read_prev | 42636 |
| Handler_read_rnd | 16602031 |
| Handler_read_rnd_next | 163339821 |
| Handler_rollback | 0 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 3031963 |
| Handler_write | 40252392 |
| Innodb_buffer_pool_pages_data | 19 |
| Innodb_buffer_pool_pages_dirty | 0 |
| Innodb_buffer_pool_pages_flushed | 0 |
| Innodb_buffer_pool_pages_free | 493 |
| Innodb_buffer_pool_pages_latched | 0 |
| Innodb_buffer_pool_pages_misc | 0 |
| Innodb_buffer_pool_pages_total | 512 |
| Innodb_buffer_pool_read_ahead_rnd | 1 |
| Innodb_buffer_pool_read_ahead_seq | 0 |
| Innodb_buffer_pool_read_requests | 77 |
| Innodb_buffer_pool_reads | 12 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 0 |
| Innodb_data_fsyncs | 3 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_data_pending_reads | 0 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_read | 2494464 |
| Innodb_data_reads | 25 |
| Innodb_data_writes | 3 |
| Innodb_data_written | 1536 |
| Innodb_dblwr_pages_written | 0 |
| Innodb_dblwr_writes | 0 |
| Innodb_log_waits | 0 |
| Innodb_log_write_requests | 0 |
| Innodb_log_writes | 1 |
| Innodb_os_log_fsyncs | 3 |
| Innodb_os_log_pending_fsyncs | 0 |
| Innodb_os_log_pending_writes | 0 |
| Innodb_os_log_written | 512 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 0 |
| Innodb_pages_read | 19 |
| Innodb_pages_written | 0 |
| Innodb_row_lock_current_waits | 0 |
| Innodb_row_lock_time | 0 |
| Innodb_row_lock_time_avg | 0 |
| Innodb_row_lock_time_max | 0 |
| Innodb_row_lock_waits | 0 |
| Innodb_rows_deleted | 0 |
| Innodb_rows_inserted | 0 |
| Innodb_rows_read | 0 |
| Innodb_rows_updated | 0 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 6998 |
| Key_blocks_used | 260 |
| Key_read_requests | 56728692 |
| Key_reads | 7278 |
| Key_write_requests | 284449 |
| Key_writes | 87659 |
| Last_query_cost | 0.000000 |
| Max_used_connections | 101 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 112 |
| Open_streams | 0 |
| Open_tables | 64 |
| Opened_tables | 2760 |
| Qcache_free_blocks | 0 |
| Qcache_free_memory | 0 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 0 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 0 |
| Questions | 4629686 |
| Rpl_status | NULL |
| Select_full_join | 139866 |
| Select_full_range_join | 23 |
| Select_range | 224024 |
| Select_range_check | 0 |
| Select_scan | 1356230 |
| Slave_open_temp_tables | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 169 |
| Slow_queries | 7 |
| Sort_merge_passes | 0 |
| Sort_range | 463585 |
| Sort_rows | 25211442 |
| Sort_scan | 487169 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | NONE |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 0 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Table_locks_immediate | 4287406 |
| Table_locks_waited | 468 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 0 |
| Threads_connected | 41 |
| Threads_created | 215205 |
| Threads_running | 1 |
| Uptime | 586236 |
+-----------------------------------+------------+
Uptime: 586236 Threads: 41 Questions: 4629687 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 64 Queries per second avg: 7.897
mysqladmin Ver 8.41 Distrib 5.0.22, for redhat-linux-gnu on i686
Copyright (C) 2000 MySQL AB & MySQL Finland AB & TCX DataKonsult AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Server version 5.0.22
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/lib/mysql/mysql.sock
Uptime: 6 days 18 hours 50 min 36 sec

Threads: 41 Questions: 4629687 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 64 Queries per second avg: 7.897
MySQL Report
MySQL 5.0.22 uptime 6 18:50:47 Thu Jul 5 18:59:07 2007

__ Key __________________________________________________ _______________
Buffer used 260.00k of 8.00M %Used: 3.17
Current 1.17M %Usage: 14.58
Write hit 69.18%
Read hit 99.99%

__ Questions __________________________________________________ _________
Total 4.63M 7.9/s
DMS 3.45M 5.9/s %Total: 74.53
Com_ 1.03M 1.8/s 22.30
COM_QUIT 212.68k 0.4/s 4.59
-Unknown 65.92k 0.1/s 1.42
Slow 7 0.0/s 0.00 %DMS: 0.00
DMS 3.45M 5.9/s 74.53
SELECT 3.24M 5.5/s 69.87 93.76
UPDATE 105.44k 0.2/s 2.28 3.06
INSERT 56.22k 0.1/s 1.21 1.63
DELETE 50.21k 0.1/s 1.08 1.45
REPLACE 3.56k 0.0/s 0.08 0.10
Com_ 1.03M 1.8/s 22.30
set_option 761.46k 1.3/s 16.45
change_db 203.82k 0.3/s 4.40
admin_comma 66.37k 0.1/s 1.43

__ SELECT and Sort __________________________________________________ ___
Scan 1.36M 2.3/s %SELECT: 41.92
Range 224.06k 0.4/s 6.93
Full join 139.90k 0.2/s 4.32
Range check 0 0/s 0.00
Full rng join 23 0.0/s 0.00
Sort scan 487.24k 0.8/s
Sort range 463.64k 0.8/s
Sort mrg pass 0 0/s

__ Table Locks __________________________________________________ _______
Waited 468 0.0/s %Total: 0.01
Immediate 4.29M 7.3/s

__ Tables __________________________________________________ ____________
Open 64 of 64 %Cache: 100.00
Opened 2.76k 0.0/s

__ Connections __________________________________________________ _______
Max used 101 of 100 %Max: 101.00
Total 215.38k 0.4/s

__ Created Temp __________________________________________________ ______
Disk table 136.12k 0.2/s
Table 310.53k 0.5/s
File 5 0.0/s

__ Threads __________________________________________________ ___________
Running 2 of 41
Cached 0 of 0 %Hit: 0.06
Created 215.25k 0.4/s
Slow 169 0.0/s

__ Aborted __________________________________________________ ___________
Clients 5.38k 0.0/s
Connects 122 0.0/s

__ Bytes __________________________________________________ _____________
Sent 3.02G 5.1k/s
Received 754.31M 1.3k/s

__ InnoDB Buffer Pool __________________________________________________
Usage 304.00k of 8.00M %Used: 3.71
Read ratio 0.156
Pages
Free 493 %Total: 96.29
Data 19 3.71 %Drty: 0.00
Misc 0 0.00
Latched 0 0.00
Reads 77 0.0/s
From file 12 0.0/s 15.58
Ahead Rnd 1 0.0/s
Ahead Sql 0 0/s
Writes 0 0/s
Flushes 0 0/s
Wait Free 0 0/s

__ InnoDB Lock __________________________________________________ _______
Waits 0 0/s
Current 0
Time acquiring
Total 0 ms
Average 0 ms
Max 0 ms

__ InnoDB Data, Pages, Rows ____________________________________________
Data
Reads 25 0.0/s
Writes 3 0.0/s
fsync 3 0.0/s
Pending
Reads 0
Writes 0
fsync 0

Pages
Created 0 0/s
Read 19 0.0/s
Written 0 0/s

Rows
Deleted 0 0/s
Inserted 0 0/s
Read 0 0/s
Updated 0 0/s9. Other Scripts: Joomla & OpenAds (phpAdsNew)

10. Average Users: 200 - Max: 300

11. phpinfo()
http://beta.wapa.tv/test.php

12. Apache httpd.conf

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 20
StartServers 2
MaxClients 150
MaxRequestsPerChild 013. Large Files:
[root@www wapa]# find / -size +2000000k
find: /proc/27335/task/27335/fd/4: No such file or directory
find: /proc/27335/fd/4: No such file or directory
[root@www wapa]#14. Output of commands

uname -a
Linux www.wapa.tv 2.6.18-8.1.6.el5 #1 SMP Thu Jun 14 17:46:09 EDT 2007 i686 i686 i386 GNU/Linuxulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 65536
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@www wapa]#cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.60GHz
stepping : 1
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl est tm2 cid cx16 xtpr
bogomips : 7203.70

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.60GHz
stepping : 1
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl est tm2 cid cx16 xtpr
bogomips : 7199.28

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.60GHz
stepping : 1
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl est tm2 cid cx16 xtpr
bogomips : 7199.41

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.60GHz
stepping : 1
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl est tm2 cid cx16 xtpr
bogomips : 7199.40Any help appreciated... thanks!

pulsorock
Fri 6th Jul '07, 8:39pm
I usually don't bump my threads, but this performance situation is worrying me. And having passed the 24 hours limit for bumping and no reply yet, I got no choice. :(


So that being said.... bump. :o


Thanks! :)

eva2000
Sat 7th Jul '07, 9:16am
I've been reading up on OpenAds and some versions seem to have high load issues compared to the older phpadsnew versions. You might want to browse http://forum.openads.org/ forums to see if there's any outstanding issues in that area. Make sure you disable delayed inserts and persistent connections in OpenAds.

For vB, try the following in this exact order. You can ignore any of the suggestions that you have already done.

1. Upgrade to MySQL 5.0.41/44 if possibel and then upgrade PHP to 5.2.3 or if you have problems install PHP 4.4.7 (in either case install as ISAPI module and NO CGI)
2. Edit /etc/my.cnf and place the following mysql server settings in /etc/my.cnf and restart mysql server afterwards


[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
old_passwords
back_log = 100
skip-innodb
max_connections = 800
key_buffer = 128M
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 4M
table_cache = 2000
thread_cache_size = 384
wait_timeout = 45
connect_timeout = 10
tmp_table_size = 256M
max_heap_table_size = 256M
max_allowed_packet = 64M
max_connect_errors = 10
thread_concurrency = 4
concurrent_insert = 2
table_lock_wait_timeout = 35
read_rnd_buffer_size = 1M
bulk_insert_buffer_size = 8M
query_cache_limit = 8M
query_cache_size = 128M
query_cache_type = 1
query_prealloc_size = 131072
query_alloc_block_size = 65536
default-storage-engine = MyISAM

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
nice = -10
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M


3. Install APC Cache lastest version from http://pecl.php.net/package/APC. Read install guide at http://www.vbulletin.com/forum/showthread.php?t=165367 OR try the alternative PHP caching software, Xcache which seems to be a bit faster than APC Cache - http://www.vbulletin.com/forum/showthread.php?t=213267

4. If you just upgraded to vB 3.5.x/3.6.x try to disable these 4 options:

Admin CP -> vBulletin Options -> Forums Home Page Options -> Display Logged in Users?

Admin CP -> vBulletin Options -> Forum Display Options (forumdisplay) -> Show Users Browsing Forums

Admin CP -> vBulletin Options -> Thread Display Options -> Show Users Browsing Thread

Admin CP -> vBulletin Options -> Message Searching Options -> Automatic Similar Thread search

5. Edit httpd.conf values to following and restart apache

KeepAlive: On
MaxKeepAliveRequests: 100
KeepAliveTimeout: 1
MinSpareServers: 15
MaxSpareServers: 20
StartServers: 15
MaxClients: 200
MaxRequestsPerChild: 1000

6. Ensure vB admincp gzip is disabled since you already have Apache with mod_deflate which does the same thing

pulsorock
Sun 8th Jul '07, 5:04am
Thanks eva!

I will do the changes and see how it goes... our peak times is on weekdays, so on weekend is a good moment to do these changes and see how it goes when we reach peak.

Thanks

pulsorock
Mon 9th Jul '07, 12:26pm
Something is wrong... did the changes I was able to (except the upgrades to mysql and php), but I did the changes to the configuration as requested. Now at 11AM is a peak time for the site, and the load increased as never seen before, making it unavailable.

Take a look:

top - 11:20:39 up 10 days, 11:12, 1 user, load average: 183.47, 142.01, 77.82
Tasks: 358 total, 191 running, 167 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.3%us, 0.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3632116k total, 3498120k used, 133996k free, 35208k buffers
Swap: 2031608k total, 72k used, 2031536k free, 556008k cached
I'll gather all the information in a few minutes....

pulsorock
Mon 9th Jul '07, 12:47pm
the load now...


top - 11:43:59 up 10 days, 11:36, 1 user, load average: 152.08, 156.99, 134.76
Tasks: 358 total, 148 running, 210 sleeping, 0 stopped, 0 zombie
Cpu(s): 98.7%us, 1.2%sy, 0.0%ni, 0.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3632116k total, 3353424k used, 278692k free, 15712k buffers
Swap: 2031608k total, 1072k used, 2030536k free, 227244k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27787 apache 18 0 75820 21m 9164 R 65 0.6 0:40.83 httpd
27623 apache 17 0 76108 21m 9084 R 47 0.6 0:40.93 httpd
27901 apache 19 0 75796 21m 9076 R 27 0.6 0:30.01 httpd
27680 apache 19 0 75764 19m 7132 R 6 0.5 0:40.03 httpd
27505 apache 20 0 76364 26m 13m R 3 0.8 1:09.68 httpd
27576 apache 20 0 76136 21m 9368 R 3 0.6 0:42.51 httpd
27577 apache 25 0 80660 25m 8872 R 3 0.7 0:53.09 httpd
27586 apache 21 0 75948 22m 10m R 3 0.6 0:43.18 httpd
27601 apache 25 0 80680 26m 9588 R 3 0.7 0:47.31 httpd
27606 apache 25 0 80648 24m 7984 R 3 0.7 0:38.81 httpd
27616 apache 25 0 80720 26m 9144 R 3 0.7 1:08.88 httpd
27618 apache 25 0 80608 24m 7668 R 3 0.7 0:52.10 httpd
27621 apache 25 0 80728 25m 8876 R 3 0.7 0:40.58 httpd
27631 apache 25 0 75872 21m 8940 R 3 0.6 0:52.58 httpd
27632 apache 25 0 80752 24m 7648 R 3 0.7 0:40.18 httpd
27644 apache 21 0 75764 19m 7116 R 3 0.5 0:31.49 httpd
27665 apache 25 0 80644 24m 7120 R 3 0.7 0:46.58 httpd
27670 apache 25 0 80764 27m 10m R 3 0.8 1:02.43 httpd
27675 apache 25 0 80664 25m 8364 R 3 0.7 0:51.84 httpd
27676 apache 25 0 75800 19m 7108 R 3 0.5 0:42.90 httpd
27677 apache 25 0 80796 29m 11m R 3 0.8 0:47.69 httpd
27678 apache 25 0 80740 26m 9108 R 3 0.7 0:47.79 httpd
27682 apache 20 0 75836 21m 9064 R 3 0.6 0:35.69 httpd
27687 apache 25 0 80832 26m 9316 R 3 0.7 0:47.18 httpd
27688 apache 25 0 76100 20m 7696 R 3 0.6 0:38.53 httpd
27710 apache 20 0 75792 20m 8324 R 3 0.6 0:41.99 httpd
27711 apache 25 0 80652 24m 7368 R 3 0.7 0:34.48 httpd
27715 apache 20 0 75956 19m 7324 R 3 0.6 0:46.51 httpd
27716 apache 25 0 80748 28m 10m R 3 0.8 0:47.99 httpd
27717 apache 25 0 75756 19m 7108 R 3 0.5 0:31.59 httpd
27739 apache 25 0 80684 24m 7216 R 3 0.7 0:38.23 httpd
27742 apache 25 0 80648 24m 7100 R 3 0.7 0:44.99 httpd
27751 apache 22 0 75756 19m 7636 R 3 0.6 0:46.43 httpd
27754 apache 25 0 80716 24m 7356 R 3 0.7 0:37.80 httpd
27769 apache 25 0 80708 24m 7240 R 3 0.7 0:32.69 httpd
27770 apache 25 0 80804 27m 10m R 3 0.8 0:32.94 httpd
27773 apache 20 0 75944 20m 8020 R 3 0.6 0:40.89 httpd

pulsorock
Mon 9th Jul '07, 1:08pm
Here is an update if the info as of now....

mysql variables
+---------------------------------+--------------------------------------------------------+
| Variable_name | Value |
+---------------------------------+--------------------------------------------------------+
| auto_increment_increment | 1 |
| auto_increment_offset | 1 |
| automatic_sp_privileges | ON |
| back_log | 100 |
| basedir | /usr/ |
| bdb_cache_size | 8388600 |
| bdb_home | /var/lib/mysql/ |
| bdb_log_buffer_size | 1024000 |
| bdb_logdir | |
| bdb_max_lock | 10000 |
| bdb_shared_data | OFF |
| bdb_tmpdir | /tmp/ |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| completion_type | 0 |
| concurrent_insert | 2 |
| connect_timeout | 10 |
| datadir | /var/lib/mysql/ |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| div_precision_increment | 4 |
| engine_condition_pushdown | OFF |
| expire_logs_days | 0 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
| group_concat_max_len | 1024 |
| have_archive | NO |
| have_bdb | YES |
| have_blackhole_engine | NO |
| have_compress | YES |
| have_crypt | YES |
| have_csv | NO |
| have_example_engine | NO |
| have_federated_engine | NO |
| have_geometry | YES |
| have_innodb | DISABLED |
| have_isam | NO |
| have_ndbcluster | NO |
| have_openssl | DISABLED |
| have_query_cache | YES |
| have_raid | NO |
| have_rtree_keys | YES |
| have_symlink | YES |
| init_connect | |
| init_file | |
| init_slave | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 8388608 |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_io_threads | 4 |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_open_files | 300 |
| innodb_support_xa | ON |
| innodb_sync_spin_loops | 20 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| innodb_thread_sleep_delay | 10000 |
| interactive_timeout | 28800 |
| join_buffer_size | 2093056 |
| key_buffer_size | 134217728 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| language | /usr/share/mysql/english/ |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_error | |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 67107840 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 1073741824 |
| max_connect_errors | 10 |
| max_connections | 800 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 268434432 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| multi_range_count | 256 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 67108864 |
| myisam_stats_method | nulls_unequal |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | ON |
| open_files_limit | 4810 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| pid_file | /var/run/mysqld/mysqld.pid |
| prepared_stmt_count | 0 |
| port | 3306 |
| preload_buffer_size | 32768 |
| protocol_version | 10 |
| query_alloc_block_size | 65536 |
| query_cache_limit | 8388608 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 134217728 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 131072 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 2093056 |
| read_only | OFF |
| read_rnd_buffer_size | 1044480 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| server_id | 0 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_compressed_protocol | OFF |
| slave_load_tmpdir | /tmp/ |
| slave_net_timeout | 3600 |
| slave_skip_errors | OFF |
| slave_transaction_retries | 10 |
| slow_launch_time | 2 |
| socket | /var/lib/mysql/mysql.sock |
| sort_buffer_size | 4194296 |
| sql_mode | |
| sql_notes | ON |
| sql_warnings | ON |
| storage_engine | MyISAM |
| sync_binlog | 0 |
| sync_frm | ON |
| system_time_zone | AST |
| table_cache | 2000 |
| table_lock_wait_timeout | 35 |
| table_type | MyISAM |
| thread_cache_size | 384 |
| thread_stack | 196608 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| timed_mutexes | OFF |
| tmp_table_size | 268435456 |
| tmpdir | |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| updatable_views_with_limit | YES |
| version | 5.0.22 |
| version_bdb | Sleepycat Software: Berkeley DB 4.1.24: (May 25, 2006) |
| version_comment | Source distribution |
| version_compile_machine | i686 |
| version_compile_os | redhat-linux-gnu |
| wait_timeout | 45 |
+---------------------------------+--------------------------------------------------------+
+-----------------------------------+------------+
| Variable_name | Value |
+-----------------------------------+------------+
| Aborted_clients | 24302 |
| Aborted_connects | 2 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 0 |
| Bytes_received | 359724418 |
| Bytes_sent | 3001675012 |
| Com_admin_commands | 12127 |
| Com_alter_db | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 54515 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_checksum | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_dealloc_sql | 0 |
| Com_delete | 20002 |
| Com_delete_multi | 0 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_drop_user | 0 |
| Com_execute_sql | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 21276 |
| Com_insert_select | 0 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 0 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 2700 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 61265 |
| Com_set_option | 51627 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 1 |
| Com_show_charsets | 12 |
| Com_show_collations | 12 |
| Com_show_column_types | 0 |
| Com_show_create_db | 0 |
| Com_show_create_table | 3 |
| Com_show_databases | 12 |
| Com_show_errors | 0 |
| Com_show_fields | 12 |
| Com_show_grants | 5 |
| Com_show_innodb_status | 0 |
| Com_show_keys | 3 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_ndb_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_privileges | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 4 |
| Com_show_storage_engines | 0 |
| Com_show_tables | 25 |
| Com_show_triggers | 0 |
| Com_show_variables | 27 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_close | 0 |
| Com_stmt_execute | 0 |
| Com_stmt_fetch | 0 |
| Com_stmt_prepare | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_send_long_data | 0 |
| Com_truncate | 9 |
| Com_unlock_tables | 0 |
| Com_update | 26129 |
| Com_update_multi | 0 |
| Com_xa_commit | 0 |
| Com_xa_end | 0 |
| Com_xa_prepare | 0 |
| Com_xa_recover | 0 |
| Com_xa_rollback | 0 |
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 64576 |
| Created_tmp_disk_tables | 32821 |
| Created_tmp_files | 0 |
| Created_tmp_tables | 35299 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 147 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 2056 |
| Handler_discover | 0 |
| Handler_prepare | 0 |
| Handler_read_first | 16761 |
| Handler_read_key | 6101725 |
| Handler_read_next | 4609514 |
| Handler_read_prev | 185 |
| Handler_read_rnd | 2530214 |
| Handler_read_rnd_next | 117470647 |
| Handler_rollback | 0 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 52907 |
| Handler_write | 13055164 |
| Innodb_buffer_pool_pages_data | 0 |
| Innodb_buffer_pool_pages_dirty | 0 |
| Innodb_buffer_pool_pages_flushed | 0 |
| Innodb_buffer_pool_pages_free | 0 |
| Innodb_buffer_pool_pages_latched | 0 |
| Innodb_buffer_pool_pages_misc | 0 |
| Innodb_buffer_pool_pages_total | 0 |
| Innodb_buffer_pool_read_ahead_rnd | 0 |
| Innodb_buffer_pool_read_ahead_seq | 0 |
| Innodb_buffer_pool_read_requests | 0 |
| Innodb_buffer_pool_reads | 0 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 0 |
| Innodb_data_fsyncs | 0 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_data_pending_reads | 0 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_read | 0 |
| Innodb_data_reads | 0 |
| Innodb_data_writes | 0 |
| Innodb_data_written | 0 |
| Innodb_dblwr_pages_written | 0 |
| Innodb_dblwr_writes | 0 |
| Innodb_log_waits | 0 |
| Innodb_log_write_requests | 0 |
| Innodb_log_writes | 0 |
| Innodb_os_log_fsyncs | 0 |
| Innodb_os_log_pending_fsyncs | 0 |
| Innodb_os_log_pending_writes | 0 |
| Innodb_os_log_written | 0 |
| Innodb_page_size | 0 |
| Innodb_pages_created | 0 |
| Innodb_pages_read | 0 |
| Innodb_pages_written | 0 |
| Innodb_row_lock_current_waits | 0 |
| Innodb_row_lock_time | 0 |
| Innodb_row_lock_time_avg | 0 |
| Innodb_row_lock_time_max | 0 |
| Innodb_row_lock_waits | 0 |
| Innodb_rows_deleted | 0 |
| Innodb_rows_inserted | 0 |
| Innodb_rows_read | 0 |
| Innodb_rows_updated | 0 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 115730 |
| Key_blocks_used | 262 |
| Key_read_requests | 11329398 |
| Key_reads | 658 |
| Key_write_requests | 75133 |
| Key_writes | 32192 |
| Last_query_cost | 0.000000 |
| Max_used_connections | 60 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 137 |
| Open_streams | 0 |
| Open_tables | 78 |
| Opened_tables | 103 |
| Qcache_free_blocks | 618 |
| Qcache_free_memory | 128170344 |
| Qcache_hits | 594597 |
| Qcache_inserts | 43553 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 17843 |
| Qcache_queries_in_cache | 4297 |
| Qcache_total_blocks | 9252 |
| Questions | 884507 |
| Rpl_status | NULL |
| Select_full_join | 75 |
| Select_full_range_join | 1 |
| Select_range | 7833 |
| Select_range_check | 0 |
| Select_scan | 83811 |
| Slave_open_temp_tables | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 0 |
| Sort_merge_passes | 0 |
| Sort_range | 3048 |
| Sort_rows | 2612104 |
| Sort_scan | 30900 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | NONE |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 0 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Table_locks_immediate | 187158 |
| Table_locks_waited | 19 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 44 |
| Threads_connected | 16 |
| Threads_created | 60 |
| Threads_running | 1 |
| Uptime | 115819 |
+-----------------------------------+------------+
Uptime: 115819 Threads: 16 Questions: 884508 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 78 Queries per second avg: 7.637
mysqladmin Ver 8.41 Distrib 5.0.22, for redhat-linux-gnu on i686
Copyright (C) 2000 MySQL AB & MySQL Finland AB & TCX DataKonsult AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Server version 5.0.22
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/lib/mysql/mysql.sock
Uptime: 1 day 8 hours 10 min 19 sec

Threads: 16 Questions: 884508 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 78 Queries per second avg: 7.637



mysqlreport

MySQL 5.0.22 uptime 1 8:11:33 Mon Jul 9 11:35:12 2007

__ Key __________________________________________________ _______________
Buffer used 262.00k of 128.00M %Used: 0.20
Current 14.98M %Usage: 11.71
Write hit 57.15%
Read hit 99.99%

__ Questions __________________________________________________ _________
Total 885.19k 7.6/s
QC Hits 595.13k 5.1/s %Total: 67.23
DMS 131.42k 1.1/s 14.85
Com_ 118.50k 1.0/s 13.39
COM_QUIT 52.44k 0.5/s 5.92
-Unknown 12.30k 0.1/s 1.39
Slow 0 0/s 0.00 %DMS: 0.00
DMS 131.42k 1.1/s 14.85
SELECT 61.27k 0.5/s 6.92 46.62
UPDATE 26.13k 0.2/s 2.95 19.88
INSERT 21.28k 0.2/s 2.40 16.19
DELETE 20.05k 0.2/s 2.26 15.25
REPLACE 2.70k 0.0/s 0.31 2.05
Com_ 118.50k 1.0/s 13.39
change_db 54.57k 0.5/s 6.16
set_option 51.67k 0.4/s 5.84
admin_comma 12.13k 0.1/s 1.37

__ SELECT and Sort __________________________________________________ ___
Scan 83.82k 0.7/s %SELECT: 136.81
Range 7.83k 0.1/s 12.79
Full join 75 0.0/s 0.12
Range check 0 0/s 0.00
Full rng join 1 0.0/s 0.00
Sort scan 30.90k 0.3/s
Sort range 3.05k 0.0/s
Sort mrg pass 0 0/s

__ Query Cache __________________________________________________ _______
Memory usage 5.77M of 128.00M %Used: 4.51
Block Fragmnt 6.66%
Hits 595.13k 5.1/s
Inserts 43.55k 0.4/s
Insrt:Prune 43.55k:1 0.4/s
Hit:Insert 13.66:1

__ Table Locks __________________________________________________ _______
Waited 19 0.0/s %Total: 0.01
Immediate 187.21k 1.6/s

__ Tables __________________________________________________ ____________
Open 78 of 2000 %Cache: 3.90
Opened 103 0.0/s

__ Connections __________________________________________________ _______
Max used 60 of 800 %Max: 7.50
Total 64.63k 0.6/s

__ Created Temp __________________________________________________ ______
Disk table 32.82k 0.3/s
Table 35.31k 0.3/s
File 0 0/s

__ Threads __________________________________________________ ___________
Running 1 of 35
Cached 25 of 384 %Hit: 99.91
Created 60 0.0/s
Slow 0 0/s

__ Aborted __________________________________________________ ___________
Clients 24.37k 0.2/s
Connects 2 0.0/s

__ Bytes __________________________________________________ _____________
Sent 3.00G 25.9k/s
Received 359.80M 3.1k/s

__ InnoDB Buffer Pool __________________________________________________
Usage 0 of 0 %Used: 0.00
Read ratio 0.000
Pages
Free 0 %Total: 0.00
Data 0 0.00 %Drty: 0.00
Misc 0 0.00
Latched 0 0.00
Reads 0 0/s
From file 0 0/s 0.00
Ahead Rnd 0 0/s
Ahead Sql 0 0/s
Writes 0 0/s
Flushes 0 0/s
Wait Free 0 0/s

__ InnoDB Lock __________________________________________________ _______
Waits 0 0/s
Current 0
Time acquiring
Total 0 ms
Average 0 ms
Max 0 ms

__ InnoDB Data, Pages, Rows ____________________________________________
Data
Reads 0 0/s
Writes 0 0/s
fsync 0 0/s
Pending
Reads 0
Writes 0
fsync 0

Pages
Created 0 0/s
Read 0 0/s
Written 0 0/s

Rows
Deleted 0 0/s
Inserted 0 0/s
Read 0 0/s
Updated 0 0/s

pulsorock
Mon 9th Jul '07, 3:26pm
I had to revert the changes back to their previous settings and the load dropped a lot.

top - 14:22:23 up 10 days, 14:14, 1 user, load average: 0.23, 0.19, 1.17
Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 0.3%sy, 0.0%ni, 98.9%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 3632116k total, 768076k used, 2864040k free, 38752k buffers
Swap: 2031608k total, 68k used, 2031540k free, 370812k cached
I only reverted the changes made to the httpd.conf and my.cnf. I left the xcache installed. I think maybe it had something to do with the changes made to the apache conf and the max connections or something. Because the load was higher than 180 and all I see was httpd instances (as I posted above in post #5).

So I changed from
KeepAlive: On
MaxKeepAliveRequests: 100
KeepAliveTimeout: 1
MinSpareServers: 15
MaxSpareServers: 20
StartServers: 15
MaxClients: 200
MaxRequestsPerChild: 1000to the original
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 20
StartServers 2
MaxClients 150
MaxRequestsPerChild 0And reverted the my.cnf to their original as posted in the first post.

What could had caused this server load even higher after these changes? I'd really like to optimize the settings, but I guess we need to find what change caused the load to get this high.

Any idea eva?

eva2000
Tue 10th Jul '07, 9:05am
try playing with these 3 values independently one at a time

KeepAlive Off or On
MaxClients 150 or 200
MaxRequestsPerChild 0 or 1000

and see what happens