Brian Cruz
Mon 27th Oct '03, 1:01am
I thought I had my server set fine, but then 600+ people decided to log in at once (for a cartoon-related event Sunday night) and everything came to a crashing halt. CPanel sent me a few e-mails right as the server became jammed. The first one said:
Apache has reached the MaxClients
limit. cPanel has increased the MaxClients limit to 202 (10 higher). About a minute later I got a second e-mail saying that apache had failed, and then another one saying that "Apache on has exceeded the maximum amount of openfiles".
So even though the server load never got past 4, pages were taking forever to load. Obviously I need my settings tweaked a bit more. So here is my info again:
Dual Xeon 2.0 GHz
2GB of RAM (upgraded from 1GB earlier this week)
Two 73 GB 10K RPM Ultra 320 SCSI Drives (MySQL is on the 2nd drive)
RedHat Linux 7.3
CPanel 8.5.4-RELEASE 7
Apache 1.3.28
PHP 4.3.3
MySQL 4.0.15
mod_gzip enabled
Top / MySQL Status:
http://forums.toonzone.net/mysqlinfo.php
PHP Info:
http://forums.toonzone.net/index.php?do=phpinfo
Apache:
Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 32
MaxSpareServers 128
StartServers 128
MaxClients 202
MaxRequestsPerChild 0
Here is the my.cnf file (some paths altered to point to 2nd drive):
[client]
port = 3306
#socket = /var/lib/mysql/mysql.sock
socket = /backup/mysqldata/mysql.sock
[mysqld]
datadir = /backup/mysqldata
port = 3306
#socket = /var/lib/mysql/mysql.sock
socket = /backup/mysqldata/mysql.sock
skip-locking
max_connections = 500
key_buffer = 96M
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 5M
table_cache = 1024
thread_cache_size = 128
wait_timeout = 100
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 2M
query_cache_size = 32M
query_cache_type = 1
thread_concurrency=4
skip-innodb
[mysqld_safe]
open_files_limit = 8192
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M
Here is the MySQL extended status info from when things "calmed down" a bit to 500 simultaneous users:
Mon Oct 27 00:04:44 EST 2003
12:04am up 4 days, 21:15, 2 users, load average: 3.59, 3.61, 3.15
211 processes: 205 sleeping, 6 running, 0 zombie, 0 stopped
Mem: 2070016K av, 2056000K used, 14016K free, 0K shrd, 44064K buff
Swap: 2048276K av, 256536K used, 1791740K free 884500K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
28225 nobody 14 0 14640 13M 9224 R 23.3 0.6 0:22 httpd
24908 nobody 14 0 15084 13M 9600 R 20.9 0.6 0:27 httpd
22390 nobody 14 0 15412 14M 9588 R 18.5 0.6 0:25 httpd
23793 nobody 15 0 15128 13M 9584 R 16.1 0.6 0:21 httpd
20580 nobody 17 0 1896 1896 1588 R 13.7 0.0 0:00 top
6278 nobody 10 0 14680 13M 9828 R 0.8 0.6 0:19 httpd
Http processes currently running = 204
Mysql processes currently running = 2
Netstat information summary
+---------------------------+-----------------+
| Variable_name | Value |
+---------------------------+-----------------+
| Aborted_clients | 499490 |
| Aborted_connects | 0 |
| Bytes_received | 2205904335 |
| Bytes_sent | 2626487783 |
| Com_admin_commands | 901921 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 1221630 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 62894 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 321743 |
| Com_insert_select | 7112 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 17047 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 23326 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 104 |
| Com_savepoint | 0 |
| Com_select | 4317003 |
| Com_set_option | 0 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 0 |
| Com_show_databases | 14 |
| Com_show_fields | 265 |
| Com_show_grants | 0 |
| Com_show_keys | 23 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 1407 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 296 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 878 |
| Com_show_variables | 43 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 17047 |
| Com_update | 964058 |
| Connections | 473906 |
| Created_tmp_disk_tables | 2618 |
| Created_tmp_tables | 194426 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 1 |
| Delayed_writes | 12186 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 240727 |
| Handler_read_first | 504528 |
| Handler_read_key | 335035932 |
| Handler_read_next | 273608959 |
| Handler_read_prev | 6348118 |
| Handler_read_rnd | 56339709 |
| Handler_read_rnd_next | 518512143 |
| Handler_rollback | 0 |
| Handler_update | 1630095 |
| Handler_write | 44017653 |
| Key_blocks_used | 93763 |
| Key_read_requests | 747537577 |
| Key_reads | 189559 |
| Key_write_requests | 1194618 |
| Key_writes | 834717 |
| Max_used_connections | 242 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 401 | 39% of table_cache in use
| Open_files | 630 |
| Open_streams | 0 |
| Opened_tables | 407 |
| Questions | 8722663 |
| Qcache_queries_in_cache | 3854 |
| Qcache_inserts | 2231699 |
| Qcache_hits | 1544769 |
| Qcache_lowmem_prunes | 19777 |
| Qcache_not_cached | 2085304 |
| Qcache_free_memory | 19930424 |
| Qcache_free_blocks | 2674 |
| Qcache_total_blocks | 10830 |
| Rpl_status | NULL |
| Select_full_join | 788 |
| Select_full_range_join | 36353 |
| Select_range | 2053119 |
| Select_range_check | 0 |
| Select_scan | 710132 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 29 | (execution time > 10 secs)
| Sort_merge_passes | 0 |
| Sort_range | 542736 |
| Sort_rows | 69848152 |
| Sort_scan | 194377 |
| Table_locks_immediate | 8976948 |
| Table_locks_waited | 4717 |
| Threads_cached | 12 |
| Threads_created | 751 |
| Threads_connected | 193 |
| Threads_running | 1 |
| Uptime | 422108 | 4 days 21 hrs 15 mins 8 secs
+---------------------------+-----------------+
Key Reads/Key Read Requests = 0.000254 (Cache hit = 99.999746%)
Key Writes/Key Write Requests = 0.698731
Connections/second = 1.123 (/hour = 4041.766)
KB received/second = 4.968 (/hour = 17885.810)
KB sent/second = 4.968 (/hour = 17885.810)
Temporary Tables Created/second = 0.461 (/hour = 1658.186)
Opened Tables/second = 0.001 (/hour = 3.471)
Slow Queries/second = 0.000 (/hour = 0.247)
% of slow queries = 0.000%
Queries/second = 20.665 (/hour = 74392.304)
This is a pretty busy server with a large dynamic site (mostly PHP, some cgi) and over a dozen hosted sites (mostly static). There are also two Invision boards and one phpBB on the server, plus phpAdsNew banners loading on much of the site including the vB.
I'm running vB3 Beta 6. At peak times there are over 400 users on the board, and server load is around 2.00. Persistant connections are on.
I have tried phpAccelerator and TurckMMcache in the past, but for some reason it caused more harm than good (high load, slow loading, etc.)
Thanks!
Apache has reached the MaxClients
limit. cPanel has increased the MaxClients limit to 202 (10 higher). About a minute later I got a second e-mail saying that apache had failed, and then another one saying that "Apache on has exceeded the maximum amount of openfiles".
So even though the server load never got past 4, pages were taking forever to load. Obviously I need my settings tweaked a bit more. So here is my info again:
Dual Xeon 2.0 GHz
2GB of RAM (upgraded from 1GB earlier this week)
Two 73 GB 10K RPM Ultra 320 SCSI Drives (MySQL is on the 2nd drive)
RedHat Linux 7.3
CPanel 8.5.4-RELEASE 7
Apache 1.3.28
PHP 4.3.3
MySQL 4.0.15
mod_gzip enabled
Top / MySQL Status:
http://forums.toonzone.net/mysqlinfo.php
PHP Info:
http://forums.toonzone.net/index.php?do=phpinfo
Apache:
Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 32
MaxSpareServers 128
StartServers 128
MaxClients 202
MaxRequestsPerChild 0
Here is the my.cnf file (some paths altered to point to 2nd drive):
[client]
port = 3306
#socket = /var/lib/mysql/mysql.sock
socket = /backup/mysqldata/mysql.sock
[mysqld]
datadir = /backup/mysqldata
port = 3306
#socket = /var/lib/mysql/mysql.sock
socket = /backup/mysqldata/mysql.sock
skip-locking
max_connections = 500
key_buffer = 96M
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 5M
table_cache = 1024
thread_cache_size = 128
wait_timeout = 100
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 2M
query_cache_size = 32M
query_cache_type = 1
thread_concurrency=4
skip-innodb
[mysqld_safe]
open_files_limit = 8192
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M
Here is the MySQL extended status info from when things "calmed down" a bit to 500 simultaneous users:
Mon Oct 27 00:04:44 EST 2003
12:04am up 4 days, 21:15, 2 users, load average: 3.59, 3.61, 3.15
211 processes: 205 sleeping, 6 running, 0 zombie, 0 stopped
Mem: 2070016K av, 2056000K used, 14016K free, 0K shrd, 44064K buff
Swap: 2048276K av, 256536K used, 1791740K free 884500K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
28225 nobody 14 0 14640 13M 9224 R 23.3 0.6 0:22 httpd
24908 nobody 14 0 15084 13M 9600 R 20.9 0.6 0:27 httpd
22390 nobody 14 0 15412 14M 9588 R 18.5 0.6 0:25 httpd
23793 nobody 15 0 15128 13M 9584 R 16.1 0.6 0:21 httpd
20580 nobody 17 0 1896 1896 1588 R 13.7 0.0 0:00 top
6278 nobody 10 0 14680 13M 9828 R 0.8 0.6 0:19 httpd
Http processes currently running = 204
Mysql processes currently running = 2
Netstat information summary
+---------------------------+-----------------+
| Variable_name | Value |
+---------------------------+-----------------+
| Aborted_clients | 499490 |
| Aborted_connects | 0 |
| Bytes_received | 2205904335 |
| Bytes_sent | 2626487783 |
| Com_admin_commands | 901921 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 1221630 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 62894 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 321743 |
| Com_insert_select | 7112 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 17047 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 23326 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 104 |
| Com_savepoint | 0 |
| Com_select | 4317003 |
| Com_set_option | 0 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 0 |
| Com_show_databases | 14 |
| Com_show_fields | 265 |
| Com_show_grants | 0 |
| Com_show_keys | 23 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 1407 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 296 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 878 |
| Com_show_variables | 43 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 17047 |
| Com_update | 964058 |
| Connections | 473906 |
| Created_tmp_disk_tables | 2618 |
| Created_tmp_tables | 194426 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 1 |
| Delayed_writes | 12186 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 240727 |
| Handler_read_first | 504528 |
| Handler_read_key | 335035932 |
| Handler_read_next | 273608959 |
| Handler_read_prev | 6348118 |
| Handler_read_rnd | 56339709 |
| Handler_read_rnd_next | 518512143 |
| Handler_rollback | 0 |
| Handler_update | 1630095 |
| Handler_write | 44017653 |
| Key_blocks_used | 93763 |
| Key_read_requests | 747537577 |
| Key_reads | 189559 |
| Key_write_requests | 1194618 |
| Key_writes | 834717 |
| Max_used_connections | 242 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 401 | 39% of table_cache in use
| Open_files | 630 |
| Open_streams | 0 |
| Opened_tables | 407 |
| Questions | 8722663 |
| Qcache_queries_in_cache | 3854 |
| Qcache_inserts | 2231699 |
| Qcache_hits | 1544769 |
| Qcache_lowmem_prunes | 19777 |
| Qcache_not_cached | 2085304 |
| Qcache_free_memory | 19930424 |
| Qcache_free_blocks | 2674 |
| Qcache_total_blocks | 10830 |
| Rpl_status | NULL |
| Select_full_join | 788 |
| Select_full_range_join | 36353 |
| Select_range | 2053119 |
| Select_range_check | 0 |
| Select_scan | 710132 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 29 | (execution time > 10 secs)
| Sort_merge_passes | 0 |
| Sort_range | 542736 |
| Sort_rows | 69848152 |
| Sort_scan | 194377 |
| Table_locks_immediate | 8976948 |
| Table_locks_waited | 4717 |
| Threads_cached | 12 |
| Threads_created | 751 |
| Threads_connected | 193 |
| Threads_running | 1 |
| Uptime | 422108 | 4 days 21 hrs 15 mins 8 secs
+---------------------------+-----------------+
Key Reads/Key Read Requests = 0.000254 (Cache hit = 99.999746%)
Key Writes/Key Write Requests = 0.698731
Connections/second = 1.123 (/hour = 4041.766)
KB received/second = 4.968 (/hour = 17885.810)
KB sent/second = 4.968 (/hour = 17885.810)
Temporary Tables Created/second = 0.461 (/hour = 1658.186)
Opened Tables/second = 0.001 (/hour = 3.471)
Slow Queries/second = 0.000 (/hour = 0.247)
% of slow queries = 0.000%
Queries/second = 20.665 (/hour = 74392.304)
This is a pretty busy server with a large dynamic site (mostly PHP, some cgi) and over a dozen hosted sites (mostly static). There are also two Invision boards and one phpBB on the server, plus phpAdsNew banners loading on much of the site including the vB.
I'm running vB3 Beta 6. At peak times there are over 400 users on the board, and server load is around 2.00. Persistant connections are on.
I have tried phpAccelerator and TurckMMcache in the past, but for some reason it caused more harm than good (high load, slow loading, etc.)
Thanks!