Dual Xeon struggling with 800-1000 users online...

[LanciaStratos]

My Dual Xeon server has been having a hard time. It's not that the server is having trouble handling this number of users online - it's just that it keeps crashing and burning once a day. What can I do? Here's the specs:





Dedicated Server
Dual Intel Xeon 2.4 Ghz



2 GB of RAM



2x 80 GB ATA hard drives (non raid)



RedHat Fedora Core 2 i686



WHM 9.9.7 / cPanel 9.9.8 - R5



Apache 1.3.33



PHP 4.3.10



MySQL 4.0.22-standard
Not sure about innodb...
MySQL compiled/installed with WHM/cPanel.
Top Stats (after a recent server crash and reboot, there is usually swap)







top - 15:34:55 up 1:02, 1 user, load average: 4.93, 3.89, 3.93



Tasks: 324 total, 3 running, 321 sleeping, 0 stopped, 0 zombie



Cpu(s): 60.1% us, 8.3% sy, 0.0% ni, 23.6% id, 6.1% wa, 0.2% hi, 1.7% si



Mem: 2075088k total, 1797864k used, 277224k free, 35888k buffers



Swap: 1807304k total, 0k used, 1807304k free, 840572k cached







PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND



6178 nobody 16 0 18732 9028 4224 S 22.8 0.4 0:01.23 httpd



6127 nobody 17 0 23344 13m 4288 R 19.9 0.7 0:01.17 httpd



5675 nobody 15 0 20852 11m 4460 S 15.2 0.5 0:09.62 httpd



6187 nobody 15 0 17996 8264 4188 S 13.9 0.4 0:00.42 httpd



5984 nobody 15 0 18736 9108 4292 S 13.6 0.4 0:05.07 httpd



6089 nobody 16 0 19732 9.8m 4224 S 13.2 0.5 0:00.71 httpd



5759 nobody 15 0 19384 9764 4300 S 12.9 0.5 0:02.82 httpd



6114 nobody 15 0 18940 9116 4224 S 12.6 0.4 0:00.78 httpd



4424 mysql 15 0 99.2m 51m 1432 S 12.3 2.5 0:05.74 mysqld



6153 nobody 15 0 19644 9948 4228 S 12.3 0.5 0:00.40 httpd



6028 nobody 15 0 18032 8380 4268 S 10.3 0.4 0:04.68 httpd



5626 nobody 15 0 18944 9324 4300 S 8.3 0.4 0:08.87 httpd



5690 nobody 15 0 18696 9068 4292 S 8.3 0.4 0:04.34 httpd



5987 nobody 15 0 18668 8980 4232 S 8.3 0.4 0:01.48 httpd



6084 nobody 15 0 18708 9008 4224 S 8.3 0.4 0:01.55 httpd



5368 nobody 15 0 19392 9736 4308 S 7.6 0.5 0:14.62 httpd



5983 nobody 15 0 18852 9216 4284 S 6.3 0.4 0:02.30 httpd



5678 nobody 15 0 19088 9452 4284 S 5.3 0.5 0:05.12 httpd



5627 nobody 15 0 22392 12m 4300 S 5.0 0.6 0:14.00 httpd



4463 mysql 16 0 99.2m 51m 1432 S 4.3 2.5 0:05.25 mysqld



4465 mysql 15 0 99.2m 51m 1432 S 4.3 2.5 0:05.91 mysqld



4449 mysql 16 0 99.2m 51m 1432 S 3.6 2.5 0:05.81 mysqld



4466 mysql 15 0 99.2m 51m 1432 S 3.3 2.5 0:05.98 mysqld



4431 mysql 15 0 99.2m 51m 1432 S 3.0 2.5 0:05.07 mysqld



4426 mysql 16 0 99.2m 51m 1432 S 2.6 2.5 0:06.31 mysqld



5217 nobody 16 0 20828 10m 4300 S 2.6 0.5 0:16.27 httpd



5621 nobody 15 0 19108 9452 4288 S 2.3 0.5 0:04.55 httpd



4468 mysql 15 0 99.2m 51m 1432 S 2.0 2.5 0:06.49 mysqld



6184 nobody 15 0 15936 6176 4212 S 2.0 0.3 0:00.07 httpd



3178 mysql 15 0 99.2m 51m 1432 S 1.7 2.5 0:12.70 mysqld



4430 mysql 15 0 99.2m 51m 1432 S 1.7 2.5 0:05.48 mysqld



5607 nobody 15 0 20892 11m 4296 S 1.7 0.5 0:12.60 httpd



5622 nobody 15 0 22040 12m 4300 S 1.7 0.6 0:09.61 httpd



6044 nobody 15 0 18780 9100 4240 S 1.3 0.4 0:01.43 httpd



3985 mysql 16 0 99.2m 51m 1432 S 1.0 2.5 0:05.28 mysqld



5567 nobody 15 0 19176 9524 4268 S 1.0 0.5 0:09.36 httpd



5602 nobody 15 0 19720 9.9m 4296 S 1.0 0.5 0:08.77 httpd



5635 nobody 15 0 21152 11m 4300 S 1.0 0.6 0:10.80 httpd



5695 nobody 15 0 20196 10m 4296 S 1.0 0.5 0:08.11 httpd



5732 nobody 15 0 20544 10m 4288 S 1.0 0.5 0:04.82 httpd



6060 nobody 15 0 18760 9112 4280 S 1.0 0.4 0:01.01 httpd



6062 nobody 15 0 18604 8924 4248 S 1.0 0.4 0:01.24 httpd



6186 nobody 15 0 15400 5080 3628 S 1.0 0.2 0:00.03 httpd



3062 mysql 15 0 99.2m 51m 1432 S 0.7 2.5 0:05.76 mysqld



3135 mysql 15 0 99.2m 51m 1432 S 0.7 2.5 0:10.27 mysqld



4447 mysql 15 0 99.2m 51m 1432 S 0.7 2.5 0:05.97 mysqld



4450 mysql 15 0 99.2m 51m 1432 S 0.7 2.5 0:04.93 mysqld



5512 nobody 16 0 19336 9724 4308 S 0.7 0.5 0:15.18 httpd



5513 nobody 16 0 20788 10m 4304 S 0.7 0.5 0:15.00 httpd



5604 nobody 15 0 21812 11m 4332 S 0.7 0.6 0:14.00 httpd



5606 nobody 15 0 20280 10m 4308 S 0.7 0.5 0:08.38 httpd



5623 nobody 15 0 20844 10m 4300 S 0.7 0.5 0:08.29 httpd



5628 nobody 15 0 18960 9340 4300 S 0.7 0.5 0:07.81 httpd



5643 nobody 15 0 19028 9396 4288 S 0.7 0.5 0:05.08 httpd




my.cnf Variables



[mysqld]



skip-innodb



max_connections = 500



key_buffer = 16M



myisam_sort_buffer_size = 64M



join_buffer_size = 1M



read_buffer_size = 1M



sort_buffer_size = 2M



table_cache = 256



thread_cache_size = 64



wait_timeout = 100



connect_timeout = 10



max_allowed_packet = 1M



max_connect_errors = 10



query_cache_limit = 1M



query_cache_size = 16M



query_cache_type = 1







[mysqld_safe]



open_files_limit = 8192







[mysqldump]



quick



max_allowed_packet = 16M







[myisamchk]



key_buffer = 64M



sort_buffer = 64M



read_buffer = 16M



write_buffer = 16M




MySQL Extended-Status







Sun Feb 27 15:42:14 EST 2005











top - 15:42:15 up 1:09, 1 user, load average: 5.08, 4.20, 4.08



Tasks: 288 total, 2 running, 286 sleeping, 0 stopped, 0 zombie



Cpu(s): 44.5% us, 4.2% sy, 0.1% ni, 43.6% id, 6.3% wa, 0.1% hi, 1.1% si



Mem: 2075088k total, 2053844k used, 21244k free, 38020k buffers



Swap: 1807304k total, 0k used, 1807304k free, 1051900k cached







PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND



4432 mysql 15 0 99.3m 52m 1432 D 3.8 2.6 0:07.12 mysqld



5566 nobody 15 0 19504 9892 4308 R 3.8 0.5 0:28.56 httpd



6539 nobody 15 0 2632 932 644 R 3.8 0.0 0:00.04 top











Http processes currently running = 137



Mysql processes currently running = 71







Netstat information summary



1 CLOSING



2 CLOSE_WAIT



2 FIN_WAIT1



2 SYN_RECV



32 LISTEN



79 ESTABLISHED



230 FIN_WAIT2



257 TIME_WAIT







+--------------------------+-----------+



| Variable_name | Value |



+--------------------------+-----------+



| Aborted_clients | 151 |



| Aborted_connects | 0 |



| Bytes_received | 90019558 |



| Bytes_sent | 741125280 |



| Com_admin_commands | 0 |



| Com_alter_table | 0 |



| Com_analyze | 0 |



| Com_backup_table | 0 |



| Com_begin | 0 |



| Com_change_db | 62111 |



| Com_change_master | 0 |



| Com_check | 0 |



| Com_commit | 0 |



| Com_create_db | 0 |



| Com_create_function | 0 |



| Com_create_index | 0 |



| Com_create_table | 0 |



| Com_delete | 782 |



| Com_delete_multi | 0 |



| Com_drop_db | 0 |



| Com_drop_function | 0 |



| Com_drop_index | 0 |



| Com_drop_table | 0 |



| Com_flush | 0 |



| Com_grant | 0 |



| Com_ha_close | 0 |



| Com_ha_open | 0 |



| Com_ha_read | 0 |



| Com_insert | 11105 |



| Com_insert_select | 34 |



| Com_kill | 0 |



| Com_load | 0 |



| Com_load_master_data | 0 |



| Com_load_master_table | 0 |



| Com_lock_tables | 399 |



| Com_optimize | 0 |



| Com_purge | 0 |



| Com_rename_table | 0 |



| Com_repair | 0 |



| Com_replace | 783 |



| Com_replace_select | 0 |



| Com_reset | 0 |



| Com_restore_table | 0 |



| Com_revoke | 0 |



| Com_rollback | 0 |



| Com_savepoint | 0 |



| Com_select | 85651 |



| Com_set_option | 0 |



| Com_show_binlog_events | 0 |



| Com_show_binlogs | 0 |



| Com_show_create | 0 |



| Com_show_databases | 0 |



| Com_show_fields | 0 |



| Com_show_grants | 0 |



| Com_show_keys | 0 |



| Com_show_logs | 0 |



| Com_show_master_status | 0 |



| Com_show_new_master | 0 |



| Com_show_open_tables | 0 |



| Com_show_processlist | 15 |



| Com_show_slave_hosts | 0 |



| Com_show_slave_status | 0 |



| Com_show_status | 1 |



| Com_show_innodb_status | 0 |



| Com_show_tables | 0 |



| Com_show_variables | 1 |



| Com_slave_start | 0 |



| Com_slave_stop | 0 |



| Com_truncate | 0 |



| Com_unlock_tables | 399 |



| Com_update | 38284 |



| Connections | 55663 |



| Created_tmp_disk_tables | 23 |



| Created_tmp_tables | 4778 |



| Created_tmp_files | 2 |



| Delayed_insert_threads | 0 |



| Delayed_writes | 0 |



| Delayed_errors | 0 |



| Flush_commands | 1 |



| Handler_commit | 0 |



| Handler_delete | 5764 |



| Handler_read_first | 11882 |



| Handler_read_key | 17967748 |



| Handler_read_next | 11395260 |



| Handler_read_prev | 132517 |



| Handler_read_rnd | 8745284 |



| Handler_read_rnd_next | 16203492 |



| Handler_rollback | 0 |



| Handler_update | 43638 |



| Handler_write | 2480703 |



| Key_blocks_used | 15586 |



| Key_read_requests | 47191694 |



| Key_reads | 96731 |



| Key_write_requests | 28353 |



| Key_writes | 23642 |



| Max_used_connections | 78 |



| Not_flushed_key_blocks | 0 |



| Not_flushed_delayed_rows | 0 |



| Open_tables | 256 | 100% of table_cache in use



| Open_files | 317 |



| Open_streams | 0 |



| Opened_tables | 1065 |



| Questions | 434737 |



| Qcache_queries_in_cache | 1170 |



| Qcache_inserts | 84599 |



| Qcache_hits | 179598 |



| Qcache_lowmem_prunes | 1876 |



| Qcache_not_cached | 1047 |



| Qcache_free_memory | 9929296 |



| Qcache_free_blocks | 706 |



| Qcache_total_blocks | 3241 |



| Rpl_status | NULL |



| Select_full_join | 6 |



| Select_full_range_join | 0 |



| Select_range | 20817 |



| Select_range_check | 0 |



| Select_scan | 15622 |



| Slave_open_temp_tables | 0 |



| Slave_running | OFF |



| Slow_launch_threads | 1 |



| Slow_queries | 40 | (execution time > 10 secs)



| Sort_merge_passes | 1 |



| Sort_range | 18472 |



| Sort_rows | 9506243 |



| Sort_scan | 11248 |



| Table_locks_immediate | 277974 |



| Table_locks_waited | 3260 |



| Threads_cached | 50 |



| Threads_created | 79 |



| Threads_connected | 15 |



| Threads_running | 9 |



| Uptime | 22125 | 6 hrs 8 mins 45 secs



+--------------------------+-----------+











Key Reads/Key Read Requests = 0.002050 (Cache hit = 99.99795%)



Key Writes/Key Write Requests = 0.833845



Connections/second = 2.516 (/hour = 9057.031)



KB received/second = 3.973 (/hour = 14303.837)



KB sent/second = 32.712 (/hour = 117763.525)



Temporary Tables Created/second = 0.216 (/hour = 777.437)



Opened Tables/second = 0.048 (/hour = 173.288)



Slow Queries/second = 0.002 (/hour = 6.508)



% of slow queries = 0.009%



Queries/second = 19.649 (/hour = 70736.868)
Other Scripts on the Server



- phpAdsNew (http://www.phpadsnew.com/)



- Subdreamer (http://www.subdreamer.com/)



- Gallery (http://gallery.menalto.com/)
Users Online



800-1000
PHP Info



http://www.gtplanet.net/misc/phpinfo.php
HTTPD.conf Variables



KeepAlive On



MaxKeepAliveRequests 100



KeepAliveTimeout 5



MinSpareServers 20



MaxSpareServers 50



StartServers 30



MaxClients 175







vBulletin Version 3.0.7
Thanks for all the help anyone can provide! :)

[sross]

My site is not as big as yours but if you are that popular chances are you don't need much search index spidering anymore by search bots. I got an app that customizes my htaccess to block all spiders and then say allow 1 or two from places I like should I choose. My guest numbers went down by 80% when I started blocking bots, and my cpu usage went from 4% during peak to now 1.2% during peak. Not sure if that helps but you might see less of a struggle when there are not 50 yahoo slurp spiders doing your board over (assuming you allow it).. I also found enormous amounts of bots ignore the robots.txt file, and many bots are spam bots searching for email addy's

[LanciaStratos]

Sounds interesting, do you have a link to that htaccess tool? However, most of these visitors are legitimate people, so that probably wouldn't help too much.

[sross]

sure. by the way I am not affiliated with this company in any way. It's called robot control pro http://webcomposing.com/webcomposing/HTML/cgi-programs.htm

I just found this tool to be an absolute godsend for me. It blocks bots, unknown bots, spam attacks, hack attacks, and the cool thing is when it blocks a bad request from bots or hacks I get notified by email. For example I just got this email a few minutes ago and I get about 40 of these types of mails per day (you can turn the mail option off but I like to know what is being blocked)

A spider robot that is not listed on your access approved page
has tried to index or collect information from: http://www.apug.org (http://www.apug.org/)
The program ignored the rules in the robots.txt file
He has been stopped and banned based on his IP address

The Spider visitor details
-------------------------------------------------------------------
Date/time: Sun Feb 27 09:48:39 2005

UserAgent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at
girafa dot com; http://www.girafa.com (http://www.girafa.com/))

provider: No domainname available ...
IP address: 64.210.196.195

[sross]

oh yeah, if you get it, let the developer install it for you. When he installs it he monitors it and tweaks it to fit your site the best. After payment he had it up and running on my site a day later..

[sross]

However, most of these visitors are legitimate people, so that probably wouldn't help too much.

See, i thought my guests were legit too. Then i updated my vb settings to include a full list of search bots (the same list that AWstats uses) and after that most of my guest listings then appeared as bot listings, a huge amount. I see you have over 500 guests online, I bet 100's of them are bots but i could be wrong. If you do get that app let me know how it goes, I'd be interested to see if your guest count drops to around 300..

[BamaStangGuy]

See, i thought my guests were legit too. Then i updated my vb settings to include a full list of search bots (the same list that AWstats uses) and after that most of my guest listings then appeared as bot listings, a huge amount. I see you have over 500 guests online, I bet 100's of them are bots but i could be wrong. If you do get that app let me know how it goes, I'd be interested to see if your guest count drops to around 300..
I would look into installing http://eaccelerator.net/DownloadEacceleratorUk

Very useful at dropping cpu load.

Also search your server for very large log files.. usually bigger than 2GB as these will cause problems

[eva2000]

800-1000 users over what cookie timeout.. ? phpadsnew turn off persistent connections

install eaccelerator from http://eaccelerator.net/InstallEacceleratorUk read http://www.vbulletin.com/forum/showpost.php?p=805040&postcount=441

and use this my.cnf below and restart mysql server


[mysqld]
skip-innodb
max_connections = 650
key_buffer = 16M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 64
wait_timeout = 10800
connect_timeout = 10
tmp_table_size = 32M
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 3M
query_cache_size = 64M
query_cache_type = 1

[mysqld_safe]
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

[LanciaStratos]

Thanks, eva, I'll apply those settings. My users online are with an 30 minute cookie timeout.

[reverof]

Lancia,
I ran a server very close to the same configuration as yours with a few more people than you have on, the only problem I really had was cpanel/whm likes to run scripts at their discretion, which is what I shortly learned. The biggest problem is its automated webstats, that seemed really hammer the server for a good 15 minutes. If you can try playing with cron, also if you really dont need the stats, in whm you can turn off all the stats, defualt runs all 3 types of stats.

Another item I did, was limit the search output to 100 top findings. search.php being a huge part of my forums, just the above helped lower that load. I just recently moved to a dual 2.8 & 3gb of ram and well on the first day of that, so cant say much, but can say no more swapping so far :)

just my 2cents

[LanciaStratos]

Thanks for your input, reverof - I'll check my search results right now. If you don't mind me asking, how exactly did you disable the stats in WHM? I want to make sure I do it correctly. Also, at what point did you decide to upgrade your hardwaer?

[reverof]

Lancia.... in whm Main (https://69.93.178.240:2087/scripts/command?PFILE=main) >> Server Setup (https://69.93.178.240:2087/scripts/command?PFILE=Server_Setup) >> Tweak Settings (https://69.93.178.240:2087/scripts2/tweaksettings)
there is then a section Stats and Logs, uncheck/check the ones you want to have and dont... make sure to delete logs after each setup, as much traffic as we get the logs are HUGE daily, atleast mine pushed almost 2gb a day and that about killed the server when it updated stats.

When my server load was constantly too high for my liking, that was constant 6.0+ at mid level periods, 9.0+ at high times and 1.5 or less between 2am-5am.

First day running the new server, server load never hit above 2, was a nice feeling. The RAM is more important than anything. 24 hours up and havent touched swap.

[eva2000]

disable Analog stats in WHM it's a disk space and server resource hog, just use webalizer for stats

[MadSkill]

I just purchased this software and waiting for them to contact me. I really hope this does the trick. We have 200+ bots usually floating around as well as an insane amount of spam mail. Fingers crossed.

sure. by the way I am not affiliated with this company in any way. It's called robot control pro http://webcomposing.com/webcomposing/HTML/cgi-programs.htm

I just found this tool to be an absolute godsend for me. It blocks bots, unknown bots, spam attacks, hack attacks, and the cool thing is when it blocks a bad request from bots or hacks I get notified by email. For example I just got this email a few minutes ago and I get about 40 of these types of mails per day (you can turn the mail option off but I like to know what is being blocked)