Dedicated server newbie. High server load. George (eva2000), we need you!!

[Aaow AnD wHiTe]

Hello everyone!! We've just re-launched our website in our new dedicated server with RackShack, and we're having a high load at this time :(

Hopefully eva2000 can find the time to help us!! BTW, I wanna say a BIG thanx to him, wether it can help us or not, because he is always so nice and helpful to everyone. Thanx George, your work is really appreciated!.

Ok, so here we go!

--
1.server specs
Intel Celeron 1,3 - 512 MB RAM, 60 GB HD 7200 RPM, Red Hat Linux 7.2 and Ensim 3.1.1
PHP Version 4.2.2
mySQL 3.23.41
Apache 1.3.22

2. How was mysql compiled
I don't really know how mysql was compiled/installed, it came standar with the server

3. top stats
3:47pm up 13 days, 13:52, 2 users, load average: 25.49, 28.16, 22.57
229 processes: 195 sleeping, 20 running, 14 zombie, 0 stopped
CPU states: 28.9% user, 27.5% system, 43.4% nice, 0.0% idle
Mem: 504792K av, 413140K used, 91652K free, 1264K shrd, 6036K buff
Swap: 1020116K av, 151648K used, 868468K free 117684K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
21132 mysql 15 10 31876 25M 8180 S N 12.4 5.1 0:00 mysqld
21144 mysql 17 10 31876 25M 8180 R N 6.4 5.1 0:00 mysqld
21145 mysql 17 10 31876 25M 8180 R N 6.0 5.1 0:00 mysqld
21106 mysql 15 10 31876 25M 8180 S N 5.6 5.1 0:00 mysqld
21133 mysql 17 10 31876 25M 8180 R N 4.7 5.1 0:00 mysqld
21097 mysql 15 10 31876 25M 8180 S N 4.4 5.1 0:01 mysqld
21130 mysql 17 10 31876 25M 8180 R N 4.4 5.1 0:00 mysqld
21131 mysql 16 10 31876 25M 8180 S N 4.0 5.1 0:00 mysqld
15459 admin 9 0 1196 1124 764 R 2.7 0.2 0:40 top
12784 apache 9 0 14156 7376 6108 S 2.5 1.4 0:18 httpd
17888 apache 9 0 14640 8064 6356 S 2.5 1.5 0:06 httpd
20919 apache 9 0 14116 7600 5332 S 2.5 1.5 0:00 httpd
15560 apache 9 0 14820 8272 7420 S 2.2 1.6 0:03 httpd
21066 mysql 15 10 31876 25M 8180 S N 2.0 5.1 0:01 mysqld
6045 apache 20 0 14804 7988 7212 R 1.6 1.5 1:29 httpd
17886 apache 9 0 14144 7472 7252 S 1.2 1.4 0:04 httpd
20337 apache 9 0 14368 7868 5384 S 1.2 1.5 0:00 httpd
11454 apache 9 0 14304 7624 6980 S 1.1 1.5 0:22 httpd
13522 apache 20 0 14412 7732 6228 R 1.1 1.5 0:12 httpd
17265 apache 9 0 14696 8044 6764 R 1.1 1.5 0:05 httpd
18060 apache 9 0 14372 7812 7480 S 1.1 1.5 0:04 httpd
13410 apache 9 0 14592 7920 6544 S 0.9 1.5 0:13 httpd
20924 apache 9 0 14224 7720 5376 S 0.9 1.5 0:00 httpd
21126 mysql 15 10 31876 25M 8180 S N 0.7 5.1 0:00 mysqld
6034 apache 9 0 14168 7368 7368 S 0.5 1.4 1:37 httpd
11606 apache 9 0 14572 7932 7564 S 0.5 1.5 0:16 httpd
13510 apache 9 0 14376 7768 7568 S 0.5 1.5 0:09 httpd
20627 apache 9 0 14232 7724 5388 S 0.5 1.5 0:00 httpd
21166 mysql 15 10 31876 25M 8180 S N 0.5 5.1 0:00 mysqld
6033 apache 9 0 14332 7516 7300 S 0.3 1.4 1:50 httpd
7754 apache 9 0 14184 7364 6312 S 0.3 1.4 1:05 httpd
15820 apache 9 0 14208 7524 7524 S 0.3 1.4 0:02 httpd
20649 apache 9 0 13940 7432 5360 S 0.3 1.4 0:00 httpd
20650 apache 9 0 14032 7524 5380 S 0.3 1.4 0:00 httpd
20865 apache 9 0 14436 7932 5388 S 0.3 1.5 0:00 httpd
20932 apache 9 0 14052 7544 5376 S 0.3 1.4 0:00 httpd
21136 apache 9 0 12060 5400 5000 S 0.3 1.0 0:00 httpd

4. my.cnf
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

5. mysql extended status
+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Aborted_clients | 4 |
| Aborted_connects | 10 |
| Bytes_received | 584050014 |
| Bytes_sent | 1470554712 |
| Connections | 17736 |
| Created_tmp_disk_tables | 3366 |
| Created_tmp_tables | 11755 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_delete | 11536 |
| Handler_read_first | 6120 |
| Handler_read_key | 25971836 |
| Handler_read_next | 106541318 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 1967635 |
| Handler_read_rnd_next | 898316127 |
| Handler_update | 2374357 |
| Handler_write | 27266811 |
| Key_blocks_used | 7793 |
| Key_read_requests | 81851938 |
| Key_reads | 712781 |
| Key_write_requests | 7236020 |
| Key_writes | 796295 |
| Max_used_connections | 100 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 64 |
| Open_files | 94 |
| Open_streams | 0 |
| Opened_tables | 11512 |
| Questions | 396882 |
| Select_full_join | 865 |
| Select_full_range_join | 0 |
| Select_range | 45966 |
| Select_range_check | 0 |
| Select_scan | 79479 |
| Slave_running | OFF |
| Slave_open_temp_tables | 0 |
| Slow_launch_threads | 34 |
| Slow_queries | 1833 |
| Sort_merge_passes | 0 |
| Sort_range | 61671 |
| Sort_rows | 2156077 |
| Sort_scan | 29581 |
| Table_locks_immediate | 443915 |
| Table_locks_waited | 2462 |
| Threads_cached | 0 |
| Threads_created | 17735 |
| Threads_connected | 23 |
| Threads_running | 12 |
| Uptime | 1173610 |
+--------------------------+------------+
Uptime: 1173610 Threads: 23 Questions: 396883 Slow queries: 1833 Opens: 11512 Flush tables: 1 Open tables: 64 Queries per second avg: 0.338

6. At this time, I have vBulletin 2.2.8 with some hacks and nothing else that I can't think of, except for Ensim.

7. max concurrent users
About 80-100

8. php info
You can find it here (http://www.michaeljacksonfans.com/foro/admin/phpinfo.php)

9. http.conf
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 150

--
If you need anything else, please let me know! Hope you can help us :D Thanx anyway!! :)

[eva2000]

heh :D

things to do:

1. change my.cnf to below one and restart mysql


[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
set-variable = max_connections=350
set-variable = key_buffer=16M
set-variable = myisam_sort_buffer_size=64M
set-variable = join_buffer=1M
set-variable = record_buffer=1M
set-variable = sort_buffer=2M
set-variable = table_cache=1024
set-variable = thread_cache_size=256
set-variable = wait_timeout=14400
set-variable = connect_timeout=10
set-variable = max_allowed_packet=16M
set-variable = max_connect_errors=10

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192

[mysqldump]
quick
set-variable = max_allowed_packet=16M

[myisamchk]
set-variable = key_buffer=64M
set-variable = sort_buffer=64M
set-variable = read_buffer=16M
set-variable = write_buffer=16M


2. upgrade to MySQL 3.23.53a first, then upgrade to PHP 4.2.3

[Aaow AnD wHiTe]

Thx eva!! :D

I've tried the first step, but see... I have all these my.cnf O_O

/etc/my.cnf
/etc/virtualhosting/templates/mysql/etc/my.cnf
/etc/my.cnf.admin_appl
/home/virtual/site1/fst/etc/my.cnf
/home/virtual/site2/fst/etc/my.cnf

I've edited this one: /etc/my.cnf Is it correct?

Thanx in advance :)

PS: My vbulletin is hosted at site1 currently!

[eva2000]

Originally posted by Aaow AnD wHiTe
Thx eva!! :D

I've tried the first step, but see... I have all these my.cnf O_O

/etc/my.cnf
/etc/virtualhosting/templates/mysql/etc/my.cnf
/etc/my.cnf.admin_appl
/home/virtual/site1/fst/etc/my.cnf
/home/virtual/site2/fst/etc/my.cnf

I've edited this one: /etc/my.cnf Is it correct?

Thanx in advance :)

PS: My vbulletin is hosted at site1 currently! /etc/my.cnf is the correct one i believe but ... hmmm i wonder if /home/virtual/site1/fst/etc/my.cnf are also editable... what control panel you're using ?

you can actually test it


in this my.cnf at /home/virtual/site1/fst/etc/my.cnf

add this line into the file

set-variable = max_user_connections=2

and restart mysql

if you go to your forum and get max_user_connections reached, then you fill in /home/virtual/site1/fst/etc/my.cnf with the above recommended values (minus the max_user_connection setting of course )

[Aaow AnD wHiTe]

Thanx George, and sorry for being so late in replying :( We've make sure that the correct one isn't /home/virtual/site1/fst/etc/my.cnf

Monitoring our server today we've noticed some active httpd processes (sp?) that have been active for a looooong time. I think those are from some of our download files. They were taking 10% CPU resources each!! Is it normal? Any tip on that?

Thanx in advance!!

[eva2000]

Originally posted by Aaow AnD wHiTe
Thanx George, and sorry for being so late in replying :( We've make sure that the correct one isn't /home/virtual/site1/fst/etc/my.cnf

Monitoring our server today we've noticed some active httpd processes (sp?) that have been active for a looooong time. I think those are from some of our download files. They were taking 10% CPU resources each!! Is it normal? Any tip on that?

Thanx in advance!! how long ?

[Aaow AnD wHiTe]

Around... 458:00 or something like that O_O I think it may have something to do with this one:

http://forum.rackshack.net/showthread.php?s=&threadid=13753

I have some mysqld processes running for a long time and using around 25-30% of the CPU too O_o

I think we'll upgrade both mySQL and PHP too :) What about PHP Accelerator, do you recommend it? :)

Thanx for being so nice and helpful :)

[eva2000]

Originally posted by Aaow AnD wHiTe
Around... 458:00 or something like that O_O I think it may have something to do with this one:

http://forum.rackshack.net/showthread.php?s=&threadid=13753

I have some mysqld processes running for a long time and using around 25-30% of the CPU too O_o

I think we'll upgrade both mySQL and PHP too :) What about PHP Accelerator, do you recommend it? :)

Thanx for being so nice and helpful :) definitely use phpaccelerator :)

[Aaow AnD wHiTe]

Originally posted by eva2000
definitely use phpaccelerator :) I've installed it a couple of minutes ago!! Working great so far! ^__^ Thanx again, eva!! :D I think I'm not going to upgrade to PHP 2.2.3 yet, I've seen there are some issues with it yet :-S Any other tips you would like to share, George? :D