some help understanding top stats

[JTotten]

I have been having problems with my server recently, and if anyone is willing to take a quick look at my stats, and give any tips, it would be much appreciated.

First of all, these are the config settings I used, which Eva posted for me: http://www.vbulletin.com/forum/showthread.php?t=103574

I am running a 2.8ghz pentium 4 /w HT and 1gb of ram.
I am on Redhat 9. I have turck-mmcache installed, running the latest versions of php (4.3.8), mysql, apache, and vB 3.0.3. The only site on the server is www.wcreplays.com which has about ~400 people browsing (not necesarily on the forums) at any given time of day.

Here is a recent top stats right after a load spike and crash.

Any tips are much appreciated, causing much frustration with our admins and users at the moment =(



01:02:24 up 1 day, 7:20, 1 user, load average: 5.27, 6.97, 11.63
223 processes: 219 sleeping, 2 running, 1 zombie, 1 stopped
CPU0 states: 9.4% user 6.4% system 0.0% nice 0.0% iowait 83.0% idle
CPU1 states: 3.2% user 5.4% system 0.1% nice 0.0% iowait 90.1% idle
Mem: 1029760k av, 1009076k used, 20684k free, 0k shrd, 75628k buff
621636k actv, 159464k in_d, 22972k in_c
Swap: 1020088k av, 116960k used, 903128k free 748244k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
792 nobody 15 0 676 248 148 S 99.9 0.0 0:00 0 proftpd
19070 root 15 0 2788 1176 1092 S 99.9 0.1 0:18 0 httpd
19834 mysql 15 0 22368 19M 1184 S 8.9 1.9 1:50 0 mysqld
18135 root 34 19 556 528 96 R N 3.1 0.0 0:42 1 cp
20663 root 15 0 1280 1280 848 R 0.3 0.1 0:00 1 top
7 root 15 0 0 0 0 SW 0.1 0.0 0:49 1 kswapd
12 root 15 0 0 0 0 SW 0.1 0.0 0:03 0 kupdated
322 root 15 0 0 0 0 SW 0.1 0.0 0:06 1 kjournald
835 root 17 0 1972 1024 616 S 0.1 0.0 0:01 0 chkservd
1562 mailman 15 0 3032 992 248 S 0.1 0.0 0:14 0 python2
1 root 15 0 112 84 56 S 0.0 0.0 0:05 1 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1

[Erwin]

1) Your server loads are not great. It was over 11.0 15 minutes prior to that top. Something is causing loads to go up.

2) Looking at the top 3 CPU guzzlers:

99.9% - proftpd (FTP processes)
99.9% - httpd (Apache/ Web processes)
8.9% - mysqld (MySQL/ Database processes)

Do you run an FTP server/ site? Because that is accounting for most of your load.

Also, your site must be busy since it's very web intense - Apache is also using up a lot of your load.

Your database is not using much at all.

3) You seem to have enough memory, as your swap disk is smallish, and not all your current RAM is active anyway.

4) Your hyperthreading P4 shows 2 virtual CPUs, which are quite free when that top was taken:

CPU0 states: 9.4% user 6.4% system 0.0% nice 0.0% iowait 83.0% idle
CPU1 states: 3.2% user 5.4% system 0.1% nice 0.0% iowait 90.1% idle

But I suspect they won't be idle for too long with 99.9% usage by Apache and FTP.

If you're not running an FTP server, then I suspect your server has been hacked, and it's being used as an FTP server by a hacker, hence the high loads. If you are, then your site is too busy for your server.

Other people may have other comments.

[binkley]

I've never seen an FTP server use 99% of the CPU. That's quite an accomplishment. Even when it's tar'ing up a big file an FTP server won't use that much processing power.

Check /var/log/messages and there might be a hint as to why it crashed.

[JTotten]

Thanks for the replys

I checked the messages log and didn't notice anything unusual, except maybe a lot of proftp access by the server itself, although that might be perfectly fine. I also looked at the xferlog and also did not see any activity that was not suppose to be there. error_log just had a lot of mysql errors after mysql crashed last night.

I'm looking at top now, and proftpd is nowwhere to be seen, so maybe it was just a fluke?

11:16:47 up 1 day, 17:35, 1 user, load average: 0.69, 0.61, 0.51
229 processes: 227 sleeping, 2 running, 0 zombie, 0 stopped
CPU0 states: 9.0% user 7.0% system 0.0% nice 0.0% iowait 83.0% idle
CPU1 states: 20.0% user 6.0% system 0.0% nice 0.0% iowait 73.0% idle
Mem: 1029760k av, 977092k used, 52668k free, 0k shrd, 91012k buff
646480k actv, 132716k in_d, 29952k in_c
Swap: 1020088k av, 153676k used, 866412k free 506472k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
22473 mysql 15 0 57132 54M 1260 S 26.4 5.4 60:39 1 mysqld
19070 root 15 0 4328 1584 1136 S 10.0 0.1 2:29 0 httpd
11840 root 20 0 1284 1284 856 R 3.6 0.1 0:00 0 top

[binkley]

You don't by chance use MySQL to authenticate FTP access?

[eva2000]

what's your cookie timeout and current mysql extended-status output like as well http://vbulletin.com/forum/showthread.php?threadid=3477

[JTotten]

my cookie timeout is currently set at 900, and you can view the extended status script here:
http://www.wcreplays.com/mysql_extended_status.php