PDA

View Full Version : Download based website in need of Optimization


NightNinjaMPC
Tue 18th Sep '07, 5:56am
Hi,

I've been playing around with my server for long over 6 months trying to find out what it is that is causing this high load whenver I have 450+ users on the site. On saturday I had 600 and the server was reaching loads close to 13. I am running vBadvanced, vBSEO, and a highly modified version of vBDownloads (the major part of my website). I know that vBSEO inherently causes a higher load than average but the majority of traffic on my website is download based and not forum based and I can't quite figure it out... what can I do to optimize this situation? Is an SCSI hard drive necessary since downloads are in constant use or maybe more RAM?

1. Dedicated
2.
cpu speed/type single or dual cpus: Dual Proc AMD Opteron 248
how much memory installed: 2GB RAM
hard drive type/configuration: 250GB SATA II
linux distributor or windows version: CentOS 4.4-32
apache/IIS version: apache 1.3.37
PHP version: php 4.4.7
MySQL version: mysql 4.1.22-standard

3. vBulletin 3.6.7

4. Don't really understand this question about innodb
5. cPanel? Not sure
6. top - 03:26:26 up 80 days, 13:30, 1 user, load average: 1.18, 0.89, 0.79
Tasks: 167 total, 3 running, 164 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% us, 4.3% sy, 47.8% ni, 47.6% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074900k total, 1518084k used, 556816k free, 33516k buffers
Swap: 2096440k total, 111592k used, 1984848k free, 733248k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4568 mysql 21 6 215m 159m 2584 S 69 7.9 18159:04 mysqld
6981 nobody 19 4 25240 15m 3268 S 9 0.8 125:02.11 httpd
26028 nobody 20 4 21912 11m 2368 S 4 0.6 0:04.71 httpd
26030 nobody 19 4 22476 12m 2324 S 4 0.6 0:03.44 httpd
26117 nobody 19 4 22424 12m 2324 S 4 0.6 0:01.85 httpd
26120 nobody 19 4 21900 11m 2304 S 3 0.6 0:01.86 httpd
2270 nobody 19 4 22536 12m 2896 R 2 0.6 1:13.50 httpd
26260 nobody 19 4 21848 11m 2260 S 2 0.6 0:00.58 httpd
32181 nobody 19 4 22800 12m 2516 S 2 0.6 1:40.36 httpd
26105 nobody 19 4 21964 11m 2300 S 2 0.6 0:01.52 httpd
26107 nobody 19 4 22052 11m 2336 S 2 0.6 0:01.81 httpd
26249 nobody 19 4 21972 11m 2328 S 2 0.6 0:00.86 httpd
26027 nobody 20 4 22080 11m 2320 R 1 0.6 0:03.62 httpd
26031 nobody 19 4 21952 11m 2380 S 1 0.6 0:03.12 httpd
11985 nails 16 0 41492 20m 2088 S 0 1.0 0:35.52 scanner
26312 root 16 0 2064 1004 764 R 0 0.0 0:00.02 top
1 root 16 0 2700 484 452 S 0 0.0 0:15.14 init
Load is low cause there's only 250 people on when I posted this.

7. [mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=128M
query_cache_type=1
max_user_connections=100
max_connections=650
interactive_timeout=10
wait_timeout=10
connect_timeout=10
thread_cache_size=128
key_buffer=32M
join_buffer=1M
max_allowed_packet=64M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10
thread_concurrency=8
myisam_sort_buffer_size=64M
server-id=1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout
8. Gives me access denied error...

9. vB is the only script running on the site per say but it is modded heavily with a download script and about 90% of the site is download based rather than forum based.

10. Average 400
Max 750

12. if you run Apache and you have your own dedicated server or access to your httpd.conf (apache configuration file) can you post the values you have set for the following :

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 150

AND Maxrequestsperchild value = 0

13. Nope

14. Post output from these 3 commands

uname -a
Linux amazing.mpcdownloads.com 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 athlon i386 GNU/Linux
ulimit -aH

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 32767
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 37
model name : AMD Opteron(tm) Processor 248
stepping : 1
cpu MHz : 2205.081
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4414.01

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 37
model name : AMD Opteron(tm) Processor 248
stepping : 1
cpu MHz : 2205.081
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
bogomips : 4409.48

mlx
Tue 18th Sep '07, 7:46am
I don't know what vBDownloads is, does it store the files in a MySQL database or in the filesystem? If you have lots of static file downloads I would highly suggest to have a look at lighttpd (http://www.lighttpd.net/). lighty really rocks if it comes to image or file hosting!

NightNinjaMPC
Tue 18th Sep '07, 1:04pm
It stores the files on the server itself in a filesystem and saves information about them in the MySQL DB. Is Lighttpd a file serving system or a way to lighten the load?

mlx
Tue 18th Sep '07, 1:14pm
lighttpd is pretty much a fully featured web server. It can replace Apache, but you can also just run it as an image or file server on a different IP or port and stick with Apache for your main site.

Sites like Wikipedia and YouTube use lighttpd to run their image servers. IMHO it's perfect for file or image hosting. I guess it should help you to "lighten" your load there.

eva2000
Thu 20th Sep '07, 8:32am
File download site! Slow SATA disks will consume alot more cpu time, I'd use 15k SCSI disk or 2 and have the actual files residing on a separate disk from main OS/site files if possible.

You're also swapping to disk so so adding more memory will help.

Befoe we go any further need question #8 stats which you can alternatively obtain from method outlined in 2nd post at http://www.vbulletin.com/forum/showthread.php?t=70117

lighttpd will lessen cpu loads as well as memory consumption related to apache/web serving but would be troublesome when you have cpanel installed so put that idea aside for now..

NightNinjaMPC
Fri 21st Sep '07, 3:10am
First off, here is my MySQLReport

MySQL 4.1.22-standard uptime 83 11:8:55 Fri Sep 21 01:07:00 2007

__ Key __________________________________________________ _______________
Buffer used 28.32M of 32.00M %Used: 88.49
Current 13.09M %Usage: 40.91
Write hit 70.28%
Read hit 99.99%

__ Questions __________________________________________________ _________
Total 172.21M 23.9/s
QC Hits 105.96M 14.7/s %Total: 61.53
DMS 42.06M 5.8/s 24.42
Com_ 12.45M 1.7/s 7.23
COM_QUIT 11.92M 1.7/s 6.92
-Unknown 187.44k 0.0/s 0.11
Slow 7.00k 0.0/s 0.00 %DMS: 0.02
DMS 42.06M 5.8/s 24.42
SELECT 27.68M 3.8/s 16.07 65.80
UPDATE 9.98M 1.4/s 5.79 23.72
INSERT 3.16M 0.4/s 1.83 7.51
REPLACE 850.49k 0.1/s 0.49 2.02
DELETE 395.43k 0.1/s 0.23 0.94
Com_ 12.45M 1.7/s 7.23
change_db 11.92M 1.7/s 6.92
lock_tables 217.87k 0.0/s 0.13
unlock_tabl 191.58k 0.0/s 0.11

__ SELECT and Sort __________________________________________________ ___
Scan 11.89M 1.6/s %SELECT: 42.96
Range 2.16M 0.3/s 7.79
Full join 183 0.0/s 0.00
Range check 0 0/s 0.00
Full rng join 0 0/s 0.00
Sort scan 1.76M 0.2/s
Sort range 2.81M 0.4/s
Sort mrg pass 5.68k 0.0/s

__ Query Cache __________________________________________________ _______
Memory usage 19.97M of 128.00M %Used: 15.60
Block Fragmnt 24.84%
Hits 105.96M 14.7/s
Inserts 27.47M 3.8/s
Insrt:Prune 27.47M:1 3.8/s
Hit:Insert 3.86:1

__ Table Locks __________________________________________________ _______
Waited 59.82k 0.0/s %Total: 0.09
Immediate 63.32M 8.8/s

__ Tables __________________________________________________ ____________
Open 398 of 1024 %Cache: 38.87
Opened 16.79k 0.0/s

__ Connections __________________________________________________ _______
Max used 111 of 650 %Max: 17.08
Total 12.03M 1.7/s

__ Created Temp __________________________________________________ ______
Disk table 59.75k 0.0/s
Table 1.08M 0.2/s
File 11.37k 0.0/s

__ Threads __________________________________________________ ___________
Running 1 of 1
Cached 110 of 128 %Hit: 100
Created 111 0.0/s
Slow 4 0.0/s

__ Aborted __________________________________________________ ___________
Clients 212.98k 0.0/s
Connects 187.68k 0.0/s

__ Bytes __________________________________________________ _____________
Sent 2.47G 342.5/s
Received 965.07M 133.8/s

Secondly, of the two that you suggested (switching to an SCSI Drive and adding more RAM) which would you give more priority to?

eva2000
Fri 21st Sep '07, 8:29am
Try the following in this exact order. You can ignore any of the suggestions that you have already done.

1. Upgrade MySQL server to 4.1.22
2. Upgrade PHP to 5.2.4 if you're on 5.x or PHP 4.4.7 if you're on PHP 4.x after upgrading MySQL (ensure you install as ISAPI and NOT CGI)
3. Edit /etc/my.cnf and place the following mysql server settings in /etc/my.cnf and restart mysql server afterwards


[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
safe-show-database
old_passwords
back_log = 75
skip-innodb
max_connections = 650
key_buffer = 64M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 3000
thread_cache_size = 384
wait_timeout = 30
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 10
thread_concurrency = 4
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_limit = 3M
query_cache_size = 48M
query_cache_type = 1
query_prealloc_size = 131072
query_alloc_block_size = 65536
default-storage-engine = MyISAM

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
nice = -5
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

[mysqlhotcopy]
interactive-timeout


4. Remove Eaccelerator or APC Cache if they're installed and instead install Xcache v1.2.1 final release which seems to be a bit faster than APC Cache - http://www.vbulletin.com/forum/showthread.php?t=213267. Xcache site http://xcache.lighttpd.net/, documentation http://trac.lighttpd.net/trac/wiki/Docs and forums http://forum.lighttpd.net/forum/4

5. Upgrade to vB 3.0.17 http://www.vbulletin.com/forum/showthread.php?t=209720 if you're on vB 3.0.xx or upgrade to vB 3.5.8 http://www.vbulletin.com/forum/showthread.php?t=221903 if you're on vB 3.5.x. Upgrade to vB 3.6.8 http://www.vbulletin.com/forum/showthread.php?t=237919 if you're on vB 3.6.x

6. If you just upgraded to vB 3.5.x/3.6.x try to disable these 4 options:

Admin CP -> vBulletin Options -> Forums Home Page Options -> Display Logged in Users?

Admin CP -> vBulletin Options -> Forum Display Options (forumdisplay) -> Show Users Browsing Forums

Admin CP -> vBulletin Options -> Thread Display Options -> Show Users Browsing Thread

Admin CP -> vBulletin Options -> Message Searching Options -> Automatic Similar Thread search

7. Edit httpd.conf values to following and restart apache

KeepAlive: On
MaxKeepAliveRequests: 100
KeepAliveTimeout: 1
MinSpareServers: 10
MaxSpareServers: 15
StartServers: 10
MaxClients: 150
MaxRequestsPerChild: 1000

8. Then wait a few days then repost output for these 2 commands

top

mysqladmin -u root -p var ext ver > stats.txt

open the stats.txt file to copy and paste it's contents.

OR repost output from mysqlreport