Multiple server questions

[TheEDIGuy]

Good morning! We recently switched to a multiple-server setup (filesystem on one server, database on another), and I wondered if anyone had any advice on how to tweak things? We're maxing out the filesystem server at around 615 simultaneous users, and we really need to be able to handle upwards of 1000.

vBulletin: Version 2.2.8, with vbPortal front page, modified a bit. Serving up 4 million page views per month, with upwards of 16,000 visitors per day.

Server 1: Filesystem
AMD Duron 1gHz, 40gighd 5400rpm, 1gig RAM - 4 other small hobby sites with almost no traffic. Not enhanced with mod_gzip or PHP Accelerator.

Server 2: Database (on same IP subnet)
Celeron 1.7gHz, 60gig hd 7200rpm, 1gig RAM - nothing else on the site, running the latest phpadmin and 3.X version of MySQL. Enhanced with mod_gzip and PHP Accelerator, and a my.cnf that seems to be working well. Average loads for this server are well under 1.00, so this doesn't seem to be the problem.

Any suggestions on what I can move where to get things running more efficiently, or what I can install where?

Much appreciation in advance.

www.fansofrealitytv.com (http://www.fansofrealitytv.com)

[TheEDIGuy]

Since I posted this, I've:

1) Installed php accelerator on the Fileserver
2) Moved the smilies and avatars to the database server
3) Turned off displaying attachments (which I really hated doing)

Server loads still hover around 35 on the fileserver (which is where the load limiting is set on vB), but now we're up to 845 max users, and have been hovering around 800 or so all day. So, we've seen an improvement with php accelerator, but I still need suggestions to more evenly balance the load.

FWIW, I'm only using about 300k of my 1gig RAM on the fileserver, so I know it can be tweaked to run more efficiently.

[Raz Meister]

Upgrade to MySQL 4 to take advantage of the new caching system. This will reduce your load quite a bit.

Also post a request for eva2000 to tweak your my.cnf file.

[TheEDIGuy]

The database server is actually handling the load quite well. It's the fileserver that's causing headaches.

[Raz Meister]

Apologies, did not read your post properly.

What kernel are you using? Do a 'uname -r' to find out. If you're not using 2.4.x, you should upgrade to help with the load.

[TheEDIGuy]

2.4.18-18.7.x

I've learned a lot about database tuning (more than I ever wanted to know), but very little about Apache tuning. And I can't find too many examples to follow who're in the same boat. So, any help from Apache gurus is definitely appreciated.

[eva2000]

durons aren't suited to web serving

celerons aren't suited to database serving

both lack the amount of L2 caching to do a good job under load and it doesn't help that you have a very slow 5400 rpm hdd for your file server, that's gotta chew up cpu time by itself!

you're grossly under powered on both servers for 845 users but we need to see how many real concurrent connections you're using so please provide the following

1. is this on dedicated or shared virual server
2. your server specs. For example:

cpu speed/type single or dual cpus): dual p3 1ghz
how much memory installed: 512mb ram
hard drive type/configuration: 60GB EIDE non raid
linux distributor or windows version: redhat linux 7.3
apache/IIS version: apache 1.3.27
PHP version: php 4.3.1
MySQL version: mysql 3.23.56


3. if you use mysql 4.x instead of mysql 3.23.x, do you have any innodb type databases/tables on your server ?
4. if possible how mysql was compiled/installed
5. your top stats
6. your mysql configuration variables located at /etc/my.cnf or c:\my.cnf if on Windows server if you don't have that file you need to log into telnet and as root user type

mysqladmin -u root -p variables

copy and paste output here

7. your mysql extended-status output either still telnet as root user type

mysqladmin -u root -p extended-status

copy and paste output here

or preferred is to installed extended-status output script which is either located

- in your vB 2.2.6 or higher vB version's zip file extra's folder, upload mysqlinfo.php script to your site or if you're on an pre vB 2.2.6 install go to
- http://vbulletin.com/forum/showthread.php?threadid=3477 and install that scrip making sure to edit $mysqllogin line with your own mysqlusername and password

and post url to that here

8. oh and is your vB the only thing on the server? or other scripts? sites?

9. how many average and max concurrent users on your vB forum ?

10. create a file named phpinfo.php and place this code in it and post the url/link to it from your web site

<?
phpinfo();
?>

i.e. yourdomain.com/phpinfo.php

11. if you run Apache and you have your own dedicated server or access to your httpd.conf (apache configuration file) can you post the values you have set for the following :

KeepAlive
MaxKeepAliveRequests
KeepAliveTimeout
MinSpareServers
MaxSpareServers
StartServers
MaxClients

12. what version of vB are you running ?

[TheEDIGuy]

I'll post this info shortly, but just out of curiosity, if I were to switch back to a single-server system which is well-suited for what we're doing, and available at Rackshack, what would you recommend? I specify Rackshack, only because they don't seem to be big on SCSI drives, and I know that'll be a recommendation.

[dzeanah]

I'll post this info shortly, but just out of curiosity, if I were to switch back to a single-server system which is well-suited for what we're doing, and available at Rackshack, what would you recommend? I specify Rackshack, only because they don't seem to be big on SCSI drives, and I know that'll be a recommendation.Yep. Since you're talking about file system access, I was about to recommend SCSI drives in a hardware-based RAID configuration.

[eva2000]

well rackshack do have dual xeon with scsi hdd

[Erwin]

A quick solution for you is to uinstall vBPortal which is very query-intensive (upwards to 100 over on your index if you have lots of features on) and use something else like vBHome Lite or vBIndex. You will find that you won't need 2 servers to have 600 users online. I use a dual Xeon with the files on 1 hard disk, and database on the other, and I can host 600 users at once with no problems.

[TheEDIGuy]

Okay, I've dropped the AMD Duron, and if necessary (and it's appearing necessary so far), I will move to a dual Xeon in another month. But for now, a tweaking will be much appreciated, if you have a moment, George.

1. Dedicated Server

2. Celeron 1.7gHz, 1gig Ram, 60gig 7200rpm HD, redhat linux 7.3, Apache 1.3.27 w/ mod_gzip and phpA, php version 4.3.1, mySQL version 3.23.56

3. N/A
4. Don't know

5. Top stats, during normal usage:

1:15pm up 3 days, 37 min, 1 user, load average: 14.10, 12.41, 12.08
221 processes: 196 sleeping, 25 running, 0 zombie, 0 stopped
CPU states: 77.0% user, 11.3% system, 11.5% nice, 0.0% idle
Mem: 1026136K av, 795116K used, 231020K free, 0K shrd, 12064K buff
Swap: 1020116K av, 17572K used, 1002544K free 422032K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
21635 apache 19 0 20016 19M 16116 R 4.5 1.9 0:10 httpd
28258 apache 19 0 20064 19M 16104 R 4.4 1.9 0:12 httpd
22422 apache 14 0 18432 18M 16120 R 4.2 1.7 0:00 httpd
30571 apache 15 0 21124 20M 18100 R 4.1 2.0 0:27 httpd
31673 apache 15 0 18512 18M 15988 R 4.1 1.8 0:00 httpd
18636 apache 16 0 20140 19M 17228 R 3.9 1.9 0:15 httpd
8076 apache 14 0 20020 19M 15928 R 3.3 1.9 0:02 httpd
19538 apache 15 0 18404 17M 16064 R 2.8 1.7 0:00 httpd
6291 root 16 0 2012 2012 1632 R 2.5 0.1 1:33 top
2982 apache 16 0 18384 17M 15988 R 2.5 1.7 0:00 httpd
10989 apache 14 0 21172 20M 18000 R 1.5 2.0 0:21 httpd
31264 apache 14 0 18572 18M 16124 R 1.3 1.8 0:00 httpd
13957 apache 14 0 21524 21M 17300 R 1.2 2.0 0:24 httpd
28606 mysql 17 10 37412 36M 4616 R N 0.8 3.6 0:04 mysqld
11123 mysql 15 10 37412 36M 4616 R N 0.7 3.6 0:12 mysqld
26703 mysql 17 10 37412 36M 4616 R N 0.5 3.6 0:14 mysqld
18103 mysql 15 10 37412 36M 4616 R N 0.5 3.6 0:13 mysqld

6. my.cnf file:

[mysqld]
set-variable = max_connections=500
set-variable = key_buffer=16M
set-variable = myisam_sort_buffer_size=64M
set-variable = join_buffer=1M
set-variable = record_buffer=1M
set-variable = sort_buffer=2M
set-variable = table_cache=1024
set-variable = thread_cache_size=64
set-variable = wait_timeout=14400
set-variable = connect_timeout=10
set-variable = max_allowed_packet=16M
set-variable = max_connect_errors=10
[safe_mysqld]
open_files_limit=8192
[mysqldump]
quick
set-variable = max_allowed_packet=16M
[myisamchk]
set-variable = key_buffer=64M
set-variable = sort_buffer=64M
set-variable = read_buffer=16M
set-variable = write_buffer=16M

7. mysqlinfo.php URL: http://www.fansofrealitytv.com/forums/admin/mysqlinfo.php

8. vb is currently the only thing running on the server, although I will eventually have 3 or 4 other sites that are tiny, and may run phpAdsNew as well.

9. Average 375 concurrent users, max 845, although we will probably max more like 1000, but at 800 or so, the server freaks out.

10. PHPinfo: http://www.fansofrealitytv.com/phpinfo.php

11. Apache config:
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 15
MaxClients 250

12. vb version 2.2.8, with some vb.org mods and vpPortal (about 5% of our visitors hit the front page, the rest are using links directly into a thread).

[eva2000]

well i'd upgrade to mysql 4.0.12 and come back to this thread after so i can give you an updated my.cnf file

i'd reduce maxclients in httpd.conf from 250 to 150 and restart apache

[TheEDIGuy]

I'll reduce the connections, and I guess I'll take the plunge into 4.0.12. I just keep reading all of the bad experiences here with it, and it seems like introducing a lot of trouble in return for the decrease in server load. I appreciate all your help, eva2000!

[TheEDIGuy]

I reduced the connection, and installed 4.0.12 (whew, what fun!). I'll let it run overnight at least before checking back in. I also tweaked my my.cnf file a little to take advantage of the caching.

Tonight should be a busy night, with American Idol on. So, it should be a good test, and maybe a little tweak before Wednesday night's results show will help.

[TheEDIGuy]

Well, the overnight test results are in. We handled things slightly better, but still had loads around 30 most of the evening. Obviously, I need to move to the Dual Xeon, but for now, any my.cnf or other tweaks you can suggest are greatly appreciated. We're now on 4.0.12 MySQL, and here is my current top and my.cnf file (everything else above remains the same) (357 users on-line right now):

9:52am up 19:25, 1 user, load average: 3.87, 2.69, 2.11
293 processes: 289 sleeping, 4 running, 0 zombie, 0 stopped
CPU states: 73.7% user, 12.2% system, 0.0% nice, 13.9% idle
Mem: 1026136K av, 1015112K used, 11024K free, 0K shrd, 25228K buff
Swap: 1020116K av, 28384K used, 991732K free 339384K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
24197 apache 17 0 19676 18M 15956 R 12.2 1.8 0:01 httpd
11425 apache 20 0 19252 18M 15852 R 11.8 1.8 0:00 httpd
30790 apache 18 0 21508 20M 17860 R 10.6 2.0 2:09 httpd
31171 root 16 0 2008 2008 1632 R 6.1 0.1 0:02 top

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
max_connections = 500
key_buffer = 150M
myisam_sort_buffer_size = 64M
join_buffer = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 768K
thread_concurrency = 2
sort_buffer_size = 1M
table_cache = 512
thread_cache_size = 80
wait_timeout = 14400
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1

[safe_mysqld]
open_files_limit=8192

[mysqldump]
quick
set-variable = max_allowed_packet=16M

[eva2000]

drop

key_buffer = 150M

to

key_buffer = 16M

and restart mysql

[TheEDIGuy]

Done. Thank you for all your help, and your amazingly quick responses! Tonight is another busy night (AI Results Show), so hopefully it'll weather the storm a little better.