Large server cluster not large enough - help us choose what's next

[JonnyQuest]

Hi,

We are once again maxing out our current server solution and would love some input from you all. Here is our current status:

2 vB's:
1) serves about 15 million pages per month. 750-1200 online at a time. DB is 6+GB - Soccer related hence the upcoming World Cup boosting traffic
2) serves about 8 million pages per month. 350-600 online at a time. DB is 4+ GB

8 servers:

2 Load Balancers:
- Each a Celeron 2.4Ghz, 512MB RAM, 40GB IDE
4 Web Servers:
- Each a Pentium 4 3Ghz HT, 1GB RAM, 80GB IDE
1 DB Server:
- Dual-Xeon 2.8Ghz, 8GB RAM, 3x73GB SCSI HW-RAID5
1 Backup server
- Pentium 4 3Ghz HT, 1GB RAM, 80GB IDE

We are thinking of adding another equivalent DB server and moving to a master/slave setup. Anyone have any experience with this on vB? Is it possible to set up 2 different slave/read DB servers on a load balancer? Would we be better of splitting our two sites onto 2 DB servers? Anyone have any experience with mcluster?

How much RAM does vB like on web servers? Are we better off boosting all our web servers to 2 GB or ram or adding in a bunch more P4's? Are we better with 1 dual xeon 1 GB web server or 2 P4 1GB web servers?

Anyone have any other comments/thoughts?

Thanks!

[Joshs]

Where exactly are you bottlenecking? DB, HTTP, what? My guess is the DB. As a first step, you should definitely split the two sites up and let them each have their own DB server. Probably keep your existing one for your bigger site and get another one for the smaller one (shouldn't need near 8GB of RAM for it).

[ChrisLM2001]

Although the research was on E-Commerce server loads, this might help you better decide your options:

http://digital5.ece.tntech.edu/hexb/publications/ic2000.pdf



"In case of single PC web server with a database server, the throughput drops down quickly as workload increases. There are several
reasons for this phenomenon. First of all, heavy database
accesses place high demand on disk I/O systems. When the
number of database queries or database connections gets
large, the RAM in the PCs is not large enough to cache all
the database connections. Increasing number of database
queries beyond this point will creates an adverse effects
increasing the processing time for each database query. As
a result, we observed reduced throughput. Secondly, heavy
Servlet calls at the web server also require a lot of CPU
time as evidenced in our previous discussions. Adding an
additional PC as a web server while keeping the same
database server results in increased maximum throughput
as shown in Figure 7a. However, the throughput still drops
after the saturation point. In case of 2ws2dbs and 3ws2dbs
architectures, the performance drop is no longer as steep
because of one more database server."

So the issue is even if you throw more RAM at this, the returns will still diminish as the CPU and I/O are overloaded. You'll need to split it, so the server will have more CPU and I/O utilization. And by the looks of it, a 2 DB setup comes off better (in your case, maybe it'll be better to have 2x2 dual Xeons with 4GB of memory apiece, than a single DB server with 8GB).

So this also probably be a future heads up for you when you have to scale again -- it's the CPU and I/O you'll be needing the most for the DB, and try to match 2 web servers to 1 DB for optimal performance.

Chris

[eva2000]

yeah care to elaborate on the probems ? which is bottleneck, issues you're having etc ?

info asked at http://www.vbulletin.com/forum/showthread.php?t=70117 for servers

i'd definitely split up the 2 vB forums to each have their own web/mysql database server combination

but single cpu web servers with slow IDE drives even in cluster probably is the bottleneck i can see and raid 5 db server probably not the best option

also for web servers more ram you have the more apache concurrent connections you can handle

for high concurrency better off ditching intel/xeon and go for amd64 opteron cpus see http://www.vbulletin.com/forum/showthread.php?t=160103

without knowing all the details asked at http://www.vbulletin.com/forum/showthread.php?t=70117

I'd roughly guestimate splitting to 2 db servers one for each forum like (assuming anticipation for world cup traffic and basing prices of cpu at pricewatch.com)

vB #1 soccer related
web servers - hardware load balancer
4x dual AMD64 Opteron 246 cpus servers
OR
3x dual AMD64 Opteron 248 cpus servers
3GB REG ECC PC3200 DDRAM each server
4x 10,000rpm scsi raid 10 configuration each server
at least dual gigabit network cards each server

(probably end up dumping 1 dual opteron cpu server from either 4x or 3x cpu configs after world cup period)

db server
dual AMD64 Opteron 265 cpus (dual core meaning 4x physical cpus - quad cpu server)
8GB REG ECC PC3200 DDRAM
4x 15,000rpm scsi raid 10 hardware raid with 256MB cache (place whole database on this array except the postindex and threadsview tables)
2x 15,000rpm scsi raid 0 hardware raid with 256MB cache (place postindex and threadsview table on this raid 0 array and symlink it back to the main mysql data directory where the rest of the database is held - this will help with the massive amount of updates/reads to the postindex and threadview tables that usually cause table locking issues)

vB #2
Probably get away with
web server
dual AMD64 Opteron 246
4GB REC ECC PC3200 DDRAM
10,000rpm scsi disk

db server
dual AMD64 opteron 248
4GB REC ECC PC3200 DDRAM
15,000rpm scsi disk raid 10 as above, maybe forgo the raid 0 2nd raid array

Use MySQL 4.0.26 and PHP 4.4.1 with 2.6.x smp linux kernels. Use 64bit flavoured OS like Suse Enterprise to address large amounts of memory and database table and file sizes.

Again just a rough guestimate of what i think you'd need to handle your growth (not just current loads/forum activity) and read http://www.vbulletin.com/forum/showthread.php?t=160103 as dual opterons are dollar for dollar best value for web and db serving compared to xeon/intel cpus!