Dream servers for large forum?

[RobFerrari]

If money wasn't as much an object, what type of server configuration would you have for a large forum (25,000+ users)?

If you had two servers, would you just make one app/web and other Db?

[Andreas]

I do have two servers ;)

Web is a Dual Xeon 2.4, 4 GB, IDE HDs
DB is a Dual Xeon 2.8, 4 GB, 4x 15K SCSI HDs as RAID 10

Works well so far, the board has about 40K members.

Quad Opterons would rock ... but they are too expensive.

[Jerry]

IBM Blue Gene (http://www.top500.org/system/7747) Running vBulletin-beowulf ;)

[Zachery]

Dream Server?

Hmm, well, a dream server would be a huge cluster with a big backend mysql setup.

But currently I run several dual server setups, but I'd like to move to 3 servers

1 php/web
1 images
1 database

I want to offload as much of the static content (images mostly) that I acn from apache, to a lighter server like tux or something simliar :)

[RobFerrari]

With 2 servers, do you put images with the web/app or db server? I store all attachments, so have 40 GB or so.

[Zachery]

web server, you could run an lite fileserver on the web server and off load apache for that, which would also be easy.

[eva2000]

If money wasn't as much an object, what type of server configuration would you have for a large forum (25,000+ users)?

If you had two servers, would you just make one app/web and other Db? Why think so small guys if money wasn't an object ? :D

Opteron cpus all the way given this info at http://www.vbulletin.com/forum/showthread.php?t=160103

first question answer: for 25,000 vB users online simultaneously over 900 second default cookie timeout ??? FYI, largest on average vB forums i have seen only have between 2,500 to 5,000 vB users online at the same time over various set longer than default vB cookie timeouts.

hard to say it would be trial and error to figure out the hardware needed but i'd start with

db server would

cpus
8-way AMD Athlon64 Opteron 880 2.4ghz (dual core) (http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8796_9240,00.html) Specs (http://www.amdcompare.com/us-en/opteron/details.aspx?opn=OSA880FAA6CC) (8 physical cpus each with 2 real cores = 16 cpus on the one server :cool: )

motherboard
Motherboard not sure there aren't many boards that support 8x opteron 880 cpus some have support for upto 128GB of memory but features i'd like on the board
- 2x x16 PCI-E slots
- at least 1 or 2 64bit/133mhz slots

memory
128GB PC3200 REG ECC memory

hard drives
for mysql data partition split into 3 separate raid 10 arrays (yes 12x 15k rpm scsi disks :D )

array 1 - 4x 15,000 rpm scsi raid 10 - storing majority of vB database except post, postindex, thread, user and word tables

array 2 - 4x 15,000 rpm scsi raid 10 - storing post, thread and user tables symlinked back to main mysql data directory

array 3 - 4x 15,000 rpm scsi raid 10 or 2x 15,000rpm scsi raid 0 - storing postindex and word tables symlinked back to main mysql data directory - if raid 0 is used and drive fails the postindex/word tables can be rebuilt via vB admincp update counters

hardware raid controller
PCI-E based raid controller with 256MB minimum memory cache and battery backup module and must have online RAID level migration and capacity expansion.

For traditional SCSI based, maybe something like
- LSILogic MegaRaid 320-2E http://www.lsilogic.com/products/megaraid/megaraid_320_2e.html

Now for the more expensive but newer SAS (http://www.lsilogic.com/technologies/industry_standards/sas.html and http://www.lsilogic.com/technologies/industry_standards/sas_pages/sas_benefits.html and http://www.seagate.com/products/interface/sas)
- LSILogic MegaRaid SAS raid controller http://www.lsilogic.com/products/megaraid_sas/index.html
- Adaptec 4805SAS SAS raid controller http://www.adaptec.com/worldwide/product/proddetail.html?sess=no&language=English+US&prodkey=SAS-4805&cat=%2fTechnology%2fSerial+Attached+SCSI%2fRAID+Co ntrollers

For SAS you would need the newer released SAS SCSI 15k rpm drives i.e. Seagate Cheetah 15k.4 SAS SCSI http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,693,00.html and http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,693,00.html

dual or quad gigabit network cards

Above is for 1x mysql server - so maybe have 2 or 3 of these, one replicated and one for backup (that's a total of 32 to 48 physical cpu cores ) hehe

Web servers:
- hardware load balancer so you can add web servers as needed
- implement lighttpd, litespeed web server for static files
- APC cache
- each web server being = dual opteron 285 dual core cpus (4 physical core/cpus), 8GB PC3200 REG ECC ram each, 15k scsi or 15k SAS scsi raid 10, gigabit network cards
- probably need at least 6 of these for 25,000 vB users online ??????? anyone care to guestimate ? :D

whole network would be gigabit connection speed at least.

Guess a memcached server with enough memory to hold large segments of database would be nice i.e. postindex, word tables - would base it on dual opteron 285 cpus i guess.

second question, yes for 2 servers, one web and one db would be best :)

[RobFerrari]

Dang eva2000, that was fun to read.

I only have 25,000 users growing about 1,000 a month. 500-600 typical online users with 900 max.

[eva2000]

heh was fun to type :D

Ah 25,000 total members! Much easier to support and less costly than 25,000 vB users online at a time :)

[RobFerrari]

heh was fun to type :D

Now how much would all of that cost? :)

[eva2000]

Now how much would all of that cost? :)
alot... i.e. just 1x opteorn 880 cpu is ~US#1,469 http://www.monarchcomputer.com/Merchant2/merchant.mv?Screen=PROD&Store_Code=M&Product_Code=120779&AFFIL=pricewatch&NR=1 so 8x times that for 8 cpus on the 1 mysql server = US$11,752 just for the cpus for 1 mysql server

Adaptech 4805SAS raid PCI-E is around US$515-550

US$1000 for each 4GB of PC3200 REG ECC :eek: upto 128GB LOL

[AlfonsoC]

If money is really not a concern.... why not going for this kind of storage?

http://h18006.www1.hp.com/products/storageworks/eva/index.html

Look for the 'small' configuration EVA8000 is like killing flies with a large cannon, but... they said money is not a concern.

15k disks are sometimes prone to fail, if you don't have enough cooling, then, for ensuring some pretty reliability I would go for this.

If money is not a concern too on the web servers I'll look for a setup with Sun T1000/T2000 machines.

The same for the database... why being stuck on x86/AMD64 world? Go for an HP RP7420 or RP8420 with PA-RISC 8900 CPU's (SuperDomes could qualify here too, but... are too bulky :D ).

Think big...

Why think so small guys if money wasn't an object ? :D

hard drives
for mysql data partition split into 3 separate raid 10 arrays (yes 12x 15k rpm scsi disks :D )

array 1 - 4x 15,000 rpm scsi raid 10 - storing majority of vB database except post, postindex, thread, user and word tables

array 2 - 4x 15,000 rpm scsi raid 10 - storing post, thread and user tables symlinked back to main mysql data directory

array 3 - 4x 15,000 rpm scsi raid 10 or 2x 15,000rpm scsi raid 0 - storing postindex and word tables symlinked back to main mysql data directory - if raid 0 is used and drive fails the postindex/word tables can be rebuilt via vB admincp update counters

hardware raid controller
PCI-E based raid controller with 256MB minimum memory cache and battery backup module and must have online RAID level migration and capacity expansion.

For traditional SCSI based, maybe something like
- LSILogic MegaRaid 320-2E http://www.lsilogic.com/products/megaraid/megaraid_320_2e.html

Now for the more expensive but newer SAS (http://www.lsilogic.com/technologies/industry_standards/sas.html and http://www.lsilogic.com/technologies/industry_standards/sas_pages/sas_benefits.html and http://www.seagate.com/products/interface/sas)
- LSILogic MegaRaid SAS raid controller http://www.lsilogic.com/products/megaraid_sas/index.html
- Adaptec 4805SAS SAS raid controller http://www.adaptec.com/worldwide/product/proddetail.html?sess=no&language=English+US&prodkey=SAS-4805&cat=%2fTechnology%2fSerial+Attached+SCSI%2fRAID+Co ntrollers

For SAS you would need the newer released SAS SCSI 15k rpm drives i.e. Seagate Cheetah 15k.4 SAS SCSI http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,693,00.html and http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,693,00.html

[TheEDIGuy]

hard drives
for mysql data partition split into 3 separate raid 10 arrays (yes 12x 15k rpm scsi disks :D )

array 1 - 4x 15,000 rpm scsi raid 10 - storing majority of vB database except post, postindex, thread, user and word tables

array 2 - 4x 15,000 rpm scsi raid 10 - storing post, thread and user tables symlinked back to main mysql data directory

array 3 - 4x 15,000 rpm scsi raid 10 or 2x 15,000rpm scsi raid 0 - storing postindex and word tables symlinked back to main mysql data directory - if raid 0 is used and drive fails the postindex/word tables can be rebuilt via vB admincp update counters

Eva, how would one go about doing this? I have no idea how to separate specific tables out to specific drives, but I think this would be a good thing to do on the server we're about to move to.

[eva2000]

Eva, how would one go about doing this? I have no idea how to separate specific tables out to specific drives, but I think this would be a good thing to do on the server we're about to move to.
http://dev.mysql.com/doc/refman/5.0/en/symbolic-links.html

for databases
http://dev.mysql.com/doc/refman/5.0/en/symbolic-links-to-databases.html

for database tables (one ya want)
http://dev.mysql.com/doc/refman/5.0/en/symbolic-links-to-tables.html

:)

[Dave#]

Storage: Something from netapp (http://www.netapp.com/)
Servers: Dual Dual core Opterons (http://www-03.ibm.com/systems/opteron/ls20/index.html) For about 10k $US a piece fully loaded

Those suggesting using supercomputers for Mysql have perhaps not had the misfortune of having to do that