500,000 Daily Viistors, What do I need?

[Meltdown]

Hi everyone, shortly I'm going to be involved in a rather large project and have been asked to investigate what would be needed to handle 300,000 - 500,000 daily visitors to a VB board. It seems the company is looking at doing some intensive hacking of a VBB as it has many features they need, and they don't want to re-invent the wheel.

I've never come across a VBB that has close to those figure and don't know if it's even technically feasable.

Has anyone ever seen a VBB with 5,000-10,000 visitors online?

Any advice appreciated.

[krohnathlonman]

It would take a rather insane amount of power. The DB box would be very expensive, the dual A64's are on their way with as many as 12DIMM slots. throw tons of RAM with the top notch Athlon64 dually with some 15k SCSI RAID and you may be able to pull it off if you configure it properly. I think the hardware may be available to do it.

I'm not all that familiar with the limitations of MySQL having tons of people trying to post at the exact same time is something that MySQL doesn't handle as well as some of the other DB systems

[eva2000]

Hi everyone, shortly I'm going to be involved in a rather large project and have been asked to investigate what would be needed to handle 300,000 - 500,000 daily visitors to a VB board. It seems the company is looking at doing some intensive hacking of a vB as it has many features they need, and they don't want to re-invent the wheel.

I've never come across a vB that has close to those figure and don't know if it's even technically feasable.

Has anyone ever seen a vB with 5,000-10,000 visitors online?

Any advice appreciated.
remember a few things

1. vB user online concurrently is based on default 15 min cookie and is different from mysql concurrent connections

2. vB currently only can use mysql as a backend but later there will be ports to sybase and postgresql way after vB 3.x has been publicly released

3. mysql, provided it has the right hardware and mysql, php and apache configuration is good for theoretically 1000 - 1500 mysql concurrent connections (latter figure with the mysql rpm binaries) based on single dual cpu server setup (excluding dual, quad p4 xeons which could handle more)

4. this means you can have 1000 vB users online displayed but they actually can use 200 - 700 concurrent mysql max connections since vB shows those 1000 vB users over 15 mins and not the exact same sec, you can have any combination

i.e.

300 members + 700 guests = 1000 vb users online past 15 min, with only 300 - 500 mysql concurrent connections in use

or

500 members + 500 guests = 1000 vb users online past 15min, with 500 - 800 mysql concurrent connections in use

both examples show 1,000 vb users online past 15mins but the latter example could place more load on the server due to more mysql concurrent connection usage

5. hardware isn't the only aspect which determines vB performance, you have to have apache, php and mysql configured optimally - left unoptimised especially mysql's configuration, then even the most powerful hardware will not perform well

6. if you set your vB timeout cookie higher than default 15 mins, you will show more vB users online - so set timeout cookie in vB admin panel to 30 mins, then the vB user online display will show users on in the past 30 mins thereby inflating the vB user online display - generally the higher the cookie timeout, the higher the vB users online to mysql concurrent connections ratio

With the above being said, for real 300-500k visitors/day you'd be looking at a cluster of dedicated web and database servers working together to handle the load. That means ALOT of money invested in hardware, and server software licensing.

I've never personally seen or handled such a heavily traffic forum before but if money wasn't really an concern I'd try the following:

Using Emicnetworks.com clustering products (license priced per cpu) for:

- Apache http://www.emicnetworks.com/products/apache.html
- MySQL http://www.emicnetworks.com/products/mysql.html

and setup 2 segments

1. a dedicated Apache cluster farm of web/PHP servers acting as 1 single large apache server
2. a dedicated MySQL cluster farm of MySQL servers acting as 1 single large mysql server

I've never used Emicnetworks.com's products, but from what I understand their clustering solution allows you to group several dedicated servers together to act as one server on both Apache/web serving end and MySQL end. You can add and remove dedicated servers from the clusters as you web site/forum demands the power.

i.e.

You can start with 2 Apache web servers clustered together, each Dual Xeon 2.4ghz, 1-2GB ram, 2x 36GB SCSI 10k disks with dual Gigabit nics and have them connected via Gigabit nic to 1 or 2 dedicated MySQL servers clustered together as 1 single mysql server.

For MySQL servers, Dual AMD Opteron 64bit cpus with 4GB PC2700 Reg ECC ram and 15k rpm scsi disks in say raid 1+0 are the best for handling MySQL if you check out the one such comparison review at http://www.aceshardware.com/read.jsp?id=60000275 AMD 64bit Opterons also can address and handle insane amounts of ram which MySQL loves

Personally if cost wasn't an issue I'd only choose emicnetworks solutions as I wouldn't have the expertise in clustering or putting together such servers without having tech support to fall back on. Also would mean no need to hack vB as the clustering just means you have groups of servers acting like 1 Apache + 1 MySQL server.

I'm unsure how well emicnetworks MySQL and Apache clustering solutions scale in terms of handling the load or how well MySQL 4.x/5.x series will handle such loads.

HTH

[Meltdown]

Sorry for the delay in replying, thanks for the replies.

eva2000, many thanks for taking the time to write such a long and informative reply, much appreciated. I will contact the company you mention and see what solution they can offer.

I have found a site called http://www.big-boards.com which lists some very big forums and think I will contact a few to discuss their server setups.

Thanks again
David

[eva2000]

Meltdown, let me know how it progresses... would be interesting to see how it all holds up eventually :)

[KrON]

Be sure to check out the Xeon vs Opteron shootout at anandtech: http://www.anandtech.com/IT/showdoc.html?i=1982

I wouldn't settle for anything less than a 4 way opteron box for your db server, it is guarenteed to be your bottleneck.

We do about 50k unique visitors a day, with anywhere between 5-9 million hits and a about 1500 users online at peak usage.. If you want to share information, feel free to email me.