Server Config (Lighttpd + FastCGI + PHP + Eacellerator)

[f0urtyfive]

Well, not so much a configuration request as a configuration statement ;)

Here's what we're running right now on absolutepunk.net:

We've ditched apache all together, and are running solely lighttpd.

Lighttpd connects to FastCGI (php) for php apps, and just serves anything else out. We have 2.6 Million posts, 100k threads, and 70k members, with a minimum 1000 users on at any given moment.

Server configuration.
Servers just got done a reconfiguration, so their a bit funky right now (will probably eliminate one, not sure yet).

water.absolutepunk.net
Runs solely MySQL
Dell Poweredge 2650 (I think?)
2x 15k RPM 18 GB SCSI Drives in Raid 1
Dual Xeon 2.8 Ghz 1 GB Ram
Load is currently 0.3, usually stays below 1.

fire.absolutepunk.net
Dual Xeon 2.4 Ghz
1 GB Ram
2x 10k RPM 72 GB Drives in Raid 1 (SATA)
Load currently 1.4, stays below 2
(not fully configured yet) Runs lighttpd + FastCGI (php) (no eaccelerator yet)
After moving from apache to lighttpd, memory usage is a whopping 134 MB)

steam.absolutepunk.net
Pentium D Dual Core 3.0 Ghz
2 GB Ram
1x 250 GB 7.2k RPM SATA Drive
Load 0.4 stays below 1
Currently is just running FastCGI (php) with eaccelerator.
(Runs no webserver!)

server1.ceteranet.com
(On loan for reconfiguration)
Dell Poweredge 750
Pentium 4 2.8 Ghz
1 Gb Ram
2x 160 GB Hard Disk in Raid 1
Currently just running lighttpd (and a bunch of unrelated stuff)
Lighttpd cpu usage usually stays within 1-5% range

So basically right now, fire runs its own php, and server1.ceteranet.com runs a webserver, and outsources all the PHP running to steam. We're hoping to have just one webserver setup, with all the dynamic requests outsourced to another / two other servers.

Lighttpd is AMAZING in resource usage when compared to apache. You should definetly try it out before switching to a multi-server configuration.

a few links for stastics:
http://66.55.139.142/server-status (server1 stats)
http://66.55.139.130/server-status (Fire stats)
http://www.ceteranet.com/munin/ (stats for the whole bunch, steam == server2.ceteranet.com)

my.conf

[mysqld]
#log-bin = mysql-bin
#server-id = 1
port = 3306
socket = /tmp/mysql.sock
log_slow_queries=/var/log/slow-queries.log
max_connections = 350
key_buffer = 16M
key_buffer = 175M
#changed 3/2/05 by f0urtyfive
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 64
wait_timeout = 30
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
skip-innodb
safe-show-database
tmp_table_size = 128M
max-binlog-size = 256M
max-relay-log-size = 128M
[mysqldump]
quick
#max_allowed_packet = 16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 64M
sort_buffer_size = 64M
read_buffer = 16M
write_buffer = 16M
[myisamchk]
#key_buffer = 64M
#sort_buffer_size = 64M
#read_buffer = 16M
#write_buffer = 16M
[mysqlhotcopy]
interactive-timeout

[Zachery]

How does the one server running php work? I've been looking into bigger options for server people I know.

Untill I can get them to ditch cpanel (for their multiple products) im probally stuck but I'm looking at offloading things if I can, where I can.

[f0urtyfive]

It's fairly simple, grab a copy of lighttpd, and start playing with it

Here's my current "install" procedure:
latest copy of lighttpd,
latest copy of php-4

yum install a few packages (pcre-devel, jpeg-devel, etc)

lighttpd
./configure --prefix=/services/lighttpd/ --with-pcre
make
make install

(--with-pcre is for mod_rewrite, otherwise it doesnt work).
php
./configure --enable-fastcgi --enable-force-cgi-redirect --with-mysql --prefix=/services/php --with-gettext --with-gd --with-zlib-dir=../zlib-1.2.3/ --with-jpeg-dir=/root/jpeg-6b/
make
make install

then to get JUST php running (if your doing lighttpd and fastcgi on the same server, lighttpd can "adaptively spawn" processes, but remotely you have to spawn fcgi by hand).
cd /services/lighttpd/bin
./spawn-fcgi -f /services/php/bin/php -a 0.0.0.0 -p 1027

where -f is the path to php-fcgi, -a is address to bind to and -p is port to bind to.

lighttpd.conf looks like this for fcgi:
fastcgi.server = ( ".php" =>
(( "host" => "66.55.139.139",
"port" => 1027,
"check-local" => "disable",
"docroot" => "/path/to/docroot"
))
)

Fairly easy after you get everything running... I imagine once I have this setup working correctly, all we'll need to do is add PHP nodes as necessary to the lighttpd config (You can have multiple fcgi servers in there, it load balances) and added capacity.

*shameless plug* If anyone is looking for a setup like this let me know ;) */shameless plug*

[f0urtyfive]

BTW, Eaccelerator is a bit of a pain on fcgi, simply because you have to do 2 installations of PHP, one for fcgi, and then one without fcgi (You need a non-fcgi version of phpize of the same PHP version as your fcgi install, for it to compile) but after that its just normal install procedure.

[Zachery]

assuming you have multiple nodes, couldn't you compile it once as a DSO and then just move it out to other servers.


And, thats an awsome feature, I had heard others talking about it but never knew it was as simple as that. I figured it would be alot more work.

Seems like a firewall/vlan would be a good idea for all of the servers to communicate on in the background to ensure that they get preimum pipe to communicate on.

[f0urtyfive]

I imagine you could compile it on one node and just share it, just havent tried yet ;). As I dont know the security implications of running php like that, we're just running it on private ip space (10.0.0.0/8) so no one can get to it... Probably going to either move it to a seperate vlan or seperate switch soon enough.

[Zachery]

Whats the network usage in the vlan? is the vlan on its own set of cards?

[Erwin]

That's a sweet setup - I plan to do almost exactly the same, although I will be using FreeBSD - lighttpd is a port and easily installed.

I didn't know about the phpize issue - I assume it would be the same for APC too.

[f0urtyfive]

Whats the network usage in the vlan? is the vlan on its own set of cards?

I imagine you're curious about how much bandwidth php uses... I cant really tell you, as our MySQL is all ran over the same interface (One of the two servers with PHP / MySQL uses around 10-20 mbits inbound.

[Zachery]

Thanks for the insight on lighttpd and how to do all of this, I've got a big project coming up and am going to be doing some possibly massive scaling.

I'm trying to think how this could be applied on some massive level.