Time to squeeze water from the rock again ... (optimize help)

[Kevlar]

Time to squeeze some power from the rock again... I think the searches are what are causing the problem, but I am unsure. Loads on the low end are around 3.5 while on the upper end there are around 7 going up to 10 occasionally.

Thanks for the help :D

1.
Dedicated Forum PHP and Dedicated DB Servers (images are servered from a 3rd seperate box)

2.
Forum php Box

4x1.6Ghz Xeon 1mb cache
4GB RAM
5x36GB 10k SCSI RAID5
RH9
Apache 2.0.54
Php 4.4.1
mySQL 4.20

DB mySQL Box

4x1.6Ghz Xeon 1mb cache
8GB RAM
5x36GB 10k SCSI RAID5
RH9
Apache 2.0.52
Php 4.4.1
mySQL 4.0.25


3.
Nope...

4.
RPM installed

5.
forum box
14:45:47 up 14 days, 5:59, 1 user, load average: 3.27, 3.41, 3.73
239 processes: 233 sleeping, 5 running, 1 zombie, 0 stopped
CPU0 states: 33.0% user 6.4% system 0.0% nice 0.0% iowait 60.0% idle
CPU1 states: 30.4% user 1.0% system 0.0% nice 0.0% iowait 68.0% idle
CPU2 states: 38.1% user 3.3% system 0.0% nice 0.0% iowait 58.0% idle
CPU3 states: 24.1% user 2.3% system 0.0% nice 0.0% iowait 72.4% idle
CPU4 states: 43.4% user 1.0% system 0.0% nice 0.0% iowait 54.4% idle
CPU5 states: 44.1% user 1.2% system 0.0% nice 0.0% iowait 54.0% idle
CPU6 states: 26.1% user 1.3% system 0.0% nice 0.0% iowait 71.4% idle
CPU7 states: 32.2% user 2.2% system 0.0% nice 0.0% iowait 64.4% idle
Mem: 4010688k av, 2033068k used, 1977620k free, 0k shrd, 250524k buff
352816k active, 463204k inactive
Swap: 8185108k av, 0k used, 8185108k free 565428k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
27050 root 8 0 4132 4132 3996 S 99.9 0.1 17:41 2 httpd
7484 root 13 0 1228 1228 868 R 2.1 0.0 0:00 3 top
3 root 19 19 0 0 0 RWN 0.1 0.0 16:01 0 ksoftirqd_CPU
25728 root 9 0 556 556 476 S 0.1 0.0 3:19 4 syslogd
19702 root 9 0 1492 1492 1356 S 0.1 0.0 0:20 2 sshd
1 root 8 0 476 476 428 S 0.0 0.0 0:24 3 init
2 root 8 0 0 0 0 SW 0.0 0.0 0:00 3 keventd
4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd_CPU
5 root 19 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd_CPU

DB Box
14:42:35 up 77 days, 20:44, 1 user, load average: 0.74, 0.67, 0.60
387 processes: 386 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 7.1% user 9.4% system 0.0% nice 0.0% iowait 83.3% idle
CPU1 states: 0.0% user 0.0% system 0.0% nice 0.0% iowait 100.0% idle
CPU2 states: 4.7% user 28.8% system 0.0% nice 0.0% iowait 66.3% idle
CPU3 states: 0.0% user 0.0% system 0.0% nice 0.0% iowait 100.0% idle
CPU4 states: 1.3% user 1.3% system 0.0% nice 0.0% iowait 97.2% idle
CPU5 states: 0.0% user 0.0% system 0.0% nice 0.0% iowait 100.0% idle
CPU6 states: 4.3% user 22.1% system 0.0% nice 0.0% iowait 73.4% idle
CPU7 states: 0.1% user 0.0% system 0.0% nice 0.0% iowait 99.8% idle
Mem: 4011124k av, 3885344k used, 125780k free, 0k shrd, 15348k buff
1643880k active, 1625148k inactive
Swap: 8128880k av, 1284k used, 8127596k free 3253604k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
17812 root 16 0 1384 1384 868 R 48.0 0.0 0:13 2 top
2817 mysql 9 0 406M 406M 2660 S 30.7 10.3 18363m 0 mysqld-max
1 root 9 0 472 472 424 S 0.0 0.0 1:06 2 init
2 root 8 0 0 0 0 SW 0.0 0.0 0:00 2 keventd
3 root 19 19 0 0 0 SWN 0.0 0.0 15:22 0 ksoftirqd_CPU
4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd_CPU
5 root 19 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd_CPU
6 root 19 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd_CPU
7 root 19 19 0 0 0 SWN 0.0 0.0 0:00 4 ksoftirqd_CPU


6.
DB Server

[root@cny101 root]# cat /etc/my.cnf
[mysqld]
datadir=/home/mysql
tmpdir=/sqltmp
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
max_connections = 800
key_buffer = 96M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 600
connect_timeout = 20
tmp_table_size = 128M
max_allowed_packet = 64M
max_connect_errors = 10
thread_concurrency = 4
query_cache_limit = 2M
query_cache_size = 196M
query_cache_type = 1
query_prealloc_size = 16384
query_alloc_block_size = 16384
server-id=1
ft_min_word_len=3
[mysql.server]
user=mysql
basedir=/var/lib
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192
[mysqldump]
quick
max_allowed_packet=16M
[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[mysql]
no-auto-rehash
[mysqlhotcopy]
interactive-timeout

7.

+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Aborted_clients | 1034 |
| Aborted_connects | 170 |
| Bytes_received | 385214730 |
| Bytes_sent | 2638813191 |
| Com_admin_commands | 0 |
| Com_alter_table | 5 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 50228998 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 2 |
| Com_delete | 1583263 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 2 |
| Com_flush | 96 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 13216328 |
| Com_insert_select | 47521 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 96 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 1395592 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 110858039 |
| Com_set_option | 192 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 1 |
| Com_show_databases | 2 |
| Com_show_fields | 7 |
| Com_show_grants | 0 |
| Com_show_keys | 5 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 2 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 130 |
| Com_show_variables | 791 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 96 |
| Com_update | 45937188 |
| Com_update_multi | 0 |
| Connections | 50080736 |
| Created_tmp_disk_tables | 173692 |
| Created_tmp_tables | 1701490 |
| Created_tmp_files | 22172 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 5061667 |
| Handler_read_first | 24771442 |
| Handler_read_key | 4154564598 |
| Handler_read_next | 3038474322 |
| Handler_read_prev | 194244402 |
| Handler_read_rnd | 1569208093 |
| Handler_read_rnd_next | 2950306189 |
| Handler_rollback | 0 |
| Handler_update | 51334547 |
| Handler_write | 66226716 |
| Key_blocks_used | 93763 |
| Key_read_requests | 1485725310 |
| Key_reads | 257956863 |
| Key_write_requests | 35729767 |
| Key_writes | 29454143 |
| Max_used_connections | 325 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 1155 |
| Open_files | 1042 |
| Open_streams | 0 |
| Opened_tables | 43204 |
| Questions | 421222067 |
| Qcache_queries_in_cache | 2858 |
| Qcache_inserts | 109917599 |
| Qcache_hits | 147873677 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 940440 |
| Qcache_free_memory | 195806120 |
| Qcache_free_blocks | 787 |
| Qcache_total_blocks | 6719 |
| Rpl_status | NULL |
| Select_full_join | 63286 |
| Select_full_range_join | 181 |
| Select_range | 31159977 |
| Select_range_check | 0 |
| Select_scan | 9490529 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 7072 |
| Sort_merge_passes | 11086 |
| Sort_range | 27830697 |
| Sort_rows | 1605914847 |
| Sort_scan | 5048567 |
| Table_locks_immediate | 314761093 |
| Table_locks_waited | 3698406 |
| Threads_cached | 322 |
| Threads_created | 326 |
| Threads_connected | 4 |
| Threads_running | 3 |
| Uptime | 4201363 |
+--------------------------+------------+





8.
Yup

9.
1600-1900 w/ a cookie timeout of 2100

10.
Forum Server
http://forums.bimmerforums.com/phpinfo.php

11.
KeepAlive Off
MaxKeepAliveRequests 1000
KeepAliveTimeout 15
IfModule prefork.c
StartServers 5
MinSpareServers 5
MaxSpareServers 20
MaxClients 256
MaxRequestsPerChild 1000

/IfModule
IfModule worker.c
StartServers 5
MaxClients 256
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 1000

/IfModule
IfModule perchild.c
NumServers 5
StartThreads 5
MinSpareThreads 5
MaxSpareThreads 10
MaxThreadsPerChild 20
MaxRequestsPerChild 1000

/IfModule

12.
3.5.4

13.
Logging has been disabled

[eva2000]

raid 5 isn't optimal at all for forum databases due to slow writes, you're better off with raid 10 with 15k scsi disks on db server

but can you install on mysql server and run mysqlreport and post the output you get http://www.vbulletin.com/forum/showthread.php?t=175177

you list db server as 8GB but there's only 4GB ram showing in top stats ?

[Kevlar]

My mistake... it's only 4GB (it's 8GB of swap) ... here is the mysqlreport (I hope I did it right).

MySQL 4.0.25-Max uptime 51 9:23:13 Mon Feb 27 09:01:05 2006
__ Key __________________________________________________ _______________
Buffer usage 91.57M of 96.00M %Used: 95.38
Write ratio 0.83
Read ratio 0.07
__ Questions __________________________________________________ _________
Total 444.07M 100.01/s
DMS 182.42M 41.08/s %Total: 41.08
QC Hits 155.93M 35.12/s 35.11
Com_ 52.93M 11.92/s 11.92
COM_QUIT 52.78M 11.89/s 11.88
-Unknown 182 0.00/s 0.00
Slow 7.75k 0.00/s 0.00 %DMS: 0.00
DMS 182.42M 41.08/s 41.08
SELECT 116.88M 26.32/s 26.32 64.07
UPDATE 48.41M 10.90/s 10.90 26.54
INSERT 14.00M 3.15/s 3.15 7.68
DELETE 1.66M 0.37/s 0.37 0.91
REPLACE 1.47M 0.33/s 0.33 0.81
Com_ 52.93M 11.92/s 11.92
change_db 52.93M 11.92/s 11.92
show_variab 800 0.00/s 0.00
set_option 193 0.00/s 0.00
__ SELECT and Sort __________________________________________________ ___
Scan 10.00M 2.25/s %SELECT: 8.56
Range 32.84M 7.40/s 28.10
Full join 66.07k 0.01/s 0.06
Range check 0 0.00/s 0.00
Full rng join 183 0.00/s 0.00
Sort scan 5.32M 1.20/s
Sort range 29.33M 6.61/s
Sort mrg pass 11.89k 0.00/s
__ Query Cache __________________________________________________ _______
Memory usage 50.71M of 196.00M %Used: 25.87
Block Fragmnt 20.61%
Hits 155.93M 35.12/s
Inserts 115.88M 26.10/s
Prunes 1 0.00/s
Insrt:Prune 115.88M:1 26.10/s
Hit:Insert 1.35:1
__ Table Locks __________________________________________________ _______
Waited 3.95M 0.89/s %Total: 1.18
Immediate 331.80M 74.73/s
__ Tables __________________________________________________ ____________
Open 1.80k of 1800 %Cache: 100.00
Opened 44.20k 0.01/s
__ Connections __________________________________________________ _______
Max used 325 of 800 %Max: 40.62
Total 52.78M 11.89/s
__ Created Temp __________________________________________________ ______
Disk table 181.84k 0.04/s
Table 1.79M 0.40/s
File 23.78k 0.01/s

[Kevlar]

I don't know if this helps or not, but I noticed that some of the searches that the users are running on the forum are taking anywhere from 30-70 seconds to run and complete which could be causing the disasterous load averages. :(

[eva2000]

You using vB full text search or regular vB search function ?

Upgrade to MySQL 4.0.26. I think upgrading to Linux 2.6.x smp kernel might help too as well as changing to raid 10 configuration with 15k rpm scsi disks.

try this my.cnf

[mysqld]
datadir=/home/mysql
tmpdir=/sqltmp
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
max_connections = 800
key_buffer = 256M
myisam_sort_buffer_size = 128M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 4M
table_cache = 1800
thread_cache_size = 512
wait_timeout = 90
connect_timeout = 10
tmp_table_size = 384M
max_heap_table_size = 384M
max_allowed_packet = 64M
max_connect_errors = 10
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 16M
thread_concurrency = 8
query_cache_limit = 6M
query_cache_size = 96M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 32768
server-id=1
ft_min_word_len=3

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[myisamchk]
ft_min_word_len=3
key_buffer=384M
sort_buffer=384M
read_buffer=256M
write_buffer=256M

[mysql]
no-auto-rehash

[mysqlhotcopy]
interactive-timeout

[Kevlar]

You using vB full text search or regular vB search function ?

I'm using the regular search feature... I have been debating on whether or not to change over to fulltext searching, but I have no reached a conclusion if it would be better or not yet.

I'll have to see what I can do about getting another piece of hardware to do the DB work.

I'll try the my.cnf and let you know.

[Kevlar]

The new my.cnf file seems to be doing well... we are up from an average of 91 queries per second up to about 120 queries per second. So far so good... will have to see how it handles the mid day and afternoon rushes.

[eva2000]

see what happens if you change my.cnf line for

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192

to

[mysqld_safe]
nice = -5
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192

restart mysql and see what happens

[Kevlar]

I made the change to the my.cnf file... I'll report back with any changes I see.

[Kevlar]

Queries per second dropped down to 95... but board seems to be running faster.

Is there anyway for me to find out which queries are the 'slow-queries'? I noticed there are about 500 of them currently.

[eva2000]

in my.cnf under [mysqld] group add

log_slow_queries

restart mysql and then in mysql data directory there's a file with yourhostname-slow.log

then use mysqldumpslow command to analyse them

http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html

FYI, default value for threshold for long query is 10 seconds... which some queries for vB on large data tables would probably exceed 10 seconds

[Kevlar]

If the slow query threshld is 10 seconds... I won't even bother because when I watch the mySQL processes, some of them are running upwards of 30-45 seconds. I'm guessing those are searches gone bad...

[eva2000]

nope depending on size of forum and hardware configuration and size of the tables being queried it can sometimes take 30-45 seconds

faster memory bandwidth/more memory allocated, faster hard drives/drive configurations and faster cpus all help for queries to complete quicker

[Kevlar]

The load on the DB server is less than .3 to .5 under peak load... it is the forum/php server that is suffering as the load on that one is averaging around 5-6 under peak times. Our forum limiter is set at 7, so as you can imagine, our users are frustrated.

Right now our forums is...
Threads: 457,254, Posts: 5,885,599, Members: 71,055

So that probably accounts for most of it, but should I be looking at another DB server or another Forum/PHP type server?

[Kevlar]

Also... do you think I should switch to fulltext search in mySQL instead of using the vB search method? I'm thinking that will help some...

[eva2000]

Yes vB 3.5.x stresses more the web end than mysql needing more memory on web server but you have enough ram just cpu power might now be enough for.

I'd add a 2nd web server on load balancer - ideally dual opteron 265 with 4GB ram and scsi disks for 2nd web server - you can try disabling old web server and using new web server to see how it goes before load balancing as dual opteron 265 (dual core = quad cpus) is way more powerful than your quad xeons!

You can try full text searching to see if it helps... look in 1 million post large vB forum thread some folks say for >5m posts full text search might cause problems unless you only do full text searches on thread titles instead of posts - not sure how to do that

[Kevlar]

Interesting... so I guess all that's left for me is to get new hardware then :( That is an interesting fact about the fulltext search tho, I was reading that it works better for larger forums, but when you get to something along our size it becomes more of a headache than a benefit unless you disable the actual searching of the post so you search titles only.

[eva2000]

Yeah i read the same thing in the large vB forum thread for full text searching of just titles. Have you tried this before ?

[Kevlar]

I have not tried the fulltext search of just titles yet ... I was unable to find any real instructions to guide me in that direction. Without properly configuring it to search just titles... I feared bad things would happen.

[cscgal]

If you go to search.php, right below the textbox to enter a keyword, there is a dropdown menu to Search Entire Posts or Search Titles Only.

Just throw &titleonly=1 into your search.php query string :)

[Kevlar]

Alright... searching for titles only returns irrelevant data (thanks to my members using titles that mean nothing).

[Kevlar]

Well... new hardware is on the way... 2x dual opteron chips to replace the current dual xeon technology in the forum/php server (which of course can't get here fast enough).

After some testing... it turns out that it is the search that is killing the forum/php server, not the DB server. The DB server averages a load of .5 but when people run a search, it spikes the load on the forum/php server sometimes well up over 30-40 (no that's not a typo) and with the forum limiter set at 7, it makes users a little unhappy.

Oh... testing means putting search.php (and a few other files) on another server in my setup and running a search. The main forum/php server continues unaffected, but the server running the search turns into molasses.

[eva2000]

interesting about search.php hmmmm not sure why that would be ? this is an unhacked search.php ? 2x network cards in each web server ?

[Kevlar]

Essentially, I just grabbed the search.php and a few other files from the includes folder and transported them over to the other machine. I was able to run searches which took forever while using the other machine to browse the forum... no hiccups or slowness. Only the machine doing the search was acting slowly. The search.php is the same one that came with vBulletin... unhacked. The load on the DB server was minimal, so I figured it had to be on the web server end. Somebody mentioned to me that apache probably just couldn't handle any more traffic.

Each machine has two network cards... one handling internal network traffic while the others handle external traffic.

[eva2000]

i wonder if the 2nd web server if you used lighttpd instead of apache just for serving search.php would it help ???

[Kevlar]

I'm not sure... I don't know if apache itself is overloaded causing the queueing or if the machine is just physically unable to handle anymore requests.

The second webserver that I used to try this out was a machine running apache that hosts the vB images. It's no a strong machine by any sense of the means, but it did the job. The search was slow on that machine, but most importantly, it didn't stop the main machine from doing other tasks.

I would have to try and see if redirecting all the search queries to the other machine would be possible, but I think that once they get directed to the other machine to search, there would be no way to redirect them back to the main machine (but that's a whole different subject).

[MGSteve]

Just a quickie - have you tried a PHP accellerator? Just stuck eAccellerator onto our box and it reduced the memory load by more than half! 6mb down to around 2Mb. Times that by all the apache threads you've got running and its a hell of a lot of RAM you save!

[Scott MacVicar]

Another possible solution is create a second apache process and run it on port 8080 and strip it of all non required modules apart from PHP.

The front apache server can then have everything removed apart from what it needs like mod_dir, mod_index etc.

Then use mod_proxy to fetch PHP requests from port 8080 and reduce overhead considerably.

[Kevlar]

Just a quickie - have you tried a PHP accellerator? Just stuck eAccellerator onto our box and it reduced the memory load by more than half! 6mb down to around 2Mb. Times that by all the apache threads you've got running and its a hell of a lot of RAM you save!

I'm using eAccelerator on all my boxes... that's how I made it this far. The search however, just seems to tie up everything.

[Kevlar]

Another possible solution is create a second apache process and run it on port 8080 and strip it of all non required modules apart from PHP.

The front apache server can then have everything removed apart from what it needs like mod_dir, mod_index etc.

Then use mod_proxy to fetch PHP requests from port 8080 and reduce overhead considerably.

Ok... seeing as how that sounds way above my skill level, I'm going to have to do some research to give that a try. However, it sounds easier just to toss the search off to another box... but then I'd have the problems of changing URLs when I go from one box to another.

[Scott MacVicar]

mod_proxy could handle that your main web server could fetch the search data from another box.

Are you using Full Text btw?

[Kevlar]

Ok... I'll google mod_proxy and see what I can figure out.

I was going to convert to full_text, but after reading a few threads, I heard that it was not a good idea for forums as large as mine. It worked great for boards with over a 1 million posts, but when you get up to the 6 million post arena like mine it tends to be more headaches than it is worth.

[Scott MacVicar]

*puts on thinking hat*

How big is your database? Could you replicate it to another server and then build the fulltext index on that and perform only searching on this slave server?

mod_proxy can be used to redirect any pages elsewhere.

ProxyRemote http://www.example.com/search.php http://search.example.com/search.php

I think from reading the documentation, I could be wrong and it only matches a partial URL. In that case ProxyRemoteMatch might need to be used.

[Kevlar]

The database is 5.3GB + 7.8GB in attachments on the filesystem + 22MB of custom avatars. I thought about trying to figure out the replication thing, but the database machine is barely being used, it has a load of 0.3 to 0.5 on average. When I put the search on the other machine, the database server still doesn't even break a sweat. It seams that the forum/www server is the one that is causing the issues.

Ok... so if I understand this mod_proxy thing. I could rewrite any URLs that point to www.example.com/search.php (http://www.example.com/search.php) to search.example.com/search.php ... just that easy?

[Scott MacVicar]

mod_proxy is just like mod_rewrite but for remote servers.

Another customer was using it for fetching static web pages from one server and dynamic requests from another to reduce memory usage on the web server.

[Kevlar]

Ah, I understand now. In that case, I just have to find the module to load and then I can start playing with it and see what happens.