PDA

View Full Version : Server Optimization Request


Surfer
Tue 17th Oct '06, 6:13am
1. One dedicated web server and one dedicated MySQL server.

2.

Webb:
Dual Xeon 3.0 GHz
4GB RAM
SCSI 4x18GB RAID 1+0
Linux: 2.6.16-gentoo-r7
Apache: 2.0.55-r1
PHP version: 4.4.2-pl2

SQL:
Dual Xeon 3.4 GHz, 800MHz FSB
8GB RAM
SCSI 15K 4x36GB RAID 1+0
MySQL version: 4.0.27

3. we use skip-innodb

4. with berkdb perl ssl latin1

5.

www:
top - 10:36:50 up 146 days, 21:41, 4 users, load average: 4.09, 4.04, 4.00
Tasks: 110 total, 3 running, 107 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.9% us, 1.7% sy, 0.0% ni, 74.2% id, 0.0% wa, 0.1% hi, 1.1% si
Mem: 3895408k total, 2525448k used, 1369960k free, 346620k buffers
Swap: 1959920k total, 132k used, 1959788k free, 1449720k cached

SQL:
top - 11:12:53 up 1 day, 18:48, 3 users, load average: 1.52, 1.72, 1.86
Tasks: 68 total, 2 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 33.6% us, 11.3% sy, 0.0% ni, 53.5% id, 0.0% wa, 0.0% hi, 1.7% si
Mem: 7791656k total, 7669028k used, 122628k free, 195224k buffers
Swap: 8792392k total, 196k used, 8792196k free, 5168820k cached

6.

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[safe_mysqld]
err-log = /var/log/mysql/mysql.err

[mysqld]
skip-innodb
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
log-error = /var/log/mysql/mysqld.err
# log-bin
# set-variable = binlog-do-db=non_existant
# set-variable = binlog-ignore-db=database_name
server-id = 207
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-locking
max_connections=4000
query_cache_size=500000000
set-variable = key_buffer=4200M
set-variable = max_allowed_packet=2M
set-variable = thread_stack=256K
#set-variable = thread_stack=512K

set-variable = max_tmp_tables=512
#set-variable = open_files_limit=50000
set-variable = open_files_limit=50000
#set-variable = table_cache=15000
set-variable = table_cache=40000
set-variable = thread_cache_size=700
set-variable = tmp_table_size=400M
set-variable = sort_buffer_size=400M
set-variable = low_priority_updates=On
# keep secure by default!
#bind-address = 127.0.0.1
port = 3306

[mysqldump]
quick
set-variable = max_allowed_packet=16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completion

[isamchk]
set-variable = key_buffer=4200M
7.


+---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| back_log | 50 |
| basedir | /usr/ |
| bdb_cache_size | 8388600 |
| bdb_home | /var/lib/mysql/ |
| bdb_log_buffer_size | 20480000 |
| bdb_logdir | |
| bdb_max_lock | 10000 |
| bdb_shared_data | OFF |
| bdb_tmpdir | /tmp/ |
| bdb_version | Sleepycat Software: Berkeley DB 3.2.9a: (May 6, 2006) |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set | latin1 |
| character_sets | latin1 big5 cp1251 cp1257 croat czech danish dec8 dos estonia euc_kr gb2312 gbk german1 greek hebrew hp8 hungarian koi8_ru koi8_ukr latin1_de latin2 latin5 sjis swe7 tis620 ujis usa7 win1250 win1251ukr win1251 |
| concurrent_insert | ON |
| connect_timeout | 5 |
| convert_character_set | |
| datadir | /var/lib/mysql/ |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 254 |
| ft_max_word_len_for_sort | 20 |
| ft_min_word_len | 4 |
| ft_stopword_file | (built-in) |
| have_bdb | YES |
| have_crypt | YES |
| have_innodb | DISABLED |
| have_isam | YES |
| have_openssl | YES |
| have_query_cache | YES |
| have_raid | NO |
| have_symlink | YES |
| init_file | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_size | 8388608 |
| innodb_data_file_path | |
| innodb_data_home_dir | |
| innodb_fast_shutdown | ON |
| innodb_file_io_threads | 4 |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 4294963200 |
| language | /usr/share/mysql/english/ |
| large_files_support | ON |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | OFF |
| log_error | /var/log/mysql/mysqld.err |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_update | OFF |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | ON |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 2096128 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 1073741824 |
| max_connect_errors | 10 |
| max_connections | 4000 |
| max_delayed_threads | 20 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_tmp_tables | 512 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| myisam_max_extra_sort_file_size | 268435456 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| open_files_limit | 84010 |
| pid_file | /var/run/mysqld/mysqld.pid |
| port | 3306 |
| protocol_version | 10 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_size | 499999744 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 131072 |
| read_only | OFF |
| read_rnd_buffer_size | 262144 |
| rpl_recovery_rank | 0 |
| server_id | 207 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_net_timeout | 3600 |
| slow_launch_time | 2 |
| socket | /var/run/mysqld/mysqld.sock |
| sort_buffer_size | 419430392 |
| sql_mode | 0 |
| table_cache | 40000 |
| table_type | MYISAM |
| thread_cache_size | 700 |
| thread_stack | 262144 |
| timezone | CEST |
| tmp_table_size | 419430400 |
| tmpdir | /tmp/ |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| version | 4.0.27 |
| version_comment | Gentoo Linux mysql-4.0.27 |
| version_compile_os | pc-linux-gnu |
| wait_timeout | 28800 |
+---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+--------------------------------+------------+
| Variable_name | Value |
+--------------------------------+------------+
| Aborted_clients | 7787 |
| Aborted_connects | 1 |
| Bytes_received | 1527415895 |
| Bytes_sent | 967174276 |
| Com_admin_commands | 567813 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 577191 |
| Com_change_master | 0 |
| Com_check | 314 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 9901 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 11614 |
| Com_insert_select | 12 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 2 |
| Com_replace | 83028 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 2163035 |
| Com_set_option | 317 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 17 |
| Com_show_create | 306 |
| Com_show_databases | 14 |
| Com_show_fields | 315 |
| Com_show_grants | 12 |
| Com_show_innodb_status | 2687 |
| Com_show_keys | 0 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 450 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 3132 |
| Com_show_tables | 358 |
| Com_show_variables | 54 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 0 |
| Com_update | 562272 |
| Com_update_multi | 0 |
| Connections | 9207 |
| Created_tmp_disk_tables | 157 |
| Created_tmp_files | 0 |
| Created_tmp_tables | 66962 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 19821 |
| Handler_read_first | 176937 |
| Handler_read_key | 89889089 |
| Handler_read_next | 853447256 |
| Handler_read_prev | 55682 |
| Handler_read_rnd | 8699650 |
| Handler_read_rnd_next | 309751588 |
| Handler_rollback | 0 |
| Handler_update | 328526 |
| Handler_write | 1453622 |
| Key_blocks_used | 1365337 |
| Key_read_requests | 549646559 |
| Key_reads | 1357092 |
| Key_write_requests | 411345 |
| Key_writes | 263717 |
| Max_used_connections | 213 |
| Not_flushed_delayed_rows | 0 |
| Not_flushed_key_blocks | 0 |
| Open_files | 824 |
| Open_streams | 0 |
| Open_tables | 521 |
| Opened_tables | 531 |
| Qcache_free_blocks | 172750 |
| Qcache_free_memory | 315975064 |
| Qcache_hits | 2126111 |
| Qcache_inserts | 2111036 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 51493 |
| Qcache_queries_in_cache | 171468 |
| Qcache_total_blocks | 516055 |
| Questions | 5686404 |
| Rpl_status | NULL |
| Select_full_join | 44 |
| Select_full_range_join | 0 |
| Select_range | 158859 |
| Select_range_check | 0 |
| Select_scan | 234447 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 22 |
| Sort_merge_passes | 0 |
| Sort_range | 459775 |
| Sort_rows | 8635584 |
| Sort_scan | 92404 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | NONE |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 0 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Table_locks_immediate | 5785461 |
| Table_locks_waited | 39645 |
| Threads_cached | 115 |
| Threads_connected | 99 |
| Threads_created | 214 |
| Threads_running | 2 |
| Uptime | 45121 |
+--------------------------------+------------+
Uptime: 45121 Threads: 99 Questions: 5686440 Slow queries: 22 Opens: 531 Flush tables: 1 Open tables: 521 Queries per second avg: 126.026
8. Only one site used on these two machines. Site also contains CMS solutions and hacks made to vbulletin. Traffic is mostly centered around vb.

9.
Avg Users: ~1100
Daily Peak: ~3000
Cookie: 1800 secs

10. Available by request

11.
KeepAlive Off
ServerLimit 800
StartServers 20
MinSpareServers 30
MaxSpareServers 50
MaxClients 700
MaxRequestsPerChild 400
memcached settings: MEMUSAGE="1000" , MAXCONN="3000"

12. 3.5.3 with several hacks

13. None, post table contains 4,065,627 posts and is 2.4GB large

14. Available on request

Appreciate any advice :)

Surfer
Thu 19th Oct '06, 5:26am
Hello again, really need some help with this one.
Our server config is eating memory like crazy. After upgrading the memory to 8GB it took 48hours before we start getting out of memory error messages from MySQL again.

Really appreciate any sugestions!

thanks!

eva2000
Thu 19th Oct '06, 10:54am
Out of memory on mysql is due to you setting memory buffers waaaay too high in my.cnf i.e. 4.2GB key_buffer is way too high!

try following

1. Upgrade Apache to 2.0.59
2. Upgrade PHP to 4.4.4 and/or recompile PHP 4.4.4 on web server after upgrading MySQL (see further below)
3. Edit /etc/my.cnf and place the following mysql server settings in /etc/my.cnf and restart mysql server afterwards


[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
server-id = 207
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-name-resolve
safe-show-database
skip-innodb
max_connections = 850
key_buffer = 1024M
myisam_sort_buffer_size = 256M
join_buffer_size = 3M
read_buffer_size = 3M
sort_buffer_size = 8M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 35
connect_timeout = 10
tmp_table_size = 1024M
max_heap_table_size = 1024M
max_allowed_packet = 64M
max_connect_errors = 1000000
read_rnd_buffer_size = 1M
bulk_insert_buffer_size = 16M
query_cache_limit = 12M
query_cache_size = 256M
query_cache_type = 1
query_prealloc_size = 163840
query_alloc_block_size = 32768

[mysqld_safe]
err-log = /var/log/mysql/mysql.err
nice = -10
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

[mysqlhotcopy]
interactive-timeout


4. On web server, install APC Cache lastest version from http://pecl.php.net/package/APC. Read install guide at http://www.vbulletin.com/forum/showthread.php?t=165367 - please remove Zend Optimizer from php.ini before installing APC Cache

5. Upgrade to vB 3.5.5 http://www.vbulletin.com/forum/showthread.php?t=194061 if you're on vB 3.5.x.

6. If you just upgraded to vB 3.5.x/3.6.x try to disable these 4 options:

Admin CP -> vBulletin Options -> Forums Home Page Options -> Display Logged in Users?

Admin CP -> vBulletin Options -> Forum Display Options (forumdisplay) -> Show Users Browsing Forums

Admin CP -> vBulletin Options -> Thread Display Options -> Show Users Browsing Thread

Admin CP -> vBulletin Options -> Message Searching Options -> Automatic Similar Thread search

7. Edit httpd.conf values from

KeepAlive Off
ServerLimit 800
StartServers 20
MinSpareServers 30
MaxSpareServers 50
MaxClients 700
MaxRequestsPerChild 400

to

KeepAlive Off
ServerLimit 500
StartServers 20
MinSpareServers 20
MaxSpareServers 25
MaxClients 256
MaxRequestsPerChild 500

restart apache

8. You may need to install same mysql version on web server, and recompile PHP with the --with-mysql=/path/to/mysql option to tell PHP to use the latest mysql 4.0.x client libraries i.e. if mysqldump/mysqlcheck/mysqladmin are located at /usr/bin/ or /usr/local/mysql/bin then you need to compile PHP with --with-mysql=/usr or --with-mysql=/usr/local/mysql - just had a few issues with one vB client have issues with PHP connecting to external mysql server until i did this.

9. If you have split web + db servers, ensure web server has dual network cards as outlined at http://www.vbulletin.org/forum/showthread.php?t=111191

Surfer
Mon 23rd Oct '06, 8:59am
Hello,

Thank you for your reply. Your settings helped stabilize our server.
One question. We have eaccelerator and memcache installed. Are these used in vb 3.5.x?

Thanks!

eva2000
Tue 24th Oct '06, 12:25am
Glad it could help :)

eaccelerator is utilised for any PHP script but memcache doesn't need to be.