Mysql was running fine and has suddenly went out of control!

[alexi]

Our forums have been running fine for several weeks on servers. 2 days ago loads on the database server began to climb. We have tried database repairs, I have changed my.cnf around and nothing seems to have helped, it is just gradually getting worse. If we restart mysql it is good for several hours.
I grabbed an extended status script before a mysql restart:

+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Aborted_clients | 54 |
| Aborted_connects | 149 |
| Bytes_received | 327401873 |
| Bytes_sent | 3412284266 |
| Com_admin_commands | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 95651 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 13980 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 2 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 49072 |
| Com_insert_select | 0 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 10 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 557263 |
| Com_set_option | 1 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 0 |
| Com_show_databases | 0 |
| Com_show_fields | 0 |
| Com_show_grants | 0 |
| Com_show_keys | 0 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 21 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 1 |
| Com_show_variables | 1 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 0 |
| Com_update | 153223 |
| Connections | 96006 |
| Created_tmp_disk_tables | 414 |
| Created_tmp_tables | 38641 |
| Created_tmp_files | 2 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 3 |
| Handler_commit | 0 |
| Handler_delete | 16840 |
| Handler_read_first | 16877 |
| Handler_read_key | 2985241036 |
| Handler_read_next | 49206604 |
| Handler_read_prev | 1084442 |
| Handler_read_rnd | 5332476 |
| Handler_read_rnd_next | 788351399 |
| Handler_rollback | 0 |
| Handler_update | 2781952575 |
| Handler_write | 198493390 |
| Key_blocks_used | 15586 |
| Key_read_requests | 169692747 |
| Key_reads | 714777 |
| Key_write_requests | 569135 |
| Key_writes | 63380 |
| Max_used_connections | 450 |
| Not_flushed_key_blocks | 1496 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 1024 |
| Open_files | 1224 |
| Open_streams | 0 |
| Opened_tables | 48467 |
| Questions | 1837503 |
| Qcache_queries_in_cache | 5435 |
| Qcache_inserts | 506291 |
| Qcache_hits | 872939 |
| Qcache_lowmem_prunes | 7329 |
| Qcache_not_cached | 50707 |
| Qcache_free_memory | 22353112 |
| Qcache_free_blocks | 1659 |
| Qcache_total_blocks | 12673 |
| Rpl_status | NULL |
| Select_full_join | 379 |
| Select_full_range_join | 3 |
| Select_range | 117333 |
| Select_range_check | 0 |
| Select_scan | 63110 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 43 |
| Slow_queries | 11497 |
| Sort_merge_passes | 1 |
| Sort_range | 96534 |
| Sort_rows | 5998550 |
| Sort_scan | 53341 |
| Table_locks_immediate | 874888 |
| Table_locks_waited | 228961 |
| Threads_cached | 0 |
| Threads_created | 3299 |
| Threads_connected | 451 |
| Threads_running | 406 |
| Uptime | 19175 |
+--------------------------+------------+

Earlier today I was looking at loads in the 100's

[alexi]

We believe we have found the issue. If we are correct it was due to a small ratings program that also utilized the MySql server.

[eva2000]

hey Alex, seems qcache low memory prunes also suggest you're out of memory in the query cache too

how's memory usage for the database server ? can you afford bumping up the key_buffer value to say from 16M to 64M and add another 12M to the query cache memory allocation ?