Urgent Help Needed

[dvdtalk]

Our site is at a crawl now.....

I sent a message last night to VB support but haven't heard back.....

My post:
http://www.vbulletin.com/forum/showthread.php?s=&threadid=7952

Isn't getting any responses.

Our loads are now in the 30's and the site hardly loads.

Geoff
http://www.dvdtalk.com

[dvdtalk]

Our loads just peaked at 98! SOS

[DVD Plaza]

Geoff I just tried to pop in and visit DVD Talk, but found that even your front page took forever to come up. Whatever is going on is system wide, it's certainly not just one area of your system crawling.

Have you done all the various optimisations to your mySQL settings? eva2000 has become quite an expert at this, his forums are hit pretty hard :)

[Martin]

um

I just went there with 348 people online and it's not exactly slow.

what kind of hardware are you running?

[dvdtalk]

we just did a reboot and the load came down... STRANGE.

We are on a dual PII 650MHZ with 512MB Ram and 9GIG HD.

I can't explain it... the load peaked at 110!

[Martin]

you might have had a stuck process, or did you make changes and then forget to restart mySQL?

[dvdtalk]

Nope we've been restarting MySQL with every change...

But we're now banged up again.

load is north of 75

[Shaman]

Brutal.

I may be able to help. I don't know, I'd have to see what is going on.

I got your e-mail, can you give me more information?

[dvdtalk]

Here's our most current MySQL Stats:



Thu Feb 8 21:47:48 EST 2001
+--------------------------+-----------+
| Variable_name | Value |
+--------------------------+-----------+
| Aborted_clients | 282 |
| Aborted_connects | 0 |
| Bytes_received | 12838866 |
| Bytes_sent | 129585773 |
| Connections | 450 |
| Created_tmp_disk_tables | 0 |
| Created_tmp_tables | 3176 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_delete | 357 |
| Handler_read_first | 854 |
| Handler_read_key | 7433361 |
| Handler_read_next | 16727490 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 381075 |
| Handler_read_rnd_next | 20547907 |
| Handler_update | 5791 |
| Handler_write | 78596 |
| Key_blocks_used | 1525 |
| Key_read_requests | 8800477 |
| Key_reads | 1524 |
| Key_write_requests | 7052 |
| Key_writes | 9367 |
| Max_used_connections | 171 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 174 |
| Open_files | 189 |
| Open_streams | 0 |
| Opened_tables | 180 |
| Questions | 71525 |
| Select_full_join | 3446 |
| Select_full_range_join | 0 |
| Select_range | 1764 |
| Select_range_check | 0 |
| Select_scan | 25189 |
| Slave_running | OFF |
| Slave_open_temp_tables | 0 |
| Slow_launch_threads | 9 |
| Slow_queries | 5 |
| Sort_merge_passes | 0 |
| Sort_range | 3036 |
| Sort_rows | 931009 |
| Sort_scan | 9958 |
| Threads_cached | 0 |
| Threads_created | 449 |
| Threads_connected | 167 |
| Threads_running | 4 |
| Uptime | 1156 |
+--------------------------+-----------+

Key Reads/Key Read Requests = 0.000173 (Cache hit = 99.999827%)
Key Writes/Key Write Requests = 1.328276
Temporary Tables Created/second = 2.747 (/hour = 9890.657)
Slow Queries/second = 0.004 (/hour = 15.571)
% of slow queries = 0.007%
Queries/second = 61.873 (/hour = 222742.215)
Uptime = 19 mins 16 secs


Our PHP Info: http://www.dvdtalk.com/forum/phpinfo.php

our my.cnf



# This will be passed to all mysql clients
[client]
#password = my_password
port = 3306
socket = /tmp/mysql.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
set-variable = max_connections=250
set-variable = join_buffer=8M
set-variable = key_buffer=256M
set-variable = record_buffer=8M
set-variable = sort_buffer=8M
set-variable = max_allowed_packet=1M
set-variable = thread_stack=128K
set-variable = table_cache=256

# Start logging
# log

[mysqldump]
quick
set-variable = max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
set-variable = key_buffer=16M


Any other info helpful?

Geoff

[Shaman]

Yep.

vmstat 2

and

iostat 2

Also, a capture of:

top

[dvdtalk]

May not be able to get e-mail now.. server just went belly up... trying to get it back

[Martin]

what OS are you running?

[dvdtalk]

Linux 6.0
Kernal 2216

[Shaman]

I'm pretty sure I can cure it, but I can't do it for free. I have a racing habit to feed. :)

I know two of your problems right now.

One of them that I don't know is why in the top you sent me that httpd is working so hard (15% of CPU). That makes me believe that you're running something other than vB as well.

[dvdtalk]

Can anyone offer help that isn't going to charge by the hour?

[Shaman]

Hmm. When you say that, it makes me glad I didn't assist. Best of luck getting skilled help for nothing.

I'll give you another hint just so you don't think I'm being a jerk: whatever it is, it isn't vb and it isn't MySQL.

[DVD Plaza]

Maybe because people come to VBulletin to help one another out, from configuration to experiences to hacks to problems... I'm not only surprised someone would post here asking for money to help out, not only surprised that you are not willing to spend a whol couple of minutes explaining what it is you claim Geoff's problem is, but that you then decide to post a smart alec responce...

Jelsoft actually allows this kind of activity in their forums?

[werehere]

Originally posted by DVD Plaza
Jelsoft actually allows this kind of activity in their forums?

Well I don't speak for Jelsoft first off!

I know Shaman has helped many people out of a jam here, and is quite knowledgable with servers! It is up to him to judge whether or not he can help for free with a difficult (possibly quite time consuming) reconfiguration of this server!

Sometimes difficult time consuming things can cost money to fix, sometimes not. I hope another person will come along and help you out of your jam for free, but try not to be to down on someone because they may have to charge money to do it!

I am sorry I cannot fix it for free, but I don't have the knowledge to do it! :)

[dvdtalk]

I'm not down on Shaman and I DO realize that somethings you really do need to hire someone to fix (You think I installed PHP and MySQL myself?!?!)

I guess I was a little surprised to see the post... yeah I can help you if you pay me.....

It's one thing to contribute to a forum and say something like - Try X...Y.....Z and then say... hey I'm a professional Admin so if you don't want to do it yourslef I'd be more than willing to do it at a cut rate.

It's another to say, YEAH I know what's up, pay me and I'll fix it.

I've been running a community forum for the last 2 years and I've see a lot of people help echother out. I mean isn't that what we're here for. Don't you think when I make it through this hell I'll be there for the next guy!?!

What I am surprised at is that I've been having problems now for almost 24 hours, and still no one from the forum or from Jelsoft has jumped in to help.

If I could get the load under control, I really do LOVE VB v UBB, but if I can't get any help or support I'll have no choice but to revert back to our last working forum which was..... UBB 5.74d.

I really hope I can get some help and get this stable and be able to look back at this mess with perspective.

Geoff

[Shaman]

It's easy enough to tell you why. It's not a MySQL configuration problem, and it's not a vBulletin problem, so you need genuine trained help to solve it.

[David Copeland]

I will have to agree on both sides. Free help is great,
provided it is small-change to offer someone in a pinch.

But me too . . . we have had massive outages too in
the last year, mostly to do with a little tweaking. And
some good people here have helped out for free.

But in the end, if I could not get it for free, I would
have pay too. Why not? The only question is that
I don't like to pay for others to learn on my time.
Most pros at this software are going to be in and out
in a short period of time. If a board is making the
owner a profit (which it appears to be), then why not
pay for it? I just get the sense that DVDtalk could
not find ANY help, whether free or paid, to get him
on his way.

So if it's free, wonderful. If it's more than a helping
hand and becomes an over-haul, then pay and be
glad it's done.

David

PS: I went to www.dvdplaza.com.au/cgi-bin/Ultimate.cgi and was
disappointed NOT to see VBulletin. If you are not
running the program, and you're just a lurker looking
down the road for an upgrade to VBulletin, I think
it's fair to say so before claiming what is fair and not fair.
(That's only my opinion, though) ;)

[DVD Plaza]

Unfortunately I've hacked my UBB to buggery, it barely resembles UBB anymore, to make it work far more efficiently as well as to add sh#tloads of features we depend on behind the scenes (as well as for members) day to day.

I've been following the development of VBulletin since early last year when I first saw John talking about it on an ezboard, as it looked like the exact product I had thought of developing inhouse for myself (mySQL based). That said, after the extreme level of hacking I did to UBB I'm hesitant to ever do "much" hacking again - as it became impossible to upgrade, as my code barely resembles UBB anymore.

To that end I've been waiting all this time for VBulletin 2, as I'm extremely excited to see that if offers EVERYTHING I need! Thus I'll be able to install the stock product and configure it (software and templates) to suit our needs.

Unfortunately our UBB is so far beyond coping that half our administration functions no longer work (for example moving threads causes corruption, loss of data, etc). VBulletin2 too just a tad too long, and I've now had to put building a new server as a higher priority. I'm doing that as we speak, still waiting on some of the parts though, and once everything is running sweet for a while (and by then VBulletin 2 would no doubt be out of beta) I'm eager to make the switch. It just took a few weeks too long, it was that close, and we reached a point where the system just wasn't working anymore. My priority HAD to change and VBulletin has had to go on the back burner now.

I have, however, continued to watch here throughout this time and I've seen everyone helping each other out. I don't think it's unfair to say that this case turned out somewhat differently.

Certainly makes me suddenly think about the consequences of purchasing software that clearly has load problems, different load problems to that of UBB, yet it would appear a customer had trouble trying to get help in getting it to run properly.

[David Copeland]

What type of server do you think would be a
minimum to have to support the new version
of VB, which I believe will require less connections
and resources?

David

[DVD Plaza]

Nobody can seem to give a definite answer, as so many factors seem to affect everyone differently.

To that end I'm preparing a dual P3 1Ghz, 1GB RAM, 120GB HD, RAID 1, etc

[eva2000]

Originally posted by DVD Plaza
Nobody can seem to give a definite answer, as so many factors seem to affect everyone differently.

To that end I'm preparing a dual P3 1Ghz, 1GB RAM, 120GB HD, RAID 1, etc chris you're building that yourself ?

[DVD Plaza]

Most definitely doing it myself, I want to know this baby is built perfectly - not just software wise, but hardware too (you should see some of the juicy cooling I've bought for this baby!!!), have purchased most of the parts already just sussing out the RAM right now.

Normally I go for Kingmax, but apparently Crucial is better and I've just now found I can order it direct from the US and it'll cost me about half the price (though I'll have to order each 256MB stick separately to avoid GST charges when it arrives into Australia) than Kingmax. Am just doing some research right now to ensure that Crucial is as good as everyone says, and then I'll submit the order.

[Freddie Bingham]

Crucial is as good as you hear it is.

[DVD Plaza]

Thanks freddie, I ordered it as soon as I read your post! :)

I'll start testing VBulletin once I've been happy with the server being settled for about a month. Too much riding on her to make any changes right now.

[David Copeland]

How much would a server like that cost in parts?

(Canadian or US Dollars?)

David