email notification while database down

[firewire]

Hi there,

We have had a database crash yesterday. While approx. 150 people have been browsing the forum and me being away from the PC for one or two hours and not able to check mails, the system accumulated approx. 3000 error mails "database down" for me.

PLEASE! It's really bad when the database goes down and visitors get to see error messages. It's even worse to get my mailbox spammed and finally blocked with thousands of error notification emails.

There definitely is a need for a better handling of these error mails. Introduce a locking so that error messages are sent only every 5 minutes or whatever, but this behaviour is unacceptable for any large-scale vBulletin with several hundred or even thousand pageviews (= notification mails) in an hour.

Thanks for your feedback and consideration.
Bjoern

[Conscience]

firewire....I know the feeling!! last time it happened to me I just stopped it from mailing me for errors full stop. I just hope someone icq's me now if the database crashes

[ABLady]

Originally posted by firewire
Hi there,

PLEASE! It's really bad when the database goes down and visitors get to see error messages. It's even worse to get my mailbox spammed and finally blocked with thousands of error notification emails.

I agree . . .

I wonder if in vbulletin.org someone has written a hack that somehow stops the system from sending emails if the same error has occured once before in a give time.

[eva2000]

you can always set up a web based viewing email account SPECIFICALLY for the vB contact in config.php or do what i do... every hour on the hour i'm checking my forum and server performance taking notes and stuff (how else do i fine tune my server :) )

[firewire]

Sorry eva2000, I disagree, that's simply not enough.
Email notification should be arriving fast, and not some web-based account which I check every once in an hour or so (and then deleting a couple of thousands of mails?).

Normally I would even let the notification email be forwarded as SMS to my mobile phone. But with this whole flood of mails (and a cost of 15 Cent per SMS)...

I have though disabled Vbulletin's email notification (by putting no email in config.php) and watch my servers with webmin, but this can't be the way (disabling vBulletin features in favour of external solutions because the vB ones are simply no good).

Regards
Bjoern

[rylin]

the way we've fixed it on our forums is:
check if a file (/tmp/dbError) exists.

if it does exist, check if it's been there for more than an hour:
if it has, touch it & send a email

if it hasn't:
don't do anything

if the file doesn't exist:
create it, fill it with the error message & send a email

this way, i get at most one email per hour if the db messes up

[DVD Plaza]

Glad I'm not the only who has had to live through this - last time I had a database problem I received just over 800 e-mails!!!

[Scott MacVicar]

okidoki i done the same thing as you but i put it in my admin directory instead :P

[ABLady]

Can you guys share the hack with us?

[rylin]

Originally posted by ABLady
Can you guys share the hack with us?

just make the if ($technicalemail) { section in db_mysql.php look something like

if ($technicalemail) {

$sendEmail = false;
if(file_exists('/tmp/dbError')) {
if(filectime('/tmp/dbError') < (time() - 3600)) {
$sendEmail = true;
touch('/tmp/dbError');
}
} else {
$sendEmail = true;
}

if($sendEmail) {
$fh = @fopen('/tmp/dbError','w');
@fwrite($fh, $message);
@fclose($fh);

@mail ($technicalemail,"$this->appshortname Database error!",$message,"From: $technicalemail");
}
}


Note, our vB is hacked to bits & pieces (eg. using our web libs etc for a lot of stuff), so this code isn't tested at all (just re-wrote a old copy of db_mysql.php i had on my drive with stuff that should work

try it out, it should work
( and yes, i know this should be on vBulletin.org.. but someone else gets to post it, i'm lazy ;) )

[ABLady]

Originally posted by okidoki


Note, our vB is hacked to bits & pieces (eg. using our web libs etc for a lot of stuff), so this code isn't tested at all (just re-wrote a old copy of db_mysql.php i had on my drive with stuff that should work

try it out, it should work
( and yes, i know this should be on vBulletin.org.. but someone else gets to post it, i'm lazy ;) )

Thanks okidiki!

I just put the code on my server and tested it. It works just perfect.

[DWZ]

Originally posted by okidoki
( and yes, i know this should be on vBulletin.org.. but someone else gets to post it, i'm lazy ;) ) oohh... i never posted a hack on vb.org... <3 year old voice> Can I, can I?!? (please :)) </3 year old voice>

[Chen]

DWZ, please shorten your signature to follow the rules (http://vbulletin.com/forum/showthread.php?s=&threadid=42918). (or make the text's size=1)

[DWZ]

Originally posted by FireFly
DWZ, please shorten your signature to follow the rules (http://vbulletin.com/forum/showthread.php?s=&threadid=42918). (or make the text's size=1) oh i guess i can do that :)

[JDD]

Thx a really nice idea.. i'll try it out ASAP...

Last time our database went down, I reseived several hundreds of email to my gmx email account.. and what happened was that GMX put the Boards Emailserver onto their spam list - as a result a didn't get any Boardemails anylonger :( That about using webbased email accounts....


So long
JDD