Well that was fun... (About downtime)

[Mike Sullivan]

I got a call from Kier who was out, saying that ThePlanet said our main load balancer server was down.

Sure enough, I couldn't SSH to it from the external NIC, but I tried one of the backend servers and was able to get in via the internal NIC.

Logged in to Orbit (ThePlanet's support system) and saw 2 tickets. One about the downtime, and one saying that they noticed malicious traffic and enabled FloodGuard, DDoS protection.

Well, this is around the time when everything went pear-shaped. One of their techs said:

Your server was being hit with a 1.5gbps inbound attack.

(Emphasis mine.)
Err... wow. Though apparently it stopped so they disabled FloodGuard.

When I was in via the internal NIC, I was able to run ethtool to see that there was no link detected on the external NIC. I spent the last few hours trying to get ThePlanet to look at the switch we're connected to.

And now it's finally up. But just before it came up, they said this in a ticket:

We are still investigating this issue at this point we cannot determine whether it is a bad switch port, or bad nic.
We are taking the server down to install an extra nic to see if this can fix the issue.
-Support

Fun... Let's hope they don't take the server down.

[Shazz™]

Wow... Busy morning

[Yours Truly]

Didn't realise vBulletin.org was on the same servers as these =D

[Ohiosweetheart]

well, ouch

[Paul M]

Wow... Busy morning

Except it was late afternoon and then evening over here

Didn't realise vBulletin.org was on the same servers as these =D

Yep, has been for years.

[Yours Truly]

Except it was late afternoon and then evening over here

Yep, has been for years.

I thought it was a different part of vBulletin all together =]

[Reeve of Shinra]

DDOS'd? Man that sucks

[Shazz™]

Except it was late afternoon and then evening over here

Yep, has been for years.

It was busy for me too, I kept refreshing vb.org 1000 times

[Ohiosweetheart]

It was busy for me too, I kept refreshing vb.org 1000 times

LOL!!

[---MAD---]

That was a very annoying and LONG down time :P.

[Floris]

From 5 to 9 .. four hours. It felt even longer *my precious*

[---MAD---]

From 5 to 9 .. four hours. It felt even longer *my precious*

Agreed .

[Floris]

I've received various concerns and comments from people if this was in relation with the latest IB news: No, it was not. It has nothing to do with IB, and we're not moving servers to a different provider or anything like that. I hope this answers those questions.

[Onimua]

Well, at least we're back up and running here.

[ChipTz]

need a new F5 key here