Downtime Today [UPS fault at upstream DC]

MichaelC · Post by **MichaelC** » Sat Dec 17, 2016 7:29 pm

Hi all,

Today we experienced some issues due to an upstream data center losing power to core internet routers.

You can read more here and here.

Apologies for any inconvenience caused.

MichaelC · Post by **MichaelC** » Sat Dec 17, 2016 8:02 pm

I've been informed we're running on emergency backup generators at the moment so we're not necessarily completely out of the woods and stabilized but we're up for now.

Many thanks,
Michael

noth · Post by **noth** » Sat Dec 17, 2016 9:37 pm

thanks for the info Michael. Everytime I went to ACP it said "Failed to verify latest version" S P O O K Y !!

MichaelC · Post by **MichaelC** » Sat Dec 17, 2016 10:49 pm

Hopefully we should remain up for the time being but there may be a small outage tomorrow between 15:00 and 16:00 UTC. For those of you interested in the details:

NERO's [ISP for our DC] Corvallis PoP is presently on generator and should be stable. Electricians will address the temporary fix, possibly tomorrow. We will endeavor to provide advance notice before they attempt to cut back over to normal power. There may be an another outage as part of that process.
Thank you for your patience.

The power should be stable and we don't expect another outage to happen today or the rest of the weekend. When the power went out, everything worked as expected but for some reason the generator was out of phase and the UPS went to battery and then ran out of power. They still aren't sure why that happened but they are confident is it stable for now.

They are planning on fixing the wiring they changed today tomorrow morning between 10am-11am PST. They are planning on doing it without an outage, but there is a risk they could lose power again. The UPS will be on battery while they make the change and have about six minutes to make the change. They did it in three minutes today without an issue.

advertisements