Downtime Today [UPS fault at upstream DC]

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
Post Reply
User avatar
MichaelC
Consultant
Consultant
Posts: 3642
Joined: Mon Dec 21, 2009 3:36 pm
Location: Surrey, UK
Name: Michael Cullum
Contact:

Downtime Today [UPS fault at upstream DC]

Post by MichaelC » Sat Dec 17, 2016 7:29 pm

Hi all,

Today we experienced some issues due to an upstream data center losing power to core internet routers.

You can read more here and here.

Apologies for any inconvenience caused.
:)
Formerly known as Unknown Bliss.
Formerly Website Team Lead/Manager & Development Team.
Please don't PM me for support (or stuff that belongs in the forums or tracker) but otherwise feel free

User avatar
MichaelC
Consultant
Consultant
Posts: 3642
Joined: Mon Dec 21, 2009 3:36 pm
Location: Surrey, UK
Name: Michael Cullum
Contact:

Re: Downtime Today [UPS fault at upstream DC]

Post by MichaelC » Sat Dec 17, 2016 8:02 pm

I've been informed we're running on emergency backup generators at the moment so we're not necessarily completely out of the woods and stabilized but we're up for now.

Many thanks,
Michael
:)
Formerly known as Unknown Bliss.
Formerly Website Team Lead/Manager & Development Team.
Please don't PM me for support (or stuff that belongs in the forums or tracker) but otherwise feel free

User avatar
noth
Registered User
Posts: 2481
Joined: Fri Jan 07, 2005 7:10 pm
Location: North Surrey
Contact:

Re: Downtime Today [UPS fault at upstream DC]

Post by noth » Sat Dec 17, 2016 9:37 pm

thanks for the info Michael. Everytime I went to ACP it said "Failed to verify latest version" S P O O K Y !! :lol:

User avatar
MichaelC
Consultant
Consultant
Posts: 3642
Joined: Mon Dec 21, 2009 3:36 pm
Location: Surrey, UK
Name: Michael Cullum
Contact:

Re: Downtime Today [UPS fault at upstream DC]

Post by MichaelC » Sat Dec 17, 2016 10:49 pm

Hopefully we should remain up for the time being but there may be a small outage tomorrow between 15:00 and 16:00 UTC. For those of you interested in the details:
NERO's [ISP for our DC] Corvallis PoP is presently on generator and should be stable. Electricians will address the temporary fix, possibly tomorrow. We will endeavor to provide advance notice before they attempt to cut back over to normal power. There may be an another outage as part of that process.
Thank you for your patience.
The power should be stable and we don't expect another outage to happen today or the rest of the weekend. When the power went out, everything worked as expected but for some reason the generator was out of phase and the UPS went to battery and then ran out of power. They still aren't sure why that happened but they are confident is it stable for now.

They are planning on fixing the wiring they changed today tomorrow morning between 10am-11am PST. They are planning on doing it without an outage, but there is a risk they could lose power again. The UPS will be on battery while they make the change and have about six minutes to make the change. They did it in three minutes today without an issue.
:)
Formerly known as Unknown Bliss.
Formerly Website Team Lead/Manager & Development Team.
Please don't PM me for support (or stuff that belongs in the forums or tracker) but otherwise feel free

Post Reply

Return to “phpBB Discussion”