Discuss: The return of the database / Recent Downtime

MichaelC · Post by **MichaelC** » Sun Aug 09, 2015 1:43 am

Please discuss the announcement and blog post here.

optimusvault · Post by **optimusvault** » Sun Aug 09, 2015 1:51 am

Announcement

The requested topic does not exist.

MichaelC · Post by **MichaelC** » Sun Aug 09, 2015 1:53 am

optimusvault wrote:Announcement
The requested topic does not exist.

We post these topics to get the link, then edit it into the announcement just before approving it. You were just very quick and caught us in between those two things. It's up there now.

optimusvault · Post by **optimusvault** » Sun Aug 09, 2015 1:55 am

I see :p

Thanks for fixing all of the issues. Sucks phpBB was down as I missed you guys! But glad you're back now

Lumpy Burgertushie · Post by **Lumpy Burgertushie** » Sun Aug 09, 2015 2:51 am

I figured someone just tripped over the plug on their way out the door for another beer.

robert

**My CDB Contributions** · Post by **HiFiKabin** » Sun Aug 09, 2015 10:42 am

I still think that they don't feed the hamsters enough

SiteSplat · Post by **SiteSplat** » Mon Aug 10, 2015 12:33 am

Looks like there is most likely some DB tables corruption around. For example:

viewtopic.php?f=461&t=2286276

Shows 7 replies , but there are 2 posts in total. Older stuff seems to be in limbo

RMcGirr83 · Post by **RMcGirr83** » Mon Aug 10, 2015 12:39 am

but at least we have ssl instead of a good database backup plan.

MichaelC · Post by **MichaelC** » Mon Aug 10, 2015 1:02 am

SiteSplat wrote:Looks like there is most likely some DB tables corruption around. For example:

viewtopic.php?f=461&t=2286276

Shows 7 replies , but there are 2 posts in total. Older stuff seems to be in limbo

Yeah, that's because some posts made to those topics are missing but we'd rather leave odd post counts like that in place so that we can identify where posts disappeared from in the future (such as realising that a post is a reply to a post that is no longer there as opposed to just a random post). I don't think a few off counts hurt?

But thanks for pointing this out.

RMcGirr83 wrote: but at least we have ssl instead of a good database backup plan.

This is handled by OSL as previously stated and it was a rare situation in that the backups are taken from SQL2 and it was fixing SQL2 so backups could recommence that then caused issues to SQL1.

OSL are making numerous changes to ensure that their backup infrastructure is a lot more steadfast (As you can read about here) and we are also working on making our own daily database backups for redundancy in light of this incident (so we will have two independent backup systems backing up to different locations). OSL did have a good backup plan in place (It is considered good/common industry practice to backup from your slave/second SQL server in order to prevent load on your primary SQL1) but of course in this rare edge case that proved problematic which is why we will be working to make it an even greater backup plan.

I might add that SSL is important to us as we highly value our users security and privacy, but requires relatively little work to implement (A few uploads and a few lines added to a virtual host as you know) compared to our new backup infrastructure which we are looking to implement which requires the setting up of various new systems and a lot of man hours in order to ensure it's reliability. Our number one priority is our commitment to your security and, by extension, the security of the data you put on phpbb.com.

Ger · Post by **Ger** » Mon Aug 10, 2015 8:01 am

RMcGirr83 wrote: but at least we have ssl instead of a good database backup plan.

Well there's a constructive comment

Those things have absolutely no correlation. Why nag? Situation is bad enough as it is and the nice people providing all this for us are working to improve it.

RMcGirr83 · Post by **RMcGirr83** » Mon Aug 10, 2015 9:34 am

It isn't a "nag" it is an observation. If it were my "toy", which it isn't, I would ensure that I had a good db backup plan in place. Redundancy is nice but clearly wasn't enough in this case. I still, to this day, do not understand the need for ssl on a forum site. But that's a discussion for another topic/day.

robertm96 · Post by **robertm96** » Mon Aug 10, 2015 12:09 pm

Feed that hamster please

Khaos-Rage · Post by **Khaos-Rage** » Mon Aug 10, 2015 3:11 pm

OSL wrote:As mentioned previously, mysql1 and mysql2 were unexpectedly set up in master-master replication configuration. Therefore, though slave replication on mysql2 was stopped, mysql2 was still sending commands to mysql1. This caused databases to be dropped on both machines.

yolo?

mrgtb · Post by **mrgtb** » Tue Aug 18, 2015 9:28 am

RMcGirr83 wrote:It isn't a "nag" it is an observation. If it were my "toy", which it isn't, I would ensure that I had a good db backup plan in place. Redundancy is nice but clearly wasn't enough in this case. I still, to this day, do not understand the need for ssl on a forum site. But that's a discussion for another topic/day.

Neither do I also, but since Google hinted (said) they'd start ranking sites higher using SLL. Every man and his dog has jumped on making their forum use SLL since. Too many people follow what Google says do...

There's been a number of issues with this site going down this year, doesn't look very good for the site in general it keep happening too regular.

**My CDB Contributions** · Post by **HiFiKabin** » Tue Aug 18, 2015 9:41 am

mrgtb wrote:
RMcGirr83 wrote:It isn't a "nag" it is an observation. If it were my "toy", which it isn't, I would ensure that I had a good db backup plan in place. Redundancy is nice but clearly wasn't enough in this case. I still, to this day, do not understand the need for ssl on a forum site. But that's a discussion for another topic/day.
Neither do I also, but since Google hinted (said) they'd start ranking sites higher using SLL. Every man and his dog has jumped on making their forum use SLL since. Too many people follow what Google says do...

There's been a number of issues with this site going down this year, doesn't look very good for the site in general it keep happening too regular.

IIRC the ranking hike Google give if you have a secure certificate is less than 1%.

Hardly worth it IMHO

advertisements