Jump to content
Sign in to follow this  
9R_RyanJ

Survived By Server Outage (08/12/2018)

Recommended Posts

This is not how I was hoping my weekend would go...

On Wednesday evening we deployed a build that was supposed to drastically increase performance and stability. The network code behind the scenes had seen significant revisions. In hindsight, that an overhaul this dramatic didn't go smoothly isn't a shocker. Disconnects were common and frequent, and the backend was still reporting errors. New errors, but errors none-the-less.

24 hours later we had another build up that helped with the disconnects, but still had some work that needed to be done regarding the backend stability and the way electrum purchases were being tracked. 

The network engieers worked through the weekend, and Sunday evening we deployed another build that was finally going to bet us back to a solid, playable state. It did just the opposite. Our database got hammered beyond its capacity, which had a domino effect on the game servers, which, practically speaking, caused a ton of lag and disconnects.

The game wasn't playable, so I've made the call to take Survived By offline until we can take a closer look at the logs and ensure that our next build doesn't just make the problem worse again. If I had to guess I'd say that outage will last between 12 and 18 hours, but that's just an estimate. 

My one consolation is that we're working through these issues now rather than later. This week was always supposed to be a stress test, and in that regard it was successful. We found the limits of what our optimized servers could do, and it wasn't much. 

We'll get the game back up as quickly as we can, and, once again, thank you for your patience.

  • Like 6
  • Thanks 9
  • Haha 2
  • Sad 12

Share this post


Link to post
Share on other sites

Update: As of 12:15PM on 8/13, our first attempt at a new build has failed - we identified what bug was causing the failure and are working on a second attempt. We will not hit our "12 hour" goal as stated above, but should be within our 18 hour goal.

  • Thanks 4
  • Sad 3

Share this post


Link to post
Share on other sites

Update: As of 2PM, our servers have been restored! Thank you for your patience, ancestors, and please let us know if you experience any further instability. Welcome back to the Heartlands!

  • Like 5
  • Thanks 2

Share this post


Link to post
Share on other sites
Sign in to follow this  

×