On our outage yesterday
Yesterday (November 5th) we had our first major outage. Both our CalDAV and CardDAV servers were unresponsive from 5pm GMT until around 11pm last night.
Firstly, we'd like to offer our sincerest apologies. No data loss has occurred, and for most of you, your calendars and address books have been accessible the entire time. However, a few people have seen that their address books and calendars were temporarily empty, so we do hope that this has not caused any problems.
Our release process is almost entirely automated. This means that with the minimum amount of effort, we can safely put new versions of our software on our servers.
A little too easy.
Yesterday one of our engineers accidentally triggered the release of a new version. While the application itself was tried and tested, we unfortunately got stuck in a rut where our databases started an upgrade that was incredibly slow.
Normally we test any database changes beforehand as well, but this time we didn't get the opportunity. Just after the release was triggered, we quickly realized that the database migration was going to take some time.
While at no point during the update cycle were you at risk of data loss, stopping and reversing this process could have taken significantly longer, so we made the decision to double-down and simply let the release run its course.
What are we going to do about it?
This really was a mistake in our release process, so this is also where we need to fix things.
Some of the things that are now done automatically will be changed to a manual process. Furthermore, we will add some checks and safeguards to our system that allows the system itself to identify whether a release is expected or not.
If not, the new release will be stopped immediately with a big exclamation mark.
Need more info?
We hope that with this statement, you as the user are fully informed on why the outage occured and the steps we've taken to prevent any further disruptions occuring. If you would still like further information, then please feel free to send us a message to our support desk, where we'll be more than glad to assist.