Server Outage Aftermath
Written by quintus on 2019-03-25.
Our server is back online.
As it has been communicated via our bugtracker, our server, alexandria, went down on March 12th, 2019. The most important thing first: the server is back online, and we have brought all services back online already. Users of the forum interface on lists.secretchronicles.org or the tsc-devel mailing list should pay close attention to the section Current State below as action is required.
On March 12th, 2019, our server
alexandria.secretchronicles.org became completely unresponsive. It
neither served web pages, nor could we log in via SSH as the TCP
connections just timed out. Alexandria was a VPS server running on an
OpenVZ host and was sponsored by First-Root UG, Germany. Quintus
quickly got into contact with them by raising a support ticket on that
day and they diagnosed that the server had “hung up” and could not be
rescued. As an alternative, First-Root offered to provide us with a
KVM server on March 13th, 2019. We considered all data on
the server as lost and checked our backups.
As it turns out, we had just a few months before upgraded the operating system from Debian 8 to Debian 9. Before that upgrade, we had automatic backups in use, but due to time constraints we didn’t set them up yet on Debian 9, although we intended to do so. Murphy’s law has no mercy, so the last backup we had available was from October 2018. Given the low activity of the project in the time since October 2018, this isn’t too much of a loss, though. The source code of TSC was entirely unaffected, and because it is stored in Git, every developer had a recent copy, and all code is available via GitHub anyway. We have not lost any source code.
It was decided that the old VPS would not be set up again, because that would duplicate work for the TSC team, which is running low on free time anyway. Instead, we wanted to wait for First-Root to provide the KVM machine. During this time, which we expected to be short, all services hosted on the secretchronicles.org domain would be unavailable. On March 13th, 2019, quintus opened a ticket on the bugtracker on GitHub to announce the problem publically as the tsc-devel mailing list was unavailable due to the outage.
At this point, all communication between the TSC team members
basically happened in our IRC channel
freenode. Our IRC bot, furbot, was also hosted on our server, which
means that for this time no public chat logs are available on
Despite some status queries, First-Root did not reply to the support ticket raised with them after March 13th, nor did they provide the promised KVM server. By March 24th, 2019, we had been offline for nearly two weeks and we decided that we need to venture for an alternative. TSC team member xet7 is involved in the Wekan project, an open-source kanban board implementation. The Wekan project has a server that is used to port Wekan to arm64 and xet7 generously allowed TSC to move over to that server. Albeit this server has a rather unusual CPU architecture (aarch64), it is really powerful and is going to serve all of our hosting needs nicely. As it’s a bare metal server and not a VPS, it should be easier to fight any problems arising with the server.
On March 24th, the new server was set up and DNS RRs were updated. The old VPS is now out of use.
All services have been restored since noon of March 25th, 2019. In particular, our IRC bot is logging again to chatlogs.secretchronicles.org.
The new server provided by the Wekan project runs Ubuntu 18.04 instead of Debian 9. We took the opportunity to switch our Mailman 3 installation from a custom one over to one based on the mailman3 packages in the Ubuntu repositories, as this should lower the maintenance overhead. As the versions in use were different, it was not possible to just copy the database over from the backup of 2018-10-28. This has the following consequences:
- If you registered to the web interface or subscribed to the tsc-devel mailing list after 2018-10-28, you lost your account and subscription. You need to register and/or subscribe again.
- If you were subscribed before 2018-10-28 and use only e-mail to interact with the tsc-devel mailing list, you don’t need to do anything. You have already been subscribed again.
- If you were registered before 2018-10-28 and use the web forum interface to interact with tsc-devel, you have been switched to an e-mail-only subscription for technical reasons, i.e. the forum account was lost. You need to go to the web forum interface and re-register with the address you receive e-mails from the forum at. The forum will automatically match your subscribed e-mail address to the forum account you create. Once you have logged in, you can configure e-mail notifications of posts by clicking on “Manage Lists” on the top right (alternatively, follow this link after login). Click the red “Unsubscribe” button on the page for the tsc-devel mailing list to stop receiving e-mails from the list. You can still post via the web interface after doing this.
- If you want to delete your subscription without creating a forum
account, please join our IRC channel
#secretchronicleson chat.freenode.net and request this there.
The other mailing list, tsc-users, has not been restored, as it never saw any serious traffic. All discussion, even if it is user discussion like submission of levels, should just happen on tsc-devel from now on. All subscriptions to the tsc-users mailing list have been cancelled.
We apologize for the inconvenience caused by the Mailman version switch.
We are going to evaluate our backup strategy and quickly get automatic backups up and running for the new server.
Additionally, beyond restoring the services that we had on the old VPS, we are considering the setup of more infrastructure. For instance, the Wiki, which we set to sleep after it got spammed, might see a rennaissance. The setup of a CI service is in discussion, as are automated nightly builds. TSC team member kirbyfan suggested installation of Discourse, a rails-based web forum software.
on behalf of the TSC team.