Major Outage 10-July-2013

[MagnetON]

Today Tenths suffered its longest unplanned outage ever.

At the moment I've not attempted a root cause analysis as the focus has been on getting Tenths back and running.

At approximately 2:30pm we experienced a major server failure that could only be resolved by re-booting the server. This caused a large database corruption and also a software corruption.

Unfortunately as I was in a meeting for the "day job" (yes, Tenths is a hobby for me too!) and was unable to give the problem my full attention.

Data was restored from the backup replication server however re-starting the forum software also caused the backup to corrupt.

Using the replication server as the main database server ALSO caused the backup to corrupt.

At approximately 7pm the decision was made to abandon attempting to repair the primary server and switch to our secondary server in the US. This involved shortening the DNS cache time from 1 hour to 30 minutes in order to allow a quicker change over when the site was brought back online.

This migration work also involved checking and configuring the secondary server to act as "primary".

This work was completed around 8:30pm and a further hour was spent testing functionality from here at Ten-Tenths Towers!

DNS migration took place at approximately 10pm and by the time of this post we are seeing traffic levels increase as the DNS change propagates.