Technical debt, laziness, and Grub.

So, I run My own server. It hosts this blog. It also, amongst other things, hosts my e-mail, and a local network share. (I know, I should use separate servers, but I do use containerisation to keep a modicum of separation)

To ensure the integrity of the data I use a software raid array, 4 disks in a raid 6 set, it's not a backup, and I should know better, but it has served me well enough. There have been a few disk failures, and I've not lost any data (at least none I care about enough to look at regularly enough to know it's gone) through any of them. One of those disk failures I put in a new disk, but it was slow, and had occasional read errors. Annoyingly these prevented me from installing the grub boot loader on that disk. But that's ok, there were three more disks, it's not a major problem. Critically however, it also prevented me installing grub on any other disks. And so begins our tale of fail. Since that disk has gone into the array there have been disk failures, I can't be certain of the number (disks don't fail in easy to identify patterns) but I can be certain that it is more than two. At least one more than two. Because the server I have doesn't support hot swapping drives, rebuilding the array requiires a restart of the system. Restating the system requires a working boot loader. The last of the disks with a working boot loader failed recently. This left me with a system that wouldn't boot, and installing grub wouldn't work with the slightly faulty disk in the system. I was left with a system that I couldn't repair without putting the integrity of my data at risk, or a long wait for the array to rebuild using a boot disk (knoppix as it happens. I highly recommend having a copy available to anyone who does any sort of computer support). I chose the latter. So it takes a long time to rebuild 3TBs of data onto a shiny new disk. And so my website, my blog, my emails too, have been offline for a long time. I have now replaced the slightly faulty drive, as well as the failed drive. The array is rebuilding (again) onto the newest drive. I have ordered enough disks to have a spare on hand. And I have learnt a lot about the grub-install command's modules flag. I have also now got the motivation to not only fix the technical debt that caused me to not have a server at home for three days, but also the technical debt that means I'm hosting a server at home, and not on a hosting service (I know what I'm doing this weekend).

posted at 07:54:47 PM on 5 Aug 2016 by Craig Stewart

Tags:sysadmin fail embarrassing mistakes breaking oops 

Roller Weblogger Update

So, I use Apache Roller as the application my blog runs on. A new version of this has come out (I was on 5.0 and 5.1 has been released) so I decided to upgrade.

This has resulted in the theme I was using breaking, badly, so I have had to move to the basic theme. I can't be bothered to tweak that right now, but I don't like it much either, so I am going to have too eventually. It also appears to have broken rss feeds, such that if you do follow my blog with an rss reader you get all my blog entries again (or at least it does in tiny tiny rss) so sorry about that.

Worse than all that however is the fact that I decided to take this opportunity to update to openjdk-7 (from openjdk-6) and tomcat 7 (from tomcat 6). The server this blog is running on used to be Debian 6, but was dist-upgraded to Debian 7 (which went terribly smoothly at the time) and the older versions of java and tomcat were left over from that. This process was far more laborious than it should have been, largely due to me having forgotten all the steps I had taken to get Roller working on tomcat 6 in the first place (the java upgrade was painless mind, so I did that bit right at least).

It's a good job I'm not getting paid for looking after this server, I'm apparently not doing a very good job of it.

posted at 10:34:09 PM on 19 Jan 2015 by Craig Stewart

Tags:web sysadmin oops screwup breaking mistakes fail should-know-better embarrassing 

Further Adventures in SSL

So following my work on fixing CVE-2014-4566 on my website, it turned out that I do indeed need to use lower versions of TLS than 1.2 a revelation that is a little embarrassing to admit. So I have been doing a little playing with the settings, and have tweaked the cipher suite to support TLSv1 TLSv1.1 and TLSv1.2 and only ciphers with forward secrecy.

posted at 09:17:35 PM on 1 Dec 2014 by Craig Stewart

Tags:mistakes embarrassing web ssl sysadmin security