When your servers go nuts

Last week was the first week I’ve worked alone at work since early 2010. Back then, my colleague was on a week’s leave after our massive network migration. This time, it was because I was now on my own. The first two days of the week went smooth enough, nothing too crazy happening.

Wednesday was the start of the chaos. The day itself started out smoothly. Lunch time, our principal took the school’s admin staff (including myself) out to high tea. It was a nice way to spend 2 hours off campus doing nothing more than relaxing and chatting with colleagues. Eventually it ended and we headed off back to work. I sat down at my computer and noticed that Outlook had frozen. This was bizarre, as it has never locked up like that before. A moment later, the head of IT came to tell me that staff had not been able to access their email for a short while. I tried to Remote Desktop into the Exchange server, only to find that it wouldn’t connect. I opened up XenCenter, only to find that it couldn’t connect as well.

I proceeded into the server room to discover that the big virtual server we had bought earlier this year had frozen rock solid. Exchange, Anti-Virus and internet access/filtering was down for everyone. I had to force a reboot via the reset switch, and that’s when things got out of hand. After the power cycle, the server tried to boot from the network, which was strange to me. Network boot was the last option on the boot order list and should not have happened. I tried resetting again, only to have the same issue. Another restart and I decided to check out the BIOS. Lo and behold, the 16GB Kingston SSD drive that was the “brain” of the server was not being seen in the BIOS. Now I knew I had a real problem. I tried swapping SATA cables and changing the port on the server’s motherboard, to no avail. Our hardware supplier suggested I try the drive in a desktop pc to see if it would pick up. No luck either, and valuable time wasted.

The end of an era at work

Last week Wednesday, an era at work came to an end. My colleague finished his last day of work at the school and was about to move out again into the big wide world of business IT. Apart from a total of about 3 weeks, the two of us have been together as a team for 2 and a half years, changing things left, right and centre at work.

Stepping back a bit, my colleague resigned on Friday the 5th of August, just before the school had a long 4 day weekend. When we came back in on the Wednesday, he dropped the news to me. I was shocked, yet not entirely surprised by it. Looking back, all the signs were there that he was getting ready to move away from the school. I could write about any number of reasons why he made his choice, but I think the reality is that working in IT in a school environment tends to burn you out. Politics, budget issues, salary issues and more all contribute towards a frustration that slowly builds up. I’ve suffered some of the same problems, so I’m very sympathetic towards his decision to leave.

The biggest challenge in the couple of weeks leading up to his last day was trying to cram as much information about the school admin system into my head as possible. My colleague has been the person more or less in charge of that since the start, due to his prior experience with the product. While I’ve managed to get most of the knowledge down quite well, there are still many areas I worry about. School reports are due in a few weeks, which will be the real test of things I guess. All the procedures are in place and ready to go, but it’s usually when marks start getting entered that things go wrong.

The somewhat annoying thing is that the school hasn’t yet taken any concrete action in finding a replacement. The principal had the idea of hiring someone for the rest of this year and depending on their performance and teamwork with myself, that they may stay on permanently next year. We compiled a list of qualities needed in a replacement and sent it on to management, but no advert has yet been published. When staff have asked me about this and I’ve told them about this fact, many of them seemed puzzled by the fact that an advert hadn’t gone out yet. I dare say that I’m pretty good at what I do, but the school network is too big for one person to handle alone, no matter how automated it is or how good the person is. Sheer size and scale just make it too much.

Above all else, I’m going to miss the team spirit I had with my colleague. We argued, had different views on technology, fought on how to to handle staff amongst other things, but in the end we were a team. We covered for each other and grew together. We balanced each other out and managed the workload between us. Time will tell whether such a bond will form with the new person, but I hope it does. There’s no real other way to handle things I think.

To my colleague, I wish you all the best. I know that you are already enjoying your new job, and I hope that you will have a long, enjoyable and constructive time at your new job. Thanks for everything bud  

P.S Indeed