When Windows Update goes wrong

Windows Update is usually a very reliable method of keeping Windows based computers up to date. Rough in the early days, it’s come a long way since then. Smooth and mostly transparent in the background, it isn’t often that bad updates slip through.

Unfortunately, during after August’s Patch Tuesday, such an event occurred. After a number of updates were either automatically approved or approved by myself, we had some computers blue screen and go into a reboot loop. Thankfully, out of almost 180 computers, only 5 have suffered the problem seen below:

All of the affected computers were running Windows 7 x64 SP1 with all updates applied. The first 3 times this happened, I couldn’t find a cure for the problem and ended up wiping and redoing the computer from scratch. Later in the week, I found some instructions online on how to get out of the loop and get back into working order.

1. Get into the Recovery Console either from install media or by letting the Repair your Computer wizard run after a number of crashes.
2. Open up a Command Prompt and delete the FNTCACHE.DAT file located in C:\Windows\System32
3. Reboot the computer, and you should now be able to get back into Windows.
4. Delete the FNTCACHE.DAT file again, as it will have been recreated by Windows.
5. Lastly, go to Windows Update in the Control Panel, then view Installed Updates. Remove KB2982791 and optionally KB2970228. The other 2 updates mentioned out there on the web only apply to Windows 8.1/Server 2012 and so are irrelevant to Windows 7 computers.
6. Reboot after the patches are removed.

As I said, it’s not often anymore that bad updates slip through all of Microsoft’s testing, but it does happen. Although it’s frustrating, I don’t intend to modify how I approve patches. I’d rather take the risk of something like this happening than get hammered by Alureon or Conficker or some other nasty because I ignored security patches.

Firmware update fun

A couple of years ago, flashing any device’s firmware was often a difficult, frustrating and sometimes downright dangerous task. Always hoping that the device wouldn’t get bricked due to some unknown bug in the firmware, or worse still, a power failure right in the middle of the flash.

These days for things like motherboards, it can be as easy as flashing inside Windows, or using the built in feature on the motherboard. Generally speaking, you no longer have to use MS-DOS and try to find floppy disks or use an alternative, it just works. Intel motherboards in particular are usually very straight forward when it comes to this: run the Express Update inside Windows. Windows reboots, the motherboard flashes itself, reboots and you are back into Windows. No other intervention required.

Thus it was a bit irritating a few weeks ago when I decided to flash some of Z68 motherboards to their latest (and last) BIOS version. I ran the Express Update inside Windows as I’ve done countless other times. Computer reboots, fails to flash the firmware and then goes back into Windows. No matter what I tried, the firmware would not update. My next step was to download the *.bio file from Intel’s website, place it on a flash drive and press F7 during boot, so that I can update the BIOS. This didn’t work as well:

That leaves me only one option – use Intel’s Iflash tool. I don’t have a copy of MS-DOS lying around, and I didn’t feel like going through many hoops just to get a flash drive set up correctly. I discovered that Iflash works with FreeDOS, so I simply placed the files on a flash drive I have set up with Ultimate Boot CD, which includes FreeDOS. Run Iflash, the computer reboots, but then sits for a while doing nothing. I was about to reset the computer when I noticed the power led on the computer doing a slow pulse. I remembered that Intel motherboards generally do this when updating the firmware or when in sleep mode, so I let the process go on. Sure enough, after about 3 minutes, the computer rebooted by itself. The latest BIOS was now installed and working correctly.

Thankfully there was only about 5 computers to do this on. I’m not sure why this model motherboard was so fussy, but it’s done now.

Fixing Windows Update issues

About three weeks ago, I approved a number of updates to be downloaded into WSUS for distribution on the school network. Among those updates was an update for the Windows Update client itself. I watched the WSUS console as the computers started reporting back and after a while I began to notice an odd pattern. 36 out of 39 computers in our main computer lab were not reporting in.

Taking a look at one of the affected computers in the lab, the cause of the computer not reporting in became clear: Windows Update Agent 7.6.7600.320 was failing to install repeatedly. Since this new version was required to download and install updated from WSUS, the computers would not be able to patch themselves until this Agent issue was fixed.

I tried numerous approaches to get the issue fixed: Uninstall anti-virus software, try installing updates at shutdown instead of through Windows Update in the Control Panel, run the System Update Readiness checker tool, run System File Check from the command prompt. Nothing worked. I was on the verge of preparing to wipe the lab and reimage the computers when I came across the answer.

Thanks to some vigorous internet scouring, I came across this Knowledge Base article on Microsoft’s website: http://support.microsoft.com/kb/2887535. Thankfully, the latest update agent was available to download right there from the article. I downloaded the 64 bit version and attempted to install the update manually on my affected lab computer. After a required reboot, I had success. Windows Update connected again and proceeded to download the now missing 17 updates and installed them. With this proving to be the solution, I went to each computer and installed the new update agent by hand. One by one, these computers were cured of the issue.

One computer however refused to install the updated agent. Checking the CBS Log file found in C:\Windows\Logs\CBS revealed that it thought it needed to be rebooted before updates could be installed. However, rebooting did not solve the problem. I’ve had issues in the past with Server 2008 where it got stuck on updates and needed a certain XML file to be deleted before it would boot again. Going to the location of the XML file, I couldn’t find the usual XML file. I did however find a reboot.xml file, which I viewed. This file pointed to a registry key that I assume was supposed to be deleted after the last round of updates. Since the key wasn’t deleted, the computer still thought it needed to be restarted. Deleting this key and rebooting solved the issue – I could now install the updated agent and install updates again.

At this point in time, I’m still not exactly sure why this lab of computers failed to install the update agent while the rest of the school did so without much fuss. About the only thing I can think of is that it’s somehow related to how the lab was cloned which was somehow causing an issue. Reading through the CBS logs doesn’t shed much light on the issue, since I don’t fully understand everything that’s captured in those log files.

I suppose this serves as a good reminder that while WSUS and Windows Updates in general normally just work, sometimes things can go wrong.