When a bad anti-virus update hits you

Late on Thursday evening last week, Eset released a bad update for NOD32 Anti-Virus. Computers that received the update hung, crashed or kept displaying error messages about ekrn.exe crashing (this is the heart of NOD32.)

For the life of myself and my colleague, we couldn’t figure out what was going on initially. He was working late on the Thursday when the bad update came out, but we hadn’t linked the problems yet. Our Exchange server had fallen over, so we spent 45 minutes over the phone fixing that up. Friday morning when I got in, the Exchange server was still up, but people were complaining about error messages they were getting on screen, or the NOD32 splash screen not minimizing after log on.

We didn’t get much time to investigate the problem, as we were busy elsewhere doing other tasks. Eventually we got called to check if the servers had frozen, as the network had suddenly gone awry. When we went into the server room, it was confirmed: all our Windows servers had frozen or were in the process of freezing. We rebooted all of them, then set out to find what had happened. I checked Eset’s home page; sure enough there was a notice about a problem update. We downloaded the tool explained on the page, inoculated our servers and then proceeded to monitor the situation. The servers stayed stable. All we could do now was wait till the clients pulled in updated definitions that had corrected the problem.

It was a tense time because of this bad update. Our network became unstable and servers froze. Luckily no major data loss happened or any other serious problems. All AV vendors have released bad updates in the past, so we didn’t fault Eset. Other vendors’ botched updates took out Windows or created blue screen loops, we were lucky our problems were so mild in comparison.

When Monday came around, most of the computers had gotten new definitions, so the problem was moot. Life went back to normal. It’s the first time in almost 3 years of using NOD32 that I’ve had this problem, hopefully it won’t happen again any time soon.