Archive

Archive for the ‘Software’ Category

The slow migration to Office 365

When it comes to corporate mail servers, many would argue that Microsoft Exchange is the king of the hill. It’s a behemoth of a product that powers so many offices around the world providing vital features. If set up correctly, Exchange has been one of those products that in my experience just hums along quietly, doing its job without demanding a lot of attention.

At the end of 2009 when my first colleague and I migrated my school’s network, Exchange 2007 was our mail server of choice. Not only would it provide everything we needed, it would offer many new features to a school that was used to using a very broken Pegasus Mail/Mercury mail server/Novell Netware combination. It also helped that we got Exchange free of charge under the national SA Government agreement with Microsoft, which ended about 6 months after we installed Exchange.

Since that time, this Exchange server has processed millions of emails and survived moving from a decrepit physical machine to a XenServer implementation to finally ending up in Hyper-V. I’ve probably had 10 incidents or less with this server over the last 7 years. External internet connectivity or issues with upstream mail servers not withstanding, our mail server has done its job perfectly.

Like all things in technology however, there comes a time to move on. The web interface of Exchange 2007 has become really dated and leaves you tied to Internet Explorer for the best results. Those were the days before Microsoft became cross browser friendly, where the “lite” version of webmail was seriously crippled. While I would have upgraded us internally to a later version, we couldn’t afford an upgrade. After the government agreement expired, we were stuck. Quotes I received for updates versions made my eyes water and no one could quite work out how to price software for schools. I think every reseller I contacted only knew how to deal with the corporate world.

In the intervening years, Microsoft essentially resolved my dilemma by introducing and refining Office 365 for Education. Originally billed as Live@Edu, the product provided some nice perks – 50GB mailbox, huge (then called) Skydrive storage etc. The problem was that the product lacked unity and cohesiveness at the time. Live@Edu folded into Office 365 and things have only gone up and up since then. For no cost to us, we get access to the latest version of Exchange, albeit Exchange online, 50GB mailbox, superior spam filtering, access to Microsoft Teams and all the other applications available for education users. As long as their is competition with Google’s G-Suite for Education, we all stand to benefit from that rivalry which forces Microsoft to up their game.

Towards the end of 2015, I decided to migrate all my student mailboxes over to the cloud since students had miniscule amounts of mail compared to staff. It got their mailboxes offsite and gave me some valuable experience on how the migration process would work. It took some reading up on how to do it, but the process is something like this:

  • Sync your on site Active Directory to 365 with the new Sync Tool. The new tool is far better than the old version and what was possible in the earlier days.
  • If you want to be able to simply connect to on premises Exchange and migrate the mailbox like that, you need to have a working Outlook Web Access instance running, secured by a SSL certificate. This lets 365 sync the selected users mailbox to the cloud.
  • The process takes a while, especially over slow connections. The faster internet speeds you have, especially upload speed, the better.
  • You are limited to either a cutover or staged migration for Exchange 2007. Cutover is defined as moving everyone at once then changing DNS MX records so that mail flows directly to 365. Staged is slower, where you move some mailboxes at a time and still use the onsite server as the engine for routing mail. There’s slightly more work with staged, but it lets you be methodical and careful.
  • You can upload Outlook PST files as another method of moving mailboxes, but it’s the same issue as an online migration – you need good uploading speed.

This year I started moving staff mailboxes over for the first time. I had only planned to start once our fibre optic internet connection was in, but the unexpected delays in getting our line in has pushed me to start now already, even over our horrible ADSL connection. I’ve now synced about 10 staff mailboxes over and given staff a manual on how to use the new interface. Some are familiar with it already having had access via their universities or other institutes. The real problem is identifying users who can adapt to the new interface and give feedback on the manual. This is easier said than done when you still have some staff who can barely work with the existing system, 7 years after it went online…

Eventually my goal is to have moved all mailboxes over the cloud, with not one email having been misplaced during the journey. Once that is done, I intend to decommission my on site Exchange server, as well as the actual Windows VM it’s running in. It will be good to not have to support Server 2008 as well, one less old OS to worry about.

In short, there’s precious little reason to have an onsite Exchange server anymore if your internet connection is fast enough. Microsoft does a better job of server uptime than what we can do on our own, they have better spam filtering and they provide a package of products that is not only compelling, but free for education as well. The only real reason to have onsite Exchange anymore is because of privacy or regulatory concerns or if you need some sort of feature that Exchange Online can’t provide.

Upgrading Windows 10 via WSUS

November 15, 2016 Leave a comment

Windows 10 is supposedly the “last” consumer Windows edition Microsoft will release. While the version will stay as 10, over time the whole OS will mature, grow and mutate into something that will look and feel very different from the original release of July 2015. One side effect of this is that in a corporate environment using WSUS, it becomes possible for new versions of Windows 10 to be deployed as an in place fully automatic upgrade, the same way any other patch or service pack is installed. I was curious to see how this worked, so I approved the Anniversary Update (also known as version 1607) for installation at work and let my PC download the update.

Sure enough, the process was the same as what my home PC went through when it upgraded to the 1607 update. A couple of update screens and quite some time later, I was back at my desktop, duly upgraded. Everything was still in place, bar the RSAT pack which had to be updated to a version compatible with v1607. Overall, an extremely smooth and hands free process, just time consuming. I imagine it would easily take twice or thrice as long if the machine runs on a mechanical HD and not a SSD.

That being said, there was one major problem with the 1607 update – checking for updates from WSUS broke due to a bug in 1607. Windows 10 1607 would start to search for updates from the configured WSUS server, only to have multiple services in the background crash repeatedly, with no indication to the user. To the end user, it simply looks like the search is stuck at 1% and never moves from there. Apparently, if one leaves the process running long enough, updates will eventually download. This is obviously an unacceptable bug and Microsoft were made aware of it. They promised a fix in one of the monthly update roll ups, which was subsequently delivered and verified as having fixed the problem. Now you have a chicken and egg situation: deploy the 1607 update via WSUS, but then struggle after that, since you need the Cumulative Update to fix the problem.

You could manually install the update, but this becomes unwieldy in a large organisation. If deploying Windows 10 via deployment tools, you could make sure that the base image has the update injected already, which prevents the issue from cropping up in the first place. Sadly, the 1607 update is delivered from WSUS as an encrypted ESD file. While it is possible to decrypt this and inject the update, I don’t know if it’s possible to convert that back in to an ESD file. Even if you could, the checksum wouldn’t be valid and WSUS would probably fail to work with the modified file.

There’s always a possibility Microsoft could revise the 1607 update in WSUS so that the ESD file comes with the last Cumulative Update installed so that it works correctly out the box. I recall something like this happening with the November 1511 update, which I declined as it was another 3-4 GB download. Unfortunately, one doesn’t know when or even if this will happen. It’s also possible the problem will never be fixed. With the Creators Update due out early 2017 (March?) it’s possible that Microsoft uses that as the new baseline. If I’m correct, once the Creators Update is approved in the WSUS console, it will supersede the Anniversary Update, so the problem should be solved by bypassing the 1607 update.

I look forward to eventually rolling out Windows versions like this, though I think it will be beneficial if every computer had a SSD inside it first. Mechanical hard drives really do slow things down these days. A nice side effect of this is that Windows shouldn’t end up suffering from “Windows rot” as the Windows directory is replaced with each major upgrade. This should keep performance up compared to something like Windows 7 that gets bogged down after years worth of updates. Interesting times ahead…

Categories: Software Tags: ,

SparkPost

November 6, 2016 Leave a comment

About a month ago, I received an email at work from the company which develops our school administration software. The email advised us that the company was planning to migrate their backend email delivery provider from Mandrill to SparkPost. We were advised that if we wanted to keep mail delivery free we’d sign up for an account with SparkPost. The email was poorly worded, as both my colleague and I assumed that the change over was going to be happening in a matter of days. Since our school sends out tons of email via the admin package, I acted quickly and got us signed up for a free account.

After getting signed up, I got the company to switch the backend provider on our account over to SparkPost, which worked correctly. I was advised to set up SPF and DKIM records in our DNS zone so that mail sent via SparkPost would be far less likely to be rejected as SPAM. It took me a bit of research on the correct way to set up these records, most especially the SPF record. We have mail coming from our domain from both SparkPost and our MX records, so both need to be covered. A catch is that your SPF record cannot require more than 10 DNS lookups or it would be not be considered a valid record. It took me a bit of fiddling to find the right balance, but I got it done eventually. As a bonus, the SPF record should help get mail delivered to Gmail recipients quicker – we’ve often have long delays in mail getting delivered to Gmail in the past probably due to the lack of the SPF record.

Once term started and users started sending mail, some problems came to light, namely that a lot of mail was simply being rejected as SPAM and that pulling out the list of automatically suppressed email addresses was impossible via the web interface. The SPAM problem comes from the fact that some of the IP addresses used in SparkPost’s free tier pool have been tainted by other users. Since we have no control over which server sends the mail, it’s a crap shoot in which mail gets through and what is blocked as SPAM. One solution is to upgrade to a paid tier and buy dedicated IP addresses, but this was not something we had budgeted for and as such isn’t a viable option just yet.

Contacting their support, I asked for help. I got a reply that apologised, told me that they were terminating accounts for SPAMMING and that they had made some change that would hopefully help our account. Time will tell if that really is the case. We cannot afford to regularly have 20 odd % of our mail routinely fail to deliver because it’s identified as SPAM due to a tainted IP address.

Getting the suppression list was a challenge. I found a command on their blog which would pull it out of a command line using cURL, a Unix tool. This displays a raw bit of JSON code on the command line which includes all the suppressed emails and reasons why it was suppressed. It took me quite some time to figure out that I could echo this output using the > command to a text file with the entire output of the command. Then I needed to get this processed into something I could use, preferably a CSV file for import into Excel. Thankfully I found an website that does just that – website here. Armed with the now useful CSV file, I imported into Excel and made a spreadsheet for our registrar to follow up with the relevant parents so that we can get correct email addresses.

This whole adventure with SparkPost has taught me quite a bit about email out there on the internet, especially when you operate on a bulk scale. It’s also taught me that the spammers have really ruined email as a communication tool. I struggle to explain to staff in plain English why exactly their email isn’t getting delivered, as the concepts are not straight forward for people who don’t have the faintest clue of how email delivery actually works.

Still, SparkPost should be useful in the long run, especially if they get their tainted IP problem sorted out. I have more insight now into the process than I did when Mandrill was the backend delivery tool. I get the feeling that at this point in time, SparkPost is still very much a programmer’s tool rather than something that is geared towards end users. Hopefully in time SparkPost will make their website more user friendly and capable, which will greatly elevate the service I think, especially for a non-programmer like myself who simply needs to get something done.

Categories: Software Tags:

Those WTF moments

October 14, 2016 Leave a comment

Sometimes in the world of IT, you have moments where all you can do is scratch your head and ask WTF happened. Such was the case on Monday this week, before I even got back into work and before the term started. I received a text from my head of IT who said that he was unable to access one of our (virtual) servers to post our PowerPoint daily notice we show our learners. In-between getting dressed and packing my bags for work, I remoted in to take a look.

I couldn’t see the server on the network nor could I Remote Desktop in to it. Ping worked surprisingly, but that seemed to be about it. Going to Hyper-V manager revealed the server was on and I could connect via the console. I picked up a clue as what to what could be wrong when I noticed that the Heartbeat status wasn’t being reported to Hyper-V Manager. This indicated that the service had stopped running for some reason.

The previous Friday I had rebooted all our servers in order to finish their update cycles, as well as to prepare them for the term ahead. This particular server had come up from the reboot ok, so I didn’t do an in depth check. It’s never given me an issue like what happened before, so I made the mistake of assuming all was well. Anyway, after connecting, I could see that all the Hyper-V services were not running inside the VM. Manually trying to start them didn’t work. I tried to upgrade the Integration Components since Hyper-V indicated that my other VM’s needed an update for the components. No matter how I ran the setup file, it would not execute on the sick VM. By this time I had to leave to get to work, so the problem had to wait until I got in.

After arriving at work and settling in, I cloned the VM to my PC so I could play around more easily. Numerous attempts at a cure all failed, until I came across a post on the internet that described the same symptoms as I had. There was a link to a Microsoft KB article, which included steps on how to fix the problem. The KB dated from a few years back, so I found it incredibly bizarre that the problem only hit us now. Still, the sick server is running Server 2008, so I went ahead and made the change in the registry as documented. A reboot later and the server on my PC was suddenly working normally again. All relevant services were starting up correctly again and the server was back in action.

Since it was successful on my local cloned image, I went ahead and made the same change on the sick VM itself. Sure enough, one reboot later and we were back in business. In the aftermath, I spent a lot of time trying to figure out what caused this issue. While I did have IIS installed on the server years ago, I don’t recall there ever being a SSL certificate on that server. How exactly we ended up with the situation is probably something I’ll never fully know. As I said to my colleague, we’ve both seen random stuff over the years, but this one was really a WTF moment in a big way.

Categories: General, Software

Keeping Adobe Flash Player updated on a network

The Adobe Flash Player plugin is a pain in the arse. It’s a security nightmare, with more holes in the codebase than Swiss cheese. It seems every other week Flash makes the headlines when some or another security vulnerability is discovered and exploited. Cue the groans from network admins and users around the world as Flash has to be updated *yet* again. Unfortunately, one can’t quite get permanently rid of it just yet, as too many websites still rely on it. While you could get away with not using it at home, in a school where multiple people use a computer and visit different websites, one doesn’t have much choice really but to make sure Flash is installed.

On Windows 7 and below, the situation with Flash is a bit crazy. There’s a version for Internet Explorer (ActiveX plugin), a version for Firefox that is installed separately and Google Chrome bundles its own version – I’m not sure about smaller or niche browsers, but I think modern Opera inherits Flash via its relationship with Chrome’s engine. Thankfully with Windows 8 and above, Flash for Internet Explorer is distributed via Windows Update. It’s automatic and contains no 3rd party advertisements, anti-virus offers, browser bundling etc – all things Adobe have done in the past with their Flash installers. Trying to install Flash from Adobe’s website on Windows 8 and above will fail, which at least may help to kill off the fake Flash installer routine used by malware authors to trick unsuspecting users.

The usual method of installing Flash is highly cumbersome if you run a large network – not to mention that EXE files are much less flexible than MSI files for deployment and silent install options. Thankfully Adobe do make Flash Player in MSI format, but it’s not easy to get hold of directly. You have to sign a free enterprise deployment license to be able to legally distribute Flash and Reader in your organisation. The problem becomes how to distribute the updates especially if you aren’t running System Center or another product like that. Enter WSUS Package Publisher, indispensable if you make use of WSUS on your network.

WPP allows you to use the enterprise update catalogs Adobe and some other vendors offer. Using this, you essentially push the updates into your existing WSUS infrastructure, where it ends up delivered to the client computers like any other update. One thing you need to do is tweak the update as you publish it, so that it isn’t applicable to computers running Windows 8 upwards – if you don’t do this, the update will download on newer Windows versions, but will fail to install repeatedly and will need to be hidden. The other thing I’ve also discovered that needs to be fixed is that the silent install command line switch needs to be deleted. When a MSI file is delivered via WSUS, it is automatically installed silently. I discovered this the hard way, since one of the Flash updates I imported was failing to install on every computer. Turning on MSI logging and searching for the error code eventually lead me to discovering what was wrong, after which I corrected the problem and now know what to do with every new update that comes out for Flash.

Since using WPP, I’ve felt happier about the safety of my network, as I can usually get Flash pushed out with 2-3 days of the initial download. This is far better than having to visit each computer manually and keeping Flash up to date that way!

Updating Windows at the source

Since the release of Windows Vista, Windows has been installed by using a compressed image file, known as a WIM file. This is what allows Microsoft to ship one disk containing the home and other versions of Windows, unlike the multiple disks of the XP era. What makes a WIM file even more useful is that it can be mounted inside a running copy of Windows and have patches and drivers injected directly into the image. This is extremely handy when you realise that Windows 7 has been out for almost 6 years now and has a couple of hundred patches out there. Anything that cuts down the wait for Updates to install is a good thing, as well as having a more secure system out the box.

There are a couple of limitations however:

  1. You can’t inject all the update patches offline. Certain updates can only be installed when Windows is running.
  2. NET Frameworks cannot be injected offline. These will need to be installed and patched after Windows is up and running.
  3. You can only inject patches if they are in CAB or MSU format. EXE files are not usable here.

To update Windows 7 (or 8 or Server editions for that matter) you will need the following:

  • Windows 7 media or ISO file. I don’t have access to OEM disks so cannot say if those can be updated. What you really need is the install.wim file, found in the \Sources directory on the disk. It’s the single biggest file on the disk.
  • Windows 7 Automated Installation Kit or the later Windows 8.1 Assessment and Deployment Kit. You need this for the DISM tools which services the WIM file.
  • Access to the updates for Windows 7. There are many ways to get these, but I have found that looking the C:\Windows\SoftwareDistribution\Download folder on a patched machine to be one of the better ways to get the updates. Other tools have had mixed success for me.
  • Hard drive space and patience. Injecting updates, committing the changes to the WIM file and optionally recreating the ISO file will take time.

Here’s my step by step guide on how to do this update procedure. A note before I begin however. My guide is a little longer than strictly speaking necessary. If you have access to ISO editing software, you could just replace the install.wim file and be done. However, I am going to include the steps to rebuild an ISO image, including the option to make it UEFI boot compatible.

Updating Windows

  1. Make 3 folders on a hard drive. For example C:\Win7, C:\Updates and C:\Mount.
  2. Copy the install.wim file from your ISO or DVD to C:\Win7.
  3. Install the Windows 7 AIK or Windows 8.1 ADK. Specifically, we are looking for the Deployment Tools option. We don’t need the rest for this process.
  4. Place all the updates for Windows 7 into the C:\Updates folder.
  5. Open up the “Deployment and Imaging Tools Environment” shortcut as an Administrator. The DISM commands will only run with Admin approval.
  6. Run the command dism /get-wiminfo /wimfile:C:\Win7\install.wim
    This will tell us about the various Windows editions present in the WIM file. Depending on the disk, it may include multiple editions or only 1. Take note of the index number which corresponds to the edition of Windows you want to update, we will use it in the next command.
  7. dism /mount-wim /wimfile:C:\Win7\install.wim /index:X /mountdir:C:\Mount (replace X with the number you want from step 6) DISM will mount the image edition at the C:\Mount folder
  8. dism /image:C:\Mount /add-package /packagepath:C:\Updates
    DISM will now start to add all the MSU and CAB files it finds in the C:\Updates directory and apply them to the mounted image. This will take some time, so feel free to take a break. Some updates may cause an error; these updates are only meant to be installed when Windows is running. You will need to find out what updates caused the error and remove them. Type dism /unmount-wim /mountdir:C:\Mount /discard to discard all the changes and follow steps 7 & 8 again until the process is error free.
  9. dism /unmount-wim /mountdir:C:\Mount /commit
    This will commit the changes, save and unmount the WIM file.
  10.   If you want to update another edition of Windows 7, go back to step 7 and use another index number. Go through steps 7-9 again for all editions you want to update.

Building the new ISO for Windows 7

If you are planning to use the updated WIM file with Microsoft Deployment Toolkit, you are good to go and can use the updated install.wim file in conjunction with the rest of the Windows setup files. Otherwise, you’ll need to create a new ISO image that can be used virtually, burned to DVD or used on a USB flash drive for install purposes.

Open up the “Deployment and Imaging Tools Environment” shortcut as an Administrator again. Run the following command to make the ISO file that can boot on traditional BIOS based systems or on UEFI systems. For the most modern UEFI systems, make sure Secure Boot is disabled before you install Windows 7, as it is not Secure Boot capable.

For this step, copy all the files from your Windows 7 DVD or ISO to the Win7 directory, but leave out the old install.wim file or you will have wasted your time.

oscdimg.exe -u2 -udfver102 -bootdata:2#p0,bC:\Win7\boot\etfsboot.com#pEF,ebC:\Win7\efi\microsoft\boot\efisys.bin -o –lVOLUME_LABEL C:\Win7 C:\Win7\Win7.iso

Replace VOLUME_LABEL with something of your choice.

You can now burn the ISO file to DVD, use it on a flash drive or as an ISO with any VM software.

I have not tried this procedure with Windows 8.x, but I believe it should work the same way as the file layout of the relevant files and folders are near identical.

The Windows 10 upgrade experience

On Wednesday 29 July 2015, a new chapter opened up in the history of Microsoft’s Windows. Windows 10 was unleashed on the world, Microsoft’s attempt to remedy the largely cool reaction to Windows 8, as well as stay relevant (at least in the eyes of a lot of tech bloggers) in the new app centric world. The return of the Start Menu, an unprecedented public participation process via the Windows Insider program, free upgrades for a year, DirectX 12 for gamers and many more features all combined to build up a hype that has not been seen for a long time in the Windows world.

Like millions of other people, I reserved my copy of the Windows 10 upgrade via the app in the system tray that appeared after a round of Windows Updates a few months back. The idea was that this application would trickle download Windows 10 in the background as soon as the system went RTM, so that on launch day you’d be ready to upgrade immediately. Only problem is that the trickle download process started 1-2 days before the launch of Windows 10, which meant that with my slow ADSL speed, it would be some time before I’d be ready to go, let alone the chance that I’d be in one of the later waves of the rollout. This is probably due to the fact that build 10240 only went RTM 2 weeks before the global rollout. Either way, I was impatient to get going.

Thankfully Microsoft thought clearly and made ISO images available for direct download or via the Windows 10 Media Creation Tool. I snagged a copy of the Media Creation Tool and used it to download a copy of Windows 10 at work, where I have access to a faster internet connection. Once the ISO file was built by the tool, I burned it to DVD for 3 other staff members who were interested. It’s legal to do this by the way, since each person would be having their own upgrade key made during the upgrade process. For myself, I used the excellent Rufus utility to burn the image to a flash drive. Although the Media Creation Tool can burn the image to flash drive, I’ve come to trust Rufus, especially thanks to its ability to create properly booting UEFI capable media.

Once at home, I simply inserted the flash drive, double clicked on setup.exe and let the upgrade process run. I had previously been running Windows 8.1 with all updates applied. The installation process ran smoothly and took about half an hour to move all my files, upgrade itself and get to the desktop. All of my software remained installed and I haven’t yet had any compatibility issues software wise. I did have some issues with my PC’s built in Bluetooth adapter, but a couple of hours after the upgrade, a driver had been installed in the background and the adapter was good to go again. After the upgrade, I did manually install Nvidia’s latest graphics driver, since I already had it downloaded and couldn’t wait on Windows Update to deliver the driver.

So far, I mostly like Windows 10. It’s been stable despite the upgrade, no blue screens or crashes. As mentioned, all my software has remained in working without issue. Speed wise it feels a little faster than Windows 8.1, but not much. The speed may be more impactful on users coming from Windows 7 or earlier. My biggest real gripe at the moment with Windows 10 is the severe regression in the OneDrive client, a very well moaned about topic on the net. Windows 8 and 8.1 spoiled me in that regards with placeholder sync that let me see the files that were on my OneDrive, without actually needing to download them. The Windows 10 version basically takes us back to the Windows 7 version of the client where you have to chose which folders and files to sync, which will then chew up space on your hard drive. I am not happy at all with this change, but I am holding out that the new client that should be here by the end of the year will offer a better experience.

One small note: my copy of Windows 10 wouldn’t activate until a day after the install. While I kept thinking that somehow it was related to my Windows 8.1 key, it was simply a case of the fact that the activation servers were getting hammered into oblivion. Over 14 million people upgraded in the first 24 hours, so I am not surprised that I struggled to activate. I am assuming that now, almost 2 weeks later, activation should be happening immediately as per normal again.

It’s been a common refrain that I’ve seen on the net from reviews that if there’s one thing Windows 10 needs, it’s that it needs more polish. Lots of little fit and finish issues keep cropping up older legacy parts of Windows are moved into the modern framework. Different right click menus, a System Settings App that isn’t quite Control Panel yet, out of place icons etc. all need some time and attention before Windows 10 becomes its own unique system. With the promise of Windows as a Service, it’s likely that many of these issues will go away with time as the system keeps being updated and improved. One thing is for sure, it’s going to be an interesting ride indeed.