Archive

Author Archive

Hyper-V bug in Windows 10 v1703

November 4, 2017 Leave a comment

I encountered a nasty little bug in Windows 10 v1703 Hyper-V a.k.a the Creators Update this past week. If you create a Generation 2 virtual machine and try to PXE boot that VM, regardless of whether it’s on an internal switch or bridged to an external network, you will end up with the following screen:

Hyper-

No matter what you do, the VM will not PXE boot. Disabling Secure Boot and fiddling with other options will not help. I was very confused by this problem, as I’ve PXE booted generation 2 clients before. A few searches later revealed this link which explains the problem in greater detail.

In short, the only answer is to either downgrade to Windows 10 1607 or upgrade to Windows 10 1709, which was released little over 2 weeks ago. Generation 1 VM’s are not affected and you can PXE boot them successfully, but they do have a higher overhead than gen 2 VM’s. How this bug crept into Hyper-V is curious to say the least, but at least there’s a definitive fix. I should add that the bug has not been fixed as of the latest cumulative update for 1703 and is probably unlikely to be fixed, given the way Microsoft now releases Windows 10 updates/upgrades.

Advertisements

Goodbye Exchange 2007

It’s been a while since I last posted here. Life has been super hectic and finding free time to write something coherent has been harder than I thought it would be. I’ve also seen that my last post was actually my 200th post, which in the grand scheme of things is a pretty nice milestone considering how eratic I’ve been with posting over the years.

Anyway, I’m going to keep this one short and sweet. I’m posting a screenshot I made as I was finishing the removal of Exchange 2007 from our network ±4 months ago. Our mail platform has been running successfully on Office 365 since that time, with only the very odd head scratching moment causing some minor grief. Staff have more or less settled down into using the new platform, though they probably still need a lot more time to become fully familiar with the interface. SPAM suppression seems to be working well, though I’m not sure how many staff are actually checking their junk folders for legit mail that is incorrectly marked as junk.

I must say that the un-install process went very smoothly, it just took a long time as per the screenshot below. I chose to do things properly instead of just trashing the VM, which would have been a lot quicker. The removal process removes a lot of stuff from Active Directory, though there is still quite a bit of cruft left behind.

Categories: Software Tags:

Being a good net citizen: SPF, DKIM and DMARC records

Spam and Phishing emails are some of the more visible scourges of the modern internet. No one enjoys opening up their mailbox and seeing junk clutter up the place, or seeing a mail that tempts you to enter credentials somewhere because it looks legitimate. The war against Spam and Phishing is an on-going battle, with many tools deployed to try and keep a user’s inbox clean.

If you own or manage a domain on the internet and that domain makes use of email, it’s only right to be a good net citizen and set up SPF, DKIM and DMARC records. Together those 3 make a 3 pronged fork that can be stabbed into the heart of junk mail, but they each do a slightly different thing. Let’s take a look at them:

SPF essentially denotes who is allowed to send mail for your domain. Anything that doesn’t match the details in the record is to be considered an attempt to spoof your domain and should ideally be rejected, provided the record is set up as such. If you have a small domain with simple records, SPF is incredibly easy to set up. It becomes harder if you are a giant corporation or have lots of mail being sent from third party bulk mailers, but even those use case scenarios can be brought into line so that you have a valid SPF record. If Microsoft, Google and others can do it, why can’t we?

DKIM is a little trickier. DKIM enabled servers sign outgoing mail with a digital signature and lets receiving servers validate the signature using the published key in the DKIM DNS record. This way, mail can be verified as having been sent from domain abcdefg.com because the signature can be verified by consulting the DKIM record in abcdefg.com’s domain. If the validation fails it’s either because the mail was forged or the message modified on route. Since spammers aren’t running your mail server, they can’t validly sign outgoing messages with your private key, so when a destination server checks the signature, the check will fail.

DMARC sits on top of SPF and DKIM. While SPF contains syntax for what to do when mail fails a check, DKIM does not. DMARC essentially tells a recipient mail server what do with those mail if they fail the SPF/DKIM checks. Mail can either be allowed through to be processed as the destination sees fit, sent to the Spam/Junk folder or rejected outright. Set to reject mode and along with an –all syntax in SPF, this will ensure that spammers cannot spoof mail from your domain (in theory)

It’s not perfect though. In order for the 3 records to be effective, the destination mail server needs to check the records. If the server doesn’t and simply accepts mail as is, junk mail will make it into the inbox from forged senders. The records also don’t help if a spammer compromises a legitimate account in a domain with all 3 records, as when the mail is sent out via that domain, it will pass all checks on the destination end, as it was sent from a domain with valid records. To prevent this, you’ll need to set up rules to detect outgoing spam and block it from being sent. Each mail server will have different instructions on how to do this.

Office 365 and G-Suite all include records for SPF, while DKIM takes a few more steps to set up in Office 365. G-Suite also supports DKIM as far as I know, but since I don’t use the product, I don’t know how hard or easy it is to set up.

While nothing is ever perfect in the war against spammers, a huge amount of junk mail could be stopped cold if more domains published valid SPF, DKIM and DMARC records. Banks and financial institutes that are a favourite target of fraudsters could save themselves a lot of grief by having destination domains reject all mail that isn’t legit. IP block lists and content filtering will remain an important part of the game, but if more junk mail could be stopped at the edge before being accepted and processed, the better off the entire internet will become.

Categories: Internet, My tips and tricks Tags: , ,

Sage Pastel Xpress/Partner V12 and 64 bit Outlook

EDIT: After migrating the department to V17, there were no issues out the box. V17 has a much newer DLL file installed out of the box, which interfaces with 64 bit Outlook just fine.

The Sage Pastel Xpress and Partner products pretty much rule the South African landscape for accounting packages. Almost everywhere you go, you’ll find some Pastel product keeping the books up to date. Our school is no exception, running Partner for the 3 ladies in our accounts department.

With my recent move to Office 365 for mail, I installed Office 2016 64 bit edition on the PC of the debtors clerk to access her mail in Outlook. No problem there, everything worked as it should. However, a few days later she called me back as she was unable to send statements out of Pastel Partner, as the PC now through up an error message when Partner tried to invoke Outlook. This wasn’t a problem in the past, as we’ve only ever used the 32 bit editions of Office. Office 2016 is the first time we’ve installed the 64 bit edition of Office.

It turns out that out of the box, Partner V12 can’t interface with Outlook 64 bit. I don’t have a 32 bit edition of Office 2016 at work, so I needed to get the functionality restored. Luckily, a bit of internet searching revealed the answer: use a replacement DLL file on the Partner installation disk. The process is as follows:

  • Close Partner and Outlook.
  • Copy the NewMail.dll file from the Pastel disk\Utils\Outlook 64-bit folder to C:\Program Files (x86)\Common Files\Softline Pastel. Overwrite the existing file.
  • Run the Component Setup utility in the Pastel folder in the Start menu. This will briefly re-register files, including the replaced DLL file.
  • Try to mail any statement from inside Partner, it should now invoke Outlook correctly.

I checked a Partner V11 disk and this didn’t contain the DLL file, so I assume it only started being introduced with V12. It’s possible that using the DLL file from the V12 disk would work with previous Partner versions going back a while, but I don’t know how compatible or reliable it will be. I have yet to test Partner V14 or V17 to see if they are compatible out the box or will also need the DLL file replaced. Since V14 and V17 were digital downloads with no extra folders, it’s going to prove interesting when I migrate the accounts department.

Lessons learned from migrating to Office 365

May 30, 2017 1 comment

My migration of staff email accounts from our onsite Exchange Server to Office 365 continues as I write this, though now at a somewhat quicker pace. With just under 50 mailboxes left to move, I should be done by the the end of this school term. So far the move has been mostly trouble free, with no email being lost. There have been some small incidents that have helped to shape future mailbox moves and have provided valuable lessons. In no order, here’s some of what I’ve learnt along the way:

  • If you plan to migrate your user’s existing mailboxes up to the cloud, you absolutely need a fast internet connection. 20Mbp/s minimum in both directions, but the faster the better.
  • If possible, get your users to perform mail cleanups before you move their mailbox. The less items in a mailbox, the less time it takes to move said mailbox into the cloud. There’s also less clutter for users after the move, which usually makes people happy, since less clutter is always a good thing.
  • If you are doing a staged migration, try to move as many mailboxes as you can per batch, so that you don’t draw the process out too long. The longer you run two systems, the more risk of something breaking or going wrong along the way.
  • Watch out for user accounts that have been renamed, i.e. people with surname changes. If this isn’t cleaned up properly before being synced to the cloud, it can come back to bite you in the ass. Cue frantic searching and entering arcane commands into Powershell.
  • Users don’t always appreciate or use manuals you may have written. Write a manual anyway, so that you’ve covered your ass.
  • Mailbox moves often don’t happen as fast as you think they should. Budget extra time for a large move.
  • Modern Outlook Web App is a really nice mail client. Light years from Exchange 2007 version obviously.
  • Use Office 2016 for fixed desktop users to connect to Exchange where possible. All previous versions are not going to get the same attention and support from Microsoft in case of trouble.
  • Office 2016 perpetual (i.e. the version you volume license and uses MSI installer) won’t get feature updates over its lifespan. This means no new and cool features like Focussed Inbox.
  • Some programs that interface with Outlook don’t like the 64 bit version of Office.
  • Direct users to the stand alone Outlook apps on Android and iOS. The built in mail client should connect with too much hassles, but Android and Exchange have always had a slightly rocky relationship in my view.

I’m in the process of moving the last giant mailboxes over in the coming week. Once that’s done, the pace of migration should go up as I move other users over with more “normal” size mailboxes. Once everyone has moved, it’s a case of testing to make sure everything is ok, then changing MX records to cut over for direct email delivery to the cloud and to cut out mail coming onsite and then back out again.

The slow migration to Office 365

When it comes to corporate mail servers, many would argue that Microsoft Exchange is the king of the hill. It’s a behemoth of a product that powers so many offices around the world providing vital features. If set up correctly, Exchange has been one of those products that in my experience just hums along quietly, doing its job without demanding a lot of attention.

At the end of 2009 when my first colleague and I migrated my school’s network, Exchange 2007 was our mail server of choice. Not only would it provide everything we needed, it would offer many new features to a school that was used to using a very broken Pegasus Mail/Mercury mail server/Novell Netware combination. It also helped that we got Exchange free of charge under the national SA Government agreement with Microsoft, which ended about 6 months after we installed Exchange.

Since that time, this Exchange server has processed millions of emails and survived moving from a decrepit physical machine to a XenServer implementation to finally ending up in Hyper-V. I’ve probably had 10 incidents or less with this server over the last 7 years. External internet connectivity or issues with upstream mail servers not withstanding, our mail server has done its job perfectly.

Like all things in technology however, there comes a time to move on. The web interface of Exchange 2007 has become really dated and leaves you tied to Internet Explorer for the best results. Those were the days before Microsoft became cross browser friendly, where the “lite” version of webmail was seriously crippled. While I would have upgraded us internally to a later version, we couldn’t afford an upgrade. After the government agreement expired, we were stuck. Quotes I received for updates versions made my eyes water and no one could quite work out how to price software for schools. I think every reseller I contacted only knew how to deal with the corporate world.

In the intervening years, Microsoft essentially resolved my dilemma by introducing and refining Office 365 for Education. Originally billed as Live@Edu, the product provided some nice perks – 50GB mailbox, huge (then called) Skydrive storage etc. The problem was that the product lacked unity and cohesiveness at the time. Live@Edu folded into Office 365 and things have only gone up and up since then. For no cost to us, we get access to the latest version of Exchange, albeit Exchange online, 50GB mailbox, superior spam filtering, access to Microsoft Teams and all the other applications available for education users. As long as their is competition with Google’s G-Suite for Education, we all stand to benefit from that rivalry which forces Microsoft to up their game.

Towards the end of 2015, I decided to migrate all my student mailboxes over to the cloud since students had miniscule amounts of mail compared to staff. It got their mailboxes offsite and gave me some valuable experience on how the migration process would work. It took some reading up on how to do it, but the process is something like this:

  • Sync your on site Active Directory to 365 with the new Sync Tool. The new tool is far better than the old version and what was possible in the earlier days.
  • If you want to be able to simply connect to on premises Exchange and migrate the mailbox like that, you need to have a working Outlook Web Access instance running, secured by a SSL certificate. This lets 365 sync the selected users mailbox to the cloud.
  • The process takes a while, especially over slow connections. The faster internet speeds you have, especially upload speed, the better.
  • You are limited to either a cutover or staged migration for Exchange 2007. Cutover is defined as moving everyone at once then changing DNS MX records so that mail flows directly to 365. Staged is slower, where you move some mailboxes at a time and still use the onsite server as the engine for routing mail. There’s slightly more work with staged, but it lets you be methodical and careful.
  • You can upload Outlook PST files as another method of moving mailboxes, but it’s the same issue as an online migration – you need good uploading speed.

This year I started moving staff mailboxes over for the first time. I had only planned to start once our fibre optic internet connection was in, but the unexpected delays in getting our line in has pushed me to start now already, even over our horrible ADSL connection. I’ve now synced about 10 staff mailboxes over and given staff a manual on how to use the new interface. Some are familiar with it already having had access via their universities or other institutes. The real problem is identifying users who can adapt to the new interface and give feedback on the manual. This is easier said than done when you still have some staff who can barely work with the existing system, 7 years after it went online…

Eventually my goal is to have moved all mailboxes over the cloud, with not one email having been misplaced during the journey. Once that is done, I intend to decommission my on site Exchange server, as well as the actual Windows VM it’s running in. It will be good to not have to support Server 2008 as well, one less old OS to worry about.

In short, there’s precious little reason to have an onsite Exchange server anymore if your internet connection is fast enough. Microsoft does a better job of server uptime than what we can do on our own, they have better spam filtering and they provide a package of products that is not only compelling, but free for education as well. The only real reason to have onsite Exchange anymore is because of privacy or regulatory concerns or if you need some sort of feature that Exchange Online can’t provide.

Upgrading Windows 10 via WSUS

November 15, 2016 Leave a comment

Windows 10 is supposedly the “last” consumer Windows edition Microsoft will release. While the version will stay as 10, over time the whole OS will mature, grow and mutate into something that will look and feel very different from the original release of July 2015. One side effect of this is that in a corporate environment using WSUS, it becomes possible for new versions of Windows 10 to be deployed as an in place fully automatic upgrade, the same way any other patch or service pack is installed. I was curious to see how this worked, so I approved the Anniversary Update (also known as version 1607) for installation at work and let my PC download the update.

Sure enough, the process was the same as what my home PC went through when it upgraded to the 1607 update. A couple of update screens and quite some time later, I was back at my desktop, duly upgraded. Everything was still in place, bar the RSAT pack which had to be updated to a version compatible with v1607. Overall, an extremely smooth and hands free process, just time consuming. I imagine it would easily take twice or thrice as long if the machine runs on a mechanical HD and not a SSD.

That being said, there was one major problem with the 1607 update – checking for updates from WSUS broke due to a bug in 1607. Windows 10 1607 would start to search for updates from the configured WSUS server, only to have multiple services in the background crash repeatedly, with no indication to the user. To the end user, it simply looks like the search is stuck at 1% and never moves from there. Apparently, if one leaves the process running long enough, updates will eventually download. This is obviously an unacceptable bug and Microsoft were made aware of it. They promised a fix in one of the monthly update roll ups, which was subsequently delivered and verified as having fixed the problem. Now you have a chicken and egg situation: deploy the 1607 update via WSUS, but then struggle after that, since you need the Cumulative Update to fix the problem.

You could manually install the update, but this becomes unwieldy in a large organisation. If deploying Windows 10 via deployment tools, you could make sure that the base image has the update injected already, which prevents the issue from cropping up in the first place. Sadly, the 1607 update is delivered from WSUS as an encrypted ESD file. While it is possible to decrypt this and inject the update, I don’t know if it’s possible to convert that back in to an ESD file. Even if you could, the checksum wouldn’t be valid and WSUS would probably fail to work with the modified file.

There’s always a possibility Microsoft could revise the 1607 update in WSUS so that the ESD file comes with the last Cumulative Update installed so that it works correctly out the box. I recall something like this happening with the November 1511 update, which I declined as it was another 3-4 GB download. Unfortunately, one doesn’t know when or even if this will happen. It’s also possible the problem will never be fixed. With the Creators Update due out early 2017 (March?) it’s possible that Microsoft uses that as the new baseline. If I’m correct, once the Creators Update is approved in the WSUS console, it will supersede the Anniversary Update, so the problem should be solved by bypassing the 1607 update.

I look forward to eventually rolling out Windows versions like this, though I think it will be beneficial if every computer had a SSD inside it first. Mechanical hard drives really do slow things down these days. A nice side effect of this is that Windows shouldn’t end up suffering from “Windows rot” as the Windows directory is replaced with each major upgrade. This should keep performance up compared to something like Windows 7 that gets bogged down after years worth of updates. Interesting times ahead…

Categories: Software Tags: ,