Archive

Archive for the ‘Networking’ Category

UEFI booting observations

It’s exam time at my school, which means that things quieten down a bit during said period. This leaves me with some free time during the day to experiment and learn new things, or attempt to do things I have long wanted to do but have not had the time. I’ve used the time during this last week to play around with deploying Windows 7 and 10 to PC’s for the purpose of testing their UEFI capabilities. While Windows 7 can and does UEFI boot, it really doesn’t gain any benefits over BIOS booting, unlike Windows 8 and above. I was more interested in testing out the capabilities of these motherboards, so I could get a clearer idea of hardware issues we may have when we move to Windows 10.

Our network comprises of only Intel based PC’s, so all my experiences so far are based off of that particular point. What I’ve found so far boils down to this:

  • Older generation Intel 5 & 6 chipset series motherboards from Intel themselves are UEFI based, but present interfaces that look very much like the traditional console type BIOS. The only real clue is that under the Boot section, there is an option to turn on UEFI booting.
  • These older motherboards don’t support Secure Boot or the ability to toggle on and off the Compatibility Support Module (CSM) – the UEFI version on these boards predate these functions.
  • I have been unable to to UEFI PXE network boot the 6 series motherboard, haven’t yet tried the 5 series boards. While I can UEFI boot the 6 series to a flash drive/DVD/hard drive, I cannot do so over the network. Selecting the network boot option boots the PC into what is essentially BIOS compatibility mode.
  • The Intel DB75EN motherboard has a graphical UEFI, supports Secure Boot and can toggle the CSM on and off. Interestingly enough though, when the CSM is on, you cannot UEFI PXE boot – the system boots into BIOS compatibility mode. You can only UEFI PXE boot when the CSM is off. This is easy to tell as the network booting interface looks quite different between CSM and UEFI modes.
  • Windows 7 needs the CSM mode turned on for the DB75EN motherboards if you deploy in UEFI mode, so that it can boot, at least from what I’ve found from using PXE boot. If you don’t turn CSM on, the boot will either hang at the Windows logo or will moan about unable to access the correct boot files. I have yet to try and install Windows 7 on these boards from a flash drive in UEFI mode to see what happens in that particular scenario.
  • I haven’t yet had a chance to play with the few Gigabyte branded Intel 8 series motherboards we have. These use Realtek network cards instead of Intel NICs. I’m not a huge fan of Gigabyte’s graphical UEFI, as I find it cluttered and there’s a lot of mouse lag. I haven’t tested a very modern Gigabyte board though, so perhaps they’ve improved their UEFI by now.

UEFI Secure Boot requires that all hardware supports pure UEFI mode and that the CSM be turned off. I can do this with the boards where I’m using the built in Intel Graphics, as these fully support both CSM mode and pure UEFI. Other PC’s with Geforce 610 adapters in them don’t support pure UEFI boot, so I am unable to use Secure Boot on them, which is somewhat annoying, as Secure Boot is good for security purposes. I am probably going to need to start making use of low end Geforce 700 series cards, as these support full UEFI mode, so will support Secure Boot as well.

It’s been a while since we bought brand new computers, but I will have to be more picky when choosing the motherboards. Intel is out of the motherboard game and I am not a fan of Realtek network cards either – this does narrow my choices quite a bit, especially as I also have to be budget conscious. At least I know that future boards will be a lot better behaved with their UEFI, as all vendors have had many years now to adjust to the new and modern way of doing things.

The long hunt for a cure

At the end of March 2014, our school took ownership of a new Intel 2600GZ server to replace our previous HP ML350 G5 server which was the heart of our network. The HP had done a fantastic job over the years, but was rapidly starting to age and wasn’t officially supported by Windows Server 2012 R2. Our new server has 32GB of RAM, dual Xeon processers, dual power supplies, 4 network ports and a dedicated remote management card. Although a little pricier than what I had originally budgeted for, it matched what the HP had and would earn its keep over the next 5-7 years worth of service.

After racking and powering up the server, I installed firmware updates and then Server 2012 R2. Install was quicker than any other server I’ve done in the past, thanks to the SSD boot volume. After going through all the driver installs, Windows Updates and so on, the server was almost ready to start serving. One of the last actions I did was to bond all 4 network ports together to create a network team. My thinking was that having a 4Gb/s team would prevent any bottlenecks to the server when under heavy load, as well as provide redundancy should a cable or switch port go faulty. Good idea in theory, but in reality I’ve never had a cable or port in the server room go bad in 6+ years.

Looking back now, I’m not sure exactly why I bothered creating a team. While the server is heavily used as a domain controller, DHCP, DNS and file server, it never comes close to saturating 1Gb/s, let alone 4. Almost every computer in the school is still connected at 100Mb/s, so the server itself never really comes under too much strain.

Either way, once everything was set up, I proceeded to copy all the files across from the old HP to the new Intel server. I used Robocopy to bulk move files, and in some cases needed to let the process finish up over night since there were so many files, especially lots of small files. Data deduplication was turned on, shares were shared and everything looked good to go.

When school resumed after the holidays, the biggest problem came to light right on the first morning: users being unable to simultaneously access Office files. We have a PowerPoint slideshow that is run every morning in the register period that has all the daily notices for meetings, events, reminders, detention etc. Prior to the move, this system worked without fault for many years. After the move, the moment the 2nd or 3rd teacher tried to access the slideshow, they would get this result:

WP_20140409_001
Green bar of doom crawling across the navigation pane, while this odd Downloading box would appear and take forever to do anything and would tend to lock Explorer up. Complaints naturally came in thick and fast and the worst part is that I couldn’t pinpoint what the issue was, aside from my suspicion that the new SMB3 protocol was to blame. I had hoped that the big Update 1 update that shipped for Windows 8.1 and Server would help, but it didn’t. Disabling SMB signing didn’t help either. At one point, my colleague and I even installed Windows 8.1 and Office 2013 on some test machines to try and rule out that possibility, but they ended up doing the same thing. As a stop gap measure, I made a dedicated Notices drive on the old HP, which was still running Server 2008, which ran fine with concurrent access to the same file. Online forums weren’t any real help and none of the other admins in Cape Town I spoke to had encountered the problem either.

In the last school holidays just gone by, we finally had a decent gap between other jobs to experiment on the new server and see if we could correct the problem. I broke the network team, unplugged 3 of the 4 cables and disabled the LACP protocol on the switch. After reassigning the correct IP to the now single network port, we did some tests on opening up files on 2 and then 3 computers at the same time. We opened up 15MB Word documents, 5MB complicated Excel files, 200MB video files and more. The downloading box never showed up once. Unfortunately, without heavier real world testing by the staff, I don’t know if the problem has been resolved once and for all. I am intending to move the Notices drive during the next school holiday and we will see what happens after that.

Chalk one up for strange issues that are almost impossible to hunt down.

Bandwidth buffet

For any person now in their late 20’s to early 30’s, the sound of a dial up modem should be a familiar, yet fading memory. Connecting to the internet more often than not meant listening to those squawking devices and hoping that the connection was clean and free of noise that would slow the connection down. As time went on, ADSL arrived, 3G arrived, HSDPA arrive, VDSL and now fibre have arrived. Slowly but surely, the trickle of data coming down the pipes has become a raging torrent, at least for those that can afford the higher speed options.

Driven by the laws of supply and demand, websites have evolved to become a lot more feature rich and complex than in years gone by. Images have gone from highly compressed gif and jpeg files into higher resolution jpeg and png files. Internet video, once an incredibly frustrating experience involving downloadable clips and QuickTime, Real or Windows Media Player has largely evolved into slick, easy to use web based players. Of course, video has also crept up from thumbnail size resolutions all the way up to the current 4K. It’s become really simple: the bigger your pipe, the more you are going to drink from the fountain by consuming rich media, online streaming and more. Creating and uploading content is also more viable than ever before, which means symmetrical connections are becoming far more important to the end user than they ever were before.

Take the following picture below, taken from my school’s firewall logs for 1 January – 30 April 2015:

Stats

That’s a total of 970GB of traffic on 1 ADSL line, the speed of which has fluctuated during the course of the year due to stability issues. We have another ADSL line which is only used by a few people, some phones and tablets, but I don’t have usage stats for that line. However, taken all together, our school has definitely used over 1TB of data in 4 months. At this rate, we may end up pushing close to 3TB by the year’s end. Also keep in note that these stats are without any wide scale WiFi available to students. I shudder to think of what the numbers will be once we have WiFi going, or even if we get a faster upload so that things like Dropbox, OneDrive and so on become viable.

Here’s a second interesting picture as well:

Stats2

Of all the web traffic going through the firewall, 82.5% of the traffic was unique or uncacheable due to its dynamic nature. In earlier days, caching statistics were higher since websites were less dynamic and had far more static HTML code, less scripts etc. That being said, the cache did at least manage to serve about 15% of the total web traffic. Every little bit helps when you are on a slow connection.

In the end, it all goes to show that the more bandwidth you have, the more applications and users are going to end up making use of it. Thankfully, bandwidth prices are much lower than they have ever been, though on some connections the speed is throttled to make sure that the end user doesn’t gorge themselves to the detriment of other users.

Categories: Networking Tags:

Migrating a VM from XenServer to Hyper-V

April 29, 2014 4 comments

My last two posts have spoken about my former XenServer and the fact that I replaced XenServer with Hyper-V. What I didn’t explain was how to actually migrate the exiting virtual machines off XenServer and into Hyper-V.

If you are lucky enough to have both XenServer and Hyper-V servers running concurrently, the process goes a lot quicker than the way I had to do the task. In my case I had to migrate VM’s off the server, install and configure Server 2012 R2, install Hyper-V and then set up my VM’s. Not a difficult task, just time consuming. With two servers, you can move the exported VM hard drive directly to Hyper-V and get going a lot quicker. Moving 120+GB VHD files does take some time, even on gigabit Ethernet.

On XenServer, I had 5 VM’s. I determined that I would be migrating 3 of them, while I would rebuild the other 2 from scratch once Hyper-V was up and running. I searched the net for info on how to do a migration of the existing VM’s, but there wasn’t a lot of info out there. I certainly couldn’t find any tools to do the job automatically, which is a pity as there is plenty of tools that will convert from Hyper-V to VMware and vice versa, or do a Physical to Virtual (P2V) migration.

I did come across info that said that once I had exported the VM, the VM would be able to boot in Hyper-V without problems. I tried exporting a VM directly from XenCenter, but it wouldn’t export VHD files. Exporting an OVF file and then converting that was possible, but it would be even more time consuming while waiting for the conversion.

Eventually the solution I found was to use Microsoft’s Disk2vhd program. It’s a small program that will snapshot the VM inside XenServer and push it to a VHD or VHDX file, which is the hard drive format used by Hyper-V. The program will create one file per physical drive, not per partition. So a VM with a hard drive with 3 partitions on it will actually create only 1 file.

Disk2vhd

Store the VHD file anywhere except on the machine being captured. Depending on the size of the drives, the process could take a few minutes to a few hours. Having a gigabit network helps in this regard, as the program will max out your network connection if you save the file via the network.

If you are capturing a server running Exchange or SQL or acting as a TMG firewall, stop all those services before capturing so that you don’t have any inconsistencies after the machine is booted up again.

Once you have your VHD file, this can be copied onto your Hyper-V server’s local storage or cluster storage or whatever you are using. Set up your VM in Hyper-V using Generation 1 hardware (unless you captured Windows 8/Sever 2012) and boot up the VHD file. Hyper-V will boot the system up and once you reach the desktop, Windows will install some drivers. Before you reboot, go to Programs and Features and remove all the XenServer tools and drivers. Reboot the VM, then install the Hyper-V tools, reboot etc.

By the time you are done, the VM should be running stably inside Hyper-V. You will need to reactivate your copy of Windows due to the massive hardware changes, as well as set up any static IP addresses again, as the virtual network card from XenServer is obviously not carried over to Hyper-V. Watch out for your anti-virus product (if you had one installed previously.) After I migrated my Exchange Server over, the VM started locking up and rebooting. Turns out that the specific version of NOD32 that was on the server was conflicting with the Hyper-V tools.

Speaking of my Exchange server, that particular system has gone from being on Pentium 4 class server into XenServer, through 3-4 XenServer OS upgrades and now into Hyper-V. Pretty amazing when you think about how it’s managed to survive all these jumps. Even more surprising when you consider it’s running on Server 2008 (not R2) which is basically the Vista code base.

I hope this helps someone out there who is contemplating making the move themselves, or is simply doing research. Always helps to have someone experiment before you and take those nervous steps first.

Intel network cards rock

February 5, 2014 Leave a comment

In the past on this blog, I’ve bitched about Realtek network cards and their sometimes incredibly buggy drivers that lead to many a head scratching problem. That being said, I ran into a case last week that was even more annoying.

We have been replacing the last of our Windows XP computers with new or reshuffled Windows 7 PC’s. One of the XP computers I took out was sufficient to run Windows 7 at a useable speed. Said computer consisted of 2GB RAM, 160GB SATA hard drive, Geforce 7100 graphics card, Intel Core 2 Duo E7300 and the Asus P5K SE/EPU motherboard. By no means modern, but enough to last a few more years as a light duty Windows 7 box.

It should be noted that the motherboard isn’t Windows 7 certified, but that’s usually not too much of a major issue. The board was certified to run up to Vista 64 bit, so installing 7 wouldn’t be a stretch. In fact, I have one of these exact same systems at home that runs 7 fine.

As is now the norm, I was preparing to boot over the network to let my custom MDT Deployment task run. Went into the BIOS, made sure the network card boot ROM was enabled. Next step is to boot off the network. This is where my plan came to an abrupt halt. After going through POST, the network card would sit waiting to detect the network for a short while, before failing to PXE boot and moving on to booting off the hard drive. No matter what I did, I was unable to boot off the network. Soon as the PC got into Windows, the card was fully functional and I could use the network. Inside Windows the card is labelled an Atheros L1 Gigabit adapter, but when you are waiting for it to PXE boot, it labels itself an Attansic L1 adapter. No big deal really, since Atheros bought Attansic.

I rebooted again to try some more tricks, when something caught my eye. Up until about halfway through the Windows XP boot screen, the network card has no link light, and the port on the switch also has no link light. It didn’t take long to figure out that the network card was not initialising until it was almost in Windows, after the drivers had been loaded. This meant that this computer would never be able to PXE boot, and no suggestions off the internet helped. Someone spoke of injecting the drivers into the boot image in MDT, but this would do no good since the card does not initialise until Windows is loaded. In short, network booting is broken on this motherboard, and there is no fix for it. There is no updated BIOS to help either.

Now that I think about it, I could have made a boot ISO with MDT and used it to boot the computer, but that would entail burning a DVD which in my mind was a waste. My idea instead was to slot in a replacement dedicated network card and network boot using that. We had an old Intel Pro 100 card laying around. Installed the card, went into the BIOS and immediately had the option of being able to boot using the Intel card’s boot ROM. To cut a long story short, the Intel card just worked, and I was able to finish my task.

This incident just highlighted again why I prefer Intel network cards on my network. They work reliably, have a good set of features, solid drivers, and their network booting has always worked for me without fail.

Realtek drivers and Microsoft Deployment Toolkit

June 2, 2013 1 comment

Our school recently purchased 3 new computers using Mini ITX motherboards. I originally wanted an Intel based motherboard, but it wasn’t in stock and our supplier ended up getting us Gigabyte H77 Wifi motherboards that had more bells and whistles on it. However, said motherboard has Realtek network cards on-board, specifically the 8168 model. Nothing wrong with this card, except getting it to work with Microsoft Deployment Toolkit 2010.

These days, the computers I’m in charge of at the school are setup using MDT rather than installing from a DVD drive. The process is largely automated after the initial first steps, and my task sequence has been set up to pull in all the latest Windows patches from our WSUS server. However, this all depends on the network card driver being able to work in the Windows PE environment. The Realtek card is not natively supported by the Windows PE version of MDT 2010, so you have to get drivers installed. The same goes for any network card not supported out the box by Windows 7, as that is what the Windows PE version of MDT 2010 is built from.

Thankfully getting things like storage and network drivers injected isn’t a difficult task. I used the drivers off of the support DVD and injected them into the Windows PE image. I started up the new workstation, only to be greeted by the error message below (sorry about the poor quality of the photo, it was taken using my phone)

WP_20130517_001

I found this odd, since the drivers were off the motherboard DVD, and they work fine in Windows itself. When you double click on a driver file in MDT, you can see all the models that the driver supports. It only became clear after a while what the potential problem is. The error message has a “&REV_06” part at the end of the vendor ID. When I looked closer, I realised that though there was indeed a line with the exact same vendor ID, it was for “&REV_01”

This means that while the driver will indeed support the network card, Realtek or Gigabyte have forgotten to add this extra line into their *.inf file that comes with the driver. I copied and pasted the existing line, and modified the end part to read REV_06. This in theory should add support for this chip in Windows PE and let me image these computers.

However, my theory did not work out. Despite modifying the driver package and completely regenerating the Windows PE images, I could not get the client PC to pick up the network card at boot. Manually loading the driver using drvload at the command prompt worked, but it defeats the purpose of using MDT in the first place.

My hope is that when I upgrade to MDT 2012, the problem will be solved. Since the Windows PE version in MDT 2012 will be based off of Windows 8, I am hoping that there will be a generic or native driver built in that will avoid me having to inject Realtek drivers into the image. While I would prefer to only use Intel network cards, it sometimes simply isn’t possible – many laptops I’ve seen lately are all using Realtek network cards for their Ethernet connection.

Tutorial on how to flash the Telkom Mega 105WR

April 21, 2012 2 comments

In my previous post here, I detailed how I flashed the firmware of my old Telkom Meaga router. Per a request in the comments to that post, I’m writing a quick tutorial on how to flash your Telkom Mega. Please be aware that flashing the device with the RouterTech firmware will invalidate any remaining warranty on the router, and there is a chance you can brick the router.

Do as follows:

  1. Register at routertech.org and sign in. Follow the download links to where you are able to download the firmware. For the Telkom Mega 105WR, you will want to download the “routertech-ar7wrd-1350A-pspboot-firmware-4ports-20120130” or similar file, depending on the date. Unzip this file to a folder anywhere.
  2. Login in to the Mega 105WR’s web page. Click the Advanced menu button at the top and then on Restore to Default on the left hand side of the page. This will reset the router to factory defaults. This is a required step.
  3. Alternatively, press and hold the physical reset button on the back of the router for about 30 seconds. This will do the same thing as step 2, and is useful if you can’t get into the router’s web management page.
  4. You should now be able to log into the router’s web page at http://10.0.0.2 Once there, click on the Advanced button on the top, and then click Firmware Upgrade on the left hand side of the page.
  5. Click the Browse button to find the RouterTech firmware.
  6. Chose the “RouterTech_3.7.1B_1350A_20120130_2.97_AR7WRD_psbl_firmware.upgrade” file in the root of the unzipped folder from step 1.
  7. Click on the Update Gateway button.
  8. Walk away from your PC, or play cards for roughly 10 minutes. DO NOT attempt to do anything on the router, ping it and so forth.
  9. After 10 minutes, refresh your computer’s IP address. It should now be in the 192.168.1.x range. If this is so, then the firmware has been flashed correctly.
  10.   Log into the router’s web page at http://192.168.1.1 The default user name and password is Admin/Admin
  11.   Click on the Advanced Menu button at the top, then on Restore to Default. The router will factory reset again. This is the last time you will need to do this step.
  12.   To make the LED lights on the front work properly, you will need to change the LED configuration file. Go to the System button on the top menu, and then RT Configurations on the left. When you see the option to change the LED configuration file, use the bewan 700 (or something similar, the name escapes me right now) model. Restart the router when asked, and you are done.

Now you can further configure the router as you need. All the options of the old Mega firmware are there, along with plenty of other options. I must stress that I haven’t yet let the Mega run for extended periods, so I can’t tell if the RouterTech firmware will fix the problems the original firmware had. Still, it’s worth a shot and it may help you out. I’m always open to feedback.