Reduce (Shrink) raw image (.img) size of a Windows Virtual Machine

If you have Windows running in a virtual machine (VM), you may find yourself wanting to reduce the overall disk-footprint of the VM’s virtual disk on your host.

For example: because I had created my Windows VM by copying a raw disk in it’s entirety, I had a 120GB VM raw image (.img) file for a Windows installation that was only taking up about 20GB of actual space.

It took a bit of research for me to piece together all of the steps required to reduce my 120GB image down to a reasonable size. Here’s the steps I took.

Assumptions

  • You’re using KVM as your hypervisor. There may be some tips in here that are useful for other VMs, but I’m assuming you’re using KVM.
  • Your .img file is in a raw image file format. There may be some tips in here if you’re trying to shrink a qcow2 image file, but you should probably not follow this verbatim.
  • This process is risky – be careful and don’t blame me if things go amiss. And make backups before you start.
  • You are okay deleting the “recovery partition” on your Windows VM image. This process will most certainly delete the recovery partition (if it exists). For most people, this isn’t a big deal… if you need to recover Windows, there are other ways to do it and the partition was just wasting space.

Reduce the size of your Windows partition

This turns out to be more involved than you’d think. That’s mainly because there are several steps to take to enable Windows to pack all of your data into one end of the disk partition so it’s size can be reduced.

These steps are useful if you’ve ever encountered the “You cannot shrink a volume beyond the point where any unmovable files are located” error when trying to shrink a partition in Windows.

These steps are taken from within your running Windows VM.

Prepare your partition to be shrunk

This is a four step process:

  1. Disable hibernation
  2. Disable the pagefile
  3. Disable system protection
  4. Defrag your ‘hard disk’

For steps 1 through 3, please see this excellent post on “How to shrink a disk volume beyond the point where any unmovable files are located“.

For step 4 (Defrag your ‘hard disk’), this turned out to be a tricky step with a very simple answer. Most folks will point you to the Windows UI to “Optimize” your disk. However, if Windows thinks your C: drive is an SSD it (correctly) won’t do a defrag on it.

To get around that, simply run defrag from an elevated / admin command prompt!

defrag C: /U /V

Consider running defrag twice – once as above, and once again with the /X option is to consolidate free space.

Shrink your Windows partition

  1. Launch the Windows Disk Management console (Protip: you can get to it by right clicking in the lower left hand corner of your screen and selecting “Disk Management”)
  2. Right click on the main partition of your C drive.
  3. Select Shrink Volume
  4. After some thinking, you should be able to set the size of the disk as small as you’d like. Make your selection and click “Shrink”
  5. Make note of the final size of the of all the partitions on the disk – you’ll need that number for when you go to reduce the .img size in the next step. It’s safer to round-up than down!

Shutdown your Windows VM

You’ll want to shut down Windows now. You don’t want it running when you’re cutting down the .img file size.

Shrink the image (.img) file

This is where the magic happens… and it’s not that complicated.

  1. Locate your .img file. In my case, I’m running unRAID, so my VM images are all in /usr/mnt/domains by default.
  2. Make a copy of your VM’s .img file. You’ll want it in case this screws up.
    cp ./vdisk.img ./backup_vdisk.img
  3. Shrink the image to the size you want it to be. Remember, this has to be bigger than the total size of the Windows partitions on the disk. So, if there’s a 1GB partition and a 30GB partition, it’s probably safe to shrink the image down to 32GB to be safe.
    qemu-img resize vdisk.img 32GB
  4. Restart your VM and make sure it launches correctly.

Finishing Up

Enable Paging

Because you’re running Windows in a VM, it’s safe and probably preferred to leave hibernation disabled. You’re likely fine leaving system protection disabled as well.

But you should not go without re-enabling paging – follow the instructions here to turn back on paging: How to shrink a disk volume beyond the point where any unmovable files are located“.

Expand the size of your Windows partition

If you rounded up when shrinking your .img file, you probably have some extra unallocated space a the end of your Windows partition. To make use of it:

  1. Launch the Disk Management console
  2. Right click on your Windows partition
  3. Select Extend Volume and follow the dialog to increase the space of the partition

Consider converting from raw .img to qcow2

There are pros and cons to each format; some of the pros of qcow2 are the ability to snapshot the VM and sparse allocation (which can further save space). However raw can be faster. I’m sticking with raw for now.

Convert a Windows Installation into a unRAID (KVM) Virtual Machine

Because I was turning my Windows 10 Pro server into an Linux (unRAID) machine I already had a Windows installation running on the bare metal that I had spent a fair bit of time setting up. I didn’t want to lose that installation and the work put into it. So, rather than starting over with a fresh Windows installation I took the operating system that was already installed, copied it, and started running it as a virtual machine (VM). Here are the steps I took to convert that Windows installation into a functioning VM.

Most of these steps come from the well-written unRAID Manual on Physical to Virtual Machine Conversion – the main thing I add in here are the Windows-specific steps for reducing the size of the virtual machine image.

They’re also useful for anyone running KVM as a hypervisor – not just specific to unRAID.

Before you begin

  • Makes sure you have access to your Windows license. If you upgraded from Windows 7 or 8 to Windows 10 for free this is really important. If you haven’t already, follow the steps here in “Before you shutdown Windows”.
  • Your hardware must be capable of processor virtualization (Intel VT-x or AMD-V), and it must be enabled in your BIOS. Depending on what you plan to do with your VM, you may need to meet other hardware requirements.
  • unRAID must have VMs enabled and configured. An important step here is to download the VirtIO Drivers ISO
  • unRAID needs to have User Shares created to hold the virtual machine images; by default, these are created for you by a new installation at /mnt/user/domains/
  • Check to make sure that your User Share (above) has enough room for the full size of your Windows Operating System hard drive. You’re going to copy the entire hard drive including the empty space.

By default, most of these things were done already on my machine, but it’s good to check them before proceeding.

If you’re unfamiliar with unRAID and how to take those steps, they are described in the “System Preparation” section of the unRAID manual quite well.

Add a new Virtual Machine

From your unRAID web UI (e.g. http://tower), perform the following steps.

  1. On the VMs tab, click the Windows 10 template (or template for your version of Windows).
  2. Click on the switch for the Basic View to toggle it to the Advanced View (in the upper-right corner of the screen) if it isn’t already set.
  3. Configure the VM with the following settings:
    1. Name your VM whatever you wish: For this example, I’m using “Windows 10”
    2. CPU Mode: Host Passthrough
    3. Logical CPUs: However many you wish (at least one)
    4. Initial Memory / Max Memory: at least 2GB is recommended
    5. Machine: i440fx-* (whatever the newest is)
    6. BIOS: SeaBIOS
    7. Hyper-V: Yes
    8. OS Install ISO: Leave empty
    9. VirtIO Drivers ISO: Select the virtio-win*.iso that (should have been) downloaded as part of enabling VMs on unRAID.
    10. VirtIO Drivers CDRom Bus: IDE
    11. Primary vDisk Location: Auto
    12. Primary vDisk Bus: IDE
    13. Primary vDisk Type: raw
    14. Primary vDisk Size: 0G (You’ll be overwriting this file in the next step)
    15. Graphics Card: VNC
    16. VNC Video Driver: QXL
    17. VNC Password: Set it if you’d like – this is how you’ll access the ‘screen’ of the running VM
    18. VNC Keyboard: Set to the right language for your OS
    19. Sound Card: None (or select one if your device supports passthrough)
    20. Network MAC: this is set randomly, though you can change it.
    21. Network Bridge: br0
    22. USB Devices: I recommend leaving all unchecked for now.
    23. USB Mode: 2.0 (EHCI) is what I found worked for me.
    24. Other PCI Devices: I recommend leaving all unassigned until you confirm the VM launches correctly.
    25. Uncheck Start VM after creation
  4. Hit Create.

Copy the OS Drive to an image

Next up, you need to copy the data from your OS drive to an virtual disk image that the VM can boot off of.

  1. Identify the disk that contains your Windows Operating System. You can find it by looking in the unRAID web UI in the Main tab under Unassigned Devices. Look for the drive that you booted windows off of before you installed unRAID. Make note of the disk id (e.g. sdb, sdc, sde, etc.)
  2. SSH into your unRAID system as root. By default unRAID doesn’t have a root password, but you should have set a strong one by now… right?
  3. Enter the following command to convert the physical disk
    qemu-img convert -p -O raw /dev/sdX /mnt/user/domains/Windows\ 10/vdisk1.img
    1. Replace sdX with the drive id for your Windows OS installaton
    2. Replace Windows\ 10 with the name you gave your Virtual Machine (if you need to see what it is run the ls /mnt/user/domains command to see what it’s name is on disk)
  4. Wait. It’ll take a while, assuming you have a reasonably large OS installation disk.

Start your Windows VM & Install the VirtIO Drivers

Because Windows is now running as a Virtual Machine on KVM, it will think there is new “hardware” and will require (and benefit from) having the right drivers installed.

  1. In the unRAID VMs tab, click on your newly created VM and click Start
  2. After the VM has started, you can VNC into the machine and interact with it. You can do this one of two ways.
    1. Click on the running Windows VM and select “VNC Remote”
    2. (Preferred) Install a VNC client like TightVNC and connect to the VM directly. To do so, you’ll need to know the IP address of your server and connect to the VNC port of the VM (which is listed next to the running VM)
  3. In Windows File Explorer, navigate to the VirtIO virtual CD-Rom (likely disk D:\)
  4. Navigate to Balloon > w10 > amd64
    1. Or your specific version of Windows and Architecture
  5. Right click on the .inf file (i.e. balloon.inf) and click install. (You may need to enable the viewing of file extensions to find the right file)
  6. Repeat the above process for each of the following folders
    1. NetKVM
    2. vioserial
    3. viostor
  7. You may want to check Windows Device Manager to make sure there are no devices which have yellow warning exclamations next to them – if so, you’ll likely need to install an additional driver.
  8. When done with the driver installation, navigate to the guest-agent folder and double click on qemu-ga-x64.msi to install the QEMU/KVM guest agent.

Shutdown the VM & update disk settings

Now that you know that the VM boots and you’ve got the drivers installed, you can stop the VM and update the physical disk to use the VirtIO bus which will give you better performance.

  1. Stop the Windows VM. You can do this one of two ways – one, by initiating a Shutdown from within Windows. Or, you can click on the VM image in unRAID and select Stop which will also gracefully shut down Windows.
  2. In the unRAID web UI, in the VMs tab, select the Windows VM and select Edit
  3. Change the following settings:

    1. Primary vDisk Bus: VirtIO
    2. VertIO Drivers ISO: delete the entry
    3. Restart the VM and make sure it’s operating correctly.

Reactivate your Windows license

Windows checks it’s license validity based on your machine’s hardware. Any time you change the hardware, Windows needs you to reactivate your license. You’ll need to run through the activation steps in the VM to re-activate your Windows license because Windows will think you just changed a lot of hardware.

To do that, search for “Activation” from the Windows 10 Start menu. From there, you should be able to follow the prompts to activate your copy of Windows. Somewhere along the way, in smallish font, it’ll ask you if you recently changed hardware. Click that option and go from there – i.e. log into your Microsoft Account, select the machine/license associated with this Windows installation.

(Optional) Remove Unneeded Software & Drivers

Because your old Windows machine had a lot of device specific drivers installed on it to operate on your hardware, you may have a lot of cruft that can now be removed. Log into your Windows machine and take a stroll through the “Add or Remove Programs” menu to see if there’s anything that can be removed. For me, I could uninstall any Intel and RealTek Drivers, ASUS motherboard features, etc. that were no longer relevant. It’s a nice thing to slim up the now VM’d Windows OS.

(Optional) Optimize the Window VM

The unRAID wiki has a good set of steps to consider to optimize your Windows VM. I personally don’t believe in disabling the search indexer, but most of the other tips are worth while.

(Optional) Reduce / Shrink the size of the VM image

Now, you’re probably saying to yourself… geez, I don’t need a XXX GB image file that just happened to be the size of your old OS hard drive just to run Windows. You probably want to slim that down. Here’s instructions on exactly how to do that.

Let me know if I missed anything here.

How-to: Migrate from Windows RAID to unRAID

When I started looking at converting my Windows 10 Pro machine (which was running a two-disk, software RAID 1 in Windows) to an unRAID machine I didn’t find a lot of good how-to guides. Below are some guidelines and step-by-step instructions that will hopefully help if you’re in the same situation.

Basic Setup

Here’s my starting & ending setup. Yours is likely different.

All of this is running on a 5 year old ASUS Maximus IV Gene-Z/GEN3 motherboard with an Intel Core i5-2500k processor and 16GB of RAM.

The 4TB “Security Recordings” disk is used to hold my IP cam footage from my security software – I made space on that disk to temporarily hold the data from my RAID while I was doing the upgrade. More on that below.

Before you get started, it might be good to familiarize yourself with the unRAID manual. It’ll help to have a passing understanding of what an unRAID array and share is, and it’s a pretty good reference for how to do a general unRAID installation.

Migration Steps

These instructions assume you have a Vista-or-later Windows machine with two-disks in a Windows-managed software RAID mirrored configuration.

Before you shutdown Windows

  1. Backup everything. This is a good time to remind yourself that RAID is not a backup strategy. You should already have a 3-2-1 Backup Strategy. If you don’t, just stop reading right here and get one. Getting a good backup is really more important than this. I like Crashplan for cloud and on-site location backups. </soapbox>
  2. Save your Windows license key. If you have Windows 10, this is really really important. If you took advantage of the free Windows 10 upgrade, it’s really easy to lose that free upgrade if you don’t save your license key. The easiest way to keep the key is to create an Admin user that logs in using a Microsoft Account credentials. If you’ve done that right, you should see your PC listed when you log into account.microsoft.com and look in the Devices tab. If your device isn’t there, then you should get that resolved before you move on. Once you re-install Windows later, you’ll be able to log in using the same Microsoft Account and adopt this license key on your newly installed Windows 10 machine.

    Make sure this PC is listed under your Microsoft Account Devices
  3. Copy the data from the Windows RAID to a temporary (NTFS) drive. This has to be a drive you won’t be using as part of the initial 2-disk unRAID. We’ll use this as the source data to copy to the newly created unRAID array.In my case, I had a 4TB drive that I could copy my 1.5TB of RAID data onto temporarily so I could later use those 1.5TB drives for my unRAID array.Some tips:
    1. Protip 1 – make sure the temporary drive you’re copying to is not a Windows Dynamic Disk. You can check what type of disk it is by looking in the Windows Disk Management console. If it’s a Windows Dynamic Disk, unRAID won’t be able to mount it (neither can much else).
    2. Protip 2 – if you don’t have an extra disk big enough to hold the data from your existing array, you can use one of your disks as the source but you have to do a couple of additional steps.
      1. First, you have to break the Windows mirrored RAID apart (in the Windows Disk Management console, right click on the mirror, select “Break Mirrored Volume”). Do Not Remove the Mirror. Breaking it keeps the data in place on both disks. Removing it deletes the data from one of the disks.
      2. Second, you need to convert the disk you’ll use as the source data into a non-dynamic disk. Windows Mirrored volumes are always dynamic, so you’ll have to convert it so unRAID can read it. I used a program called TestDisk. The steps are essentially do a Create > Select Disk > select type “Intel” > Analyze > Backup > Press “Enter” > Write (Y) > Reboot. For detailed instructions, this walk through here was really useful.
      3. Third, when you go to create your unRAID array later, you’ll need to create an array that only has one disk (and no parity disk). After you’ve copied the data from the NTFS source disk to your new unRAID disk, then you can repurpose the source disk as your parity disk.
  4. Run a comparison. If you’re really paranoid, you can run a byte-by-byte comparison of the data in your source disk to your temporary disk to make sure they have the exact same data. I used Beyond Compare utilizing the binary comparison option in the settings.
    Beyond Compare Settings

    That is. You can now power down your Windows machine for the last time. Now would be a good time to do any drive swapping you intend to do. In my case, I actually pulled out one of the 1.5TB mirror drives and installed a new 4TB drive to use as my new parity drive for unRAID (future proofing for a larger array in the future).

Install unRAID

Installing unRAID should be pretty straight forward. Just follow the standard instructions to download it, get it on a USB flash drive, and boot your PC to the new OS. Once it’s loaded up, you should be able to navigate in your web browser to http://tower or http://server.ip.address.here

Setup unRAID Array & Shares

Next, setup your unRAID array and shares as you would normally. Whatever disks you assign to your array will be deleted. Be sure you pick the right disks to add to the array.

  1. Assuming you copied your data to a temporary drive, go ahead and setup your array with your two data disks. The array configuration is on the Main tab of the unRAID web UI.
    1. If you decided to use one of your Windows RAID disks as the source for your data to copy (Protip 2 above), be sure set up a single-disk array in unRAID without parity. And be sure the disk you assign to the array is the one you want to delete!
    2. If you plan to copy your physical installation of Windows to a Virtual Machine so you can boot and use it later, be sure not to assign your OS SSD as the cache device at this point.
  2. Once you get it all configured, start the array.
  3. Now that the Array is started, you can setup your shares. Setup your shares the way you’d like them, using whatever structure you see fit. I’ll assume you create a single share called “mirror” for these instructions.

Install Unassigned Devices & Mount Disk

The Unassigned Devices plugin will let you access the disk you’re using as your source data to copy. unRAID doesn’t normally mount drives which are not part of the array. This plugin makes that easy.

  1. On your unRAID web UI, navigate to Plugins > Install Plugin.
  2. Copy the URL below into the box for the “URL of remote plugin file or local plugin file”
    https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg

  3. Click “Install”
  4. Navigate to the “Main” tab – you’ll see the disks which are not part of your unRAID array listed there.
  5. Find the disk that contains your source data. Click the (+) next to the disk to find the partition that contains the data and select “Mount”. The mount location will be listed (e.g. /mnt/disks/Purple in my case below).

Copy your data onto your the array

Now that the array and shares are set up, and the source disk is mounted, it’s time to copy your data to the array. The main thing here is to make sure the copy is identical – that the dates and times of the files, folders, and relevant attributes get copied correctly.

  1. SSH into your unRAID box. If you’re unfamiliar with SSH, I recommend getting a copy of Putty and learning how to use it to log into the server. It’s pretty handy.
  2. Use rsync to copy the data. Here’s the command I used. Obviously, you’ll need to change the source and destination to match where you’re copying your data. The nice thing about rsync is that it will pick up where it left off it it gets interrupted.
    rsync -aHAXvhW --no-compress --progress --info=progress2 --stats /mnt/disks/source/ /mnt/user/destination/

Finish setting up your unRAID

Your data should be now in it’s new home on your unRAID share. Next up you should look to complete the other installation steps in the unRAID manual if you haven’t already.

Some particularly important steps to take next:

There’s a lot more you can do with unRAID, obviously, but these ones I found to be pretty important as part of my initial config.

Happy unRAIDing!