How-to: Migrate from Windows RAID to unRAID

When I started looking at converting my Windows 10 Pro machine (which was running a two-disk, software RAID 1 in Windows) to an unRAID machine I didn’t find a lot of good how-to guides. Below are some guidelines and step-by-step instructions that will hopefully help if you’re in the same situation.

Basic Setup

Here’s my starting & ending setup. Yours is likely different.

All of this is running on a 5 year old ASUS Maximus IV Gene-Z/GEN3 motherboard with an Intel Core i5-2500k processor and 16GB of RAM.

The 4TB “Security Recordings” disk is used to hold my IP cam footage from my security software – I made space on that disk to temporarily hold the data from my RAID while I was doing the upgrade. More on that below.

Before you get started, it might be good to familiarize yourself with the unRAID manual. It’ll help to have a passing understanding of what an unRAID array and share is, and it’s a pretty good reference for how to do a general unRAID installation.

Migration Steps

These instructions assume you have a Vista-or-later Windows machine with two-disks in a Windows-managed software RAID mirrored configuration.

Before you shutdown Windows

  1. Backup everything. This is a good time to remind yourself that RAID is not a backup strategy. You should already have a 3-2-1 Backup Strategy. If you don’t, just stop reading right here and get one. Getting a good backup is really more important than this. I like Crashplan for cloud and on-site location backups. </soapbox>
  2. Save your Windows license key. If you have Windows 10, this is really really important. If you took advantage of the free Windows 10 upgrade, it’s really easy to lose that free upgrade if you don’t save your license key. The easiest way to keep the key is to create an Admin user that logs in using a Microsoft Account credentials. If you’ve done that right, you should see your PC listed when you log into account.microsoft.com and look in the Devices tab. If your device isn’t there, then you should get that resolved before you move on. Once you re-install Windows later, you’ll be able to log in using the same Microsoft Account and adopt this license key on your newly installed Windows 10 machine.

    Make sure this PC is listed under your Microsoft Account Devices
  3. Copy the data from the Windows RAID to a temporary (NTFS) drive. This has to be a drive you won’t be using as part of the initial 2-disk unRAID. We’ll use this as the source data to copy to the newly created unRAID array.In my case, I had a 4TB drive that I could copy my 1.5TB of RAID data onto temporarily so I could later use those 1.5TB drives for my unRAID array.Some tips:
    1. Protip 1 – make sure the temporary drive you’re copying to is not a Windows Dynamic Disk. You can check what type of disk it is by looking in the Windows Disk Management console. If it’s a Windows Dynamic Disk, unRAID won’t be able to mount it (neither can much else).
    2. Protip 2 – if you don’t have an extra disk big enough to hold the data from your existing array, you can use one of your disks as the source but you have to do a couple of additional steps.
      1. First, you have to break the Windows mirrored RAID apart (in the Windows Disk Management console, right click on the mirror, select “Break Mirrored Volume”). Do Not Remove the Mirror. Breaking it keeps the data in place on both disks. Removing it deletes the data from one of the disks.
      2. Second, you need to convert the disk you’ll use as the source data into a non-dynamic disk. Windows Mirrored volumes are always dynamic, so you’ll have to convert it so unRAID can read it. I used a program called TestDisk. The steps are essentially do a Create > Select Disk > select type “Intel” > Analyze > Backup > Press “Enter” > Write (Y) > Reboot. For detailed instructions, this walk through here was really useful.
      3. Third, when you go to create your unRAID array later, you’ll need to create an array that only has one disk (and no parity disk). After you’ve copied the data from the NTFS source disk to your new unRAID disk, then you can repurpose the source disk as your parity disk.
  4. Run a comparison. If you’re really paranoid, you can run a byte-by-byte comparison of the data in your source disk to your temporary disk to make sure they have the exact same data. I used Beyond Compare utilizing the binary comparison option in the settings.
    Beyond Compare Settings

    That is. You can now power down your Windows machine for the last time. Now would be a good time to do any drive swapping you intend to do. In my case, I actually pulled out one of the 1.5TB mirror drives and installed a new 4TB drive to use as my new parity drive for unRAID (future proofing for a larger array in the future).

Install unRAID

Installing unRAID should be pretty straight forward. Just follow the standard instructions to download it, get it on a USB flash drive, and boot your PC to the new OS. Once it’s loaded up, you should be able to navigate in your web browser to http://tower or http://server.ip.address.here

Setup unRAID Array & Shares

Next, setup your unRAID array and shares as you would normally. Whatever disks you assign to your array will be deleted. Be sure you pick the right disks to add to the array.

  1. Assuming you copied your data to a temporary drive, go ahead and setup your array with your two data disks. The array configuration is on the Main tab of the unRAID web UI.
    1. If you decided to use one of your Windows RAID disks as the source for your data to copy (Protip 2 above), be sure set up a single-disk array in unRAID without parity. And be sure the disk you assign to the array is the one you want to delete!
    2. If you plan to copy your physical installation of Windows to a Virtual Machine so you can boot and use it later, be sure not to assign your OS SSD as the cache device at this point.
  2. Once you get it all configured, start the array.
  3. Now that the Array is started, you can setup your shares. Setup your shares the way you’d like them, using whatever structure you see fit. I’ll assume you create a single share called “mirror” for these instructions.

Install Unassigned Devices & Mount Disk

The Unassigned Devices plugin will let you access the disk you’re using as your source data to copy. unRAID doesn’t normally mount drives which are not part of the array. This plugin makes that easy.

  1. On your unRAID web UI, navigate to Plugins > Install Plugin.
  2. Copy the URL below into the box for the “URL of remote plugin file or local plugin file”
    https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg

  3. Click “Install”
  4. Navigate to the “Main” tab – you’ll see the disks which are not part of your unRAID array listed there.
  5. Find the disk that contains your source data. Click the (+) next to the disk to find the partition that contains the data and select “Mount”. The mount location will be listed (e.g. /mnt/disks/Purple in my case below).

Copy your data onto your the array

Now that the array and shares are set up, and the source disk is mounted, it’s time to copy your data to the array. The main thing here is to make sure the copy is identical – that the dates and times of the files, folders, and relevant attributes get copied correctly.

  1. SSH into your unRAID box. If you’re unfamiliar with SSH, I recommend getting a copy of Putty and learning how to use it to log into the server. It’s pretty handy.
  2. Use rsync to copy the data. Here’s the command I used. Obviously, you’ll need to change the source and destination to match where you’re copying your data. The nice thing about rsync is that it will pick up where it left off it it gets interrupted.
    rsync -aHAXvhW --no-compress --progress --info=progress2 --stats /mnt/disks/source/ /mnt/user/destination/

Finish setting up your unRAID

Your data should be now in it’s new home on your unRAID share. Next up you should look to complete the other installation steps in the unRAID manual if you haven’t already.

Some particularly important steps to take next:

There’s a lot more you can do with unRAID, obviously, but these ones I found to be pretty important as part of my initial config.

Happy unRAIDing!

 

From Windows to unRAID: choosing my next home-server OS

This is the first in several posts about getting unRAID setup for the first time. Look for the unRAID tag to see all of the related posts.

I’ve run a Windows 10 (and Vista, 7, 8, 8.1) machine as our household’s file server for many, many years. It worked well enough. I had two hard drives set up as a RAID, and an SSD to run the OS from. But I began to realize that this setup was a lil risky and pretty inflexible – most notably it would never notify me if one of my disks started to fail (or any other data-oriented problems, for that matter). And it wasn’t easy to run the open source software packages I was wanting to run (like OpenVPN or ZoneMinder).

This was a prescient worry given that my two 1.5TB drives which stored all of my family’s most important photos & files were about to fail.

I looked around at getting a dedicated box for just file serving (Synology, QNAP). I looked at software to add to the Windows install to solve some of my issues like better data protection (most notably SnapRAID). And I looked at the slew of dedicated home NAS software like FreeNAS, Nas4Free, and unRAID.

Ultimately I picked unRAID because it looked to be the easiest to get everything running, had a good and active support community, was cheap (but not free) and was based on a Linux distribution (instead of FreeBSD). The biggest thing going against unRAID to me was that (1) it didn’t have native bit rot (data degradation) protection like SnapRAID or FreeNAS and (2) it’s not open source. I decided I can live with a commercial project and that I can find a work around to protect against bit rot.

I was really close to choosing FreeNAS, but they were going through some turmoil with a failed release of their next-generation version 10 that had the Docker support I wanted… so I passed for now.

It should be clear that this isn’t intended to be a comparison of all the NAS options out there – if you’re interested in doing your own comparison, there’s a bunch of info out there and you should judge for yourself. For example, SnapRAID has a really nice comparison table of the file system capabilities that I found very useful.

What my computer did before (running Windows):

  • Store and serve files (documents, pictures, music, and video)
  • Backup those files to the cloud (via CrashPlan)
  • Provide a backup location for the computers in our extended family (via CrashPlan)
  • Run our in-house music system (via Logitech Squeezebox family of devices)
  • Run my IP security camera system
  • Monitor the UPS (backup battery) and gracefully shut down in the event of power loss
  • Occasionally watch Hulu via the web browser & the TV it was connected to.

What it didn’t do:

  • Monitor the health of the disks (and files) and inform me of impending problems
  • Provide a dashboard of system health, status, and related controls
  • Operate power-efficiently, because…
    • Both disks in the RAID spinning up when reading any file
    • Case fans ran all the time (vs. being controlled by HDD temperatures)
    • Spending time & energy updating parts of the OS I’ll never use (looking at you, Windows Modules Installer Worker)
    • Spend time/energy protecting itself from viruses (looking at you, Antimalware Service Executable)
  • Parity check the files in a way to allow for recovery in the event of bit rot (data degradation)
  • Enable me to run open source packages easily (in particular, leverage Docker to manage software installs)
    • OpenVPN, NextCloud, ZoneMinder, for example.
  • Be more resilient to ransomware attacks (e.g. WannaCry)

Choosing unRAID got me all of these missing things on my existing hardware. The only thing that is missing right now is a good parity checking solution to enable finding & fixing data degradation on unRAID.

In the next few posts I’ll cover how I did the conversion, and document some of the hassles I had to overcome to get unRAID working exactly the way I want it to.

In that process I was able to:

  1. Migrate the basic functionality and data of my Windows machine to unRAID
  2. Copy my Windows installation to a VM running on unRAID, and set it up with exclusive access to one of my disks outside of the NAS RAID array.
  3. Setup OpenVPN (which runs way better than it did on my Raspberry Pi)
  4. Verify that unRAID’s parity system does indeed work to recover a failed disk.
  5. Setup custom fan controls so the machine runs very, very quietly (which is important, because it’s in our TV room)

Look for the unRAID tag to see all of the posts related to this project.

Raspberry Pi + OpenVPN + the Baby’s Webcam – Secure Access to your Home Network

M and I have a great webcam setup for our home for the lil nugget – it’s a simple DLink webcam that costs less than $50. So much cheaper than an expensive, baby-specific monitor.

We use a tablet and our phones to check in on the baby using this great app (tinyCam Monitor Pro) – the best feature is that you can turn off your phone and still have the audio alert you to the baby starts making noise louder than the ambient room noise (i.e. squelch the volume & not have to listen all the time to the white noise we pump into the nugget’s ears to keep him happily asleep).

The problem is that we can’t view the webcam from outside the house securely. DLink provides an app that lets you view the camera when you’re not on your home WiFi, but it does so in an insecure way – basically broadcasting the video feed for anyone to intercept and see. That creeps me out.

So we needed a way to get secure access from our phones, anywhere in the world, back into our home network so we could see the camera. Well, need is a strong term… desire, perhaps.

This is what Virtual Private Networks (VPN) were built for – I’ll walk through how I set one up for our home.

This is a tech how-to. It’ll take an afternoon or a weekend to complete the project. It was a pain in the butt for me – I’m writing these instructions down in the hopes I can save someone else some time.

Continue reading Raspberry Pi + OpenVPN + the Baby’s Webcam – Secure Access to your Home Network