How to fix guest processing errors on Hyper-V virtual machine backups

So you setup a third party backup system, sticking mostly to defaults on the job settings, with “application aware” or “guest OS processing” enabled, in order to capture Hyper-V virtual machines, and keep their application data consistent (process database transaction logs, etc.) Or, perhaps you have migrated existing virtual machines to a newer piece of hardware, and need to reconfigure backups as a result. Then depending on your situation, you could run into a message similar to the following when you attempt to run certain backup jobs:

• Guest processing failed
• Guest processing skipped
• Check guest OS VSS state
• Check integration components

Let’s begin troubleshooting this error. First, on the host where the VM’s are located, open an administrative command prompt or PowerShell session and type the following:

diskshadow /L C:\temp\vss.txt
list writers

This will output the results to a text file in the C:\temp directory.  When it completes, open this text file and search for the text:

Caption: Offline

This can be run after each step in the troubleshooting process, to see whether the value is updated to the desired result. What we would like to see, in order for our backup to succeed, is “Caption: Online.” If it says “Offline” or “Saved state,” then something isn’t right and needs to be corrected. What is that something? Likely, you still need to meet one or more of the following conditions:

1. Check Guest OS Compatibility

Especially when introducing new VM’s or migrating existing VM to newer platforms, check for basic compatibility. Are you aware, for example, that you should only deploy a guest OS as new as Server 2012 on top of a 2008 R2 host? 2012 R2 is not supported, nor is 2016. Likewise, on Windows Server 2016, it is not technically supported to run a Windows 2003 or XP guest OS. Also, Windows 2008 (not R2) is now in extended support: just something to be aware of.

• Compatibility List for 2008 R2
• Compatibility List for 2012 R2
• Compatibility List for 2016

By the way, you might notice that it will also be necessary to be running the latest service pack for each Windows edition (e.g. 2008 R2 SP 1, 2008 SP2). In fact, you won’t be able to complete the next step, to update the Integration services components, until you are on the latest service pack.

2. Check all Windows updates, hotfixes and integration services

Do you have all the necessary updates & hotfixes installed on your Hyper-V host? I always start any Hyper-V project with a new client by checking for and applying these fixes if they aren’t already present.

Again, especially if you migrated virtual machines from older hardware with a previous version of Hyper-V, such as 2008 R2, then remember that your migration is not complete until you update the Integration Services on each VM. A quick way to check whether any VM’s on your host need an update is by using PowerShell:

Get-VM | select VMName, IntegrationServicesState

Remediation: On Hyper-V 2012 or 2012 R2, connect to the affected VM, go to Action > Insert Integration Services Setup Disk. Run the installation on the virtual machine, and reboot (yep, it’s a manual process–unless you have System Center VMM). In Hyper-V 2016, integration services is handled differently, via Windows Update. So just be sure that Windows Update is fully caught-up for host & guest operating systems. Note: for older guest Operating Systems including Windows 7/8, 2008 R2 SP 1, and 2012, you will also need to ensure that the VSS and Data Exchange integration service is enabled, in order for the updating to work.

Get-VM | Get-VMIntegrationService -Name "Key-Value Pair Exchange","VSS"

Finally, if you are in a situation where your 2016 Hyper-V host and/or guest VM’s are isolated from the Internet and therefore cannot access Windows Update, then you would need to download the updates (.CAB files) from Microsoft, and install them like so:

Add-WindowsPackage -Online –PackagePath <path to .CAB file>

3. Make sure you have a SCSI controller on Gen1 VM’s

In Windows Server 2012 R2, you need to have a virtual SCSI controller present in your virtual machine configuration file in order for guest VSS to work. Yes: even if there are no devices attached to it. Any Gen2 VM has these by default, but in Gen1, IDE was your primary hard disk controller. Now in 2012 R2, a newly minted VM will include a SCSI controller by default, but virtual machines that were migrated from earlier versions such as 2008 R2 might not have this present yet, so add it.

Shut down the VM, then right-click and go to Settings. From Add Hardware, select SCSI Controller, click Add. Click OK.

4. Check the location of the virtual machine files

Especially on a cluster, do not store the virtual machine files at the root of the drive. For example, don’t let your VM configuration or VHD/VHDX files live in a location that looks like one of the following:

• E:\
• C:\ClusterStorage\Volume1
• Any other root-level mounted directory/path

Strange as it may sound, this makes a difference. I normally put each VM in its own folder. You can move a virtual machines storage, configuration files or both. In Hyper-V Manager, right-click on the VM and choose Move… From Failover Cluster Manager, right-click the VM, go to Move > Virtual Machine Storage.

5. Check Shadow Copies & disks in the Guest VM

In the guest OS, open diskmgmt.msc. First, see the Properties of the system reserved partition–make sure it has at least 42 MB free. If not, try to create a new shadow copy on the Shadow Copies tab, deleting any older copies.

Right-click another volume such as C: and go to Properties. On the Shadow Copies tab, click on the volume and check settings. Be sure that the volume and storage area match (don’t use a separate volume to store shadow copies).  Note: Shadow copies may also be disabled for any/all volumes, including the reserved partition.

Last, just make sure you don’t have any FAT32 disks in here–NTFS and ReFS only.

6.  If all else fails, restart

The age-old solution. Maybe we should have started here. Reboot the guest OS to see if this improves the state from Caption: Offline to Caption: Online. If not, in some rare circumstances, it may be necessary to also restart the “Hyper-V Volume Shadow Copy Requestor”:

net stop vmicvss && net start vmicvss

Now you will want to check out the event logs for event ID 13 coming from vmicvss.  If this error is present, you may have to create a registry key as follows (open regedit):

1. Create HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
2. Create a DWORD with the Name: ScopeSnapshots and a Value of 0

Now restart the vmicvss service again.

Conclusion

One or more of the above was likely your issue. If you meet all of the above requirements, your application-aware backup should not present a guest VSS error. If it still does, you will need to engage your backup vendor for further support. If they try to point the finger at Microsoft too soon, then you are at least able to explain that you have already validated compatibility and other pre-requisites.