Challenge
Guest VMs will fail to engage VSS when application-aware image processing is enabled generating an error:
Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
Cause
This issue has multiple potential causes, usually related to the Integration Services version of the failing guest server. This issue occurs when Application-Aware image processing is utilized in the Backup or Replication job settings for transactional quiescence using Microsoft VSS
Solution
Integration ServicesMake sure integration services are up to date. The Hyper-V Integration Services log in the event viewer of the Hyper-V host that maintains the failing VM. Look for the name of the failing server with any error events referencing the integration components being out of date. The following is an example of the event text.
Hyper-V Volume Shadow Copy Requestor connected to virtual machine ‘VM-Name’, but the version does not match the version expected by Hyper-V (Virtual machine ID). Framework version: Negotiated (3.0) - Expected (3.0); Message version: Negotiated (3.0) - Expected (4.0). This is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.
If any Integration Service components are listed as out of date, update them on the guest server by following the Steps listed below. Please note this process applies to Windows Server 2008 R2, Server 2012 and Server 2012 R2
• Server 2012 http://technet.microsoft.com/en-us/library/hh846766.aspx#BKMK_step4
• Server 2008 R2 http://technet.microsoft.com/en-us/library/ee207413(v=WS.10).aspx
• Server 2008 R2 http://technet.microsoft.com/en-us/library/ee207413(v=WS.10).aspx
To determine the Integration services version on the guest:
• 2008R2: On the Hyper-V host server, browse to C:\windows\system32\vmms.exe > open the proprieties of the file and go to the ‘details’ tab.
• 2012 & 2012 R2: Run this powershell command on the host containing the VMs as Administrator:
• 2012 & 2012 R2: Run this powershell command on the host containing the VMs as Administrator:
o Get-VM | ft Name, IntegrationServicesVersion
Click here for a list of Hyper-V Integration Services and their corresponding build numbers.
Guest OS
The following steps and checks are to be performed within the Guest OS of the VM that is failing to backup.
NOTE:Server 2012 R2 guests are not supported for VSS-enabled (AAIP) backups on 2008 R2 hosts due to integration components incompatibility. Please see http://technet.microsoft.com/en-us/library/cc794868(v=ws.10).aspx for the list of supported guest OSes.
• Make sure Windows is running the most recent service pack.
• Open Disk Management (diskmgmt.msc), check for the following:
1. View the properties of a volume, then goto the Shadow Copies tab.
Check the settings for each Volume in the Shadow Copies section to make sure the Volume and the Storage area match.
2. No Dynamic disks. Dynamic disks cannot utilize VSS and be backed up live. (The VHD(X) itself can be dynamic, the volume within the Guest OS cannot.)
3. No FAT or FAT32 Volumes. All volumes must be NTFS or ReFS. The System Reserved partition must have more than 42MB of space free. If less than 42MB is free, try creating a shadow copy for the volume.
3. No FAT or FAT32 Volumes. All volumes must be NTFS or ReFS. The System Reserved partition must have more than 42MB of space free. If less than 42MB is free, try creating a shadow copy for the volume.
• If the VM Guest OS is 2012 and the Host is 2008R2:
View the Guest OS Application Event logs for errors with source:
vmicvss
Event ID: 13.
If this error is present, open the Registry Editor (regedit) and navigate to:
HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
Within that key create a new DWORD:
Name: ScopeSnapshots
Value: 0
Note: If the SystemRestore key is not present, you will have to create it.
vmicvss
Event ID: 13.
If this error is present, open the Registry Editor (regedit) and navigate to:
HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
Within that key create a new DWORD:
Name: ScopeSnapshots
Value: 0
Note: If the SystemRestore key is not present, you will have to create it.
• If all other troubleshooting fails, restart the virtual machine. Once the VM has powered on and all services are started, restart the “Hyper-V Volume Shadow Copy Requestor” service.
VM Configuration
The following steps and checks are to be performed from the Hyper-V host level.
• Make sure the VM has a SCSI controller. One must be present, even if there are no devices to use the SCSI controller.
The error will present itself in the Hyper-V VMMS > Admin Event log. Example:
“The virtual machine ' Vm-Name' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID).”
“The virtual machine ' Vm-Name' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID).”
• Within the Virtual Machine settings verify that the “Backup (Volume Snapshot)” option is enabled.
Make sure that the virtual machine files (VHD files) are not sitting in the root folder of the CSV, as this may prevent correct snapshot creation due to permission restrictions. We strongly recommend to keep virtual machine files in separate folder.
Verification
Use these steps to verify whether your troubleshooting has worked without having to run a Veeam backup or replication job. If the Hyper-V writer caption for the VM displays the wrong state, you can contact Veeam Support to verify your troubleshooting based on this article, or contact Microsoft support for additional troubleshooting steps.
Perform the following steps on the Hyper-V host that the VM is running on.
1. Open an Administrator Command Prompt
2. Run the command:
Diskshadow /L C:\writers.txt
3. Run the command:
list writers
When this completes you can then open writers.txt and locate the VM name in the section for the Hyper-V writer. For each VM there's a "Caption" entry that should read "online" or "child partition", and not "offline" or "saved state".
More Information
PowerShell script to tell you if you need to update your Integration Services
https://blogs.msdn.microsoft.com/virtual_pc_guy/2013/04/29/powershell-script-to-tell-you-if-you-need-to-update-your-integration-services/
Recommended hotfixes for Hyper-V servers:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2009/02/02/looking-at-the-hyper-v-event-log.aspx