Hyper-V Guest processing skipped (check guest OS VSS state and integration components version)

Challenge

Guest VMs will fail to engage VSS when application-aware image processing is enabled generating an error:

Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)

Cause

This issue has multiple potential causes, usually related to the Integration Services version of the failing guest server. This issue occurs when Application-Aware image processing is utilized in the Backup or Replication job settings for transactional quiescence using Microsoft VSS

Solution

Please Note:

• A list of supported Guest OSes for each version of Hyper-V can be found here. 
• Important! Server 2012 R2 and higher guests are not supported for VSS-enabled (AAIP) backups on 2008 R2 hosts due to integration components incompatibility.
• If the VM Guest OS is 2012 and the Host is 2008R2:
  • View the Guest OS Application Event logs for errors with source: vmicvss and Event ID: 13.
  • If this error is present, open the Registry Editor (regedit) and navigate to:
    HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
    Within that key create a new DWORD
    Name: ScopeSnapshots
    Value: 0
  • Note: If the SystemRestore key is not present, you will have to create it.

The solution section of this knowledge base article has been split into four different sections. Please review each step of the following sections.

1. Integration Services
2. Within the Guest
3. VM Configuration
4. Verification

1. Integration Services

Click here for a list of Hyper-V Integration Services and their corresponding build numbers.

How to identify…

1. Using Event Viewer

This section will provide directions on ensuring the Hyper-V Integration Services are up to date within the Guest OS of the VM that is failing to backup.

If Hyper-V Integration Services are out of date you will find Event ID 4000 within the Hyper-V-Integration section of the Event Viewer on the Hyper-V host where the VM is located.

2. Using PowerShell (Server 2012 or newer)

Run the follow command in an Administrative PowerShell prompt on the host where the VM that failed to backup resides.
 

Get-VM | select VMName, IntegrationServicesState

The results of which will list out each present VM, and the state of its integration services.

How to correct:

Note: A reboot of the virtual machine will most likely be required to complete the Integration Services update.

The following information is from the Microsoft page found here.

Install or update integration services

For hosts earlier than Windows Server 2016 you'll need to manually perform install the update from within the guest operating system. These steps cannot be automated or executed within a Windows PowerShell session.

1. Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.
2. Connect to the virtual machine. Right-click the virtual machine and click Connect.
   
3. From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive.
   
   Depending on the guest operating system, you may need to start the installation manually.
   
4. After the installation finishes, you may be prompted to restart.
   

2. Within the Guest

The following steps and checks are to be performed within the Guest OS of the VM that is failing to backup.

1. Make sure Windows is running the most recent service pack.
    
2. Ensure that the VSS Storage Area and Volume Match
   1. From an Administrative Command Prompt, run the command ‘VSSUIRUN.exe”
      
   2. Check the [Settings] for each Volume and ensure that the Volume and the Storage area match.
      
       
3. Ensure there are no Dynamic disk or FAT\FAT32 volumes
   1. Open Disk Management (diskmgmt.msc)
   2. Review the Type and File System columns
   3. VSS is not compatible with Dynamic disks or FAT\FAT32
      Note: The VHD(X) that the VM runs off can be dynamic. The volume within the Guest OS cannot.
      
      All volumes must be NTFS or ReFS
       
4. Check System Reserved partition free space.
   1. System Reserved partition must have more than 42MB of space free.
   2. If less than 42MB is free, try creating a shadow copy for the volume.
      
   3. Remember to delete the Shadow Copy if it is successful.
       
5. Start\Restart the “Hyper-V Volume Shadow Copy Requestor
   1. Open Services (services.msc)
   2. Find the “Hyper-V Volume Shadow Copy Requestor” service.
   3. If it is ‘Stopped’, Start it. (If the service will not start, check in the VM settings under Integration Services, and ensure the "Backup (volume Checkpoint)" option is enabled.)
   4. If it is ‘Running’, restart it.

3. VM Configuration

The following steps and checks are to be performed from the Hyper-V host level.

1. Make sure the VM has a SCSI controller. One must be present, even if there are no devices to use the SCSI controller.
   •  The error will present itself in the Hyper-V VMMS > Admin Event log. Example:
     “The virtual machine ' Vm-Name' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID).”
2. Within the Virtual Machine settings verify that the “Backup (Volume Snapshot)” option is enabled.
3. Make sure that the virtual machine files (VHD files) are not sitting in the root folder of the CSV, as this may prevent correct snapshot creation due to permission restrictions. We strongly recommend keeping virtual machine files in separate folder. More information regarding this can be found here.

Verification

Performing the following will allow you to determine if the steps performed above have corrected the issue without the need to run the Veeam Backup & Replication job again.

1. 1. Open an Administrator Command Prompt.
   2. Run the command:

      Diskshadow /L C:\writers.txt

   3. Run the command:

      list writers

Open the file that was created and and locate the VM name in the section for the Hyper-V writer. For each VM there's a "Caption" entry. The caption should read "online" or "child partition", and not "offline" or "saved state".

More Information

PowerShell script to tell you if you need to update your Integration Services
https://blogs.msdn.microsoft.com/virtual_pc_guy/2013/04/29/powershell-script-to-tell-you-if-you-need-to-update-your-integration-services/

Recommended hotfixes for Hyper-V servers:

http://www.veeam.com/kb1838
http://blogs.msdn.com/b/virtual_pc_guy/archive/2009/02/02/looking-at-the-hyper-v-event-log.aspx
http://blogs.technet.com/b/virtualization/archive/2008/08/29/backing-up-hyper-v-virtual-machines.aspx