Using the Diskshadow Utility to manually test VSS operations

Challenge

This article describes the steps needed to manually create a volume shadow copy using the Diskshadow command line utility in Windows.

Cause

Veeam products use the Microsoft Volume Shadow Copy Service (VSS) for a variety of tasks:

• In VMware environments, VSS ensures transaction consistency when backing up Windows VMs.
• In Hyper-V on Windows Server 2008R2, 2012, and 2012R2, VSS is the primary component of the backup API.
• Veeam Agent for Windows (formerly Veeam Endpoint Backup) uses VSS to help make sure data is consistent.
• VSS is used to trigger transaction log truncation for Microsoft Exchange.

When troubleshooting, it is sometimes necessary to test if the Volume Shadow Copy Service is operating correctly. A quick test for a single volume can be performed using the Shadow Copies utility (see KB1794). Windows Server Backup is generally sufficient for isolation tests (and may be preferred by Microsoft Support), but it does not support all possible configurations.

The Diskshadow command line utility provides a greater degree of support, and is a better match for how Veeam products use the VSS API. However, it is only available on Windows Server 2008 or newer; it is not available for client versions of Windows.

Solution

Test shadow copy creation, then review Interpreting the Results (below).  

VMware Environments or Agent Backups

Copy-Only Mode

Use this method when transaction log truncation is not relevant. For best comparison, specify copy-only in the Veeam job settings.
From an administrator command prompt run the following commands:
 

​diskshadow/l <output path for log file> (the /l parameter is optional)
set verbose on
set context volatile

Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.

Examples:
 

add volume \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx}\
add volume X:
add volume C:\ClusterStorage\Volume1

See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

create

Normally, shadow copies created this way will be automatically deleted when Diskshadow exits or is reset, but the delete shadows command is available if needed.

Note: Some issues may be isolated to a single volume. To identify which volume is responsible for an error, add only one volume to the backup set, create the shadow copy, then exit or reset Diskshadow before starting over with the next volume.
 

Full Mode

Full mode is the default setting for Veeam jobs. This method will notify VSS writers that a backup has occurred. Some applications, including Microsoft Exchange, will truncate transaction logs automatically in response to this notification. Other applications, such as Microsoft SQL Server, record that a backup has been taken but do not truncate transaction logs.

To test log truncation along with the creation of the shadow, run the following commands from an administrator command prompt:

diskshadow /l <output path for log file> (the /l parameter is optional)
set verbose on
set context volatile

Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.

Examples:

add volume \\?\Volume{3c8d1bcb-9d95-11e5-80b0-806e6f6e6963}\
add volume X:
add volume C:\ClusterStorage\Volume1

See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

begin backup
create

 
Once the shadow copy is created, inform the VSS writers that the backup has completed. For some writers, this will trigger transaction log truncation:
 

end backup

 
Normally, shadow copies created this way will be automatically deleted when the end backup command is completed, but the delete shadows command is available if needed.
 

Hyper-V Environments

Note: The following information is applicable to Server 2008R2, 2012, and 2012R2.

Veeam Backup & Replication creates shadow copies of volumes containing virtual machines. For transactionally-consistent backups (Application-Aware Image Processing or Hyper-V Quiescence), the Hyper-V VSS Writer triggers the Hyper-V Volume Shadow Copy Requestor service in each VM to create shadow copies of volumes within the VM. Error messages generated by shadow copy creation failure generally do not indicate whether the problem is isolated to a hypervisor volume or an in-guest volume.

Start by isolating whether the failure occurs only when using Application-Aware Image Processing or Hyper-V Quiescence. If the problem does not occur when both options are disabled, test shadow copy creation from within the virtual machine as described in the VMware Environments or Agent Backups section, above.

If snapshot creation is failing for crash-consistent backups, or if in-guest shadow copy creation does not result in any errors, it may be necessary to test shadow copy creation on the Hyper-V host. Because Hyper-V backup can be complex, a scripted approach is recommended. Veeam Support can run a scripted test that also collects a significant amount of useful logging. Contact support to schedule a remote session, and ask about hv_mkshadow.

Warning: Attempting to perform a shadow copy of a volume containing virtual machines will trigger in-guest shadow copy creation on all VMs on that volume unless specifically excluded. This may have negative effects on performance or responsiveness of these VMs and the Hyper-V host. Veeam Backup and the hv_mkshadow script exclude unnecessary VSS writer components from the shadow copy.
 

 

Finding Volume GUIDs

To add volumes that lack a mount point (such as the System Reserved Partition), use the volume GUID obtained through either the mountvol command-line tool, or the Shadow Copies utility.

To access the Shadow Copies utility, right click any volume and choose Configure Shadow Copies. On older operating systems, this may instead be a tab in Properties. In the Shadow Copies utility:

1. Select the volume displayed as a GUID;
2. Click Settings…
3. Copy the volume GUID from the settings window and paste it into DiskShadow.

Alternatively, run “mountvol” from a command prompt; below the usage information will be a list of GUIDs and their associated mount points, if any.

 

More Information

Interpreting the Results
 
Typically, if shadow copy creation fails via both Diskshadow and Veeam products, that indicates that the Veeam application is functioning correctly, and the problem is isolated to shadow copy creation. Troubleshoot the errors reported by Diskshadow and any events appearing in Event Viewer.
 
Two examples of failures reported directly by Diskshadow are as below. The first is an instance where the Volume Shadow Copy service fails, hangs, or is shut down midway through the creation process. The “RPC server” error can be a bit misleading; in this case a call was sent to the Volume Shadow Copy service and it was not there to receive it.
 
COM call "m_pVssBackup->StartSnapshotSet" failed.
 
The last operation failed.
                - Returned HRESULT: 800706ba
                - Error text: The RPC server is unavailable.
 
Another example shows a more common issue, that of the volume having insufficient free storage for the shadow copy creation.
 
The last operation failed.
                - Returned HRESULT: 8004231f
                - Error text: VSS_E_INSUFFICIENT_STORAGE
 
These messages will usually immediately follow the list of included writers, and should be the last output returned in the Diskshadow process.
 
In some cases, shadow copy creation will succeed via Diskshadow, but error or warning events will be generated in the event log. If similar errors are generated when Veeam backup or replication jobs fail to create a shadow copy, that usually indicates that the Veeam application is aborting the shadow copy in response to the errors. In this situation, the Veeam application can usually be assumed to be functioning correctly, so it is best to focus troubleshooting on any events appearing in Event Viewer.
 
Troubleshooting shadow copy creation or transaction log truncation may require the assistance of Microsoft technical support. The error
“COM call "lvssObject4->GetRootAndLogicalPrefixPaths" failed”
can be ignored. For more information, see Microsoft KB3025158.