VSS wait timeout

Challenge

Either of the following errors occurs in the job statistics:

"VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec."

"VSSControl: Failed to freeze guest, wait timeout"

Cause

These errors look similar but their causes are unrelated.

"VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec"

Refers to VSS preparation, which consists largely of enumerating VSS writers, their components, and associated files and volumes. The timeout may be exceeded because of performance limitations of the VM guest, or because a specific writer is enumerating a very large number of files. This timeout is configurable.

"VSSControl: Failed to freeze guest, wait timeout"

Refers to the limit imposed by Microsoft VSS writers on the duration of a freeze. This timeout is not configurable. Veeam uses VSS to freeze applications immediately prior to creating the VMware snapshot, and then sends the thaw command as soon as snapshot creation is complete. VSS will only hold a freeze on the writers for up to 60 seconds (20 for Exchange), so several steps must fit within this timeframe:

1. Verification of freeze state¹
2. Snapshot creation request via VIM API²
3. Snapshot creation on the ESXi host
4. Return of snapshot information via VIM API²
5. Thaw request to Microsoft VSS¹
6. Thawing of VSS writers’ I/O

¹ If a network connection to the guest OS is not available, VIX API will be used, which introduces additional latency.
² These steps should usually be near-instantaneous, but if the vCenter is heavily loaded or has a high latency to the ESXi hosts, the delay may be significant.

Solution

Review the relevant section below:

Note: Improving in-guest performance will help in resolving either error.

Failed to prepare guest for freeze, wait timeout 900 sec.

You can change this timeout by modifying the value in the registry of the server where Veeam Backup & Replication is installed as follows:

1. Go to registry key HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
2. Add a DWORD (32-bit) value named VssPreparationTimeout. The value is in milliseconds (decimal), the default timeout is 900000, which equals 15 minutes. Set the value to 1800000. This will change the timeout to 30 minutes.
3. Stop any running jobs or wait for them to complete, then restart the Veeam Backup Service.

Notes:

• Certain configurations (such as MS SQL Server with FILESTREAM) or third-party VSS writers may require significant time for preparation. The timeout can be extended to several hours.
• On Exchange servers, this error typically indicates a large number of transaction logs. You can increase the timeout or truncate the logs by another method. Compare the dates on the transaction logs to the date of the last successful Veeam backup to verify transaction logs are being truncated correctly.
• On SharePoint servers, you may need to run PSCONFIG. For more info see Potential issues after installing SharePoint Foundation 2010 SP1. Unusual search scopes have also sometimes been implicated.

Identification of Problem Writer:

• Collect the logs in KB1789 and provide them when opening a technical support case.
• Alternatively, use the Diskshadow utility from an administrator command prompt within the VM guest OS:

  diskshadow /l <path to output file>.txt
  list writers detailed

• The output file will contain a list of files associated with each writer. If a particular writer has a very large number of associated files, it is probably the culprit.

Failed to freeze guest, wait timeout

• On Exchange servers, this is a variation on “Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].” See KB1680.
• If the VM is a vCenter server with a local database (local instance of SQL), or a SQL server which hosts a vCenter database, you may be encountering a problem with the automatic vCenter database exclusion. See KB1051.
• Otherwise, this usually indicates excessive VMware snapshot creation time, an in-guest performance issue, delays related to VIX API, or a combination of factors. The list of possible solutions is identical to KB1680, but the timeout is 60 seconds instead of 20. This often occurs in combination with VSS-related event log errors containing the word “abort”.
• This error message also sometimes appears when there are other problems with VSS that should generate additional logging. Collect the logs in KB1789 and provide them when opening a technical support case.

More Information

Both errors are related to Application Aware Image Processing. As a temporary workaround, you can disable AAIP in order to obtain crash-consistent backups. For more information, consult the user guide overview of Transaction Consistency.

Overview of Processing a Backup Under VSS has a visual explanation of the freeze timeout (the green box).