Troubleshooting VDDK error 16053

Challenge

VMware backup jobs fail after update 3, whereas prior to update they have been working.

• The jobs are using Network transport mode
• Source Proxy is Windows 2008SP1/Windows 2008R2

The error message for all jobs in the GUI looks like this one:

Processing VM_Name Error: Failed to open VDDK disk [datastore] VM_Name/VM_Name.vmdk] ( is read-only mode - [true] ) Logon attempt with parameters [VC/ESX: [hosname.local];Port: 443;Login: [administrator@vsphere.local];VMX Spec: [moref=vm-ref];Snapshot mor: [snapshot-ref];Transports: [nbd];Read Only: [true] failed because of the following errors: Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

Solution

This issue is related to some of the dll files in the Windows ZIP file for VDDK 6.5 with incorrect manifests, making them depend on two Visual C++ 9.0 versions, both 9.0.32709.4148 and 9.0.21022.8. This causes a failure to load vixDiskLibVim.dll with error code = 0x36b1, which means side-by-side configuration is incorrect.

Workaround

Edit failing jobs and configure them to use a proxy running Windows 2012/2012R2/2016 proxy.

Note: The steps to Verify and Correct this issue below are to be performed on the proxy which was used by the failing job.

Verify issue matches this KB:

1. On the source proxy navigate to Veeam logs location (default %ProgramData%\Veeam\Backup)
2. Open the folder that matches the name of the failing Job.
3. Find the log file matching the nomenclature Agent.Job_Name.Source.VM_Name.vmdk.log.
4. Open the file with any text editor and search for:
   VDDK error: 16053 (One or more required subsystems failed to initialize)
   
   Example snippet:

   [DD.MM.YYYY HH:MM:SS] < 12620> cli| ERR |Command 'Open' has failed.
   [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|VDDK error: 16053 (One or more required subsystems failed to initialize). Value: 0x0000000000003eb5
   [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|--tr:Failed to open virtual disk [Test_Vol1_LUN3] TEST/TEST.vmdk (flags: 4)
   [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|An exception was thrown from thread [12620].

5. To further confirm, open event viewer and in Application logs search, Source: 'SideBySide' Event ID: '35'
   ​example event:
   

How to correct this issue:

1. Open Programs and Features and verify that Microsoft Visual C++ 2008 Redistributable Package (x64) is not installed.
2. Download and install Microsoft Visual C++ 2008 Redistributable Package (x64) (https://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15336)
3. Ensure that no other jobs are running, and reboot the proxy.
4. Retry failed jobs.

More Information

Note: This KB article only applies if VDDK error 16053 is found in the logs. If this error is not found in the logs, please contact Veeam support for further troubleshooting.

See also: https://vdc-download.vmware.com/vmwb-repository/dcr-public/2ab18705-4960-45f8-88a5-830c5a308d9e/23ca4c1c-821f-4093-91f6-e38912977e24/vsphere-vddk-651-release-notes.html