Challenge
VMware backup jobs fail after update 3, whereas prior to update they have been working.- The jobs are using Network transport mode
- Source Proxy is Windows 2008SP1/Windows 2008R2
Processing VM_Name Error: Failed to open VDDK disk [[datastore] VM_Name/VM_Name.vmdk] ( is read-only mode - [true] ) Logon attempt with parameters [VC/ESX: [hosname.local];Port: 443;Login: [administrator@vsphere.local];VMX Spec: [moref=vm-ref];Snapshot mor: [snapshot-ref];Transports: [nbd];Read Only: [true]] failed because of the following errors: Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Solution
This issue is related to some of the dll files in the Windows ZIP file for VDDK 6.5 with incorrect manifests, making them depend on two Visual C++ 9.0 versions, both 9.0.32709.4148 and 9.0.21022.8. This causes a failure to load vixDiskLibVim.dll with error code = 0x36b1, which means side-by-side configuration is incorrect.Workaround: set the failing jobs to Windows 2012/2012R2/2016 proxy.
How to confirm and fix:
1. On the source proxy navigate to Veeam logs location (by default C:\ProgramData\Veeam\Backup), find %Issuing_Job_Name% folder and find Agent.Job_Name.Source.VM_Name.vmdk.log.
2. Open the file with any text editor and search for VDDK error: 16053 (One or more required subsystems failed to initialize)Example snippet:
[28.12.2017 14:24:51] < 12620> cli| ERR |Command 'Open' has failed.
[28.12.2017 14:24:51] < 12620> cli| >>|VDDK error: 16053 (One or more required subsystems failed to initialize). Value: 0x0000000000003eb5
[28.12.2017 14:24:51] < 12620> cli| >>|--tr:Failed to open virtual disk [Test_Vol1_LUN3] TEST/TEST.vmdk (flags: 4)
[28.12.2017 14:24:51] < 12620> cli| >>|An exception was thrown from thread [12620].
3. To double-check, open event viewer and in Application logs for the same timestamp search for SideBySide event ID 35.
example event:
4. Open Programs and Features and verify that Microsoft Visual C++ 2008 Redistributable Package (x64) is not installed.
5. Download and install Microsoft Visual C++ 2008 Redistributable Package (x64) (https://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15336)
6. Ensure that no other jobs are running, and reboot Source proxy if needed.
7. Retry failed jobs.
More Information
Please note that this is ONLY about VDDK error 16053, in case this error is not found in the logs, please contact Veeam support for further troubleshooting.See also: https://vdc-download.vmware.com/vmwb-repository/dcr-public/2ab18705-4960-45f8-88a5-830c5a308d9e/23ca4c1c-821f-4093-91f6-e38912977e24/vsphere-vddk-651-release-notes.html