Challenge

The following conditions are met:

1. Nutanix AOS is upgraded up to 5.8.0.1 within STS (Short Term Support) release branch;
2. Affinity rules are configured for the affected machines:

​​ 

Cause

Solution

Challenge

Guest VMs will fail to engage VSS when application-aware image processing is enabled generating an error:

Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)

Cause

This issue has multiple potential causes, usually related to the Integration Services version of the failing guest server. This issue occurs when Application-Aware image processing is utilized in the Backup or Replication job settings for transactional quiescence using Microsoft VSS

Solution

Please Note:

• A list of supported Guest OSes for each version of Hyper-V can be found here. 
• Important! Server 2012 R2 or later guests are not supported for VSS-enabled (AAIP) backups on 2008 R2 hosts due to integration components incompatibility.
• If the VM Guest OS is 2012 and the Host is 2008R2:
  • View the Guest OS Application Event logs for errors with source: vmicvss and Event ID: 13.
  • If this error is present, open the Registry Editor (regedit) and navigate to:
    HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
    Within that key create a new DWORD
    Name: ScopeSnapshots
    Value: 0
  • Note: If the SystemRestore key is not present, you will have to create it.

The solution section of this knowledge base article has been split into four different sections. Please review each step of the following sections.

1. Integration Services
2. Within the Guest
3. VM Configuration
4. Verification

1. Integration Services

Click here for a list of Hyper-V Integration Services and their corresponding build numbers.

How to identify…

1. Using Event Viewer

This section will provide directions on ensuring the Hyper-V Integration Services are up to date within the Guest OS of the VM that is failing to backup.

If Hyper-V Integration Services are out of date you will find Event ID 4000 within the Hyper-V-Integration section of the Event Viewer on the Hyper-V host where the VM is located.

2. Using PowerShell (Server 2012 or newer)

Run the follow command in an Administrative PowerShell prompt on the host where the VM that failed to backup resides.
 

Get-VM | select VMName, IntegrationServicesState

The results of which will list out each present VM, and the state of its integration services.

How to correct:

Note: A reboot of the virtual machine will most likely be required to complete the Integration Services update.

The following information is from the Microsoft page found here.

Install or update integration services

For hosts earlier than Windows Server 2016 you'll need to manually perform install the update from within the guest operating system. These steps cannot be automated or executed within a Windows PowerShell session.

1. Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.
2. Connect to the virtual machine. Right-click the virtual machine and click Connect.
   
3. From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive.
   
   Depending on the guest operating system, you may need to start the installation manually.
   
4. After the installation finishes, you may be prompted to restart.
   

2. Within the Guest

The following steps and checks are to be performed within the Guest OS of the VM that is failing to backup.

1. Make sure Windows is running the most recent service pack.
    
2. Ensure that the VSS Storage Area and Volume Match
   1. From an Administrative Command Prompt, run the command ‘VSSUIRUN.exe”
      
   2. Check the [Settings] for each Volume and ensure that the Volume and the Storage area match.
      
       
3. Ensure there are no Dynamic disk or FAT\FAT32 volumes
   1. Open Disk Management (diskmgmt.msc)
   2. Review the Type and File System columns
   3. VSS is not compatible with Dynamic disks or FAT\FAT32
      Note: The VHD(X) that the VM runs off can be dynamic. The volume within the Guest OS cannot.
      
      All volumes must be NTFS or ReFS
       
4. Check System Reserved partition free space.
   1. System Reserved partition must have more than 42MB of space free.
   2. If less than 42MB is free, try creating a shadow copy for the volume.
      
   3. Remember to delete the Shadow Copy if it is successful.
       
5. Start\Restart the “Hyper-V Volume Shadow Copy Requestor
   1. Open Services (services.msc)
   2. Find the “Hyper-V Volume Shadow Copy Requestor” service.
   3. If it is ‘Stopped’, Start it. (If the service will not start, check in the VM settings under Integration Services, and ensure the "Backup (volume Checkpoint)" option is enabled.)
   4. If it is ‘Running’, restart it.

3. VM Configuration

The following steps and checks are to be performed from the Hyper-V host level.

1. Make sure the VM has a SCSI controller. One must be present, even if there are no devices to use the SCSI controller.
   •  The error will present itself in the Hyper-V VMMS > Admin Event log. Example:
     “The virtual machine ' Vm-Name' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID).”
2. Within the Virtual Machine settings verify that the “Backup (Volume Snapshot)” option is enabled.
3. Make sure that the virtual machine files (VHD files) are not sitting in the root folder of the CSV, as this may prevent correct snapshot creation due to permission restrictions. We strongly recommend keeping virtual machine files in separate folder. More information regarding this can be found here.

Verification

Performing the following will allow you to determine if the steps performed above have corrected the issue without the need to run the Veeam Backup & Replication job again.

1. 1. Open an Administrator Command Prompt.
   2. Run the command:

      Diskshadow /L C:\writers.txt

   3. Run the command:

      list writers

Open the file that was created and and locate the VM name in the section for the Hyper-V writer. For each VM there's a "Caption" entry. The caption should read "online" or "child partition", and not "offline" or "saved state".

More Information

PowerShell script to tell you if you need to update your Integration Services
https://blogs.msdn.microsoft.com/virtual_pc_guy/2013/04/29/powershell-script-to-tell-you-if-you-need-to-update-your-integration-services/

Recommended hotfixes for Hyper-V servers:

http://www.veeam.com/kb1838
http://blogs.msdn.com/b/virtual_pc_guy/archive/2009/02/02/looking-at-the-hyper-v-event-log.aspx
http://blogs.technet.com/b/virtualization/archive/2008/08/29/backing-up-hyper-v-virtual-machines.aspx

Challenge

Backup/Replication jobs fail with:

VDDK error: 13.You do not have access rights to this file

Solution

Below is a list of possible solutions to this issue sorted by what transport mode was being using when this error occurred.
Note: There are many causes for VDDK 13, this list is not intended to be all encompassing.

Using Transport Mode: Network Mode

• Solution 1:
  In some instances, you may receive this error when permissions are not correctly set. To confirm, please follow the steps outlined in the following KB article to test ports, permissions, and DNS resolution
  http://www.veeam.com/kb1198

Using Transport Mode: Hotadd Mode

• Solution 1:
  https://www.veeam.com/kb1054
  VMware Tools must be installed and up-to-date on the Veeam server and any Veeam proxies
• Solution 2:
  https://www.veeam.com/kb1775
  Stuck Hotadded Disks – Review all virtual proxies for disks attached that do not belong to the VM in question, remove the disks not owned by this VM without deleting from the datastore. Once deleted, consolidate snapshots on the affected VMs.
• Solution 3:
  kb.vmware.com/kb/2003638
  Orphaned Snapshots – Consolidate snapshots on affected VMs
• Solution 4:
  http://www.veeam.com/kb1875
  Permissions - Please configure the account leveraged to access the vCenter as an Administrator within vCenter
• Solution 5:
  http://www.veeam.com/kb1882
  Automount – In some cases, it may be necessary to perform an automount scrub to clear any existing mount points that are no longer in the system.
  
  Run the following via CMD Prompt on the proxy server:

  diskpart
  automount disable
  automount scrub
  exit

• Solution 6:
  http://www.veeam.com/patches.html
  Patching – Please ensure that all VMware and Veeam components are fully up to date.
• Solution 7:
  In some rare instances, performing a storage vMotion has been known to resolve the issue due to a locked VM file on the datastore where the VM files reside.
• Solution 8:
  In some rare instances, restarting the vCenter appliance or alternatively restarting the ESXi management agents (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490) may yield positive results if all above testing does not resolve the issue.
• Solution 9:
  Veeam Proxy has a Veeam replica with the same bios UUID within the vCenter Server. The following can be observed in 
  
  Job.Job_name.log:

  [time stamp] < threadID> Info VM VeeamProxy _replica(vm-100001) has the same bios uuid as proxy VeeamProxy.local in the DB

  Agent.Job_name.VM_name.Hotadd.Attacher.log:

  [time stamp] < threadID> vdl| [vddk] time stamp error -[07132] [Originator@6876 sub=transport] Cannot use mode hotadd to access [datastore_name] VM_name/VM_name.vmdk: Cannot mount using this method. (Mounting VM vm-101 using transport hotadd failed : Cannot access datastore for one of the disks of Virtual Machine VM_name..)

  The behavior is observed when a Veeam Proxy is replicated with Veeam B&R, the replication process preserves the original VM bios UUID for restore purposes. To work around the issue setup a dedicated VM as a Veeam Proxy, replicating Veeam Proxies is not usually necessary due to ease of deployment.

• Solution 10:
  In some rare cases you may encounter a limitation for the amount of disks attached to virtual SCSI controller (max. 15 disks). In this case you may try to add another SCSI controller to proxying server in use by the job. Please keep in mind that there's a limitation for the amount of SCSI controllers - 4 per VM.
   
• Solution 11:
  Known issue with vSphere 6 or later when a proxy has a SATA Controller present.
  (https://kb.vmware.com/s/article/2151091)

Using Transport Mode: Direct SAN

More Information

Additional Troubleshooting(All versions):

• Manual Hotadd - http://www.veeam.com/kb1184
• Hotadd Requirements - http://www.veeam.com/kb1054
• Deploy an additional standalone proxy to isolate failures to a single specified proxy
• To isolate a specific transport mode as the point of failure, you may also test the other transport modes to ensure that connectivity is correctly established and backup jobs work as intended via the other transport mode.

Additional Troubleshooting (vSphere 5.1 and below):
http://www.veeam.com/kb1777
Backup via Standalone host – Configure backup job to backup via Standalone host rather than through vCenter; this will isolate vCenter as the point of failure. If backup via standalone host is successful, you can attempt restarting the vCenter as well as the management agents on the host(s) where the proxies/VM(s) reside.

Information: Hotadd will only work when the proxy is on the same host as the object being backed up

Additional Troubleshooting (vSphere 5.5):
When attempting to hotadd leveraging a standalone host configuration (that is managed by a vCenter object) with vSphere 5.5 you will receive VDDK Error 13, as well as the error listed below:

Error message: [00680 error 'transport'] Reconfigure attempt failed for VM "2" with Vmomi::MethodFault::Exception ("vim.fault.HostAccessRestrictedToManagementServer")\n

To resolve this particular issue, please leverage Network Mode or Direct SAN transport mode as this configuration is not supported due to limitations of VMware

Information: For more information regarding ‘HostAccessRestrictedToManagementServer’ please see below VMware documentation.

http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.fault.HostAccessRestrictedToManagementServer.html

Challenge

Solution

1. Close the Veeam Backup & Replication console.
2. Open Regedit.exe
3. Navigate to the key HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
4. Add a new DWORD ‘EnableDBExclusions’
5. Set the Value to 1

More Information

Challenge

Cause

Solution

More Information

• VMware Cloud on AWS and Veeam - VMware KB
• VMware Cloud on AWS SDDC 1.6 - VMware Compatibility Guide Listing
• VMware Cloud on AWS SDDC 1.5 - VMware Compatibility Guide Listing
• VMware vSAN and Veeam - VMware KB
• VMware vSAN 6.7 U1 - VMware Compatibility Guide Listing
• VMware vSAN 6.0-6.5 - VMware Compatibility Guide Listing

Challenge

This article covers two most popular scenarios of adding SMB repository to Veeam Backup for Office365
Note: Such repository type is on experimental support in the current version. So i.e. if a storage device supports both SMB 3.0 and iSCSI protocol, then the best practice would be to use the latter due to its ability to use Disk Queue which acts as a buffer space for data to land on before being sent off to disk.

If it’s a Windows-based shared folder, then it’s recommended to add the server it’s based on as a proxy and connect the same storage as a directly attached disk. 

Cause

Solution

Scenario 1: Microsoft Windows SMB share with authorization

Prerequisites:

Consider that Microsoft Windows 7 SP1 and Microsoft Windows 2008 R2 SP1 do not support the SMB 3.0 share as a storage system. To use the SMB 3.0 share as a storage system, make sure you are using Microsoft Windows 8 or Microsoft Windows 2012 or later. Both Veeam Servers and SMB share location must be deployed within the same or a trusted domain.

• Open folder sharing settings for planned repository location and add all Veeam servers to it using their computer names. In example if your Veeam server is called “WIN-F4N92CDLKET” the list should look like this:

• Add repository in Veeam console as shown in this article

Scenario 2: Standalone storage appliance

In this scenario we will use Synology appliance as an example, but the same method should work for any standalone device that supports Microsoft Active Directory and its computer groups. 

• Add Veeam server and all remote backup proxies (if any) to Active Directory domain.
• Create an Active directory group in this domain and add all Veeam servers to it as “computer” objects:

• Add the storage device to the same domain:

• Make sure that SMB 3.0 protocol is enabled and previous versions are not allowed to use:

• Create SMB shared folder and grant read/write access to it for the domain group that was created earlier:

• Add new repository in Veeam console and manually type SMB share name in “Path” field as described here

Bonus: Adding standalone storage appliance using iSCSI protocol
In this scenario we will use Synology appliance as an example, but the same method would work for storage appliance that supports iSCSI protocol.

• Open iSCSI manager and create new LUN and Target:

• On Veeam server: open iSCSI initiator, enter your storage appliance IP address, click “Quick Connect…” and then click “Done”:

• Open Disk Management, bring iSCSI disk online and initialize it using default parameters:

• Create a new simple volume using default parameters:

• Create a new repository on this disk as shown here

Challenge

Job may report warning: "Failed to finalize guest processing. Details: Failed to process 'TruncateSQLLog' command. Failed to truncate transaction logs for SQL instances: <instance name>. Possible reasons: lack of permissions, or transaction log corruption."

Solution

SQL logs truncation is done under user account specified in AAIP in Job settings, if it fails then GuestHelper tries to truncate transaction logs under LocalSystem account.

In order to understand why SQL logs truncation failed you will need to open the GuestHelper log on the Guest VM, and search for "Truncation Statistics".

• Windows 2008 or later
  \\GUESTVM\c$\ProgramData\Veeam\Backup\VeeamGuestHelper_%date%.log
• Windows 2003
  \\GUESTVM\c$\Documents and Settings\All Users\Application Data\Veeam\Backup\VeeamGuestHelper_%date%.log

Known Errors and Solutions

1. Error: OpenFromInitializationString failed. [Login failed for 'DOMAIN\user'.]
   Solution: give DOMAIN\user permissions on SQL instance and add db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.
    
2. OLEDB Error: 'The server principal "DOMAIN\user" is not able to access the database "DATABASE" under the current security context.', HelpCtx: '0'
   Solution: give DOMAIN\user db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.
    
3. OLEDB Error: 'BACKUP detected corruption in the database log. Check the error log for more information.', HelpCtx: '0'
   Solution: error points to possible corruption and issues with SQL server
    
4. OLEDB Error: 'BACKUP LOG cannot be performed because there is no current database backup.'
   
   As a rule this is an issue with the secondary node of the SQL always on cluster. You can solve this by making a backup of the DB in question via SQL Management Studio. Otherwise, you can set the secondary node as primary for just one run of your backup job. As a result all its DBs will be backed up without "copy only" flag and the error will disappear.
   
   The issue occurs when the secondary node has always been backed up with "copy only" flag and its standalone DBs do not have any full backup. Thus during the truncation of the standalone DB logs we get the above-mentioned message.
   
   The same solution applies if you get this message with regard to the excluded vCenter database / Veeam database.
    
5. "Query timeout expired" If you see this entry in VeeamGuestHelper log, it usually means that we couldn't truncate SQL logs in allotted time (by default timeout is only 60 seconds). Usually you might experience such issues with rather large databases, and with large amount of transaction logs
   Solution: Implement the following registry value in affected VMs in [HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\] and [ HKLM\SOFTWARE\Wow6432Node\VeeaM\Veeam Backup and Replication] (if either location does not exist, create it):
   • SqlExecTimeout
   • Type: REG_DWORD
   • Default value: 60 (in seconds, decimal)
   Try to expand that value and run a backup afterwards, safe guess here is to set it for 600 seconds.

More Information

If you observe the following warning "Failed to truncate transaction logs for SQL instances: MICROSOFT WID" on Veeam B&R version 8.0.0.2084, please contact Veeam Support for the hot-fix.

Challenge

Product Information:

Company name: Red Hat, Inc.
Product Family: Red Hat Ceph Storage
Status: Veeam Ready – Object
Classification Description: Verified object storage solutions that has been tested with Veeam Backup & Replication Cloud Tier features.

Solution

Product Details:

Model number: 3.1
Storage Category: Software Defined Storage
Drive quantity, size, type: 7 - 1TB SATA hard drives per cluster node, for a total of 21 drives across the cluster.
Storage configuration: Hard drives were all used in single, standalone mode. No hardware or software RAID used.
Firmware version: ceph version 12.2.5-59.el7cp
Connection protocol and speed: 1GbE
Additional support: All models and configurations of Red Hat Ceph 3.1 or later for S3 compatible storage with specifications equivalent or greater than the above.

General product family overview:
Red Hat Ceph Storage 3.1 running on a 3-node cluster, with all nodes running the Object Storage Daemons (OSDs), and one node dedicated for the Monitor daemon (MON), Manager daemon (MGR), and Object Gateway daemon (RGW).

Veeam testing configuration:

Veeam Build Number: 9.5.4.2282

Vendor recommended configuration:

Vendor Settings:

• No hardware deduplication used in testing
• No hardware compression used in testing

Challenge

Failed to execute SOAP command "CInvSvcCreateCategory". Inventory Service was not created.
Can not Create tag with EXCEPTION: CreateCategory failed.
Failed to dispose inventory service proxy EXCEPTION: The remote server returned an error: (401) Unauthorized.

Cause

Solution

• Open Windows registry editor snap-in and navigate to the following registry path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Reporter Enterprise\
• Create a new registry key or modify the existing one:

More Information

Challenge

Product Information:

Product Family: ES1640dc v2
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: ES1640dc v2
Storage Category: RAID Storage Array
Drive quantity, size, type: 15 – 6TB SAS drives
Storage configuration: RAID6
Firmware version: 2.1.0
Connection protocol and speed: 10GbE
Additional support: All models and configurations of ES1640dc v2 with specifications equivalent or greater than the above.

Veeam testing configuration:

Veeam Build Number: 9.5.0.1536

Job Settings:

• Deduplication: Enabled (Default)
• Compression: Optimal (Default)
• Storage Optimization: Local Target (Default)

• Repository Type: Shared Folder
• Per-VM Backup Files: Disabled (Default)
• Decompress before storing: Disabled (Default)
• Align backup file blocks: Disabled (Default)

Vendor recommended configuration:

Hardware Settings:

• Jumbo frames used in testing environment
• Multi-pathing using SMB3 used in testing environment
• No array compression or deduplication was used during testing

More Information

Company Information:

Company name: QNAP Systems, Inc
Company overview: QNAP Systems, Inc., as its brand promise "Quality Network Appliance Provider", aims to deliver comprehensive offerings of cutting edge network attached storage (NAS) and network video recorder (NVR) solutions featured with ease-of-use, robust operation, large storage capacity, and trustworthy reliability. QNAP integrates technologies and designs to bring forth quality products that effectively improve business efficiency on file sharing, virtualization applications, storage management and surveillance in the business environments, as well as enrich entertainment life for home users with the offering of a fun multimedia center experience. Headquartered in Taipei, QNAP delivers its solutions to the global market with nonstop innovation and passion.

Challenge

Connecting to the Office 365 organization fails with the “Check Exchange plan: No Exchange plan found” or “Check SharePoint plan: No SharePoint plan found” error.
 

Cause

Some of the Office 365 Exchange or SharePoint service plans assigned to the Office 365 organization cannot be automatically identified as valid by Veeam Backup for Microsoft Office 365. This happens because assigned Office 365 plans are missing from the list of valid ones.

Solution

To resolve this issue, add missing Plans IDs to the list:

1. Get the current Office 365 plan IDs assigned for the organization. You can do this via ‘MSOnline’ AzureAD module and the following PowerShell One-Liners: *

   Note: This script is provided as is as a courtesy for Gathering Plan IDs from MSOnline. Please be aware that the commands and parameters may be altered by Microsoft in the future, which may cause the script to function incorrectly. There is no support provided for this script, and should it fail, we ask that you please contact Microsoft support. This script may be updated in the future to reflect changes to the Office 365 Powershell Environment

   • To call "get-msolaccountsku" (use the CMDLET), call "Connect-MsolService" first. This requires the following modules to be installed and activated:

     install-module AzureAD
     install-module AzureADPreview
     install-module MSOnline

   • To get the list of Exchange Online service plans IDs:

     (get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "Exchange"} | ft ServiceName,ServicePlanId

   • To get the list of SharePoint Online service plans IDs:

     (get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "SharePoint"} | ft ServiceName,ServicePlanId

2. Make sure that no backup jobs or restore sessions are running.
3. Stop the Veeam.Archiver.Service and Veeam.Archiver.Proxy services.

4. Navigate to the C:\ProgramData\Veeam\Backup365 directory on the VBO365 server, create a backup of the Congif.xml file by copying it to another location, then (Only after making the pre-edit copy) edit the Config.xml file by adding the following lines between <Archiver> tags:

   <Options>
   <O365Plans AdditionalExchangePlans="your-Exchange-plans-IDs" AdditionalSharePointPlans="your-SharePoint-plans-IDs" />
   </Options>

    

   If you do not have any sharepoint plans, do not add the AdditionalSharePointPlans to the line mentioned

   NOTE: You will have to add the Plans as the examples shown in the following screenshot. The GUIDs are entered in Coma Separated Value within the Quotation after each of the key values in the <O365Plans line.

5. Start the Veeam.Archiver.Service and the Veeam.Archiver.Proxy services.
6. Open ‘Add organization’ wizard and retry the procedure.

More Information

In case the issue persists after these changes, please, contact Veeam Support.

Challenge

This article explains how to collect the diagnostic information required for a Veeam Agent for Microsoft Windows support case.

Solution

To encourage quicker support case processing, attach the following logs to the support case:

• Veeam Agent logs
• VSS diagnostics
• Event logs
• File system minifilter diagnostics

You can collect logs in the following ways: 

• Automatically using the script that is attached to this article.
• Manually in case the script fails to retrieve required logs.

Collect Logs Automatically

[[DOWNLOAD|DOWNLOAD SCRIPT|https://storage.veeam.com/tmp/File_kb3025_713f5f0e22.zip]]

Download the script from the link above.

1. Extract the script to any folder on the computer where Veeam Agent for Microsoft Windows is installed (for example, C:\temp\).
2. Open the folder with the script. Depending on the OS settings, you might be prompted to make execution policy changes to run the script.

4. Run the script with PowerShell. The script executes simple select queries, no system changes are made.

If the script runs successfully, the warning message will appear. 

After the script execution, the message about the successful execution will appear. The script creates a new folder with the VeeamAgentLogs_<DDMMYYYY> name, where <DDMMYYYY> is the current date, and stores the collected information into this folder. 

5. Attach the folder with all the contents to the support case.

If the script fails to collect the logs, proceed to collect the information manually as outlined below.

 

Collect Logs Manually

The following steps are to be done on the computer where Veeam Agent for Microsoft Windows is installed.

1. Collect Veeam Agent Logs

1. Navigate to the following folder: C:\%Programdata%\Veeam\Endpoint (this folder is hidden by default, copy and paste the provided path). 
2. Archive the content of this folder.

2. Collect VSS Diagnostics

1. Run the following commands using the administrative command prompt:

• vssadmin list writers > C:\vss_writers.log
• vssadmin list providers > C:\vss_providers.log
• vssadmin list shadows > C:\vss_shadows.log
• vssadmin list shadowstorage > C:\vss_shadow_storage.log

2. Archive the resulting files.

3. Collect Event Logs

1. Run the Event Viewer (eventvwr.msc).
2. In the left menu, find the Application logs.
3. Right-click the Application logs and select Save All Events As.
4. Enter the file name that includes the log type and the computer name the logs were exported from (for example, when exporting Application logs from a computer named HV01, enter Application_HV01).
5. In Save as type, select Event Files (*.evtx).
6. Include the display information.

7. Repeat steps 3—7 for the System logs.
8. Archive the logs. Include the LocaleMetaData folder in the archive as well.

4. Collect File System Minifilter Diagnostics

1. Run the fltmc instances > C:\filter.log command using the administrative command prompt.

Send Logs to the Veeam Support

1. Create a support case.
2. U​​pload the logs to the FTP server provided by the Veeam Support engineer. If you need a Secure FTP, see this article.
3. After the upload, notify the Veeam Support engineer.

More Information

Challenge

Incompatible Veeam Backup for Office 365 server version (received: 9.6.5.XXX, expected: 9.6.4.YYY).

Cause

Solution

1. Close all Veeam Explorers.
2. Download the latest version of Veeam Backup for Office 365 with Veeam Explorers here
3. Install the following MSI packages: 
   • VeeamExplorerForExchange_X.X.X.XXX.msi.
   • VeeamExplorerForSharePoint_X.X.X.XXX.msi.
4. Restore data using Veeam Explorer for Microsoft Exchange, Veeam Explorer for Microsoft SharePoint or Veeam Explorer for Microsoft OneDrive for Business.

Challenge

Failed to start backup job Error: Unknown invoker method: 2757

Cause

Solution

Challenge

Product Information:

Company name: Amazon Web Services
Product Family: Amazon Simple Storage Service (Amazon S3)
Status: Veeam Ready – Object
Classification Description: Verified object storage solutions that has been tested with Veeam Backup & Replication Cloud Tier features.

Solution

Product Details:

Object Storage Tiers: 

• Amazon S3 Standard
• Amazon S3 Standard-Infrequent Access

General product family overview:
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2753

Challenge

A VM that was processed by Veeam is found to require consolidation, yet consolidation attempts fail.

or

A VM that was processed by Veeam is found to have an orphaned snapshot.

Cause

This can occur when Virtual Appliance Mode (HOTADD) is used due to:

• The command to release the virtual machines's disk from the proxy is not processed by VMware.
• The Veeam server crashes while processing a virtual machine.
• Jobs running within Veeam are forced to stop, and those jobs were using Virtual Appliance Mode (HOTADD).
• A backup or replication job creating a snapshot on the backup proxy while the proxy is being used by another job. (Note: It is advised to build dedicated Proxies, and not backup or replicate them) 

Solution

Before beginning ensure that all jobs have stopped. This way the only disks that will be found attached to proxies will be the proxy's own disks and ones that need to be removed.
Identify Problem VMs

1. Using a vSphere Client find any virtual machines which are displaying a warning that consolidation is needed.
   
    
2. Perform consolidation, and note which virtual machines continue to have a warning regarding consolidation.
   
   
   

Find Stuck HOTADDed Disks

Review the settings of each Veeam Proxy looking for any disks which match the following:

• Name of VMDK does not match the proxy, and matches a VM identified in the previous section.
• Disk has a Disk Mode of [Independent - NonPersistent]
  Note: If the proxy is used as a Target Proxy for replication, the replica disks will be attached as Dependant.
  
  

Release Stuck HOTADDed Disks

Once you have found a disk HOTADDed to a proxy which needs to be released perform the following steps to release it.

1. Click the ⦻ symbol to select the disk for removal.
   
   Note: Do not check the box to delete files from datastore!
   
2. Click [OK] to Confirm removal.
3. After the disk is released from the proxy, attempt Consolidation again.

More Information

If you find that a specific proxy continues have issues with disks becoming stuck HOTADDed to it, please review: https://www.veeam.com/kb1882

For more information regarding orphaned snapshots please read: https://kb.vmware.com/s/article/1005049

Challenge

Data transfer is failing across a connection that is protected by an advanced firewall. The firewall uses signature-based detection. Relevant features may have names like antivirus, anti-spyware, intrusion prevention, or application control.

This may affect any make or model of firewall with similar features, but support cases have been opened for:

• Dell SonicWALL
• Check Point
• Cisco
• Palo Alto
• Fortinet appliances.

Any type of data transfer may be affected, including backup, replication, and backup copy jobs. This problem may manifest in a wide variety of error messages and failure patterns. It may appear random, or completely consistent. One common failure pattern is for transfer of specific VM disks to consistently fail at or around the same percentage of completion.

Common error messages include:
 

An existing connection was forcibly closed by the remote host.

Unstable connection: unable to transmit data.

Depending on the job type and the version of Veeam Backup & Replication, connection failures may cause the job to fail immediately, or the connection may retry several times. While the connection is retried, the job may appear to be frozen because it is unable to transmit data. For more information, see:
Resume on Disconnect
Backup Copy: Automatic Job Retries
WAN Acceleration: Data Transport on WAN Disconnect

Cause

Disk images contain a potentially unlimited variety of data blocks. Because the traffic is compressed (and in most cases, encrypted), data blocks analyzed by a firewall will be different from data as it exists on the virtual disk. Over the long term, this approximates feeding random data into the signature-based threat detector: false positives are inevitable.

Data transfer is not actually random: a particular data block will always have the same signature after compression and encryption. If the source data does not change, the same blocks will be re-sent on every reconnect attempt and every retry of the job. In this case, the firewall will close the network connection every time Veeam Backup & Replication tries to transfer that data block, because it incorrectly detects a pattern of data within that block as matching the signature of a known threat.

Solution

Create exclusions for Veeam data traffic. In most cases, the relevant traffic will be between proxies or repositories over a default port range of TCP 2500-5000. This range can be configured for each managed server in the backup infrastructure settings.

Cloud Connect Service Providers should create exclusions for data traffic sent to Cloud Gateways on port 6180 (TCP and UDP).

For more information on port ranges, find the relevant product or component user guide in the Help Center and consult the Used Ports page.

For information on how to configure exclusions on a specific firewall appliance, please contact your vendor.

More Information

To isolate whether or not an issue is caused by a firewall, temporarily disable all signature-based features in the firewall’s configuration. For best results, do this while data transfer appears to be frozen – traffic should resume in no more than a few minutes. In some cases, the firewall may allow you to selectively disable specific sites or zones; this can be a useful as a solution, but it is not a good isolation step because such features are easily misconfigured.

Reset packets generated by firewall appliances can usually be distinguished from normal traffic by their IP time to live. For example, if most packets in a TCP stream have a TTL of 128, but the reset packet that closes the stream has a TTL of 64, the connection was closed by a firewall.

Firewall features that block encrypted key exchange will block the majority of WAN connections used by Veeam Backup & Replication.

Challenge

Cause

Solution

1. Download archive with u4b_FIX_Licensing.Reporting.CC.GetInstancesInfo.sql file, extract and copy it to the SQL server, which hosts the Veeam Cloud Connect database or any other machine that can connect to the SQL instance in question.
2. Connect to the SQL instance by using Microsoft SQL Studio. When connecting, please provide valid credentials with full permissions.
3. Find your Veeam Cloud Connect database (its default name is VeeamBackup). Right-click it and select New Query.
4. Open file u4b_FIX_Licensing.Reporting.CC.GetInstancesInfo.sql by using the text editor, copy all its content and paste it to the query window from step 3.
5. Run the query by clicking Execute or pressing F5. Make sure that the query is executed successfully.
6. The fix is applied immediately, restart of Veeam services is not required.

More Information

Challenge

Cause

Solution

• Public role (default permissions)
• CREATE ANY DATABASE permissions
• db_owner role on the Veeam ONE database
• db_datareader permissions on the master database
• public, db_datareader, SQLAgentUserRole permissions on the msdb database
• [For Always-On Availability Groups] VIEW SERVER STATE permissions

More Information

Challenge

Cause

Solution

1. Download archive with u4b_FIX_Licensing.Reporting.CC.GetInstancesInfo.sql file, extract and copy it to the SQL server, which hosts the Veeam Cloud Connect database or any other machine that can connect to the SQL instance in question.
2. Connect to the SQL instance by using Microsoft SQL Studio. When connecting, please provide valid credentials with full permissions.
3. Find your Veeam Cloud Connect database (its default name is VeeamBackup). Right-click it and select New Query.
4. Open file u4b_FIX_Licensing.Reporting.CC.GetInstancesInfo.sql by using the text editor, copy all its content and paste it to the query window from step 3.
5. Run the query by clicking Execute or pressing F5. Make sure that the query is executed successfully.
6. The fix is applied immediately, restart of Veeam services is not required.

More Information