Challenge

Cause

Solution

Challenge

Direct-SAN transport fails or configuration assistance is needed.

Solution

You must disable automount in diskpart on the proxy that the volumes are presented to, to ensure that VMFS volumes are not modified. (http://kb.vmware.com/kb/2002227) 

Requirements:

• Only VMFS type datastores are supported.
• The volume for each datastore containing a VM that is intended to be backed up over Direct-SAN must be presented to the Backup Proxy.
• The volume must appear in Disk Management within the Backup Proxy Server.

Known Issues:

• As the number of volumes presented to the Backup Proxy increases the delay for starting of the VM task may increase. This is due to the job scanning each volume. This can be mitigated by manually specifying in the Backup Proxy setting which datastores are connected.
• Under certain circumstances volumes may not be automatically discovered and necessitate manually specifying which datastores are connected in the Backup Proxy settings.

More Information

• Veeam Backup & Replication User Guide - Data Backup in Direct SAN Access Mode
• How to setup Direct-SAN Access vee.am/kb1446
• Enabling MPIO breaks Direct-SAN mode vee.am/kb1107
• Direct-SAN stops working vee.am/kb1115
• Direct SAN access tips for iSCSI VMFS volumes and backup proxies

Challenge

Jobs fail with an error message indicating incorrect user name or password. An account associated with Veeam is being locked out in Active Directory.

Cause

Solution

Run services.msc in order to access details of services on the Veeam Backup & Replication and/or Backup Enterprise Manager servers. Sort by service name, and scroll down to find the Veeam services.

Key services (highlighted below) should have administrator credentials entered. This frequently is done with a Veeam service account, but can be any administrator account. If the logged-in user is an administrator, “Local System” can suffice, but in environments with multiple backup operator users, it is recommended to use a service account.


To change the account of a service, right-click the service and select Properties, then the Log On tab. You can enter the account or browse for it in AD, then provide the password.


Credentials are also specified in Veeam for:

• Any Managed Server (aside from “This Server”):

• ​Credentials for CIFS shares

• Credentials for EMC Data Domain and HPE StoreOnce deduplicating appliance repositories

• Guest credentials for Application Aware processing

 

More Information

Challenge

This article documents general performance expectations, best practices, and configuration advice, when using an EMC Data Domain appliance with deduplication as a repository for Veeam Backup & Replication.

Solution

For further information regarding how Veeam Backup & Replication works with EMC Data Domain DDBoost please review: https://helpcenter.veeam.com/docs/backup/vsphere/emc_dd.html

Performance Expectations

EMC Data Domain Deduplication Storage Systems provide both high compression and deduplication ratios so that data can be kept for extended periods. When a Data Domain is configured as a repository for Veeam Backup & Replication, write performance may vary depending upon the particular EMC Data Domain Deduplication System model, protocol, and backup infrastructure architecture.

When attempting to read data from a Data Domain it must rehydrate and decompress each block, for this reason operations which read from the Data Domain will perform slower than non-deduplicated storage, this is more noticeable with operations which use random I/O. All restores will occur as fast as the environment can accept new information, and as fast as the Data Domain can decompress and rehydrate the blocks.

For quick recovery you may consider using fast primary storage and keeping a several restore points (3-7) for quick restore operations such as Instant Recovery, SureBackup, Windows or Other-OS File restores since they generate the highest amount of random reads. Then use the DataDomain as a secondary storage to store files for long term retention. If an EMC Data Domain Deduplication System will be used as primary storage, it is strongly suggested to leverage alternative restore capabilities within Veeam Backup & Replication such as Entire VM restore and VM files restore. This may result in faster recovery capabilities when used with EMC Data Domain Deduplication Systems than Instant Recovery and File Level Restore operations.

Instant Recovery

• This type of restore can be effected adversely by the aforementioned limitations of a Data Domain appliance, and also the type of VM being restored. Highly transactional VMs will require more IOPS from the Data Domain during the Instant Recovery than others. With this in mind you can expect to only be capable of running only a few Instant Recoveries simultaneously. Instant Recovered VMs that are started from a backup file stored on a Data Domain may react or start slowly as the majority of their read operations will be hindered by the Data Domain.
• For VMware users it is highly advised when performing an IR that the user select to have virtual disk updates redirected to a high performance Datastore. This will improve performance by caching written blocks to low latency storage.
• It is advised that if the VM is intended to be made permanent that the VM be migrated to production storage as soon as possible after the Instant Recovery has begun.

File Level Restores

When performing Veeam Backup & Replication File Level Restore (FLR) capabilities slow recovery times may be experienced. During Veeam FLR recovery capabilities, a significant amount of read activity occurs when accessing the Veeam “service data” metadata for each individual file as the Veeam backup files are not arranged in sequence. This read activity must be performed to determine the location of the data block(s) associated with each file during granular restore sessions. This significant level of random access is not recommended with archive tier storage devices because they are designed for optimal performance with sequential read operations. Veeam recommends implementing EMC Data Domain Deduplication Storage Systems as a secondary target for these use cases as the more random read operations, the slower the restore will be with EMC Data Domain Deduplication Systems.

• The backup browser may take longer than usual to open if an increment is selected and furthermore by that increments distance from the full restore point.
• Navigating between folders within the Backup Browser may take additional time as each folder’s content must retrieved from the backup file to display it.

Backup

• Reverse Incremental performance will be very poor due to its highly random I/O.
  Note: When the Backup Job is configured with a DDBoost repository, Veeam Backup & Replication will prevent Reversed Incremental from being selected by the user.
• Synthetic Full creation will be very slow to a Data Domain, unless using DDBoost.
• Synthetic Full with Transforms are not advised.

Backup Copy

• A retention longer than 30 is not advisable as restore operations will diminish in performance.
• The Health Check option may take a very long time as it is performing a read operation.

Replication

• Using the Datadomain to store Replica metadata is not advisable.
  Note: Veeam Backup & Replication will prevent the user from selecting a DDBoost repository.

Veeam Backup & Replication Configuration

Parallel processing (global option):

This option significantly accelerates the backup process and decreases the backup window since virtual disk data is gathered simultaneously. It also dramatically increases fragmentation in the backup files causing high random read for any restore operation. The greater volume of VMs or disks processed simultaneously will increase fragmentation and result in slow restore times. You may consider disabling Parallel Processing, this will decrease backup performance, but increase restore performance.

Storage optimization (job option):

Setting the storage optimization to Local 16TB+ has been shown to improve the effectiveness of Data Domain’s deduplication. The larger this value is, the smaller the preparation phase will be for a backup task and less memory will be used to keep storage metadata in memory.

Inline-deduplication (job option):

Since EMC Data Domain Deduplication Systems have excellent hardware deduplication and compression capabilities, it is highly advised that Veeam built-in deduplication be disabled to decrease load on the backup proxy.

Decompress backup block before storing(repository option):

Veeam strongly recommends enabling this option so that raw data is sent to the EMC Data Domain Deduplication System, leveraging its global deduplication and compression capabilities. Leaving Veeam compression enabled may significantly impact EMC Data Domain deduplication capabilities resulting in high load and slow backup jobs.

Use Per-VM Backup Files(repository option):

Veeam recommends enabling this option so that there is improved performance writing and reading data from the EMC Data Domain Deduplication System. This option is enabled by default when adding the repository as a Deduplicating Storage Appliance.

Backup Mode

• For CIFS/NFS presented repositories Forward incremental mode with periodic Active full backups is recommended to avoid the rehydration penalty during synthetic operations.
• For DDboost enabled repositories Forward incremental mode with either Active full or Synthetic full backups is recommended. Synthetically produced full backups will generally have the best restore performance and reduce the time VM is run off of a snapshot during the backup job run. However in some environments an Active Full job may run faster.
• Transforming previous backup chains into rollbacks is not advisable for both repository types.
• For forever forward incremental backup and backup copy on DDboost enabled repositories, the option “Defragment and compact full backup file” should be enabled if available. In most cases a weekly schedule is appropriate. This helps to avoid excessive growth of pre-compression data size for the full backup file.

Repository Performance Expectations and Configuration

If DDBoost is not licensed on the Data Domain system it must be added as a CIFS type or Linux type repository. It is advised to use a Linux server with the volume mounted via NFS as a relay server to help improve performance. Under some circumstances, CIFS or NFS communication may perform better than DDBoost with Veeam Backup & Replication v8 because of the limitation of a single thread per backup job when using DDBoost. DDBoost has been shown to improve performance when performing Synthetic Fulls.

With Veeam Backup & Replication v9, support for EMC Data Domain Boost is enhanced with the introduction of the following capabilities:

• Support EMC Data Domain Boost 3.0
• Reduced impact of storage fragmentation during restore operations even with enabled parallel processing. This feature allows Veeam to store the VM backup in the dedicated backup chain so that fragmentation ratio will be minimum.
• Reduce the impact of the block size so you may define any block size without impact on the restore process. Veeam will be able to read data granularly so amount of the redundant will be minimum.

With DDBoost

If the Data Domain System is licensed for DDBoost please proceed to configure it using the following steps.

1. Launch the creation of a new Repository, on the Type tab select Deduplication storage appliance.
2. Select the deduplication storage as EMC Data Domain.
3. On the next tab configure the information to for connecting to the Data Domain appliance.
4. On the Repository tab click Browse and select the necessary location from the list of available paths.
5. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Without DDBoost

CIFS

1. Launch the creation of a new Repository, on the Type tab select CIFS
2. On the next tab configure the path to which the Repository will write to, and set credentials to access that share.
3. On the Repository tab within the advanced section, enable “Decompress backup data blocks before storing”
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

NFS

The Data Domain will need to be configured for NFS access, and configure a Linux server to mount the volumes from the Data Domain via NFS. Please refer to the following links for further information regarding connecting Linux to the Data Domain via NFS:

http://forums.veeam.com/veeam-backup-replication-f2/veeam-datadomain-and-linux-nfs-share-t8916.html

http://tsmith.co/2014/veeam-and-datadomain/

1. Launch the creation of a new Repository, on the Type tab select Linux
2. On the next tab select the Linux server that we be connected to. If it is not present in the list select “Add New…”
3. On the Repository tab specify the path on the Linux server that leads to where you mounted the Data Domain via NFS. On this tab in the advanced section enabled “Decompress backup data blocks before storing”.
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Challenge

Cause

Solution

More Information

Challenge

Cause

Please confirm that you are running version 9.0 Update 2 (build 9.0.0.1715) or later prior to installing this update. You can check this under Help | About in Veeam Backup & Replication console. After upgrade, your build number will be 9.5.4.2753.

You will also see the vendor name changed from Veeam Software AG to Veeam Software Group GmbH, reflecting the legal company name change. Please note that the name change has also resulted in a change of the code signing certificate.

Solution

• VMware vSphere 6.7 U2 readiness including support for VMs with virtual hardware version 15. Please note that the testing has been performed on pre-release 6.7 U2 builds, while the official support will still require a comprehensive testing of the GA build.
• Microsoft System Center Virtual Machine Manager 2019 is now supported as a source for Microsoft Hyper-V VMs backup jobs.
• Dell EMC Data Domain using DD OS 6.2 is now supported as the backup repository via both DDBoost and CIFS/NFS protocols.
• NSS trustee rights on files and folders are now restored during file-level recovery from NSS volumes hosted on Micro Focus Open Enterprise Server (OES).
• Experimental support for keyboard-interactive authentication can be enabled by creating the LinuxKeyboardInteractiveAuth (DWORD) = 1 registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. By default, we will look for the "password" substring in the SSH console prompts. You can override this using the LinuxKeyboardInteractiveAuthPrompt (REG_MULTI_SZ) value, where multiple substrings can be specified one per line.
• New Japanese era compatibility. 新元号の変更検証済み

• Certificate verification issues cause the backup console taking too long to open, or failing to open with "Object reference not set to an instance to an object" from backup console or "Could not load file or assembly" error in the log.
• Upgrade to Update 4 fails with the "Database version downgrade detected ... Reboot and restart the setup" error on machines where system time regional format uses a.m. and p.m. (lowercase with dots).
• SureBackup jobs start failing after upgrading to Update 4 due to virtual lab settings incorrectly updated by the configuration upgrade, in case of lack of free disk space in the vPower NFS cache folder.
• On Hyper-V versions prior to 2016, jobs with application-aware processing disabled fail with "The system cannot find the file specified" error.
• Backup of Windows XP SP1 VMs fails with the "Failed to call RPC function Vss.GetFileFromGADir" error.
• Various primary storage integration issues with HPE Nimble, NetApp, IBM and Dell EMC VNX storage.
• Tape support reliability issues, including a number of UI performance issues with tape management functionality.
• HPE StoreOnce Catalyst library memory leak issue (solved by updating the library to version 24.90.0.8).
• VeeamZIP™ operation incorrectly consumes a license for VMs that were previously licensed by regular backup jobs.

More Information

If you have 9.5 Update 4 installed (RTM build 9.5.4.2399 or GA build 9.5.4.2615), you can upgrade to Update 4a using the following update package:

[[DOWNLOAD|DOWNLOAD UPDATE|https://www.veeam.com/download_add_packs/vmware-esx-backup/9.5.4.2753.update4a/]]
MD5: 9E7438488FFD10B8F804EE2B8C8C03CF
SHA1: D5F888D81426AC3648E6E0BB0412F63003A26517

For new installations and upgrades from the previous versions, use the 9.5 Update 4a ISO image:
[[DOWNLOAD|DOWNLOAD ISO|https://www.veeam.com/send_license/vmware-esx-backup/]]
MD5: 098F4B4048D6E2B29F08D0386B7FA7FE
SHA1: 4B8EF9F6F3C7AFA013DFA248687F62F322A0D180

Should you have any questions, contact Veeam Support.

Challenge

You need to move Veeam Backup & Replication or Enterprise manager software and/or database to a new server.

Solution

Veeam Backup & Replication migration

Note: This section will refer to the server where Veeam Backup & Replication is installed prior to the move as the “old server”, and the location after the move as the “new server”.

There are two methods to perform Veeam Backup & Replication migration task:

• Migrate using Configuration Backup (recommended)
• Migrate using SQL Backup

Migrate using Configuration Backup

Perform the following steps on the old server:

1. Stop and disable all jobs.
2. Manually perform a configuration backup.
3. Close Veeam GUI and stop all Veeam Services.

Perform the following steps on the new server:

1. Install Veeam Backup & Replication with a brand new DB.
2. Apply the latest patch to Veeam Backup & Replication: http://www.veeam.com/patches.html
3. Make sure all local drives that were being used as repositories on the old server are now attached with the same drive letters on the new server.
4. Perform configuration backup restore:
   Note: It will be necessary to re-enter the password for every account during restore of non-encrypted configuration backups.
5. Run a test job to make sure everything moved correctly.

Migrate using SQL Backup

IMPORTANT: Use this option only if Configuration Backup is not available. If encryption was used on the previous installation, a newly installed backup server will not be able to decrypt such information if attached to the existing database, so any encrypted information will have to be supplied manually – existing encrypted backups have to be re-imported to provide the password.

Perform the following steps on the old server:

1. Stop and disable all jobs.
2. Close Veeam GUI and stop all Veeam Services.
3. Create backup of the SQL database. (http://vee.am/kb1471)
   
   To find out where the VeeamBackup database is located, open regedit on the old server and navigate to: [HKLM\SOFTWARE\Veeam\Veeam Backup and Replication]
    
   There you will find 3 dword values:
   • SqlServerName
   • SqlInstanceName
   • SqlDatabaseName

Perform the following steps on the new server:

1. Install Veeam Backup & Replication with a brand new DB.
2. Apply the latest patch to Veeam. http://www.veeam.com/patches.html
3. Make sure all local drives that were being used as repositories on the old server are now attached with the same drive letters on the new server.
4. Restore the Veeam database to the instance on the new server or on a SQL server which will host Veeam DB.
   Click (here) for information regarding restoring databases.
5. To associate Veeam with the restored database, please use the DBConfig Utility to change the Database that Veeam is connecting to.
6. Restart Veeam Services to apply the changes.
7. Open the Veeam console and using the “Manage Credentials” button from the main menu and check the passwords for every account to confirm they are up to date.
8. Run a test job to make sure everything moved correctly.

Enterprise manager migration

Note: This section describes how to move the Enterprise Manager (EM) database to a different SQL if the EM is installed separetly from the Backup & Replication.

1. Relocate VeeamBackupReporting DB to the new SQL server. See http://vee.am/kb1250
2. On the Veeam Backup & Replication server, open the following directory:
   C:\Program Files\Veeam\Backup and Replication\Backup
3. On the EM server, open the following directory:
   C:\Program Files\Veeam\Backup and Replication\Enterprise Manager
4. Сopy the following files from the VBR folder to the EM one:
   • Veeam.Backup.Configuration.Tool.exe
   • Veeam.Backup.DBConfig.exe
   • Veeam.Backup.Tool.Core.dll
   • Veeam.Backup.Backup.UI.dll
5. Run the copied ‘Veeam.Backup.DBConfig.exe’ as Administrator
6. Follow the steps described here to connect the EM to the new DB location: https://helpcenter.veeam.com/docs/backup/hyperv/using_dbconfig_utility.html?ver=95

More Information

In situations where physical local storage is used please see the following KB that details how to relocate Veeam Backup and Replication backup file locations: https://www.veeam.com/kb1729

For steps on migrating the Veeam database to another SQL Instance or Server, please refer to KB1250.

Challenge

Product Information:

Company name: QNAP Systems, Inc.
Product Family: ES1686dc
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: ES1686dc
Storage Category: RAID Storage Array
Drive quantity, size, type: 16 – 2TB SAS HDD
Storage configuration: RAID 6, no SSD caching
Firmware version: 2.1.0.0711
Connection protocol and speed: SMB 10GbE
Additional support: All models and configurations of ES1686dc with specifications equivalent or greater than the above

The ES1686dc is powered by dual Intel® Xeon® processors with RAM expandable to 1TB and with a server-grade hardware design that optimizes mission-critical tasks and boasts unrivaled performance for Big Data computing.

Veeam testing configuration:

Veeam Build Number: 9.5.0.1536

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing
• Jumbo frames used in testing (MTU=9000)

Challenge

Failed to execute SOAP command "CInvSvcCreateCategory". Inventory Service was not created.
Can not Create tag with EXCEPTION: CreateCategory failed.
Failed to dispose inventory service proxy EXCEPTION: The remote server returned an error: (401) Unauthorized.

Cause

Solution

• Open Windows registry editor snap-in and navigate to the following registry path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Reporter Enterprise\
• Create a new registry key or modify the existing one:

More Information

Challenge

During SureBackup, Instant Recovery, or Other-OS FLR operation, you receive the error: 

"Error during the configuration of the host: NFS Error: Unable to Mount filesystem: Unable to connect to NFS server"

The NFS datastore cannot be mounted to the host, so vPower NFS based restores fail.

Cause

There are several possible causes for this:

• The Veeam vPower NFS Service is not started. (http://www.veeam.com/kb1094)
• The Veeam server is on a subnet that does not have access to a VMkernel port on the ESXi host.
• The Veeam server cannot reach the VMkernel port on the ESXi host due to a firewall configuration.
• Datastores within the environment that appear as (Invalid).
• An issue within the VMware environment is preventing the NFS datastore from being mounted.

Solution

More Information

Challenge

Failed to execute SOAP command "CInvSvcCreateCategory". Inventory Service was not created.
Can not Create tag with EXCEPTION: CreateCategory failed.
Failed to dispose inventory service proxy EXCEPTION: The remote server returned an error: (401) Unauthorized.

Cause

Solution

• Open Windows registry editor snap-in and navigate to the following registry path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Reporter Enterprise\
• Create a new registry key or modify the existing one:

More Information

Challenge

Solution

 Implementation step 1 - Backup & Replication  

1. Use a new Windows Server and install Veeam Backup & Replication v9.5 Update 4a or newer if you do not have a Veeam Backup Server. The Server can run within any VMware cloud on AWS SDDC, AWS S3 or on-premises environments, if the network connection to the VMC vCenter/Veeam Servers potentially the VMs (for Guest processing) is possible.
2. Add DNS network settings so that this Server can resolve Internet DNS names to be able to resolve the fully qualified domain name of the VMC vCenter server
3. Check the below information carefully for any known limitations and configuration steps before you proceed.

Implementation step 2 - VMware Cloud on AWS

Firewall Configuration for vCenter connection

1. Usage of the vCenter public IP for customers with NSX-v (default) 
   • Open Port TCP 443 from Backup Server and Proxy Server to the predefined vCenter object on the Compute Network.
   • Allow the Compute Gateway Public IP to communicate over TCP 443 with the predefined vCenter object on the Management Network.
     
2. Usage of a VPN tunnel for customers with VMware NSX-v 
   To be able to directly access the vСenter within VMC, please follow the VMC internal guidelines to create a VPN tunnel from the compute network to the management network: https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.getting-started/GUID-30BED7B3-D312-4DF3-BD7A-66F8D1C619DC.html
   Please update your DNS Servers to resolve the FQDN of the vСenter to its private IP address. If you want to use hosts entries on the Veeam Server for it, add them on all Veeam Backup and Proxy Servers.
   If your Backup & Replication (Management) Server is outside of the VMC cluster, please implement the same VPN connection for it.
3. Usage of the local connection for customers with VMware NSX-t 
   NSX-t allows VMC customers to directly access the management network over the built-in firewall. TCP Port 443 needs to be opened from all Veeam Backup and Veeam Proxy Servers as a Source with the vCenter internal IP as a target.
   a) Configure DNS entry of the vCenter for local IP address usage.

   Go to your SDDC Management – Settings – vCenter FQDN and select the Private vCenter IP address.
   Hint: If you configure the vCenter DNS record for the internal IP address, you will lose VMC connection from Backup and Replication Server outside of VMC. You can use the local hosts file or any other DNS method to resolve the vCenter FQDN with the public IP address on the Veeam Server outside of VMC. Optionally, use the Public IP address for the VMC internal and external Veeam Server.

   b) Open firewall ports for vCenter Server access

   On the Management Network
   
   On the Compute Gateway
   

1. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
2. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

1. On the Compute Network:
   1. Open TCP 22 (SSH) port from Veeam Backup server and Veeam proxy server to the Amazon VPC where the EC2 Server was installed. You can as well define the exact IP addresses of the repository server as Destination.
   2. Open TCP 2500-5000 ports for Veeam Data Transport in both directions for same servers. It is recommended to use the VMware Cloud on AWS integrated high throughput/low latency ENI network connection to avoid any traffic costs.
2. Open the same ports on the Inbound Firewall of the Amazon EC2 server used as a repository server. As the Firewall Rule Source you should add all Veeam Backup Servers (including Proxy/Repository/MountServer/Console/…) instead of 0.0.0.0/0                                                                                                                                                                 

1. Veeam Backup Copy Job to a second EC2 Server used as an additional Repository. The second EC2 Server can be placed on another AWS Availability Zone or AWS Geo Location.
2. AWS Storage Gateway Software in VTL mode can be used to emulate a Tape Library to write data to S3. Veeam Backup to Tape Jobs can be used with it. For details see: https://www.veeam.com/wp-using-aws-vtl-gateway-deployment-guide.html
3. Veeam Backup Copy Job to on premises or Veeam Cloud Connect (Enterprise). There is no special configuration needed for this use case beside network and firewall connections. For standard Repository usage on premises it is recommended to create a VPN tunnel from VMware Cloud on AWS to the on premises datacenter. This can be done by the VMC integrated VPN functionality, by Veeam PN or Third Party.

1. VMware Cloud on AWS used as Restore target. 
   1. Implementation steps 1-4 are needed.
2. Veeam VM Replication.
   1.  Implementation steps 1-5 are needed. The Repository Server (when NOT used for Backups, can run within the VMware Cloud on AWS SDDC to store the Veeam Replication data. On premises to VMC, VMC to VMC and VMC to on premises is possible. Usage of Veeam Availability Orchestrator is possible in specific scenarios, see VAO deployment guide: https://helpcenter.veeam.com/docs/vao/deployment/welcome.html?ver=10

More Information

VMware Cloud on AWS specific problems and solutions:
 

Problem

Solution

1. After restore, go to the VM configuration within the vCenter
2. Delete the network card and save the configuration
3. Create a new network card for the VM
4. Apply the needed Network card settings within the Operating System as usual

Problem

Solution

5. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
6. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

Problem

Solution

• vsanDatastore datastore
• Management VMs folder
• Mgmt-ResourcePool resource pool

Problem

Solution

Problem

Solution

Challenge

Cause

Solution

Challenge

Cause

Solution

• Constant
• Carp
• Cwd
• Data::Dumper
• Encode
• Encode::Alias
• Encode::Config
• Encode::Encoding
• Encode::MIME::Name
• Exporter
• Exporter::Heavy
• File::Path
• File::Spec
• File::Spec::Unix
• File::Temp
• List::Util
• Scalar::Util
• SOAP::Lite
• Socket
• Storable
• Threads

More Information

Challenge

Cause

• Veeam Backup&Replication related reports in Veeam Management Pack result in displaying incorrect data or “No relevant data found” message. Protected VMs report show all VMs as Unprotected. For example, all or part of VMs included in Protected VMs report shown as Unprotected.
• Backup Jobs data is missing in Veeam MP views in SCOM for particular Veeam Backup & Replication servers.
• "Veeam Backup Repository Maintenance Mode Renewal" monitor changes its state from Healthy to Warning every interval (1 hour, by default).

Solution

1. Unpack the archive.
2. Import the Management Packs to SCOM: SCOM > Administration > Management Packs > Import MPs.
3. Ensure that the build number for Veeam Backup Monitoring and Veeam Backup Discovery management packs became 8.0.0.2474.

1. Open SCOM > Authoring > Object Discoveries > Veeam Backup Job Discovery (Discovered type: Veeam Backup Abstract Job).
2. Right-click and select "Override for all objects of class: Veeam Backup Jobs".
3. In the Override Properties change the Discovery SQL Job parameter value from True to False.
4. Save the overrides to the dedicated Management Pack.

More Information

Challenge

Solution

More Information

Challenge

Guest VMs will fail to engage VSS when application-aware image processing is enabled generating an error:

Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)

Cause

This issue has multiple potential causes, usually related to the Integration Services version of the failing guest server. This issue occurs when Application-Aware image processing is utilized in the Backup or Replication job settings for transactional quiescence using Microsoft VSS

Solution

Please Note:

• A list of supported Guest OSes for each version of Hyper-V can be found here. 
• Important! Server 2012 R2 and higher guests are not supported for VSS-enabled (AAIP) backups on 2008 R2 hosts due to integration components incompatibility.
• If the VM Guest OS is 2012 and the Host is 2008R2:
  • View the Guest OS Application Event logs for errors with source: vmicvss and Event ID: 13.
  • If this error is present, open the Registry Editor (regedit) and navigate to:
    HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SystemRestore\
    Within that key create a new DWORD
    Name: ScopeSnapshots
    Value: 0
  • Note: If the SystemRestore key is not present, you will have to create it.

The solution section of this knowledge base article has been split into four different sections. Please review each step of the follow sections.

1. Integration Services
2. Within the Guest
3. VM Configuration
4. Verification

1. Integration Services

Click here for a list of Hyper-V Integration Services and their corresponding build numbers.

How to identify…

1. Using Event Viewer

This section will provide directions on ensuring the Hyper-V Integration Services are up to date within the Guest OS of the VM that is failing to backup.

If Hyper-V Integration Services are out of date you will find Event ID 4000 within the Hyper-V-Integration section of the Event Viewer on the Hyper-V host where the VM is located.

2. Using PowerShell (Server 2012 or newer)

Run the follow command in an Administrative PowerShell prompt on the host where the VM that failed to backup resides.
 

Get-VM | select VMName, IntegrationServicesState

The results of which will list out each present VM, and the state of its integration services.

How to correct:

Note: A reboot of the virtual machine will most likely be required to complete the Integration Services update.

The following information is from the Microsoft page found here.

Install or update integration services

For hosts earlier than Windows Server 2016 you'll need to manually perform install the update from within the guest operating system. These steps cannot be automated or executed within a Windows PowerShell session.

1. Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.
2. Connect to the virtual machine. Right-click the virtual machine and click Connect.
   
3. From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive.
   
   Depending on the guest operating system, you may need to start the installation manually.
   
4. After the installation finishes, you may be prompted to restart.
   

2. Within the Guest

The following steps and checks are to be performed within the Guest OS of the VM that is failing to backup.

1. Make sure Windows is running the most recent service pack.
    
2. Ensure that the VSS Storage Area and Volume Match
   1. From an Administrative Command Prompt, run the command ‘VSSUIRUN.exe”
      
   2. Check the [Settings] for each Volume and ensure that the Volume and the Storage area match.
      
       
3. Ensure there are no Dynamic disk or FAT\FAT32 volumes
   1. Open Disk Management (diskmgmt.msc)
   2. Review the Type and File System columns
   3. VSS is not compatible with Dynamic disks or FAT\FAT32
      Note: The VHD(X) that the VM runs off can be dynamic. The volume within the Guest OS cannot.
      
      All volumes must be NTFS or ReFS
       
4. Check System Reserved partition free space.
   1. System Reserved partition must have more than 42MB of space free.
   2. If less than 42MB is free, try creating a shadow copy for the volume.
      
   3. Remember to delete the Shadow Copy if it is successful.
       
5. Start\Restart the “Hyper-V Volume Shadow Copy Requestor
   1. Open Services (services.msc)
   2. Find the “Hyper-V Volume Shadow Copy Requestor” service.
   3. If it is ‘Stopped’, Start it. (If the service will not start, check in the VM settings under Integration Services, and ensure the "Backup (volume Checkpoint)" option is enabled.)
   4. If it is ‘Running’, restart it.

3. VM Configuration

The following steps and checks are to be performed from the Hyper-V host level.

1. Make sure the VM has a SCSI controller. One must be present, even if there are no devices to use the SCSI controller.
   •  The error will present itself in the Hyper-V VMMS > Admin Event log. Example:
     “The virtual machine ' Vm-Name' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID).”
2. Within the Virtual Machine settings verify that the “Backup (Volume Snapshot)” option is enabled.
3. Make sure that the virtual machine files (VHD files) are not sitting in the root folder of the CSV, as this may prevent correct snapshot creation due to permission restrictions. We strongly recommend keeping virtual machine files in separate folder. More information regarding this can be found here.

Verification

Performing the following will allow you to determine if the steps performed above have corrected the issue without the need to run the Veeam Backup & Replication job again.

1. 1. Open an Administrator Command Prompt.
   2. Run the command:

      Diskshadow /L C:\writers.txt

   3. Run the command:

      list writers

Open the file that was created and and locate the VM name in the section for the Hyper-V writer. For each VM there's a "Caption" entry. The caption should read "online" or "child partition", and not "offline" or "saved state".

More Information

PowerShell script to tell you if you need to update your Integration Services
https://blogs.msdn.microsoft.com/virtual_pc_guy/2013/04/29/powershell-script-to-tell-you-if-you-need-to-update-your-integration-services/

Recommended hotfixes for Hyper-V servers:

http://www.veeam.com/kb1838
http://blogs.msdn.com/b/virtual_pc_guy/archive/2009/02/02/looking-at-the-hyper-v-event-log.aspx
http://blogs.technet.com/b/virtualization/archive/2008/08/29/backing-up-hyper-v-virtual-machines.aspx

Challenge

Cause

Last-minute kernel patches for Red Hat Enterprise Linux and its derivatives may cause Veeam Agent for Linux to go unstable (veeam.com/kb2786).
 

Challenge

In some cases a SP backup server receives "Success" status of a tenant job even when the job has failed on the Tenant's server.

Cause

• All cloud gateways are unavailable
• There is not enough space on the disk
• An existing connection was forcibly closed by the remote host
• Repository quota exceeded

Challenge

Cause

Solution

• Additional Hypervisor support vSphere 6.7. VMware vSphere 4.0 and later (including vSphere 6, vSphere 6.5, vSphere 6.7 and vSphere 6.7 U2) 
• Monitoring SQL Transaction Log Backup Job. Veeam MP separately monitors SQL Transaction Log Backup jobs running inside backup jobs with the enabled application-aware processing option. For this purpose, Veeam MP includes a new Veeam SQL Transaction Log Backup Jobs view and Veeam Backup: SQL Transaction Log Job Status monitor.
• Improved performance of Morning Coffee Dashboards. In previous versions, Morning Coffee Dashboards tended to load slowly for large-scale environments. In Veeam MP 8.0 Update 6, the performance of Morning Coffee Dashboards has been significantly improved.
• A new metric for Veeam VMware Collector. Due to the Ops Mgr Health Service (Monitoring Agent Service) limitations that appear when a Veeam VMware Collector processes a large amount of data, it is required to distribute monitoring load among multiple Veeam VMware Collectors in large environments. To assess the monitoring load on Veeam VMware Collectors more precisely, Veeam MP provides a new % Object Monitoring Load metric.

• No monitoring for Hyper-V hosts that exceeded the license limit. Veeam MP license limits the number of Hyper-V hosts enabled for monitoring in Ops Mgr. Earlier, if the number of licensed Hyper-V hosts was exceeded and new hosts were added to the Ops Mgr infrastructure, Ops Mgr automatically stopped monitoring the old infrastructure and started monitoring the newly added hosts. In Veeam MP 8.0 Update 6, the issue was resolved, and Ops Mgr continues monitoring the old infrastructure.
• Ops Mgr event log shows excessive BackupCopyJobSessionFinished events. In previous versions, the Ops Mgr event log contained a large number of 3101 events caused by errors in the backup copy job status collection process. In Veeam MP 8.0 Update 6, the issue was resolved.

• Collector Autodeploy limitations. If you use the Collector Auto-Deployment feature to automatically deploy Veeam VMware Collectors on Management Servers, keep in mind that it will install clear 8.0 version without patches. When the installation completes, run the ISO\Update\VeeamMP80_Update6.exe installation file on each Veeam Collector to install patches.  
• Veeam MP for Hyper-V stops collecting storage metrics. If storage migrates to another volume, the storage performance collection will halt and Veeam MP will stop collecting the following metrics: Storage IOPS, Storage KB Total/sec, Storage Flush Count and Storage Error Count. Note that this issue does not affect migration between hosts. To work around the issue, restart the Ops Mgr agent after storage migration.

More Information