Challenge

Cause

Solution

Information regarding solutions and workarounds are discussed in the following VMware article.

Deploying Windows 2008 R2 and Windows 7 templates with vmxnet3 renames the NIC as #2 (1020078)
 

1. Delete the current VM replica.

2. Verify Re-IP settings in the properties of a replication job.

3. Re-run the replication job.

4. When a new replica is created, test failover to make sure that Re-IP settings are applied properly.

Challenge

Product Information:

Product Family: Fujitsu ETERNUS CS
Status: Veeam Ready - Tape
Classification Description: Tape solution where available hardware features have been tested to work with Veeam.

Solution

Product Details:

Model number: CS8000 v7.0 VTL
Library Firmware Version V7.0A4010-SP02 Virtual juke box (VJUK) emulation
Drive firmware version: V7.0A4010-SP02 Virtual juke box (VJUK) emulation
Driver for tape drive: Microsoft IBM Ultrium-TD7 version 10.0.14393.1794
Driver for media changer: N/A
Media type tested: LTO7
General product family overview:
CS8000 is a virtual tape library with emulated tape drive at the front end connected to real library with real tape drive at the back end. The data are backup into virtual volumes on internal disks RAID’s. The virtual volumes are stored on physical volumes. https://www.fujitsu.com/emeia/products/computing/storage/data-protection/cs8000/

Veeam Details:

Veeam Build Number: 9.5.0.1922
 

More Information

Company Information:

Company name: Fujitsu Limited
Company overview: Fujitsu is the leading Japanese information and communication technology (ICT) company, offering a full range of technology products, solutions and services. Approximately 156,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers.

Challenge

Product Information:

Company name: Zadara
Product Family: VPSA (Virtual Private Storage Array)
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: VPSA Storage Array
Storage Category: Software Defined Storage
Drive quantity, size, type: 4 x 1.6TB SSD
Storage configuration: RAID10
Firmware version: 18.11
Connection protocol and speed: 40 GbE iSCSI
Additional support: All models and configurations of VPSA running v18.11 and above with hardware specifications equivalent or greater than the above

Zadara VPSA Storage Array provides:

• Unified storage with both Block volumes and File shares exposure
• Data Protection (RAID-1, 6, 10, 60)
• Advanced Data management (Thin Volumes, Snapshots & Clones)
• Management GUI and Rest API
• Flexibility to grow and shrink resources such as CPU, RAM, cache and drive allocations.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2753

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• Multipathing used in testing – yes (MPIO)
• All VPSA volumes should be connected using Zadara’s best practices for VMware

Challenge

Product Information:

Company name: QNAP Systems, Inc.
Product Family: TS-2888x
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: TS-2888x
Storage Category: RAID Storage Array
Drive quantity, size, type: 8 – 1TB SATA HDD
Storage configuration: RAID 6, no SSD caching
Firmware version: 4.4.0.0820
Connection protocol and speed: SMB 10GbE
Additional support: All models and configurations of TS-2888x with specifications equivalent or greater than the above

The TS-2888X AI-Ready NAS is designed to optimize AI workflows, and is also positioned as a Complete Backup Solution. Up to 18 cores with 36 threads and up to 4.5 GHz with Intel® Turbo Boost Technology 2.0, combined with up to 512 GB DDR4 ECC RDIMM 2666 MHz memory to deliver rapid workload loading and processing. The TS-2888X utilizes a hybrid architecture with high-capacity HDDs and high-performance SSDs to meet your simultaneous needs for scale and performance. Flexible snapshot protection assure your data while also allowing efficient disaster recovery.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2615

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing
• Jumbo frames used in testing (MTU=9000)

Challenge

Cause

• Cloud Copies counter on the Overview dashboard does not return any data.
• Backup agents managed by Veeam Backup & Replication report 0 restore points available in the backup policy drill down view. 

• Portal administrators can log in only when using a full domain name of the Veeam Availability Console server. 
• Users belonging to a domain user group which is added to a Veeam Availability Console security group cannot log in to the portal.
• Users belonging to a nested domain group cannot log in to the portal.

• Under certain conditions, Veeam.MBP.Service might be consuming excessive memory.
• Under certain conditions, data collection job fails to retrieve information from a Cloud Connect server that has lots of task sessions on cloud gateway servers. This results in displaying these tenants’ companies as deleted in the Veeam Availability Console UI.
• Tenant password changed in Veeam Cloud Connect is not synchronized with Veeam Availability Console.

• Office 365 and Gmail SMTP servers are not supported for notifications.  

• Inaccessible computers cannot be located when using CSV and IP discovery rule options.

• Full Access UI mode cannot be applied to a backup agent.  
• When using Select All checkbox on the Managed Backup Agents tab agent settings are applied to all computers of all managed companies regardless of the selected scope in the Company\Reseller filter.  
• Licensing attribute is not updated for managed backup agents which leads to license expiration on the backup agent.    
• Backup agent files are not validated when the installation is performed from a local folder without using a master management agent. This may result in a corrupted installation/upgrade of the backup agent. 

• Under certain conditions, management agents fail to upgrade with the following error message “The system cannot find the file specified.”

• Workstation and server backup counters in the Veeam Cloud Connect usage report are mixed up.
• Under certain conditions License usage report for Veeam Backup & Replication servers is not generated. 

• Under certain conditions, all VMs with a cloud copy which belong to different locations are reported as VMs from the default location in the invoice. 

• Querying tenant information with SetLinks parameter may take considerable time.
• The vMsBackudUpToCloud counter does not display data about VM and Agent backup copy jobs targeted to a cloud repository.  

• Cloud repository usage is not updated in ConnectWise Manage agreements when there are unknown files (not related to Veeam backups) in the repository. 
• Tickets state synchronization may stop working when triggering a mass-resolve operation in the ConnectWise Manage application. 

Solution

msiexec /update c:\VAC.ApplicationServer.x64_3.0.0.2725.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

msiexec /update c:\VAC.WebUI.x64_3.0.0.2725.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACWebUI.txt

More Information

Challenge

Cause

• Backup of failover clusters is supported in Veeam Agent for Microsoft Windows managed by Veeam Backup & Replication only. You cannot process a failover cluster by Veeam Agent for Microsoft Windows operating in the standalone mode.
• Agent license with a server counter must be installed in Veeam Backup & Replication to enable Agent Management features including failover cluster support.
• Veeam Agent for Microsoft Windows supports Windows Server Failover Clusters running Microsoft Windows Server 2008 R2 and later.
• Workgroup clusters, multi-domain clusters, and mixed OS version clusters are not supported.
• Cluster Shared Volumes (CSV) are skipped automatically during the backup.
• The following Microsoft Failover Cluster applications are supported: Microsoft SQL Server Failover Cluster Instances (Microsoft SQL Server 2008 SP4 or newer), Microsoft SQL AlwaysOn Availability Groups, Microsoft Exchange Database Availability Groups (Microsoft Exchange Server 2010 or newer).
• AlwaysOn Clusterless Availability Groups are not supported.
• Managed by backup server job with failover cluster type does not support the file-level backup mode.
• Recovery must be performed via Veeam Backup & Replication console.

Solution

Microsoft Windows Server Failover Clusters, Microsoft SQL Server Failover Cluster Instances, SQL Always On Availability Groups

There may be various infrastructure changes required to grant that access: network configuration, firewall and security software settings. For instance, if the cluster is deployed On Azure, then it is important to check that all the required ports are opened in the network interface settings - https://docs.microsoft.com/bs-latn-ba/azure/virtual-machines/windows/nsg-quickstart-portal

Specific fault tolerance configurations may have Azure load balancer deployed - https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ha-ports-overview
In this scenario, make sure that the corresponding rules are added the load balancer configuration, otherwise part of the Veeam traffic will be dropped - https://docs.microsoft.com/en-us/azure/load-balancer/configure-load-balancer-outbound-cli

Microsoft Exchange Database Availability Group

Starting with Veeam Backup & Replication 9.5 Update 4, Microsoft Exchange database availability group (DAG) nodes are now automatically processed in a sequential manner, so it is possible to use Veeam Agent for Microsoft Windows to process Microsoft Exchange Database Availability Groups (DAGs).

The procedure of adding a DAG to a Veeam Agent backup job differs depending on the type of the DAG that you want to process:

• For a regular DAG, the backup job configuration procedure is the same as for any failover cluster mentioned above, so all the steps above are relevant
• For an IP Less DAG (a DAG without an Administrative Access Point), the backup job configuration procedure is similar to the same procedure for standalone servers. To process an IP Less DAG, you must create a Veeam Agent backup job for servers in the Veeam backup console and add  all nodes of the IP Less DAG (the steps below show it in detail) to this job.

Then add servers that act as DAG nodes to a Veeam Agent backup job for servers configured in Veeam Backup & Replication.

During the backup process, Veeam Backup & Replication detects that servers in the backup job are part of a DAG, and creates a transactionally consistent backup of Microsoft Exchange databases running on these servers (only passive database copies are processed, active ones are excluded from VSS freeze operations; however, the job still copies all the correspondent EDB-files of all databases from all the nodes).

Application-aware processing should be enabled for a consistent backup, log truncation and application-item restore possibility.

Challenge

Cause

Last-minute kernel patches for Red Hat Enterprise Linux and its derivatives may cause Veeam Agent for Linux to go unstable (veeam.com/kb2786).
 

Challenge

Solution

1. Download .zip archive from and transfer it to your Veeam Availability for Nutanix AHV backup proxy (sftp can be used as shown on this page of the User Guide)
2. Install the script dependencies (make sure the backup proxy has an internet connection)
   • Login to the backup proxy shell
   • Run the following commands:

   $ sudo apt-get -y install python python-pip
   $ pip install aenum
   

3. Unpack .zip archive using unzip command

   $ unzip <the_zip_file_with-script>
   

4. Edit van-mail-notificator.py script file using your favorite text editor. Go to line 53 and change the necessary values where toMail value is an account which will receive backup job report. Save changes in the file.

   fromMail = "me@example.com"
   toMail = "me@example.com"
   fromServer = "localhost"
   

5. Change file permissions, add executable permission:

   $ chmod +x van-mail-notificator.py
   

6. Run the script

   $ ./van-mail-notificator.py
   

More Information

1. Login to the backup proxy shell
2. Issue the following command:

   $ crontab -e
   

3. Insert the following line at the end of the file (assuming van-mail-notificator.py is located in /home/veeam location on a backup proxy)

   0 1 * * * /home/veeam/van-mail-notificator.py &>> /home/veeam/emailReporter.log
   

Challenge

Cause

Solution

• Linux user account used by Veeam Backup and Replication server for any Veeam Agent for Linux deployment and management operations must have /bin/bash shell set by default.
• /etc/sudoers file should contain a line with ‘requiretty’ parameter negated explicitly for desired Linux user account
• The account must have either ‘root’ permissions or permissions to execute a specific list of commands as a sudoer on the target Linux machine.

An example of /etc/sudoers file containing proper settings for SLES 11, Veeam Agent for Linux v3.0 (or later)/Veeam Backup & Replication v9.5 update 4 (or later):

Defaults:username !targetpw  

username ALL=(root) /bin/hostname 
username ALL=(root) /bin/uname 
username ALL=(root) /bin/arch 
username ALL=(root) /bin/cat 
username ALL=(root) /usr/bin/test 
username ALL=(root) /bin/mkdir 
username ALL=(root) /bin/rmdir 
username ALL=(root) /bin/chown 
username ALL=(root) /usr/bin/whoami 
username ALL=(root) /usr/bin/id 
username ALL=(root) /tmp/*/veeamagentid 
 
# Agent config/backup 
username ALL=(root) /usr/bin/veeamconfig 
  
# Agent deployment 
username ALL=(root) /usr/bin/zypper 
username ALL=(root) /sbin/modinfo 
username ALL=(root) /bin/rpm 
  
# FLR Restore via console 
username ALL=(root) /bin/sh 
username ALL=(root) /bin/touch 
username ALL=(root) /bin/chmod
username ALL=(root) /tmp/VeeamAgent*-*-*-*
username ALL=(root) /bin/rm 
username ALL=(root) /bin/ps 
username ALL=(root) /bin/mv

An example of /etc/sudoers file containing proper settings for SLES 11, Veeam Agent for Linux v2.0.1/Veeam Backup & Replication v9.5 update 3a:

Defaults:username !requiretty  

username ALL=(root) NOPASSWD: /bin/hostname 
username ALL=(root) NOPASSWD: /bin/uname 
username ALL=(root) NOPASSWD: /bin/arch 
username ALL=(root) NOPASSWD: /bin/cat 
username ALL=(root) NOPASSWD: /usr/bin/test 
username ALL=(root) NOPASSWD: /bin/mkdir 
username ALL=(root) NOPASSWD: /bin/rmdir 
username ALL=(root) NOPASSWD: /bin/chown 
username ALL=(root) NOPASSWD: /usr/bin/scp
username ALL=(root) NOPASSWD: /tmp/*/veeamagentid 
 
# Agent config/backup 
username ALL=(root) NOPASSWD: /usr/bin/veeamconfig 
  
# Agent deployment 
username ALL=(root) NOPASSWD: /usr/bin/zypper 
username ALL=(root) NOPASSWD: /sbin/modinfo 
username ALL=(root) NOPASSWD: /bin/rpm 
  
# FLR Restore via console 
username ALL=(root) NOPASSWD: /bin/sh 
username ALL=(root) NOPASSWD: /bin/touch 
username ALL=(root) NOPASSWD: /bin/chmod
username ALL=(root) NOPASSWD: /tmp/VeeamAgent*-*-*-*
username ALL=(root) NOPASSWD: /bin/rm 
username ALL=(root) NOPASSWD: /bin/ps 
username ALL=(root) NOPASSWD: /bin/mv

Challenge

Failed to connect to the Veeam Backup & Replication server:
No connection could be made because the target machine actively refused it <IP>:9392

Cause

Solution

More Information

Challenge

Connecting to Office 365 organization fails with the “Check Exchange plan: No Exchange plan found” or “Check SharePoint plan: No SharePoint plan found” error.

Cause

Some of the Office 365 Exchange or SharePoint service plans assigned to the Office 365 Organization cannot be automatically identified as valid by Veeam Backup for Microsoft Office 365. This happens because assigned Office 365 plans are missing from the list of valid ones.

Solution

1. First, get your current service plan ID. For example, you can do it via ‘MSOnline’ AzureAD module and the following PowerShell One-Liners: 

(get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "Exchange"} | ft ServiceName,ServicePlanId

(get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "SharePoint"} | ft ServiceName,ServicePlanId

2. Ensure that no jobs and restore sessions are running.
3. Stop the Veeam.Archiver.Service and the Veeam.Archiver.Proxy services.
4. Navigate to C:\ProgramData\Veeam\Backup365 and edit the Config.xml file by adding the following lines between <Archiver> tags:

<Options>

<O365Plans AdditionalExchangePlans="your-Exchange-plans-IDs" />

</Options>

<Options>

<O365Plans AdditionalSharePointPlans="your-SharePoint-plans-IDs" />

</Options>

5. Start the Veeam.Archiver.Service and the Veeam.Archiver.Proxy services.
6. Open ‘Add organization’ wizard and re-try the procedure.
    

More Information

In case the issue persists after these changes, please, contact Veeam Support.

Challenge

Failed to connect to the Veeam Backup & Replication server:
No connection could be made because the target machine actively refused it <IP>:9392

Cause

Solution

More Information

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

Challenge

Cause

• When adding an Organization there are 11 Verification checks that happen. Below lists each of the verification and what it means if it fails. In addition any permission assigned in Office 365 Online can take 15-60 minutes to apply on Microsoft's end.
   

Connection to Microsoft Graph: This meaning can change based on if you are using Modern authentication or Basic authentication.

Modern authentication: This would mean that the Application ID and secret failed to authenticate.

• Make sure the Application ID and Secret were properly entered
• Check the Application APIs from the "Azure AD Application" section below
• Try generating a new Application Secret from Azure AD

Basic authentication: This would mean that your username does not have permissions to authenticate with the Microsoft Graph Online.

• If your Organization is Federated try creating a new cloud user account from Microsoft Azure AD for authentication.
• Make sure that the user belongs to the Organization.
• Check that user has all permissions assigned from the above SharePoint and Exchange section

Connect to EWS: This is a connection to the Exchange Web Service.

Modern authentication: This uses a duel authentication leveraging the Application ID and Username.

• For the Application ID check in Microsoft Azure AD that the correct APIs were assigned as Application and not Delegated.
• For the Username check Microsoft Exchange Admin center that all permissions have been assigned to the user as documented below.

Basic authentication:

• For the Username check Microsoft Exchange Admin center that all permissions have been assigned to the user as documented below.

Connection to PowerShell: This step checks that we can connect to Exchange Online PowerShell. Only the username is used for this verification. Require AllowBasicAuthPowershell and AllowBasicAuthWebServices to be allowed in the group policy this user is added to.

• In Azure AD check Conditional Access- Polices for anything blocking legacy authentication and make sure the service account is being excluded from these policies: https://aad.portal.azure.com/#blade/Microsoft_AAD_IAM/ConditionalAccessBlade/Policies
• Check through PowerShell to see if the policies are allowed for the user:
  1. Connect to Exchange Online PowerShell module: https://docs.microsoft.com/en-us/powershell/exchange/exchange-server/connect-to-exchange-servers-using-remote-powershell?view=exchange-ps
  2. Next you will run the "Get-AuthenticationPolicy" command. If there is a blank return then there is no policy in place.
  3. If the current policy does not have true for AllowBasicAuthPowershell and AllowBasicAuthWebService contact your Domain admins to update the policy with the following Microsoft Document
  4. https://docs.microsoft.com/en-us/powershell/module/exchange/organization/set-authenticationpolicy?view=exchange-ps

Check Exchange plan and SharePoint plan: If either of these fail then your plan cannot automatically identified as Valid. Use the below KB to add the proper plans to your configuration

Service plan is not found when adding Office 365 organization: https://www.veeam.com/kb2946

Check Required cmdlets access, Mailbox Search role, and Applicationimpersonation role: The roles are all related to Exchange Online.

• Check that the user has been properly added to the role group with all necessary permissions as shown above in the Exchange section

Check SharePoint Online Administrators role: This refer the SharePoint Administrator role that you assign to the user in the Admin Center.

• Refer to the SharePoint section above and make sure that the service has the SharePoint Administrator role assigned

Check LagacyAuthProtocalsEnabled: This is a setting in SharePoint Admin center to enable legacy Authentication. 

• Refer to the SharePoint section above to Allow Apps that don't use modern authentication.
• “Unauthorized” error is thrown by SharePoint Online and/or OneDrive for business backup jobs: https://www.veeam.com/kb2714

Solution

Exchange
SharePoint
User App Password
Azure AD Application

Exchange

• First log into the Exchange Admin Center: https://outlook.office365.com/ecp/
• Select Permissions and add a New Role Group

• Create your role Group: -Name your role group appropriately (Example: VBO Permissions) -Add Roles: ApplicationImpersonation, View-Only Configuration, View-Only Recipients, Mailbox Search, and Mail Recipients -Add the Veeam Service account under Members and Save

SharePoint

• Add the SharePoint Administrator Role to user In Azure Admin Center: https://admin.microsoft.com/ Select: Users > Active Users >  Select Backup Service account

• Allow Apps that don't use modern authentication In the SharePoint Admin Center: https://[ORGANIZATION NAME]-admin.sharepoint.com/ Select: Access control > Apps that don't use modern authentication > Allow Access

User App Password

• Set up user to leverage Multi Factor Authentication: https://account.activedirectory.windowsazure.com/UserManagement/MultifactorVerification.aspx? Select: Service account > Enable.

• Once the account have been enabled for Multi-factor authorization you will need to sign-in with the service account and follow the steps for configuring Multi Factor authentication.
• Generate an application password for the user: https://portal.office.com/account/#security Select: Additional security verification> Create and manage app password:

Azure AD Application

• Register a new App in Azure AD: https://aad.portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade Can also be found under: All services> App Registrations-

• Name the Application, add Redirect URL (URL does not need to be real. Example: "http://localhost/"), and Click Register

• Select API permissions and Add permission. Please keep in mind that all permissions must be added as Application permissions and not Delegated permissions

• First API will be at the top of the page: Microsoft Graph> Application Permissions

• Check Directory.Read.All and Group.Read.All then select Add permissions

• Second API will be halfway down the page: SharePoint > Application Permissions

• Check Sites.Fullcontrol.All Then select Add permission

• Last API will be at the bottom of the page Under Supported legacy APIs: Exchange> Application Permissions

• Check full_access_as_app Then select Add Permission

• After all APIs have been added you will need to Grant consent:

• Select Certificates & secrets and New client Secret

• Add a Description> Choose Expiration> Add

• Copy Secret Value because it will no longer be available once you close the window:

• Locate Application ID: Overview> Application (client) ID

Challenge

After installing Veeam Agent for Microsoft Windows and configuring a backup job, the settings are not shown in the Control Panel, and Veeam Agent for Microsoft Windows keeps prompting to configure backup settings.

Cause

The Windows OS was installed with non-English region and culture settings. As a result, these settings were applied to the Windows system account.
Veeam Agent for Microsoft Windows is based on Microsoft .NET Framework which has some limitations in term of particular OS cultures

 

Solution

You will need to set Format to English (United States) for the system account.

Go to the Region settings (press Win + R > type intl.cpl > press Enter)

1. On the Formats tab
   1. set Format to English (United States)
2. On the Administrative tab
   1. click Copy Settings
      1. check that Format is set to English for Current user
      2. Enable the Welcome screen and system accounts checkbox
      3. Click OK to save settings and return to the Administrative tab.
3. Click OK to close the Region settings and save settings.
4. Restart your machine

Once the necessary changes are applied to the system account, a non-English Format can be set for the Current user account. Note, that the Format has to remain English (United States) for the system account.

More Information

Here is an example of a working configuration:

Challenge

This article documents general performance expectations, best practices, and configuration advice, when using an EMC Data Domain appliance with deduplication as a repository for Veeam Backup & Replication.

Solution

For further information regarding how Veeam Backup & Replication works with EMC Data Domain DDBoost please review: https://helpcenter.veeam.com/docs/backup/vsphere/emc_dd.html

Performance Expectations

EMC Data Domain Deduplication Storage Systems provide both high compression and deduplication ratios so that data can be kept for extended periods. When a Data Domain is configured as a repository for Veeam Backup & Replication, write performance may vary depending upon the particular EMC Data Domain Deduplication System model, protocol, and backup infrastructure architecture.

When attempting to read data from a Data Domain it must rehydrate and decompress each block, for this reason operations which read from the Data Domain will perform slower than non-deduplicated storage, this is more noticeable with operations which use random I/O. All restores will occur as fast as the environment can accept new information, and as fast as the Data Domain can decompress and rehydrate the blocks.

For quick recovery you may consider using fast primary storage and keeping a several restore points (3-7) for quick restore operations such as Instant Recovery, SureBackup, Windows or Other-OS File restores since they generate the highest amount of random reads. Then use the DataDomain as a secondary storage to store files for long term retention. If an EMC Data Domain Deduplication System will be used as primary storage, it is strongly suggested to leverage alternative restore capabilities within Veeam Backup & Replication such as Entire VM restore and VM files restore. This may result in faster recovery capabilities when used with EMC Data Domain Deduplication Systems than Instant Recovery and File Level Restore operations.

Instant Recovery

• This type of restore can be effected adversely by the aforementioned limitations of a Data Domain appliance, and also the type of VM being restored. Highly transactional VMs will require more IOPS from the Data Domain during the Instant Recovery than others. With this in mind you can expect to only be capable of running only a few Instant Recoveries simultaneously. Instant Recovered VMs that are started from a backup file stored on a Data Domain may react or start slowly as the majority of their read operations will be hindered by the Data Domain.
• For VMware users it is highly advised when performing an IR that the user select to have virtual disk updates redirected to a high performance Datastore. This will improve performance by caching written blocks to low latency storage.
• It is advised that if the VM is intended to be made permanent that the VM be migrated to production storage as soon as possible after the Instant Recovery has begun.

File Level Restores

When performing Veeam Backup & Replication File Level Restore (FLR) capabilities slow recovery times may be experienced. During Veeam FLR recovery capabilities, a significant amount of read activity occurs when accessing the Veeam “service data” metadata for each individual file as the Veeam backup files are not arranged in sequence. This read activity must be performed to determine the location of the data block(s) associated with each file during granular restore sessions. This significant level of random access is not recommended with archive tier storage devices because they are designed for optimal performance with sequential read operations. Veeam recommends implementing EMC Data Domain Deduplication Storage Systems as a secondary target for these use cases as the more random read operations, the slower the restore will be with EMC Data Domain Deduplication Systems.

• The backup browser may take longer than usual to open if an increment is selected and furthermore by that increments distance from the full restore point.
• Navigating between folders within the Backup Browser may take additional time as each folder’s content must retrieved from the backup file to display it.

Backup

• Reverse Incremental performance will be very poor due to its highly random I/O.
  Note: When the Backup Job is configured with a DDBoost repository, Veeam Backup & Replication will prevent Reversed Incremental from being selected by the user.
• Synthetic Full creation will be very slow to a Data Domain, unless using DDBoost.
• Synthetic Full with Transforms are not advised.

Backup Copy

• A retention longer than 30 is not advisable as restore operations will diminish in performance.
• The Health Check option may take a very long time as it is performing a read operation.

Replication

• Using the Datadomain to store Replica metadata is not advisable.
  Note: Veeam Backup & Replication will prevent the user from selecting a DDBoost repository.

Veeam Backup & Replication Configuration

Storage optimization (job option):

Setting the storage optimization to Local 16TB+ has been shown to improve the effectiveness of Data Domain’s deduplication. The larger this value is, the smaller the preparation phase will be for a backup task and less memory will be used to keep storage metadata in memory.

Inline-deduplication (job option):

Since EMC Data Domain Deduplication Systems have excellent hardware deduplication and compression capabilities, it is highly advised that Veeam built-in deduplication be disabled to decrease load on the backup proxy.

Decompress backup block before storing(repository option):

Veeam strongly recommends enabling this option so that raw data is sent to the EMC Data Domain Deduplication System, leveraging its global deduplication and compression capabilities. Leaving Veeam compression enabled may significantly impact EMC Data Domain deduplication capabilities resulting in high load and slow backup jobs.

Use Per-VM Backup Files(repository option):

Veeam recommends enabling this option so that there is improved performance writing and reading data from the EMC Data Domain Deduplication System. This option is enabled by default when adding the repository as a Deduplicating Storage Appliance.

Backup Mode

• For CIFS/NFS presented repositories Forward incremental mode with periodic Active full backups is recommended to avoid the rehydration penalty during synthetic operations.
• For DDboost enabled repositories Forward incremental mode with either Active full or Synthetic full backups is recommended. Synthetically produced full backups will generally have the best restore performance and reduce the time VM is run off of a snapshot during the backup job run. However in some environments an Active Full job may run faster.
• Transforming previous backup chains into rollbacks is not advisable for both repository types.
• For forever forward incremental backup and backup copy on DDboost enabled repositories, the option “Defragment and compact full backup file” should be enabled if available. In most cases a weekly schedule is appropriate. This helps to avoid excessive growth of pre-compression data size for the full backup file.

Repository Performance Expectations and Configuration

If DDBoost is not licensed on the Data Domain system it must be added as a CIFS type or Linux type repository. It is advised to use a Linux server with the volume mounted via NFS as a relay server to help improve performance. Under some circumstances, CIFS or NFS communication may perform better than DDBoost with Veeam Backup & Replication v8 because of the limitation of a single thread per backup job when using DDBoost. DDBoost has been shown to improve performance when performing Synthetic Fulls.

With Veeam Backup & Replication v9, support for EMC Data Domain Boost is enhanced with the introduction of the following capabilities:

• Support EMC Data Domain Boost 3.0
• Reduced impact of storage fragmentation during restore operations even with enabled parallel processing. This feature allows Veeam to store the VM backup in the dedicated backup chain so that fragmentation ratio will be minimum.
• Reduce the impact of the block size so you may define any block size without impact on the restore process. Veeam will be able to read data granularly so amount of the redundant will be minimum.

With DDBoost

If the Data Domain System is licensed for DDBoost please proceed to configure it using the following steps.

1. Launch the creation of a new Repository, on the Type tab select Deduplication storage appliance.
2. Select the deduplication storage as EMC Data Domain.
3. On the next tab configure the information to for connecting to the Data Domain appliance.
4. On the Repository tab click Browse and select the necessary location from the list of available paths.
5. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Without DDBoost

CIFS

1. Launch the creation of a new Repository, on the Type tab select CIFS
2. On the next tab configure the path to which the Repository will write to, and set credentials to access that share.
3. On the Repository tab within the advanced section, enable “Decompress backup data blocks before storing”
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

NFS

The Data Domain will need to be configured for NFS access, and configure a Linux server to mount the volumes from the Data Domain via NFS. Please refer to the following links for further information regarding connecting Linux to the Data Domain via NFS:

http://forums.veeam.com/veeam-backup-replication-f2/veeam-datadomain-and-linux-nfs-share-t8916.html

http://tsmith.co/2014/veeam-and-datadomain/

1. Launch the creation of a new Repository, on the Type tab select Linux
2. On the next tab select the Linux server that we be connected to. If it is not present in the list select “Add New…”
3. On the Repository tab specify the path on the Linux server that leads to where you mounted the Data Domain via NFS. On this tab in the advanced section enabled “Decompress backup data blocks before storing”.
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Challenge

This article covers two most popular scenarios of adding SMB repository to Veeam Backup for Office365
Note: Such repository type is on experimental support in the current version. So i.e. if a storage device supports both SMB 3.0 and iSCSI protocol, then the best practice would be to use the latter due to its ability to use Disk Queue which acts as a buffer space for data to land on before being sent off to disk.

If it’s a Windows-based shared folder, then it’s recommended to add the server it’s based on as a proxy and connect the same storage as a directly attached disk. 

Cause

Solution

Scenario 1: Microsoft Windows SMB share with authorization

Prerequisites:

Consider that Microsoft Windows 7 SP1 and Microsoft Windows 2008 R2 SP1 do not support the SMB 3.0 share as a storage system. To use the SMB 3.0 share as a storage system, make sure you are using Microsoft Windows 8 or Microsoft Windows 2012 or higher. Both Veeam Servers and SMB share location must be deployed within the same or a trusted domain.

• Open folder sharing settings for planned repository location and add all Veeam servers to it using their computer names. In example if your Veeam server is called “WIN-F4N92CDLKET” the list should look like this:

• Add repository in Veeam console as shown in this article

Scenario 2: Standalone storage appliance

In this scenario we will use Synology appliance as an example, but the same method should work for any standalone device that supports Microsoft Active Directory and its computer groups. 

• Add Veeam server and all remote backup proxies (if any) to Active Directory domain.
• Create an Active directory group in this domain and add all Veeam servers to it as “computer” objects:

• Add the storage device to the same domain:

• Make sure that SMB 3.0 protocol is enabled and previous versions are not allowed to use:

• Create SMB shared folder and grant read/write access to it for the domain group that was created earlier:

• Add new repository in Veeam console and manually type SMB share name in “Path” field as described here

Bonus: Adding standalone storage appliance using iSCSI protocol
In this scenario we will use Synology appliance as an example, but the same method would work for storage appliance that supports iSCSI protocol.

• Open iSCSI manager and create new LUN and Target:

• On Veeam server: open iSCSI initiator, enter your storage appliance IP address, click “Quick Connect…” and then click “Done”:

• Open Disk Management, bring iSCSI disk online and initialize it using default parameters:

• Create a new simple volume using default parameters:

• Create a new repository on this disk as shown here

Challenge

Product Information:

Company name: SUSE
Product Family: SUSE Enterprise Storage
Status: Veeam Ready – Object
Classification Description: Verified object storage solutions that has been tested with Veeam Backup & Replication Cloud Tier features.

Solution

Product Details:

Model number: Enterprise Storage 5.5
Storage Category: Software Defined Storage
Drive quantity, size, type: 144x 7.2k SATA w/RocksDB+WAL
Storage configuration: Erasure coded 3+2
Firmware version: 5.5
Connection protocol and speed: 100 GbE
Additional support: All models and configurations of Enterprise Storage 5.5 for S3 compatible storage with specifications equivalent or greater than the above.

General product family overview:
SUSE Enterprise Storage provides unified block, file and object. Object for Veeam is an S3 interface and is compatible with a large subset of the Amazon S3 RESTful API.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2399

Vendor recommended configuration:

Vendor Settings:

• On Veeam Proxy, disable Windows Defender
• Use PVSCSI
• Monitor node is used as the Object Gateway (RGW)
• ESX 6.5 Tuning (to improve performance to VMFS data stores)
  esxcli system module parameters set -m iscsi_vmk -p iscsivmk_LunQDepth=512
  esxcli system settings advanced set -o /ISCSI/MaxIoSizeKB -i 512

Challenge

Product Information:

Company name: Fujitsu
Product Family: ETERNUS CS
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: CS8000
Drive quantity, size, type: 10 - SAS HDD
Firmware version: V6.1A7120-SP03
General product family overview:

ETERNUS CS8000 is a datacenter solution for backup storage for mainframe and open systems. Its intelligent process automation and pooling of storage capacity reduces the TCO for complex backup environments by 40%. Unique scalability in capacity and performance makes ETERNUS CS8000 a viable solution, delivering ongoing cost savings.

Veeam testing configuration:

Veeam Build Number: 9.5.x

Job Settings:

Repository Settings: