Challenge

Cause

New fixes:

• “/v2/tenants” get request does not display the “deleted” status for removed companies. 
• Querying “v2/tenants/{id}” and “/v2/tenants/{id}/backupResources” information may take considerable amount of time. 
• A backup policy cannot be created if there are “space” symbols in the encryption hint. 
• A backup policy cannot be created when “processingEnabled” parameter is set to True, and “Credentials” parameter is used.

• After applying a backup policy to a managed Agent, job scheduling settings are displayed incorrectly in the VAC UI, but the settings pushed to the Agent are correct. 
• Site administrators can select Public IP Addresses belonging to a different site when creating a company. 
• Under certain conditions, “Used Storage” counter may display incorrect data on the “Overview” tab. 

• Management agent installed on a Cloud Connect server is incorrectly displayed on the Backup Agents tab. 
• Under certain conditions management agent restarts every hour. 

• Network Extension IP addresses are not filtered by a reseller. Reseller can select IP addresses belonging to unmanaged companies. 
• Reseller can change his or her backup quota to “unlimited” when creating a managed company with “unlimited” quota.

• Only the first 10 000 products are displayed in the mapping area of the plugin. 
• Plugin services may consume excessive memory when there are a lot of tickets created in the ConnectWise Manage application.

• Platform type cannot be detected for old computers and is reported as “Other.”

• Under certain conditions after installing a private hotfix Veeam.MBP.Service fails to connect to the existing database with the following error message ”Cannot change configuration database: Database version is invalid.” 
• Veeam.MBP.Service fails to start when managed backup agents have duplicate IDs (due to cloning operation) in the configuration database.

• “VM without backup” alarm is triggered for VMs that are migrated between vCenter Servers or are re-registered in the virtual infrastructure.

• Under certain conditions, the management agent may connect to VAC server without authentication.

• Computer renaming leads to duplicated values in the Protected Computers report.

• Under certain conditions, usage report for managed Veeam Backup & Replication servers may not be created within first ten days of a month.

Fixes included from previous patches:

• Cloud Copies counter on the Overview dashboard does not return any data.
• Backup agents managed by Veeam Backup & Replication report 0 restore points available in the backup policy drill down view. 

• Portal administrators can log in only when using a full domain name of the Veeam Availability Console server. 
• Users belonging to a domain user group which is added to a Veeam Availability Console security group cannot log in to the portal.
• Users belonging to a nested domain group cannot log in to the portal.

• Under certain conditions, Veeam.MBP.Service might be consuming excessive memory.
• Under certain conditions, data collection job fails to retrieve information from a Cloud Connect server that has lots of task sessions on cloud gateway servers. This results in displaying these tenants’ companies as deleted in the Veeam Availability Console UI.
• Tenant password changed in Veeam Cloud Connect is not synchronized with Veeam Availability Console.

• Office 365 and Gmail SMTP servers are not supported for notifications.  

• Inaccessible computers cannot be located when using CSV and IP discovery rule options.

• Full Access UI mode cannot be applied to a backup agent.  
• When using Select All checkbox on the Managed Backup Agents tab agent settings are applied to all computers of all managed companies regardless of the selected scope in the Company\Reseller filter.  
• Licensing attribute is not updated for managed backup agents which leads to license expiration on the backup agent.    
• Backup agent files are not validated when the installation is performed from a local folder without using a master management agent. This may result in a corrupted installation/upgrade of the backup agent. 

• Under certain conditions, management agents fail to upgrade with the following error message “The system cannot find the file specified.”

• Workstation and server backup counters in the Veeam Cloud Connect usage report are mixed up.
• Under certain conditions License usage report for Veeam Backup & Replication servers is not generated. 

• Under certain conditions, all VMs with a cloud copy which belong to different locations are reported as VMs from the default location in the invoice. 

• Querying tenant information with SetLinks parameter may take considerable time.
• The vMsBackudUpToCloud counter does not display data about VM and Agent backup copy jobs targeted to a cloud repository.  

• Cloud repository usage is not updated in ConnectWise Manage agreements when there are unknown files (not related to Veeam backups) in the repository. 
• Tickets state synchronization may stop working when triggering a mass-resolve operation in the ConnectWise Manage application. 

Solution

msiexec /update c:\VAC.ApplicationServer.x64_3.0.0.2762.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

msiexec /update c:\VAC.WebUI.x64_3.0.0.2762.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACWebUI.txt

msiexec /update c:\VAC.ConnectorService.x64_1.0.1.610.msp /l*v C:\ProgramData\Veeam\Setup\Temp\ConnectorService.txt

msiexec /update c:\VAC.ConnectorWebUI.x64_1.0.1.610.msp /l*v C:\ProgramData\Veeam\Setup\Temp\ConnectorWebUI.txt

More Information

Challenge

• Event 3502 "BackupRepositoryDiscovery.js : Access denied"
• Event 3510 "JobDiscovery.js : Access denied"

Cause

Solution

• It must be a member of the Backup Administrators group in Veeam Backup & Replication.
• It must be a member of the Administrators group on the Veeam Backup & Replication server.

Challenge

Cause

Solution

More Information

Challenge

This article documents antivirus exclusions that have been found to help prevent antivirus software from interrupting Veeam Backup & Replication communications with components such as proxies and repositories or with the guest OS of VMs that are being backed up or replicated.

Cause

In some rare instances antivirus can prevent certain executables, processes, or agents from being correctly deployed or leveraged from the Veeam server, proxy, repository, or even the guest OS.

Solution

Antivirus Exclusions:

Folders on the Veeam Server:

• C:\Program Files\Veeam
• C:\Program Files (x86)\Veeam
• C:\Program Files\Common Files\Veeam
• C:\Program Files (x86)\Common Files\Veeam
• VBRCatalog ( [HKLM\SOFTWARE\Veeam\Veeam Backup Catalog] CatalogPath value)
• NFS (Configured in each repository, stored in [HKLM\SOFTWARE\Wow6432Node\Veeam\Veeam NFS] RootFolder value)
• C:\VeeamFLR\*
• C:\Windows\Veeam
• %programdata%\Veeam

Folder on Guest OS for VSS:

• C:\Windows\VeeamVssSupport (folder might be missing, depends on backup job settings)
• C:\Windows\VeeamLogShipper (folder might be missing, depends on backup job settings)

Folders on VMware Backup Proxies and CIFS Repository Gateway:

• C:\Program Files (x86)\Veeam
• C:\Windows\Veeam

Folders on Windows Repositories:

• C:\Program Files (x86)\Veeam
• C:\Windows\Veeam
• %programdata%\Veeam
• *Also include the path the backup files*
• *Also include the path to 'ArchiveIndex' for capacity tier*

Folders on WAN Accelerator:

• C:\Program Files (x86)\Veeam
• C:\Windows\Veeam
• *Path of WAN cache on WAN accelerators*

Repository Files:

• VeeamAgent.exe
• VeeamAgent64.exe
• *.vmdk
• *.flat
• *.vbm
• *.vbk
• *.vib
• *.vrb
• *.vsb
• *.vlb
• *.erm

Due to the complex nature of antivirus software some additional exclusions may be needed.
If your antivirus has a logging or history system please review this to see actions it may have taken that may affect Veeam Backup & Replication operations.

Challenge

Cause

Last-minute kernel patches for Red Hat Enterprise Linux and its derivatives may cause Veeam Agent for Linux to go unstable (veeam.com/kb2786).
 

VAL build 3.0.1.1046

VAL build 3.0.2.1185

RHEL 7.7 and derivatives (kernel 3.10.0-1062 or higher) are ONLY supported by VAL build 3.0.2.1190.

For standalone installations you can obtain build 3.0.2.1190 from repository.veeam.com.

For instructions on how to deploy build 3.0.2.1190 on installations managed by Backup & Replication please open a support ticket.

Challenge

Cause

Solution

More Information

Challenge

Solution

• New resource manager added for optimizing parallel task processing.
• Improved speed of the full backup for some guest filesystems.
• Fixed rare service crashes caused by memory leaks.
• Fixed rare service crashes on source clusters with high loads.
• Optimized deleted VM retention for configurations with large number of VMs.
• Persistent job retries have been updated to continue jobs interrupted by the reboot of the proxy appliance.

More Information

• To deploy brand new Veeam Availability for Nutanix AHV
• To update existing Veeam Availability for Nutanix AHV

Challenge

Cause

Solution

IMPORTANT: there must be at least 1 storage snapshot that wasn’t transferred to the target. If you specify a snapshot in -source-snapshot that was already transferred, such relationship will not appear in the get-destination-iter output. 

Challenge

Product Information:

Company name: Qumulo
Product Family: Qumulo C-Series
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: C-168T
Storage Category: Hybrid Storage
Drive quantity, size, type: 12 – 14TB SATA HDDs and 4 – 960GB SATA SSDs per node
Storage configuration: 4 node configuration, erasure encoded
Firmware version: Qumulo Core 2.14.0
Connection protocol and speed: SMB – 25GbE
Additional support: All models and configurations of Qumulo C-Series with specifications equivalent or greater than the above. Qumulo C-series includes multiple capacity points (C72T and C168T) with identical performance requirements.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2866

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing

Challenge

Product Information:

Company name: Qumulo
Product Family: Qumulo K-Series
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: K-168T
Storage Category: Hybrid Storage
Drive quantity, size, type: 12 – 14TB SATA HDDs and 3 – 960GB SATA SSDs per node
Storage configuration: 4 node configuration, erasure encoded
Firmware version: Qumulo Core 2.14.0
Connection protocol and speed: SMB – 10GbE
Additional support: All models and configurations of Qumulo K-Series with specifications equivalent or greater than the above. Qumulo K-series includes multiple capacity points (K144T, and K168T) with identical performance requirements.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2866

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing

Challenge

Product Information:

Company name: Qumulo
Product Family: Qumulo P-Series
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: P-92T
Storage Category: All Flash Storage
Drive quantity, size, type: 24 – 3.8TB NVMe SSDs per node
Storage configuration: 4 node configuration, erasure encoded
Firmware version: Qumulo Core 2.14.0
Connection protocol and speed: SMB – 10GbE
Additional support: All models and configurations of Qumulo P-Series with specifications equivalent or greater than the above. Qumulo P-series includes multiple capacity points (P23T, P92T, and P184T) with identical performance requirements.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2866

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing

Challenge

Product Information:

Company name: Qumulo
Product Family: Qumulo HPE Apollo 4200 Series
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: Qumulo HPE Apollo 336T
Storage Category: Hybrid Storage
Drive quantity, size, type: 24 – 14TB SATA HDDs and 4 – 1.9TB SATA SSDs per node
Storage configuration: 4 node configuration, erasure encoded
Firmware version: Qumulo Core 2.14.0
Connection protocol and speed: SMB – 25GbE
Additional support: All models and configurations of Qumulo HPE Apollo 4200 Series with specifications equivalent or greater than the above. Qumulo HPE Apollo 4200 series includes multiple capacity points (90T, 180T, 192T, 288T, and 336T) with similar performance characteristics.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2866

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing

Challenge

Cause

Solution

• Make sure the Key path exists.
• The WriteCacheDirectory key must be specified without backslash ("\").

Challenge

Incompatible Veeam Backup for Office 365 server version (received: 9.6.5.XXX, expected: 9.6.4.YYY).

Cause

Solution

1. Close all Veeam Explorers.
2. Download the latest version of Veeam Backup for Office 365 with Veeam Explorers here
3. Install the following MSI packages: 
   • VeeamExplorerForExchange_X.X.X.XXX.msi.
   • VeeamExplorerForSharePoint_X.X.X.XXX.msi.
4. Restore data using Veeam Explorer for Microsoft Exchange, Veeam Explorer for Microsoft SharePoint or Veeam Explorer for Microsoft OneDrive for Business.

Challenge

Cause

Solution

• Make sure the Key path exists.
• The WriteCacheDirectory key must be specified without backslash ("\").

Challenge

Product Information:

Company name: Nexsan
Product Family: Beast P
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: BT60P
Storage Category: RAID Storage Array
Drive quantity, size, type: 8 – 4TB SATA 7200RPM
Storage configuration: RAID5, 2GB Write cache, 16GB read cache
Firmware version: S011.1304
Connection protocol and speed: iSCSI - 10 GbE
Additional support: All models and configurations of Beast P with specifications equivalent or greater than the above

BEAST P lets you get back your rack when space is tight, designed for 60 drives in 4U of a standard rack. That’s up to 720TB of capacity in 4U and up to a total of 2.16PB in a fully scaled 12U configuration. Our systems are built to withstand some pretty tough storage environments outside the data center like ships, subway stations, and storage closets that have less than ideal temperatures and vibration. Nexsan leverages an innovative design with Cool-Drive Technology and Anti-Vibration Design to significantly reduce heat and vibration in the unit.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2866

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing
• Jumbo frames used in testing (MTU=9000)
• MPIO used in testing - round robin using 2 x 10Ge physical ports, 4 iSCSI sessions per port
• Additional storage settings – sequential optimization used

Challenge

When backing up to a deduplication storage appliance from Veeam Backup & Replication, performance or deduplication ratio is low without using best practice settings. 

Solution

The default options of Veeam Backup & Replication are intended for non-deduplication storage.

If you use deduplication storage systems, you need to additionally configure the backup architecture and Veeam settings to achieve best performance. Below are some hardware and software configuration options that can be used with deduplication storages. In some cases, you may need to combine more than one option, but in all cases, it is strongly encouraged that you consult your deduplication appliance vendor and/or a value added reseller to discuss individual needs.

For the general architecture planning, Veeam’s best practices is to follow the industry 3-2-1 backup rule:

3: Maintain at least three copies of your data and applications. That’s the one copy you’re using and two backups. This way, if one of your backups is unavailable for any reason (for example, if you store the backup in the same locations with the copy you use and both go down), you can still recover what you need.

2: Store your backups on at least two different types of media or storage controller logics. One reason for this is that each type of media has its own vulnerabilities, and you don’t want both of your backups susceptible to the same problem. 

For example: run your business on a general purpose fast storage system and store your backups on a storage system that is specialized for this purpose, or a separate server with local disks.

1: Keep one of the backups in a different location. Consider a crisis in your primary data center, such as a fire or power failure. If all your copies are collocated, they can all be affected, taking down your organization.

To achieve this, use the Veeam best practices. However, each deduplication storage vendor has specific unique feature sets that allow the usage of their storages in different ways. Please check the following links for vendor specific guidelines:

• Best Practices for HPE
• Best Practices for ExaGrid
• Best Practices for Dell EMC DataDomain
• Best Practices for Quantum

General Deduplication Storage Best Practices

Step 1:  Creating backups on the first site with long term retention.

Step 2: Add offsite target.

In addition to the above scenarios, you can use the following options to create an offsite copy of your data.

3. Place a deduplication storage on primary side and use object storage or tape on second site. Use Veeam backup to tape jobs or Veeam scale-out backup repository cloud tier (connection to object storages) to store data offsite.

The scenario above reflects general best practices. Please contact your deduplication storage vendor for further guidance and check the vendor links provided above for additional usable scenarios with the specific storage.
 

Best Practices For Deduplication Storage Systems

Backup Chains

• General Deduplication Storages / ExaGrid / Quantum DXI + Backup Copy Job That Reads From Source Backup Chain:

 

•  HPE StoreOnce Catalyst / Dell EMC DataDomain DDboost  + Copy Over WAN + Backup Copy Job:

The previous described option provides fast processing in situations that do not involve transport over WAN links. Also, optimized integration with Catalyst and Dell EMC DDBoost allows using synthetic processing. For these environments uncheck the “Read the entire restore point from source backup instead of synthesizing it from increments” but leave the weekly GFS backup enabled.

• Special Scenarios Based on Vendor Own Guidelines + Backup Job With Active Full or Synthetic Full Backup:

Inline Deduplication

However for deduplication devices that have a non deduplicated landing zone (Like ExaGrid) and backup only VMs, you can save some space: to do this, enable inline deduplication



 

Compression

Block Size

• Storage optimization: Local target (large blocks)

• Storage optimization: Local target 

Use this option for ExaGrid.

Block Alignment

Encryption

Health Check

  
Veeam Defragmentation And Compact Settings

More Information

Challenge

[31.08.2019 00:00:00] Warning Failed to create SSH connection to host: '<hostname>', port: 22, user: '<username>', elevation to root: 'yes', autoSudo: no, use su if sudo fails: no, auth type: 'Password', host name: <hostname>, IPs: [<ipaddress>].. Server does not support diffie-hellman-group1-sha1 for keyexchange

Cause

• Granados SSH — this library does not support FIPS 140-2 compliant key exchange methods. Veeam uses this library if FIPS mode was not enabled on the Linux server at the time when the server was added in the Veeam backup console. 
• Renci SSH — this library supports FIPS 140-2 compliant key exchange methods. Veeam uses this library if FIPS mode was enabled on the Linux server at the time when the server was added in the Veeam backup console. 

Solution

1. In the Veeam Backup console, click Backup Infrastructure > Managed Servers > Linux.
2. Right-click the server in the list and click Properties....
3. Follow the steps of the Edit Linux Server wizard without changing the server properties — just click Next, and then click Finish.

More Information

Challenge

Cause

Solution

More Information

Challenge

• The following error appears during the backup job session, then the backup job fails: This functionality requires using paid Veeam Agent edition.

• The following error appears during the agents discovery session, then the job fails: Remote deployment and management is available for licensed agents only. Please change your backup server settings to allow managed agents to consume the license, then perform a protection group rescan.

Cause

Solution