Challenge

Cause

Apr 19 11:26:45 hostname kernel: ERR veeamsnap:page_array_alloc Cannot allocate page. Schedule.

Solution

More Information

• User Guide: Installing Veeam Agent for Linux
• User Guide: Installing Veeam Agent for Linux in Offline Mode
• KB 2260: Failed to load module [veeamsnap] (express version)

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

msiexec /update c:\VAC.ApplicationServer.x64_2.0.2.1807.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

msiexec /update c:\VAC.ConnectorService.x64_1.0.0.254.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

More Information

Challenge

"Specified database is not valid. Please choose Veeam ONE database or create a new database to proceed."

Cause

Solution

More Information

Challenge

Cause

Solution

Enable the rules

CMD (Command line)

To enable these rules via the command line, launch the command prompt console with elevated permissions (as Administrator) and execute the command below:

netsh advfirewall firewall set rule name="Remote Event Log Management (NP-In)" new enable= Yes
netsh advfirewall firewall set rule name="Remote Event Log Management (RPC)" new enable= Yes
netsh advfirewall firewall set rule name="Remote Event Log Management (RPC-EPMAP)" new enable= Yes
netsh advfirewall firewall set rule name="COM+ Network Access (DCOM-In)" new enable= Yes

PowerShell

To enable these rules via the PowerShell, launch the PowerShell console with elevated permissions (as Administrator) and execute the command below:

Set-NetFirewallRule -Name 'RemoteEventLogSvc-NP-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'RemoteEventLogSvc-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'RemoteEventLogSvc-RPCSS-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'ComPlusNetworkAccess-DCOM-In' -Enabled True

More Information

Challenge

To allow communications between Veeam Agents and VBR, TLS certificates are used. By default, Veeam Backup & Replication uses a self-signed certificate.

Solution

In order to use a certificate signed by Certification Authority (CA), the following requirements should be met:

• Veeam Agents must trust the Certification Authority and the VBR signed certificate (they must be added to the Trusted Root Certification Authority store on the clients)
• Certificate revocation List (CRL) should be accessible from Veeam Agents and VBR server

A certificate signed by Certification Authority should have the following key usage to sign and deploy child certificates on Veeam Agents:

• Digital Signature
• Certificate Signing
• Off-line CRL Signing
• CRL Signing (86)

For example a subordinate CA Certificate template in Windows has the required key usages:

After applying the signed certificate on the VBR server according to the User Guide, on the next job run Veeam Agents will receive child certificates. The resulting certification path will look like this:

More Information

Note: Veeam Agent for Microsoft Windows version 2.1 has a known issue with CRL check if a signed certificate is installed on the VBR server. Please contact technical support in order to obtain a fix.

Challenge

Cause

Solution

Enable the rules

CMD (Command line)

To enable these rules via the command line, launch the command prompt console with elevated permissions (as Administrator) and execute the command below:

netsh advfirewall firewall set rule name="Remote Event Log Management (NP-In)" new enable= Yes
netsh advfirewall firewall set rule name="Remote Event Log Management (RPC)" new enable= Yes
netsh advfirewall firewall set rule name="Remote Event Log Management (RPC-EPMAP)" new enable= Yes
netsh advfirewall firewall set rule name="COM+ Network Access (DCOM-In)" new enable= Yes

PowerShell

To enable these rules via the PowerShell, launch the PowerShell console with elevated permissions (as Administrator) and execute the command below:

Set-NetFirewallRule -Name 'RemoteEventLogSvc-NP-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'RemoteEventLogSvc-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'RemoteEventLogSvc-RPCSS-In-TCP' -Enabled True
Set-NetFirewallRule -Name 'ComPlusNetworkAccess-DCOM-In' -Enabled True

More Information

Challenge

Cause

Solution

• CREATE ANY DATABASE permissions
• db_owner role on the Veeam ONE database
• db_datareader permissions on the master database
• public, db_datareader, SQLAgentUserRole permissions on the msdb database
• [For Always-On Availability Groups] VIEW SERVER STATE permissions

More Information

Challenge

1. Veeam ONE 9.5 server may lose a connection to the SQL Server for runtime operations
2. Veeam ONE 9.5 server may lose a connection to the SQL Server during the initial installation
3. “Veeam Licensing” web server is unavailable when only TLS 1.2 protocol is enabled
4. Veeam ONE Reporter website doesn’t work when only TLS 1.2 protocol is enabled

Cause

Solution

More Information

Challenge

Solution

1. Log on to the Veeam Customer Portal and go to Notifications >> Promos (https://my.veeam.com/#/promotions)
2. If you are the License Administrator for a Nutanix AHV or Agent license you will see this:

       3. Select “Download a License now”.  The download will begin automatically.  The file name is Veeam_backup_subscription_0.lic.

More Information

Challenge

The following error can be found in the Windows Application log or in VeeamGuestHelper.log file (in earlier versions this log file is called VeeamVSSSupport.log) located in C:\ProgramData\Veeam\Backup on the guest:

Cause

Solution

Depending on Veeam B&R you need to apply the following settings either for NT AUTHORITY\SYSTEM (pre-v8) or for the user specified in the backup job (v8):

Challenge

Product Information:

Product Family: Data Protection Appliance Series
Status: Veeam Ready - Repository 
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: DPA M (M24)
Storage Category: Integrated Data Protection Appliance
Drive quantity, size, type: 4, 8TB, SAS HDD | (32TB total)
Storage configuration: RAID5 configured HDDs (12Gbps SAS) with read/write cache
Firmware version: V2
Connection speed and protocol: 10GbE ISCSI
Additional support: All models and configurations of DPA M with specifications equivalent or greater than the above: M, L & XL

General product family overview:
Adistec Data Protection Appliances for VEEAM combine: customized Intel hardware platform, Veeam Availability Suite software, and Adistec Enterprise Cloud services resulting in a robust solution to protect your customer’s data supported by Adistec Total Support and Veeam. Simplify the Backup Storage’s infrastructure, eliminate complexity and maximize data availability and protection with this turnkey solution at an affordable price. Three models available in one and two RUs, with single Quad-Core or dual Octa-Core processors configuration. Starting with four 3.5” disks up to twelve, getting 16TB to 120TB of usable capacity.

Veeam testing configuration:

Veeam Build Number: 9.5.0.1536
Veeam Settings:

• Repository Type: Windows
• Deduplication: Yes
• Compression: Optimal
• Storage Optimization: Local Target
• Per-VM Backup Files: Yes
• Decompress before storing: No
• Align backup file blocks: No

Vendor recommended configuration:

Vendor Settings:

• HDD – 1GB read cache on RAID controller, write back policy using backup battery
• Jumbo Frames used during testing (9000 MTU)
• Multipathing using Windows Server NIC teaming

More Information

Company Information:

Company name: Adistec
Company overview:
Adistec delivers enterprise Hybrid Cloud Solutions that facilitate companies be more agile and flexible. We sell our solutions through channels, with focus in Latin America and the Caribbean.Our offering includes integrated hardware, software, Professional and Cloud Services solutions. We have our own DC Infrastructure in USA and in main countries of Latin America, from where we deliver local Cloud Services, addressing the need of Data Locality and Invoicing. Scaling production, testing and Disaster Recovery workloads, are all part of our strengths and benefits offered. Thanks to many years of experience Adistec guarantees end-to-end excellence and professionalism.

Challenge

The following error can be found in the Windows Application log or in VeeamGuestHelper.log file (in earlier versions this log file is called VeeamVSSSupport.log) located in C:\ProgramData\Veeam\Backup on the guest:

Cause

Solution

Depending on Veeam B&R you need to apply the following settings either for NT AUTHORITY\SYSTEM (pre-v8) or for the user specified in the backup job (v8):

Challenge

25.7.2018 10:14:12    6 (5940)   Upgrading backup jobs...
25.7.2018 10:14:12    4 (4588) Error: Job with the specified name already exists.
25.7.2018 10:14:12    4 (4588) Type: System.Exception
25.7.2018 10:14:12    4 (4588) Stack:
25.7.2018 10:14:12    4 (4588)    at Veeam.Archiver.Controller.JobConfigVerifier.VerifyCreatedJob(JobConfig config, BackupScopeConfig

Cause

Solution

Challenge

To fulfill the 3-2-1 rule’s requirement for an offsite backup, a repository has been pointed to removable storage media (such as USB hard drives or RDX disks). The action of swapping to a new medium causes Veeam Backup & Replication jobs to fail because the job cannot find a file.

Cause

By default, Veeam Backup & Replication assumes that if recent files are missing, the job should fail. This way, the job provides a notification that something is wrong instead of taking any action.

Solution

For an introduction to Veeam Backup & Replication’s options for rotated media, consult the user guide. The following notes assume the option for rotated media is enabled in the repository advanced settings.
 
Retention Options on Windows-type Repositories
 
There are three options:

1. Default backup job behavior;
2. Default backup copy job behavior;
3. Deleting files when disks are rotated.

• A forward incremental backup job creates a full backup file and an increment on each of the three disks over the first three days. On the fourth day, the first disk is re-used. A new full backup file is created. There are now 7 points across all disks, so retention policy is met, but the initial full backup cannot be deleted because an incremental file is dependent on it. After another restore point is created, the older two files on disk 1 are deleted by retention policy.
• A backup copy job also creates a full backup file and an incremental file on each of the three disks over the first three days. On the fourth day, the first disk is re-used. A new incremental backup file is created, and the oldest incremental file is merged into the full backup file.

 
If a disk does not contain enough space for a new backup file, the job will fail instead of deleting old files. This can be avoided by deleting old files as soon as the disks are swapped. This can be done manually, via pre-job script, or with the registry setting described below.
 
 
Retention on All Other Repositories
 
By default, repositories configured for rotated media do not delete any backup files when disks are swapped. If a disk containing a previous backup is to be re-used, but lacks sufficient available space for new backup files, the old files must be deleted manually, or by a pre-job script.
 
Retention policy is enforced, but only on the current backup chain. For example, consider a backup copy job that creates restore points every hour, with disks swapped once per day, and retention policy set to 6 restore points. Once there are 7 restore points on the current disk (a full backup file and 6 incremental backup files), the oldest increment is merged with the full backup file so that there are 6 restore points on disk. The disk is swapped out for a new one, and the process repeats. When the first disk is re-used, the 6 backup files still on the disk are ignored. A new full backup file is created, and a new chain of incremental files. At the end of the day, there are 12 restore points on disk, with only the most recent 6 visible in the Veeam Backup & Replication console.
 
An alternative behavior is available as a registry setting (below).
 
Deleting Files When Disks are Rotated
 
Please note

When this registry setting is enabled (set to 3 or 5), jobs using a repository configured for rotated media will maintain retention normally until the job detects that previously-available files are missing.
 
Create this value on the Veeam Backup and Replication server:
 
HKLM\Software\Veeam\Veeam Backup and Replication
ForceDeleteBackupFiles (DWORD)
 
Set to 3 to make the job delete the entire contents of the backup job’s folder only.
Set to 5 to make the job delete the entire contents of the root backup repository folder, potentially deleting any files belonging to other jobs. Note that if the backup repository points at the volumes root folder, the entire volume's contents are erased.
 
The Veeam Backup Service must be restarted after creating this registry value. Make sure no jobs or restores are running before restarting this service.
 

Challenge

This document provides details for resetting CBT of a VMware VM.

Under certain circumstances CBT cannot be used please see: http://www.veeam.com/kb1163

In Veeam Backup & Replication v7 and older:

Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

In Veeam Backup & Replication v8 and newer:

CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss.

The following entries can be seen in the job log:

[timestamp] Info [AP] (Client) output: Soap fault. Error caused by file /vmfs/volumes/4b953b81-76b37f94-efef-0010185f132e/name/name.vmdkDetail: '', endpoint: ''\n
[timestamp] Info [AP] (Client) output: --tr:Failed to enumerate changed areas of the disk using CTK. Device key: [2000], size: [26843545600]. VM ref: [vm-xx]. Change ID: [*]\n

Solution

Reset CBT as detailed by VMware KB 2139574 

More Information

Note: Resetting CBT is procedure which occurs within the VMware environment. If you require assistance with this procedure, please contact VMware Support.

When resetting CBT on a virtual machine, the next time the Backup or Replication job runs, the entirety of the disks on the Virtual Machine is read. It will not however create a new full (*.VBK) as blocks are compared to create a normal sized incremental.

Typically, a single job session with CBT warnings does not indicate a malfunction. VMware CBT will be automatically reset due to certain operations, such as Storage vMotion. Note that the presence of snapshots will prevent successful reset of CBT. For more information see https://kb.vmware.com/kb/2048201

Challenge

When attempting to backup a SQL server that is hosting a significant amount of databases the following error is thrown:
 

Unable to release guest. Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{}]. Instance ID: [{}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{}]. Instance ID: [{}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Cause

In SQL Server, the snapshot backup of each database uses five threads in the Sqlservr.exe process. Additionally, other activities may also use threads in the Sqlservr.exe process. Depending on the configuration of SQL Server, the available threads may be used up if you create a snapshot backup of many databases at the same time.

It is possible to process the VM in a crash-consistent manner by disabling Application-Aware Image Processing within the job.

Solution

It is advised to create another SQL Server and move the databases that are causing failure to the other SQL server.
In certain scenarios changing the max worker threads Server Configuration Option may help negate the issue. However, it is advisable to contact Microsoft support for verifying this step.

Please refer to the following articles for details:

• https://support.microsoft.com/en-us/kb/943471
• http://blogs.msdn.com/b/psssql/archive/2009/03/03/how-it-works-sql-server-vdi-vss-backup-resources.aspx
• https://msdn.microsoft.com/en-us/library/ms190219.aspx?f=255&MSPPError=-2147217396
• https://support.microsoft.com/en-us/kb/2157129

More Information

Note: Veeam Support is not able to provide assistance with modifying the number of threads within SQL. Please contact Microsoft Support for assistance.

Challenge

1. To add such organization into Veeam Backup for Microsoft Office 365, the SharePoint Administrator role must still be assigned to a user you add an organization with. Otherwise you would see the following error: “Role is not assigned”
   
2. Backup jobs will fail with the error ““The remote name could not be resolved: 'example-admin.sharepoint.com'”.

Solution

1. To solve the “Role is not assigned” issue:

1. To solve the “The remote name could not be resolved: 'example-admin.sharepoint.com'” issue:

Challenge

Solution

1. Creating a custom role on Microsoft Azure

$role = [Microsoft.Azure.Commands.Resources.Models.Authorization.PSRoleDefinition]::new()
$role.Name = 'Veeam Restore Operator'
$role.Description = 'Permissions for Veeam Direct Restore to Microsoft Azure'
$role.IsCustom = $true
 
$permissions = @(

'Microsoft.Storage/storageAccounts/listkeys/action',
'Microsoft.Storage/storageAccounts/read',
'Microsoft.Network/locations/checkDnsNameAvailability/read',
'Microsoft.Network/virtualNetworks/read',
'Microsoft.Network/virtualNetworks/subnets/join/action',
'Microsoft.Network/publicIPAddresses/read',
'Microsoft.Network/publicIPAddresses/write',
'Microsoft.Network/publicIPAddresses/delete',
'Microsoft.Network/publicIPAddresses/join/action',
'Microsoft.Network/networkInterfaces/read',
'Microsoft.Network/networkInterfaces/write',
'Microsoft.Network/networkInterfaces/delete',
'Microsoft.Network/networkInterfaces/join/action',
'Microsoft.Network/networkSecurityGroups/read',
'Microsoft.Network/networkSecurityGroups/write',
'Microsoft.Network/networkSecurityGroups/delete',
'Microsoft.Network/networkSecurityGroups/join/action',
'Microsoft.Compute/locations/vmSizes/read',
'Microsoft.Compute/locations/usages/read',
'Microsoft.Compute/virtualMachines/read',
'Microsoft.Compute/virtualMachines/write',
'Microsoft.Compute/virtualMachines/delete',
'Microsoft.Compute/virtualMachines/start/action',
'Microsoft.Compute/virtualMachines/deallocate/action',
'Microsoft.Compute/virtualMachines/instanceView/read',
'Microsoft.Compute/virtualMachines/extensions/read',
'Microsoft.Compute/virtualMachines/extensions/write',
'Microsoft.Resources/checkResourceName/action',
'Microsoft.Resources/subscriptions/resourceGroups/read',
'Microsoft.Resources/subscriptions/resourceGroups/write',
'Microsoft.Resources/subscriptions/locations/read'

)


$role.Actions = $permissions
$role.NotActions = (Get-AzureRmRoleDefinition -Name 'Virtual Machine Contributor').NotActions
$subs = '/subscriptions/00000000-0000-0000-0000-000000000000'
$role.AssignableScopes = $subs
New-AzureRmRoleDefinition -Role $role

2. Add the user with custom role to Veeam Backup and replication software.

1. Start Windows command line with Administrative privileges
2. Navigate to Veeam Backup installation folder (by default: C:\Program Files\Veeam\Backup and Replication\) and then to “Backup” subfolder, so the full path should look like: C:\Program Files\Veeam\Backup and Replication\Backup
3. Type following command:

   Veeam.backup.manager.exe REGISTERAZUREACCOUNT <username> <password>

Challenge

Solution

1. Using the netsh command, a system proxy must be configured in order for Exchange Online PowerShell to work properly. Please start an elevated cmd.exe and input:

   Netsh
   Winhttp
   Set proxy <proxyFQDNorIP>

2. Ensure that no Backup or Restore jobs are running, then stop both the Veeam Backup for Microsoft Office 365 Service and the Veeam Backup Proxy for Microsoft Office 365 Service
3. Navigate to %ProgramData%\Veeam\Backup365
4. Edit Config.xml, adding <Ews UseSystemProxy="True" /> under <Veeam>, but above <Archiver>
   
5. Do the same thing with Proxy.xml
   
6. Navigate to the Veeam Backup for Microsoft Office 365 installation path (by default %ProgramFiles%\Veeam\Backup365), edit the file Veeam.Archiver.Service.exe.config,and add the following lines under <configuration>:
    

   <system.net>
   <defaultProxy>
   <proxy proxyaddress="http://yourproxyFQDNorIP:port" usesystemdefault="true" />
   </defaultProxy>
   </system.net>

   
   
7. And the same thing for Veeam.Archiver.Proxy.exe.config
   
8. Pay attention to proxy address format – ‘http://yourproxyFQDNorIP’ and ‘port’ are mandatory, otherwise you will see errors like this one