Challenge

Cause

Solution

1. Download the hotfix here.
2. Open Veeam ONE Reporter web interface and navigate to the Configuration tab > Extension modules node.
3. Use the Install option to apply the hotfix.

More Information

Challenge

A job fails with an error related to NFC connectivity. For example:
 
“…Client error: NFC storage connection is unavailable…Failed to create NFC download stream.”



 

Cause

The cause of the majority of NFC errors fall in to 3 primary categories:

• Port (902)
• Permissions*
• DNS
 
*If the account that Veeam Backup & Replication is using to communicate with the VMware Environment has granular permissions set please confirm all permissions are set according to the Granular Permissions Guide.
https://www.veeam.com/veeam_backup_9_0_permissions_pg.pdf
 
There is also the potential that the issue lies within the VMware Infrastructure itself.

Solution

It is suggested that while attempting to resolve this issue the job be configured to use a single proxy. This is done to isolate an issue specific to a single proxy, as well as make it easier to identify which logs are needed.
 
──────────────────────────────────────────────────────────
Port, Permission, or DNS: Investigation
──────────────────────────────────────────────────────────
To investigate the potential of a Ports, Permissions, or DNS issue you will need to review the Agent logs.
1.  Navigate to the following location, on the Veeam Proxy

%programdata%\Veeam\Backup

2.  Open the folder that matches the name of the job that is having an issue.
3.  Within this folder you will need to find the agent log for the specific VM.

For a Backup Job:
Agent.<JobName>.Source.<VMName>.log
For a Replication Job:
Agent.<JobName>.Source.<VMName>.log
Agent.<JobName>.Target.<VMName>.log

4.  Search for the following:

Creating NFC download stream

5.  Here you will want to look for the following items that could point to the cause of the error.

a.       Port
If there is an issue with ports, there will be an error after the following:
nfc|           Establishing connection with the host [esx1]. Port: [902]. Failed.
b.      Permissions
If there is an issue with permissions, there will be an error after the following:
nfc|           Sending authd message: [SESSION ID NUMBER].
nfc|           Sending authd message: [Name of Host].
nfc|           Waiting for the authd reply message...
c.       DNS
If there is an issue with DNS, the following error will be found.
nfc|             Resolving host name (esx1) to IP address...
nfc|             Resolving host name (esx1) to IP address... Failed.

 
──────────────────────────────────────────────────────────
NFC Connectivity Troubleshooting
──────────────────────────────────────────────────────────
The Management Agents for that host may need to be restarted. https://kb.vmware.com/kb/1003490
 

More Information

• DNS Resolution of target host
• Port 902 is open to/from Backup Server/Proxy to ESX(i) host
• Permissions to download files via ESX(i) host and/or vCenter

Challenge

Solution

If there are any remote proxies in use:

During .msp install, the default proxy is updated automatically. But if there are any remote proxies in the environment, use the procedure below to upgrade them.

1. When .msp is installed, run Veeam Backup for Microsoft Office 365 application, go to Backup Infrastructure, right-click on the Backup Proxies and select Upgrade.
2. In the wizard choose the remote proxies that should be upgraded.
3. When the upgrade is done, the upgrade statistics will be shown.
4. Confirm the backup from the remote proxies can be browsed.

Challenge

The following error message occurs when attempting to create a VMware snapshot.

File <unspecified filename> is larger than the maximum size supported by datastore <unspecified datastore>

The Veeam Backup & Replication job fails with the following error:

CreateSnapshot failed, vmRef "vm-xxx", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
File <unspecified filename> is larger than the maximum size supported by datastore

Cause

This error is displayed by Veeam Backup & Replication when creation of a snapshot fails within the user vSphere environment.

The user can recreate this error by manually creating a snapshot on the target virtual machine through the VMware vSphere Client.

Solution

Challenge

Some of VMware features and permissions are not granted by default at the start of VMware Cloud on AWS (VMC). Thus, some depending Veeam Backup & Replication features will be limited or not operating. Depending on VMware update releases for VMware Cloud on AWS, the situation may change and the features from the table below may become available. Please contact your VMware administrator for timely update.

Solution

Problem

Impossible to add the VMware Cloud on AWS vCenter server to the managed server, or VMs within this vCenter are not visible in the list of VMs.

Solution

1. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
2. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmc.vmware.com.

Problem

When working with Restore or VM Replication wizard, users may face some issues accessing VMware Cloud on AWS vCenter server.

Cause

By design, VMware does not provide customers access to the background infrastructure and used datastores.

Solution

For proper operation, you can select the specific areas marked as “Workload” or “Compute”. Avoid using the non-accessible areas, for example:

• vsanDatastore datastore
• Management VMs folder
• Mgmt-ResourcePool resource pool

More Information

VMware Cloud on AWS Firewall Configuration

Challenge

Cause

Solution

In Veeam 9.5 we removed the setting for disabling Hosts right from the menu. Now we do it through the Managed VM’s tab. Excluding a host this way will also exclude the sockets that were used for that host. You should see the licensed socket count change after this KB has been finished. 

The first thing to do is go into the Server Settings for Veeam One.



After the server settings window is open we are going to go into the Monitored VM’s tab and hit the ‘Create New…’



This will open a new window where we will put in the information for this new exclusion rule. In this window name and describe the rule in any way you want. Click next when you are finished. 



In the next window we will click ‘Add…” and then ‘Infrastructure Tree’.



When the tree shows up in a new window, go ahead and find the host you no longer wish to be in Veeam One.



Once the host is selected click ‘OK’ and hit ‘Next’. This will send you to the last window which will give you an overview of the change. Just hit finish and your all done. 

More Information

Challenge

Cause

Solution

Database migration (permanent solution)

The best way to resolve the issue would be to move Veeam ONE database to a Standard or Enterprise edition of MS SQL Server. The procedure for configuring Veeam ONE to use a new SQL Server connection is described in the following KB article: http://www.veeam.com/kb1599

Purging old data (temporary workaround)

You can also address the issue by purging old performance data as described below.

You can delete past performance data via a custom SQL script that should be run against Veeam ONE database. Follow these steps to reduce the database size:

• make sure to properly backup Veeam ONE database;
• be aware that this change will touch all of Veeam ONE components: for example in Veeam ONE Reporter you will not be able to build reports that rely on the data that you delete with the script presented below;
• keep in mind that all graphs in Veeam ONE Monitor that rely on the deleted data that will be unavailable;
• unless specifically instructed by Veeam Support, do not modify the SQL statement and do not execute it against other database tables. If the aforementioned workaround does not help to reduce the database size, please contact Veeam Support for further assistance.

   2017-01-01 00:00:00.001 - the point in time before which the historical performance data will be purged. Change the date accordingly.
   (year)-(month)-(date)

The LDF log file name can be checked in the database properties window (select VeeamONE database > Right-click > Properties > Files):

More Information

Challenge

Cause

Solution

1. Create simple repositories, and at least one of them should be a path where backup files to be imported are stored (e.g., if backup files are in D:\Backups\Backup_Job_1\ then create a repository referencing the path D:\Backups).
2. Re-scan the simple repository or repositories, and ensure all backups are imported.
3. Create the SOBR and add the relevant repositories as extents.
4. Within the creation process Veeam will inform via confirmation prompt that “Jobs and backups using this extent will be automatically updated to point to the scale-out backup repository.”
6. If any jobs are not mapped to the backup files now within the SOBR, map them per the standard process.
    

1. Place the backup files to be imported in a temporary path. (e.g., if backup files are in D:\Backups, either copy or move them to D:\Backups-temp. A move is usually much faster.)
2. Create a new simple backup repository that references this path (in our example, D:\Backups-temp).
3. Re-scan the repository, and ensure all backups are imported.
4. You can map the pre-existing job to the backup files’ new location now, or at the end.
5. Edit your existing SOBR and add the new temporary repository to its list of extents.
6. Place the new temporary repository extent in “Maintenance” mode.
7. Evacuate the extent. Veeam will redistribute the backup files to other extents in the SOBR.
8. If you did not map the original job in step 4, do so now.

1. If you haven't done so yet, create a new simple backup repository.
2. In the root folder of the new simple backup repository, create a folder with the name of the job folder on SOBR.
3. (Optional) To avoid automatic rescans of SOBR finding the files during the move again, place the extents with the job files in "Maintenance" mode.
4. Remove the SOBR backup from Backups->Disk using the option "Remove from configuration" (not "remove from disk"!).
5. Move all VRB/VBK/VIB backup files from the folders of the job from all extents into the newly created folder on the simple backup repository.
6. Move one of VBM files from any job folder on SOBR to the newly created job folder on the simple backup repository.
7. Remove the leftover folders with the job name and VBM files from all extents of SOBR.
8. Rescan the simple backup repository. It will import the backup into configuration.
9. (Optional) Map the job which you want to use for this backup to the newly imported Backup.

More Information

• Allowed alphanumeric characters: a-z, A-Z, 0-9
• Allowed special characters: _ - . + = @ ^
• Must not contain spaces; use underscores in their place

• Make sure to copy files from the backup chain from all extents.
• SOBR backup chains are typically per-VM backup files. Simple repositories are typically per-job backup files. The chain will retain this method until an active full is run on the job to enforce the change.
• The VBM file will exist in all extents. All VBMs for a particular backup job within a single SOBR should be identical. As a result, the target simple backup repository needs only contain one VBM.

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

1. Create simple repositories, and at least one of them should be a path where backup files to be imported are stored (e.g., if backup files are in D:\Backups\Backup_Job_1\ then create a repository referencing the path D:\Backups).
2. Re-scan the simple repository or repositories, and ensure all backups are imported.
3. Create the SOBR and add the relevant repositories as extents.
4. Within the creation process Veeam will inform via confirmation prompt that “Jobs and backups using this extent will be automatically updated to point to the scale-out backup repository.”
6. If any jobs are not mapped to the backup files now within the SOBR, map them per the standard process.
    

1. Place the backup files to be imported in a temporary path. (e.g., if backup files are in D:\Backups, either copy or move them to D:\Backups-temp. A move is usually much faster.)
2. Create a new simple backup repository that references this path (in our example, D:\Backups-temp).
3. Re-scan the repository, and ensure all backups are imported.
4. You can map the pre-existing job to the backup files’ new location now, or at the end.
5. Edit your existing SOBR and add the new temporary repository to its list of extents.
6. Place the new temporary repository extent in “Maintenance” mode.
7. Evacuate the extent. Veeam will redistribute the backup files to other extents in the SOBR.
8. If you did not map the original job in step 4, do so now.

1. If you haven't done so yet, create a new simple backup repository.
2. In the root folder of the new simple backup repository, create a folder with the name of the job folder on SOBR.
3. (Optional) To avoid automatic rescans of SOBR finding the files during the move again, place the extents with the job files in "Maintenance" mode.
4. Remove the SOBR backup from Backups->Disk using the option "Remove from configuration" (not "remove from disk"!).
5. Move all VRB/VBK/VIB backup files from the folders of the job from all extents into the newly created folder on the simple backup repository.
6. Move one of VBM files from any job folder on SOBR to the newly created job folder on the simple backup repository.
7. Remove the leftover folders with the job name and VBM files from all extents of SOBR.
8. Rescan the simple backup repository. It will import the backup into configuration.
9. (Optional) Map the job which you want to use for this backup to the newly imported Backup.

More Information

• Allowed alphanumeric characters: a-z, A-Z, 0-9
• Allowed special characters: _ - . + = @ ^
• Must not contain spaces; use underscores in their place

• Make sure to copy files from the backup chain from all extents.
• SOBR backup chains are typically per-VM backup files. Simple repositories are typically per-job backup files. The chain will retain this method until an active full is run on the job to enforce the change.
• The VBM file will exist in all extents. All VBMs for a particular backup job within a single SOBR should be identical. As a result, the target simple backup repository needs only contain one VBM.

Challenge

Cause

Solution

More Information

Challenge

Cause

vprob.storage.connectivity.lost
esx.problem.storage.connectivity.lost

Solution

More Information

Challenge

Cause

Solution

Challenge

This document provides details for resetting CBT of a VMware VM.

Under certain circumstances CBT cannot be used please see: http://www.veeam.com/kb1163

In Veeam Backup & Replication v7 and older:

Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

In Veeam Backup & Replication v8 and newer:

CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss.

The following entries can be seen in the job log:

[timestamp] Info [AP] (Client) output: Soap fault. Error caused by file /vmfs/volumes/4b953b81-76b37f94-efef-0010185f132e/name/name.vmdkDetail: '', endpoint: ''\n
[timestamp] Info [AP] (Client) output: --tr:Failed to enumerate changed areas of the disk using CTK. Device key: [2000], size: [26843545600]. VM ref: [vm-xx]. Change ID: [*]\n

Solution

Reset CBT as detailed by VMware KB 2139574 

More Information

When resetting CBT on a virtual machine, the next time the Backup or Replication job runs, the entirety of the disks on the Virtual Machine is read. It will not however create a new full (*.VBK) as blocks are compared to create a normal sized incremental.

Typically, a single job session with CBT warnings does not indicate a malfunction. VMware CBT will be automatically reset due to certain operations, such as Storage vMotion. Note that the presence of snapshots will prevent successful reset of CBT. For more information see https://kb.vmware.com/kb/2048201

Challenge

You need to remove an existing Veeam Backup and Replication (VBR), Veeam Agent for Windows (VAW) or Veeam Agent for Linux (VAL) license from Veeam Backup and Replication (VBR) or Enterprise Manager (EM).

Cause

A license has to be removed from VBR/EM.

Solution

In order to determine which section of this article to use, you must know if Enterprise Manager is controlling the licenses within Veeam Backup & Replication. If you are unsure, attempting to change the license file within Veeam Backup & Replication will tell you. If it allows you to click the Install button within the license section without a message about Enterprise Manager then Enterprise Manager is not in use.

If licenses are managed by Enterprise Manager:

1. Open an administrative command prompt.
2. Navigate to the EM installation folder (default path in the example below):

   cd C:\Program Files\Veeam\Backup and Replication\Enterprise Manager

3. Remove the license by running

   Veeam.Backup.EnterpriseService.exe -removelicense [vaw, val]

4. After running this command, restart Veeam Backup Enterprise Manager service so that the license information is updated in the UI.

If licenses are managed by Veeam Backup & Replication

1. Open an administrative command prompt.
2. Navigate to the VBR installation folder (default path in the example below):

   cd C:\Program Files\Veeam\Backup and Replication\Backup

3. Remove the license by running

   Veeam.Backup.Manager.exe -removelicense [all, vbr, vaw, val]

More Information

Note:These commands can also be run from PowerShell.

Note: specify the desired parameter without brackets.

Example commands:

• To remove the Veeam Agent for Windows license from Veeam Enterprise Manager:

  Veeam.Backup.EnterpriseService.exe -removelicense vaw

• To remove Veeam Agent for Linux from Veeam Backup & Replication:

  Veeam.Backup.Manager.exe -removelicense val

If no parameter is specified, all existing licenses will be removed.

Challenge

After installing Veeam Agent for Microsoft Windows and configuring a backup job, the settings are not shown in the Control Panel, and Veeam Agent for Microsoft Windows keeps prompting to configure backup settings.

Cause

The Windows OS was installed with non-English region and culture settings. As a result, these settings were applied to the Windows system account.

Solution

You will need to set Format to English (United States) for the system account.

Go to the Region settings (press Win + R > type intl.cpl > press Enter)

1. On the Formats tab
   1. set Format to English (United States)
2. On the Administrative tab
   1. click Copy Settings
      1. check that Format is set to English for Current user
      2. Enable the Welcome screen and system accounts checkbox
      3. Click OK to save settings and return to the Administrative tab.
3. Click OK to close the Region settings and save settings.
4. Restart your machine

Once the necessary changes are applied to the system account, a non-English Format can be set for the Current user account. Note, that the Format has to remain English (United States) for the system account.

More Information

Here is an example of a working configuration:

Challenge

Job with truncation enabled shows warning: 

Failed to truncate transaction logs for SQL instances: MICROSOFT##WID

Note: This KB article only applies if the specified instance is named MICROSFT##WID.

Cause

After applying update 3 for Veeam Backup & Replication v8, we attempt to connect to an internal Microsoft Database during log truncation.

The Task.<vmname>.<refid>.log file for the VM in the job will contain:  
 

Error VssControl: TruncateSqlLogs failed (System.Exception)
Error at Veeam.Backup.VssProvider.CVssControl.TruncateSqlLogs(Guid jobId, CCredentials creds)
Error at Veeam.Backup.Core.CViGuestVssFreezer.TruncateSqlLogs(Boolean bSafe)
Error Failed to process 'TruncateSQLLog' command.
Error Failed to truncate transaction logs for SQL instances: MICROSOFT##WID. Possible reasons: lack of permissions, or transaction log corruption. (System.Runtime.InteropServices.COMException)
Error at VeeamGuestHelperCtrlLib.IVeeamVssProvider.TruncateSqlLogs(Guid jobUid, String sqlLogin, String sqlPwd)
Error at Veeam.Backup.VssProvider.CVssControl.TruncateSqlLogs(Guid jobId, CCredentials creds)

Solution

This issue is fixed in Veeam Backup & Replication v9.

If you are unable to upgrade to v9, please contact Veeam Support and reference Bug# 59525.

More Information

This issue only applies to 8.0.0.2084, and for the MICROSOFT##WID instance.
For any other truncation issues, please see the following KB: https://www.veeam.com/kb2027

Challenge

Solution

Restores

Restores from recent backups or backup copies will utilize the ExaGrid landing zone as a high-speed disk cache, eliminating the need to rehydrate the backup data. All Veeam vPower operations such as Instant VM Recovery and SureBackup will experience VM boot times of seconds to minutes.

Use the "Per-VM backup files" option on the Storage Compatibility Settings in Repository's "Repository" page (Advanced button). After switching to this option, do a manual active full before the next scheduled backup runs. Set the repository’s “Limit maximum concurrent tasks to:" 10 and work with your ExaGrid support engineer to further tune this setting.

Utilize the ExaGrid Veeam Accelerated Data Mover option whenever possible for best performance and fastest synthetic full and backup copy jobs – since with the Data Mover option, I/O is done directly on the ExaGrid backup storage appliances rather than having to pull Veeam backup data across a network into the proxy server and back out again to the ExaGrid.

You can convert a CIFS repository to use the ExaGrid Veeam Accelerated Data Mover – contact your ExaGrid Customer Support engineer for assistance.
 

Backup Jobs

• Use “Incremental” backup mode
• Check “Create synthetic full backups periodically” – recommend one day per week
• Check “Enable inline data deduplication” – the ExaGrid will further deduplicate the Veeam backups. See below for more about combined deduplication.
• Change the Compression Level to “Dedupe-friendly”
• Change Storage optimization to “Local”
• If you need to retain more than 14 recovery points, see the section below on how to use Backup Copy Jobs for longer retention

Backup Copy Jobs

When it comes to deciding the best value for number of backup recovery points to retain, keep in mind that the daily incremental backups are, by definition, largely just data that has changed in the VM blocks, and so consume a larger amount of backup storage space since deduplication is not able to significantly reduce the size on disk. ExaGrid standard sizing expects the retention of backup recovery points to be 14 or less. To retain backups for longer than 14 recovery points, utilize Veeam Backup Copy Jobs as follows:

• Consider creating another ExaGrid repository, dedicated for Backup Copy Jobs to ensure that you maintain an isolated copy of your production data to better follow 3-2-1 rule.
• Keep the number of restore points at the minimum of 2, since you will use the backup job’s restore points for your primary source of restore points. Set the copy job interval as long as possible given the GFS settings. So, if GFS settings include weekly retention, set the copy interval to 7 days.
• Create Veeam Backup Copy job(s) to make a copy of the backups you need to retain longer, using the 2nd repository. Enable the "Read the entire restore point from source backup instead of synthesizing it from increments" option on the "Target" tab of the Backup Copy Job.
• Set the retention parameters on the Backup Copy Job to match your needs – i.e. retain 16 weeks, 6 monthlies, 1 yearly, etc.

Using the ExaGrid Veeam Accelerated Data Mover type repository will significantly speed up Backup Copy Jobs, since the reads and write I/O to make the copy all run directly on the ExaGrid backup storage appliance, using the high-speed RAID 6 disks in the ExaGrid appliance rather than having to move all backup data back and forth across between the ExaGrid and the backup proxy across a network.

Contact your ExaGrid Customer Support engineer for assistance in setting up the Veeam Backup Copy jobs for longer retention.
 

Deduplication

The ExaGrid backup storage appliance will perform deduplication of the Veeam backups, which themselves have been deduplicated by Veeam (see backup job settings above). The ExaGrid web-based administrative interface will show you the ExaGrid-achieved deduplication, but not the Veeam deduplication. For example, if Veeam is deduplicating a backup job 2:1 (typical), and ExaGrid’s GUI show 5:1, the combined dedup is 10:1.

ExaGrid Customer Support has a PowerShell script that collects Veeam deduplication information and produces a combined deduplication result for each Veeam Backup Job; contact your support engineer to obtain the script and assistance using it. Please note that for Veeam Backup and Replication v9, you will need an updated script from ExaGrid Support.

More Information

The best practice guides can be download on the Exagrid’s appliance under [Help]->[Online Library] then expand the “Name VM Backup Applications”.

Challenge

Cause

Solution

More Information

Challenge

After installing Veeam Agent for Microsoft Windows and configuring a backup job, the settings are not shown in the Control Panel, and Veeam Agent for Microsoft Windows keeps prompting to configure backup settings.

Cause

The Windows OS was installed with non-English region and culture settings. As a result, these settings were applied to the Windows system account.

Solution

You will need to set Format to English (United States) for the system account.

Go to the Region settings (press Win + R > type intl.cpl > press Enter)

1. On the Formats tab
   1. set Format to English (United States)
2. On the Administrative tab
   1. click Copy Settings
      1. check that Format is set to English for Current user
      2. Enable the Welcome screen and system accounts checkbox
      3. Click OK to save settings and return to the Administrative tab.
3. Click OK to close the Region settings and save settings.
4. Restart your machine

Once the necessary changes are applied to the system account, a non-English Format can be set for the Current user account. Note, that the Format has to remain English (United States) for the system account.

More Information

Here is an example of a working configuration: