Veeam Support Knowledge Base

Challenge

Veeam Agent for Linux may cause an operating system crash on machines running Red Hat Enterprise Linux (RHEL) or its derivatives (CentOS, Oracle Linux) with Linux kernel version 2.6.32-754.6.3.el6 (update RHSA-2018:2846).

Cause

Veeamsnap, Agent for Linux kernel module (versions 1.x and 2.x) is not compatible with Linux kernel version 2.6.32-754.6.3.el6.

Solution

The issue is specific to kernel version 2.6.32-754.6.3.el6. For details please refer to the following article: https://access.redhat.com/solutions/3658111
This issue is addressed in the following kernel updates: Red Hat Enterprise Linux 6.10.z - kernel-2.6.32-754.9.1.el6 via RHBA-2018:3763.
Workaround: boot the system from kernels earlier than kernel-2.6.32-754.6.3.el6.

Challenge

You need to map an existing Veeam Agent backup chain to a Veeam Backup & Replication repository.

Cause

There are no features in Veeam Agent that would allow mapping existing backups to Veeam Backup & Replication, however it is possible to copy a local backup chain to a Veeam Backup & Replication repository and continue using it.

Solution

Here is the list of steps for mapping a local backup chain to a Veeam Backup & Replication repository.

1. Create a backup with Veeam Agent at any temporary backup destination of your choice.

If a Veeam Backup & Replication repository was used,

remove the existing Veeam Agent backup job from jobs (right click > Delete)
remove the backup files from configuration: right click Disk (backups) > Endpoint Backups > Remove from configuration.

2. Place the backup chain into the required folder.

2.1 Example for a non-cloud repository

Backup files reside in C:\VeeamBackup\Backup Job 1 and need to be moved to a Veeam Backup & Replication repository into the following folder: C:\Backup.

Under C:\Backup you will need to create a folder named after the account that is used to access the repository. In this example, the account is DOMAIN1\user1, so the folder name would be DOMAIN1_user1.

The complete path to the backup files would be C:\Backup\DOMAIN1_user1\Backup Job 1

2.2 Example for a cloud repository

The backup files reside in C:\VeeamBackup\Backup Job 1 and need to be moved to a cloud Veeam repository into following folder: C:\Backup.

Under C:\Backup you will need to create a folder named after the tenant account that is used to access the repository. If the account name is Tenant2, the complete path to the backup files would be C:\Backup\Tenant2\Backup Job 1

If a subtenant is used with the account name Subtenant3, the path will be C:\Backup\Tenant2\users\Subtenant3\Backup Job 1

3. Rescan the Veeam Backup & Replication repository. Here is a brief guide on repository rescan.
Note: For backups that you are mapping to cloud, rescan the repository hosting the tenant backups on the provider side.

4. The Veeam Agent backup should appear under Disk > Imported.
Note: When a backup chain is mapped to cloud, it will not be displayed under Disk > Imported.

5. Point the Veeam Agent job to the required Veeam Backup & Replication repository. Do not run it immediately, just close the wizard after making the changes.

6. On the Veeam Backup & Replication machine, the backup should be moved from the Imported section to Backup > Disk. Make sure that backups are located in Backups > Disks (not Disk (imported)).

7. Run the Veeam Agent backup job. It should continue the imported backup chain.

Note: The described solution works for Veeam Backup & Replication repository and Veeam Cloud Connect repository.

More Information

NOTES: 1. If local backup encryption was enabled during the initial job configuration, additional steps may be needed to complete the mapping.

For non-cloud repositories
Importing a backup chain to a Veeam Backup and Replication server puts the imported chain in the Disk (encrypted) tab. Before the backup chain can be continued, you will need to right click on the encrypted backup chain and enter the password.

For cloud repositories
In some cases it may be necessary to enter the password in order to continue the backup chain. If entering the password is required, you will be prompted to do so when the job starts.

2. If Backup & Replication repository’s encryption is enabled at the repository, you should decrypt it using password specification (Right click > Specify password) to finish the import procedure.

3. Unfortunately, it is not possible to seed a local Veeam Agent backup to a Cloud Scale-Out repository. You can either use a simple Cloud repository to seed the backup instead or use a SOBR repository without an initial seeding.

If you have any questions, open a case with Veeam Support.

Challenge

Product Information:

Product Family: Fujitsu ETERNUS CS
Status: Veeam Ready - Tape
Classification Description: Tape solution where available hardware features have been tested to work with Veeam.

Solution

Product Details:

Model number: CS8000 v7.0 VTL
Library Firmware Version V7.0A4010-SP02 Virtual juke box (VJUK) emulation
Drive firmware version: V7.0A4010-SP02 Virtual juke box (VJUK) emulation
Driver for tape drive: Microsoft IBM Ultrium-TD7 version 10.0.14393.1794
Driver for media changer: N/A
Media type tested: LTO7
General product family overview:
CS8000 is a virtual tape library with emulated tape drive at the front end connected to real library with real tape drive at the back end. The data are backup into virtual volumes on internal disks RAID’s. The virtual volumes are stored on physical volumes. https://sp.ts.fujitsu.com/dmsp/Publications/public/ds-eternus-cs8000-v7-ww-en.pdf

Veeam Details:

Veeam Build Number: 9.5.0.1922

More Information

Company Information:

Company name: Fujitsu Limited
Company overview: Fujitsu is the leading Japanese information and communication technology (ICT) company, offering a full range of technology products, solutions and services. Approximately 156,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers.

Challenge

Any interactions with either SharePoint Online or OneDrive for Business within Veeam Backup for Microsoft Office 365 fail with “For security reasons DTD is prohibited in this XML document”

Cause

To communicate with SharePoint online Veeam Backup for Microsoft Office 365 uses Microsoft CSOM library.
When the library receives an authorization request it, among other things, tries to resolve and reach msoid.onmicrosoft.com and msoid.<your-organization-domain>.onmicrosoft.com.
Most internet service providers cannot resolve those names.
CSOM library can ignore those sites being unreachable and keep working on your request, but only if it has received an HTTP error code (e.g. 404, 500, 503 etc).
However, some ISPs redirect unresolved DNS calls to their own DNS helper page which then returns status code 200 OK.
The library tries to authenticate through this "helper" page and fails with error “For security reasons DTD is prohibited in this XML document”.

Solution

While the issue is out of Veeam’s support scope, there are a few workarounds you can try.

Depending on your ISP you can request to have this DNS relay feature disabled.
Add 2 records to C:\Windows\System32\drivers\etc\hosts binding msoid domain with localhost IP 127.0.0.1
Add a CNAME record on your DNS server as seen below:

Challenge

Product Information:

Company name: Panzura, Inc.
Product Family: Freedom NAS
Status: Veeam Ready - Repository*
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

*This Veeam Ready test was performed with an SSD/Flash configuration. As such, this classification only applies to configurations in which all SSD drives are used. The use of non-SSD drives may impact performance.

Solution

Product Details:

Model number: 5500
Storage Category: Cloud-out NAS Storage
Drive quantity, size, type: 12 - 800GB, SAS SSD
Storage configuration: RAID50 via automatic software configuration
Firmware version: 7.0.2.5
Connection protocol and speed: SMB, 2 x 10GbE
Additional support: All models and configurations of Freedom NAS with specifications equivalent or greater than the above

General product family overview:

Panzura FreedomTM is an intelligent hybrid cloud storage solution underpinned by Panzura CloudFS. Panzura CloudFS is a scale-out, distributed file system purpose-built for the cloud. Programmatically accessible through REST APIs, it utilizes high performance flash in the data center, and integrates the cloud for economics, scalability, and durability. Unlike NAS, it scales globally across multiple sites and multiple clouds.

Veeam testing configuration:

Note: The following settings were used by the vendor to meet Veeam Ready testing requirements and should not be considered best practices. Additional changes or settings may be needed to meet the storage efficiency or performance needs for each environment. For each setting, reference links are provided for further clarification.

Veeam Build Number: 9.5.0.1536

Job Settings:

Deduplication:	Enabled (Default)
Compression:	Optimal (Default)
Storage Optimization:	Local Target (Default)

Repository Settings:

Repository Type:	Shared Folder
Align backup file blocks:	Enabled
Decompress before storing:	Disabled (Default)
Per-VM Backup Files:	Enabled

Vendor recommended configuration:

Hardware Settings: The default configuration enables auto-tuning for all Panzura settings, which work well with the Veeam application. The caching policy should be set to pinned (as opposed to autotune) to ensure the administrator controls which backup sets are available locally. All Veeam backup data must be pinned locally to ensure fast restores and effective data operations within Veeam, without cloud reads.

Challenge

Product Information:

Company name: DataCore
Product Family: SANSymphony
Status: Veeam Ready - Repository*
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: SANSymphony
Drive quantity, size, type: 4 - 450GB, SSD
Firmware version: 10.0
Connection protocol and speed: iSCSI - 10GbE

General product family overview:

SANsymphony
https://www.datacore.com/products/datacore-sansymphony
DataCore™ Hyper-converged Virtual SAN
https://www.datacore.com/products/datacore-hyper-converged-virtual-san

Both products (above) are share a common code base, implementation use case and license keys differentiate products. For this test, SANsymphony and Hyperconverged Virtual SAN were configured using a 10GB iSCSI SAN connection. However, all popular line speeds found in typical iSCSI SAN environments are available (e.g. 25, 40, 100 Gb/s). In addition, the SANsymphony family of products is also compatible with the Fibre Channel SANs as deployed in customer environments at all popular speeds (4, 8, 16, 32 Gb/s).

See the DataCore technical support site for All Qualification and Host Configuration Guides (Previous hyper link references https://datacore.custhelp.com/app/answers/detail/a_id/838) to check DataCore Software's minimum pre-requisites in the General category and the Qualified hardware components in the DataCore Server category for more information.

Veeam testing configuration:

Veeam Build Number: 9.5.0.1038

Job Settings:

Deduplication:	Enabled (Default)
Compression:	Optimal (Default)
Storage Optimization:	Local Target (Default)

Repository Settings:

Repository Type:	Windows
Align backup file blocks:	Enabled
Decompress before storing:	Disabled (Default)
Per-VM Backup Files:	Enabled

Challenge

The following message has been found to occur on both backup and volume restore from Veeam Agent for Windows:

The media is write protected. Failed to write data to file

Cause

During Backup:

This issue can appear when backing up a system with OneDrive automatic folder sync in Windows 10.
There are Operating System limitations preventing VSS backups of OneDrive folders using Files On-Demand among other possible issues.

During Restore:

This issue can appear if the “Choose How Much Of Your Drive To Encrypt” BitLocker setting was changed and saved (e.g. changed Encrypt Used Disk Space Only to Encrypt Entire Drive or vice versa).

Solution

The following workarounds are available.

For issues on Backup:

Make sure OneDrive “Files On-Demand” functionality is disabled in settings.
Check that the OneDrive client software is up-to-date.
If issues continue, try volume level backup, or try switching to file level backup and exclude the OneDrive sync folder
Other workarounds are to try to re-establish the OneDrive connection:
1. Disconnect your local OneDrive from the cloud sync (click on the system tray right below your screen).
2. Click on the three dots right at the top of the window that appears and close the application.
3. Reconnect with OneDrive, but now you re-establish the connection (login).
Or try to re-sync the OneDrive data (these steps only need taken on affected folders):

Make a copy the affected folders to another location on your HDD which is not synchronized with OneDrive.
Remove affected files and folders from OneDrive local folder (and, if they are not removed automatically - from the online interface of OneDrive).
Copy your files back, let OneDrive sync the files (keep "Files On-Demand" disabled).
Re-try Veeam Agent backup.

There are other workarounds available as well. If the above does not resolve the issue, please see “More Information” to open a case with support.

For issues on Restore:

Disable BitLocker on the restore target drive:
1. In File Explorer, right-click the drive in question and then click Manage BitLocker.
2. On the BitLocker Drive Encryption page, click Turn Off BitLocker, and then click Turn off BitLocker again when prompted.
3. When decryption is finished, click Close.

Format the restore target drive:
1. Right-click the drive in File Explorer.
2. Choose Format.
3. Click Start.

More Information

If you have further questions or issues continue, please open a support case as follows:

Right-click the Veeam Endpoint Backup / Veeam Agent for Windows icon in the system tray and select Control Panel.
Click the Support link at the top of the window.
Click Technical Support to submit a support case to the Veeam Support Team directly from the product.

Challenge

The vCenter database in most cases will be automatically excluded from the Application-Aware Image Processing process, preventing the database from being frozen or quiesced. If this does not happen automatically backing up the VM where the vCenter database is stored may fail with the following error.

"VSSControl: Failed to freeze guest, wait timeout"

More information regarding this error can be found in http://www.veeam.com/KB1377

Solution

The solution below will instruct you on how to enable and use the DBExclusion menu that is built-in to Veeam Backup & Replication v7 and greater.

Close the Veeam Backup & Replication console.
Open Regedit.exe
Navigate to HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
Add a new DWORD ‘EnableDBExclusions’
5. Set the Value to 1
Close the Registry Editor and open Veeam Backup & Replication
Open “Database Exclusions” from the Main Menu
In the window that pops up, click Add.
Specify the DNS name (case sensitive) or IP address, and Instance Name (case sensitive) that you wish to exclude from VSS Freezing and being Quiesced.
Specify the Database name (case sensitive) that you wish to exclude from application-aware image processing.

Allow the writer service account access to the Volume Shadow Copy service via the registry:

On the VM you are backing up, open regedit.exe and navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl
If a DWORD value for the account assigned to the writer service is present in this key, it must be set to 1. In most cases, the relevant service account appears in services.msc as the log-on account for the service listed in KB2041. However, some writers have special accounts. For example, in MS SQL Server 2012 and later, if the DWORD value “NT SERVICE\SQLWriter” is present in the VssAccessControl key, it must be set to 1.

On next job run the specified database will be excluded from the Application-Aware Image Processing freeze task.

Note: If you are using a default or unnamed SQL instance (example: MSSQLSERVER) then only the server name (or IP address) should be added in the "Add Database" screen.

Challenge

When opening a support ticket, Veeam Support might request log files.

Cause

These log files might potentially contain information which might fall under the regulation of data protection laws or internal company policies.

Solution

The below are the types of sensitive information that can be found in the Veeam log files:

Usernames;
Customer specific infrastructure object names: hosts, datastores, VMs, clusters;
VM file names and paths;
FQDN/Hostname or NetBIOS names;
IP addresses: IPv4 and IPv6;
Customer specific paths to backup files;
Backup file names;
SharePoint information: DB names, paths to DBs, SP URL;
Exchange information: DB names, paths to DBs;
SQL information: DB and Instance Name, paths to DBs;
Oracle information: DB name, Home Name, Oracle SID, DB global name, paths to DBs;
File Copy: Customer specific paths and names of files;
File to tape: Customer specific paths and names of files.

More Information

For more information on how we process the collected data, please visit: https://vee.am/processing_of_personal_data

Challenge

If you are a Cloud Provider gathering logs from your infrastructure please see: https://www.veeam.com/kb2096

This article covers how to properly compile your logs in Veeam Backup & Replication v8.x and v9.x, as well as what to include when submitting a Support case for technical assistance.

Solution

The following animation demonstrates how to compile logs in Veeam Backup & Replication.

User-added image

To compile your logs via the log compilation wizard, click the Menu button in the top left-hand corner of the gui and go to Help -> Support Information in the Veeam Backup & Replication console.
This wizard will allow you various methods to compile logging. For Backup, Replication, and other jobs, select “export logs for this job.” If multiple jobs are affected, you can multiple-select. For problems with restore or replica failover, you should select “Export logs for this VM” and choose the production VM that was backed up or replicated. For issues with the GUI or problems that don’t fall into any of the above categories, select “Export all logs for selected components” and choose “This server.”
If possible, compile the logging for each job with issues for at least the last week.
You may save these logs to any location. Ensure that the location referenced has free space for compilation.
When selecting "Next", the wizard will notify of its progress in the compilation process. Please attach these to your case, or in the event that the total attachment size is under 3.5MB, you may attach via email reply. If none of these options for upload are working, please notify Veeam Support and an engineer will provide FTP credentials for the case.

Below is an animation demonstrating how to upload logs to the Veeam Support FTP.

Note: Please use the FTP account provided to you by the engineer for your case. When submitting logging data for analysis by request of an engineer, ensure that the case is replied to when the log submission has completed. In this reply, please include the names of any tasks and associated VMs with issues.

User-added image

Note: SFTP is now supported when uploading to supportftp2, please place files in the \upload\ folder.

More Information

Below are the relative log path lists for Veeam Backup & Replication. If there are difficulties compiling the logging requested via the wizard-based instructions above, please obtain the files from the appropriate directory below and provide Veeam Support a .zip file copy.

Windows 2003, XP – C:\Documents and Settings\All Users\Application Data\Veeam\Backup\
Windows Vista, 7 - C:\Users\All Users\Veeam\Backup\
Windows 2008/2008 R2/2012 - C:\ProgramData\Veeam\Backup\
Linux - /var/log/VeeamBackup/

Typical log files requested for a manual log compilation will be:

The task's folder located in the parent directory, zipped. This will contain all relevant job/task/agent data.
All logs from the parent directory named svc.*.log, util.*.log, as well as the VeeamBackupManager.log and VeeamShell.log

Please note that information provided to Veeam for purposes of customer support may be considered protected under certain data protection regulations and/or internal company policies. A list of potentially sensitive data types is available in the following KB article. Veeam will only utilize this information for the specific purpose of providing customer support to you and will protect the information in accordance with its privacy policy. In any event, the information provided will be securely deleted after 45 inactive days as of the last upload to the FTP. If you would like it deleted sooner, please express this request directly to your support contact once the issue has been resolved. For more information on how we process the collected data, please visit: https://vee.am/processing_of_personal_data

Challenge

When a computer is being added to protection group, Veeam Agent for Microsoft Windows deployment starts and fails with the following:

Info [UploadManager] Checking windows package 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' signature on host 'HOSTNAME'
Error Failed to call RPC function 'PckgCheckSignature': Signature of module 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' is invalid..
Error Signature of module 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' is invalid Error --tr:Failed to call DoRpc. CmdName: [PckgCheckSignature].

Cause

The certificate which is used to sign the Veeam Agent installation package is not installed in the Trusted Root Certification Authority store on the client computer.

Solution

For SHA256

Download DigiCert Assured ID Root CA here (right click on Download button - save as)
Double click the downloaded .srt file
Click Install Certificate
Choose Local Machine and click on Next
Place the certificate into Trusted Root Certification Authorities by clicking the Browse button.

For SHA1

Download R1 GlobalSign Root Certificate (Thumbprint B1BC968BD4F49D622AA89A81F2150152A41D829C) from here
Double click the downloaded .srt file
Click Install Certificate
Choose Local Machine and click on Next
Place the certificate into Trusted Root Certification Authorities by clicking the Browse button.
Download Code Signing SHA-1 G3 Intermediate Certificate (Thumbprint F1E7B6C0C10DA9436ECC04FF5FC3B6916B46CF4C) from here
Double click the downloaded .srt file
Click Install Certificate
Choose Local Machine and click on Next
Place the certificate into Intermediate Certification Authorities by clicking the Browse button.

Challenge

Veeam ONE raises Events data collection failure against Veeam Backup and Replication OR Hyper-V server.

Cause

Permissions, UAC configuration and firewall settings need to be configured.

Solution

Verify that the account used to connect to Veeam Backup and Replication OR Hyper-V server in Veeam ONE has correct permissions. To check the account name, go to Veeam ONE Monitor > Data Protection View > right-click on the Veeam Backup and Replication object. Note the specified account. You can check the same for the Hyper-V server: Veeam ONE Monitor > Infrastructure tree > right-click on the Hyper-V object.
On the Veeam Backup and Replication OR Hyper-V server, confirm that this account as well as the Veeam ONE service account are added to the Event Log Readers Group and Administrators group in Local Systems and Groups
Even if the Firewall is off, activate the following firewall rules on the Veeam Backup and Replication OR Hyper-V server side: Remote Event Log Management and COM+ Network Access (DCOM-In)
Check UAC settings.

If Veeam Backup&Replication/Hyper-V and Veeam ONE are installed on the same server, try one of the following solutions in addition to the steps above:

Change the type of the all Public network connections to Private using the following Powershell cmdlet:

Set-NetConnectionProfile -NetworkCategory Private

Or using the registry editor (machine restart will be required):

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\Profiles\
0 = public
1 = private
2 = domain

2. If changing the network type does not help, put Built-in Administrator account credentials for the Veeam Backup&Replication/Hyper-V server in the Connection Settings using Veeam ONE Monitor client.

More Information

If the issue persists, contact Veeam Support for further assistance.

Challenge

When opening a support ticket, Veeam Support might request log files.

Cause

These log files might potentially contain information which might fall under the regulation of data protection laws or internal company policies.

Solution

The below are the types of sensitive information that can be found in the Veeam log files:

Usernames;
Customer specific infrastructure object names: hosts, datastores, VMs, clusters;
VM file names and paths;
FQDN/Hostname or NetBIOS names;
IP addresses: IPv4 and IPv6;
Customer specific paths to backup files;
Backup file names;
SharePoint information: DB names, paths to DBs, SP URL;
Exchange information: DB names, paths to DBs;
SQL information: DB and Instance Name, paths to DBs;
Oracle information: DB name, Home Name, Oracle SID, DB global name, paths to DBs;
File Copy: Customer specific paths and names of files;
File to tape: Customer specific paths and names of files.

More Information

For more information on how we process the collected data, please visit: https://vee.am/processing_of_personal_data

Challenge

Solution

The following animation demonstrates how to compile logs in Veeam Backup & Replication.

User-added image

To compile your logs via the log compilation wizard, click the Menu button in the top left-hand corner of the gui and go to Help -> Support Information in the Veeam Backup & Replication console.
This wizard will allow you various methods to compile logging. For Backup, Replication, and other jobs, select “export logs for this job.” If multiple jobs are affected, you can multiple-select. For problems with restore or replica failover, you should select “Export logs for this VM” and choose the production VM that was backed up or replicated. For issues with the GUI or problems that don’t fall into any of the above categories, select “Export all logs for selected components” and choose “This server.”
If possible, compile the logging for each job with issues for at least the last week.
You may save these logs to any location. Ensure that the location referenced has free space for compilation.
When selecting "Next", the wizard will notify of its progress in the compilation process. Please attach these to your case, or in the event that the total attachment size is under 3.5MB, you may attach via email reply. If none of these options for upload are working, please notify Veeam Support and an engineer will provide FTP credentials for the case.

More Information

Windows 2003, XP – C:\Documents and Settings\All Users\Application Data\Veeam\Backup\
Windows Vista, 7 - C:\Users\All Users\Veeam\Backup\
Windows 2008/2008 R2/2012 - C:\ProgramData\Veeam\Backup\
Linux - /var/log/VeeamBackup/

Typical log files requested for a manual log compilation will be:

The task's folder located in the parent directory, zipped. This will contain all relevant job/task/agent data.
All logs from the parent directory named svc.*.log, util.*.log, as well as the VeeamBackupManager.log and VeeamShell.log

Challenge

Job will fail with the error Failed to call RPC function 'Vss.FinishSnapshot': Error code: 0x800401fd

In the guest helper logs you will see the below occur before Veeam sends the VSSControl: FinishSnapshot started command in the task log.

18/10/2018 22:39:41 2688 INFO RPC: KeepSnapshotEx. Ok.
18/10/2018 23:00:00 21004 WARN Snapshot timeout occurred.

Cause

Vss snapshot is timing out before Veeam is able to send the finish command.

Solution

Add the below key to the registry location “HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\” on the Veeam server. After applying the key make sure no jobs are running and restart the Veeam backup service.

The default value is 1200 so attempt doubling that value when creating the key.

VSSGuestSnapshotTimeout
Type: REG_DWORD
Default value: 1200 (in seconds)
Description: Found in job log as "VSSControl: KeepSnapshot started, ttl 1200 sec". Registry setting was added in v9 update 1. Prior to that, this timeout was hard-coded to 600 seconds. This registry key goes to the Veeam Server

Challenge

This article describes making manual firewall changes for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing.

IMPORTANT: With Cisco HyperFlex 3.0 the needed Firewall changes have been implemented in the OS image and make sure the following is not restricted on the ESXi hosts security profile: “vSphere Web Client > Hosts and Clusters > Select Host > Configuration > Security Profile > Edit > NFS access (please don’t confuse with NFS Client)”. Please follow the KB below only if you are running a HyperFlex version below 3.0. For new customers, we recommend to install the HyperFlex cluster with HX 3.0 and for existing customers we recommend to upgrade to HX 3.0 to benefit from the new Firewall changes.

Solution

To achieve the optimal balancing within the Cisco HyperFlex data network at Backup from Storage Snapshot processing, it is needed to change the ESXi host firewall. See more background information here.

One of the methods to change the ESXi host firewall is by manual creation of an ESXi host firewall rule. This configuration is reset by an ESXi host reboot and can be used for test environments.

To open ports on ESX(i) hosts for Cisco HX < 2.5, add the following firewall rule to the services.xml file on an ESX(i) host.

<ConfigRoot>
   <service id='9230'>
     <id>VeeamCiscoFirewall</id>
     <rule id='0000'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>
         <begin>0</begin>
         <end>65535</end>
       </port>
     </rule>
</service>
</ConfigRoot>

To open ports on ESX(i) hosts for Cisco HX >= 2.5, add the following firewall rule to the services.xml file on an ESX(i) host.

<ConfigRoot>
   <service id='9230'>
       <id>VeeamCiscoHXFirewall</id>
       <rule id='0000'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>111</port>
       </rule>
       <rule id='0001'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>2049</port>
       </rule>
       <rule id='0002'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>2449</port>
       </rule>
       <enabled>true</enabled>
       <required>false</required>
       </service>
</ConfigRoot>

The following example shows all steps required to open the firewall at an ESXi host SSH connection:
1. Back up the current services.xml file by running the command:

cp /etc/vmware/firewall/service.xml /etc/vmware/firewall/service.xml.bak

2. Modify the access permissions of the service.xml file to allow writes by running the chmod command:

chmod 644 /etc/vmware/firewall/service.xml

chmod +t /etc/vmware/firewall/service.xml

3. Open the service.xml file in a text editor:

vi /etc/vmware/firewall/service.xml

4. Add the rule to the service.xml file (see example above)
5. Revert the access permissions of the service.xml file to the read-only default by running the command:

chmod 444 /etc/vmware/firewall/service.xml

6. Refresh the firewall rules for the changes to take effect by running the command:

esxcli network firewall refresh

7. Enable the new firewall rule:

esxcli network firewall ruleset set -r "VeeamCiscoFirewall" -e true -a false

8. Bind the firewall rule to all Veeam proxy server data network IPs. This is the IP on the HyperFlex “Storage Controller Data Network”. Repeat the command for each proxy server:

esxcli network firewall ruleset allowedip add -r "VeeamCiscoFirewall" -i "<yourVeeamProxyIP>"

9. Check the IP binding

esxcli network firewall ruleset allowedip list | grep -v "All"

10. Check if the firewall rule is enabled

esxcli network firewall ruleset list

More Information

For more information about custom firewall rule creation, click here.

Challenge

After the upgrade to Veeam MP 8.0 Update 5 (8.0.0.2451) or Veeam MP 8.0 Update 6 (8.0.0.2467), Veeam Backup&Replication related reports in Veeam Management Pack result in displaying incorrect data or “No relevant data found” message.

For example, all or part of VMs included in Protected VMs report, shown as Unprotected.

Cause

In some cases, one of the database tables related to Veeam Backup monitoring is not upgraded properly during the applying Veeam Management Pack 8.0 Update 5 or Veeam Management Pack 8.0 Update 6.

Solution

In case Veeam Management Pack 8.0 Update 5 (8.0.0.2451) is currently installed in your environment, upgrade to Veeam Management Pack 8.0 Update 6 (8.0.0.2467) first following this KB article, then proceed with troubleshooting.

In case Veeam Management Pack 8.0 Update 6 (8.0.0.2467) is already installed in your environment:

1) Download the archive file with the hotfix.
2) Unzip the archive.
3) Import Veeam Backup Monitoring management pack via SCOM > Administration > Management pack > Import Management Packs.
4) Ensure that Veeam Backup Monitoring management pack imported successfully and its build number changed to 8.0.0.2473 in SCOM view.

Please note that this fix is compatible with Veeam Management Pack 8.0 Update 6 (8.0.0.2467) only.
Please note that after applying the hotfix you should wait until daily backup jobs finish +1 hour, the new data will be inserted into the database and report will display correct results.

More Information

[[DOWNLOAD|DOWNLOAD THE HOTFIX|https://www.veeam.com/download_add_packs/vmware-microsoft-esx-monitoring/kb2826/]]

MD5 checksum for KB2826.zip is 20877D282BEAFA52E29C3C950F44352D

Should have any questions, contact Veeam Support.

Challenge

Any jobs configured to backup groups fail with “A parameter cannot be found that matches parameter name 'GroupMailbox’”

User-added image

Cause

To be able to backup group mailboxes with Veeam Backup for Office 365 version 2.0 a new permission is required. So, the job fails because the account used to add the organization has insufficient privileges to access the group mailbox. This applies to any type of job so long as the group mailbox is enabled for backup.

Solution

Add either Mailbox Search or Mail Recipients permission on the account used to add your organization as shown below:

User-added image

Challenge

This KB provides information on items to check when logs are not being truncated, even when Veeam Backup & Replication is stating that everything is successful.

Cause

There are various causes for failure of truncation operations, both in the Veeam Backup & Replication application, and within the Exchange Windows guest itself.

Solution

Make sure that the job is set to use Application-Aware Image Processing and proper credentials are being used:

User-added image

You will then want to check to make sure that transaction logs are set to be truncated on successful backup:

User-added image

If this is set correctly, make sure that the command is being sent to the Exchange server. You can verify this by analyzing the VeeamVSSSupport.log (VeeamGuestHelper.log in VBR v8+) file that can be found in the following location:

Windows 7/8 Generations: C:\ProgramData\Veeam\Backup
Windows XP/2k3: %allusersprofile%\Application Data\Veeam\Backup

If the command is being sent, you should see the following in the log file:
INFO                Processing external FinishSnapshot command.
                         Notifying writers about the 'BACKUP FINISH' event.
                         Truncate logs: [true].
                         Processing external FinishSnapshotEX command, job id ‘{job ID}’, is logs truncation enabled yes.. ok.

If this is being sent, we will want to look at Windows event logs for any possible errors as well as the VSS writers.

Run the following in an elevated CMD prompt and verify the Exchange writer is present:

vssadmin list writers

If the writer is not being listed or is in a failed state, the Exchange server should be rebooted, then it should be verified that the writer returns and shows as Stable/No Error. Once this has been done, retry the Veeam job and verify log truncation.

Within the event logs, look for any Exchange based errors. If the Exchange configuration is utilizing Distributed Availability Groups (DAG), the replication process between the different nodes can prevent log truncation if there are any errors. These will show with the event source MSExchangeRepl. If there are errors here, generally these are not affected by Veeam Backup & Replication processes and should be addressed in the Exchange and/or Windows configuration. To isolate whether the issue is caused within Veeam Backup & Replication processes or within VSS itself, enabling the Windows Backup role on the server and running a native Windows Backup (system state) will generally isolate the issue to the operating system.

More Information

To manually test shadow creation and truncation please review KB1980: Using the Diskshadow Utility to manually test VSS operations.

Information on Application-Aware Processing and Log Truncation:

vSphere:
Application-Aware Processing
Log Truncation

Hyper-V:
Application-Aware Processing
Log Truncation

MSExchangeRepl errors:
http://technet.microsoft.com/en-us/library/hh994877(v=exchg.141).aspx

Challenge

The logging level needs to be changed.

Solution

Logging detail level is changed by altering the value data of the DWORD ‘LoggingLevel’ on the Veeam server in the registry

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
Value: LoggingLevel
Type: DWORD (32-bit) Value (REG_DWORD)
Data: 4

This value can be set from 1 to 6.
Logging level:

Logging level: [1 (LowDetailed)]
Logging level: [2 (BelowNormal)]
Logging level: [3 (Normal)]
Logging level: [4 (AboveNormal)]*
Logging level: [5 (HighDetailed)]
Logging level: [6 (UltimateDetailed)]

*The value of 4 for the LoggingLevel DWORD is the default setting.

Version 8.x and earlier: After changing the value, the Veeam Backup Service must be restarted. Make sure no jobs or restores are running before restarting this service.
Version 9.x: The Veeam Backup Service periodically rescans the registry. The logging level will adjust automatically after no more than 15 minutes.