Challenge

Product Information:

Company name: Seagate Technology
Product Family: Exos X Series
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: Exos X 2U12 Enclosure with Exos X 3005 iSCSI RAID Controllers
Storage Category: RAID Storage Array
Drive quantity, size, type: 12 – 12TB SAS
Storage configuration: RAID6, 8+2 drives and 2 dedicated spares.
Firmware version: GT265R009-03
Connection protocol and speed: ISCSI 10 GbE
Additional support: All models and configurations of Exos X Series with specifications equivalent or greater than the above

Seagate produces disk drives, disk enclosures and disk controllers. The enclosures differ according to the quantity and size of disk drive that they take, while controllers range from entry-level to high performance. The RAID controllers and enclosures can be used in almost any combination. The smallest enclosure option, containing the fewest number of drives and the entry-level controller, was used for the testing.

Veeam testing configuration:

Veeam Build Number: 10.0.0.4461 P2

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• Default storage settings used, other than:
  • RAID type to RAID 6 (8+2)
  • chunk size to 64KB

Challenge

Cause

A license file installed to your Veeam software replaces the information from the previous license of that type. A merged license file allows for increased socket or instance capacity by combining licenses from multiple Support IDs.

Please note that the merged license key will use the earliest support expiration date of all product keys.

Solution

NOTE! Licenses can be merged only when they have the same:

• Account Name
• Product Name and Edition
• Product Version
• Licensing Terms
• License Administrator
• They must all be Active on Support.

Veeam Backup & Replication v10

Version 10 of Veeam Backup & Replication is designed to accept two separate license files; one for sockets and one for instances. If you have multiple license keys for either workload, you will need to merge them together before installation.

Your License Administrator may obtain a merged license file by following the steps below.

1. Log on to the Veeam My Account dashboard and go to License management >> Production Licenses (https://my.veeam.com/#/licenses/production)
2. If you are managing multiple accounts, please select the account you are performing the merge for.
   
   Once you have made the selection all the products under the account will be visible.
   
3. To start the merge process, please use the License merge link:
   
4. Now you can select the Products and the SupportID’s you want to merge. You can select several compatible product licenses to merge into one license file.
   
5. After you have selected the SupportID’s you want to merge; you will have to click the Preview button to view the new licenses information. You will need to select which version of Veeam the requested merged file is for.
   
6. After you have verified the information’s of the page you will have to use the “Merge Licenses” button to finalize the merge process. You will then be presented with an option to download the license or send a copy to the license administrator’s email.
   

Veeam Backup & Replication 9.5 Update 4b

Veeam Backup & Replication 9.5 U4\U4a\U4b (Build numbers: 9.5.4.x) is designed to accept only a single license key. If you have multiple license keys or keys for different workloads (sockets and instances); they must be merged into a single license file via the Customer Portal prior to installation. The action of installing a license file will replace the previously installed license file.

More Information

• To revert the License Key merge operation, Customers may choose “Cancel merge” to destroy the merged license and revert to original product License Keys.
   
• License Key split option on the Customer portal is available to select Customers only. Splitting licenses within one SupportID requires opening a licensing support case.
  https://my.veeam.com/#/open-case/step-1?type=1
   
• Please note that the license auto-update feature does NOT correctly update merged licenses created on the license management portal. If you have enabled auto-update, disable it before installing a merged license from the license management portal.
   
• More info about the merge policy here. 

Challenge

Cause

Last-minute kernel patches for Red Hat Enterprise Linux and its derivatives may cause Veeam Agent for Linux to go unstable (veeam.com/kb2786).

Solution

Veeam Agent for Linux v4.0.1:

Veeam Agent for Linux v4:

Veeam Agent for Linux v3:

Challenge

As this is an error with VSS and Veeam's Guest Processing technique for Domain Controllers it is relevant to all backup jobs for both virtual and physical Domain Controllers

Veeam Backup & Replication job using Application-Aware Processing to process a Domain Controller fails with one the following errors:

Unable to release guest. Details: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [NTDS]. Class ID: [{b2014c9e-8711-4c5c-a5a9-3cf384484757}]. Instance ID: [{66fddc15-0e4c-4a2a-ad31-32eaf6dae8a3}]. Writer\'s state: [VSS_WS_FAILED_AT_POST_SNAPSHOT]. Error code: [0x800423f4].]

Error: VSSControl: 0 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
Cannot prepare the [NTDS] data to a subsequent restore operation.
Cannot process NTDS data.
Cannot create a backup copy of the BCD.

Error: VSSControl: -1 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
Cannot prepare the [NTDS] data to a subsequent restore operation.
Cannot process NTDS data.
Cannot create a backup copy of the BCD. Cannot get [BcdStore] object. COM error: Code: 0xffffffff

Solution

The actions listed in this section are to be performed within the Guest OS of the DC that is having these issues.

The cause is most likely one of the following. If a listed troubleshooting step requires attention, test the Veeam job after performing that troubleshooting task, before proceeding on to further troubleshooting steps.

Verify that the NTDS VSS writer is stable.

From an elevated command prompt run the following command:

vssadmin list writers

The results will appear as:

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {ee24b741-eaf7-4663-8f95-b92ae8c5e164}
State: [1] Stable
Last error: No error

If the NTDS writer is not listed as "State: [1]Stable", reboot the DC (Domain Controller).

Note: If the NTDS writer does not appear in the list, it is advisable to contact Microsoft support for their assistance in investigating why the writer is not present.

Verify that Automatic mounting of new volumes enabled.

From a Run prompt (Win+R) run the program 'diskpart'.

From within DiskPart run the following command.

automount

If the results do not show “Automatic mounting of new volumes enabled.” Run the following command:

automount enable

Verify that there are no .bak keys in the ProfileList within the Registry.

Open the Registry Editor (regedit.exe)

Within the registry navigate to:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList

There will be a list of Keys, you must remove any ending in .bak

More Information

The Majority of cases are resolved uses the above 3 troubleshooting steps. However in rare cases users have reported having to rebuild the WMI repository.

Challenge

Cause

Solution

• NAS Backup functionality can be used to gain unprivileged access to files on managed servers.
• Weak credentials encryption issue in guest processing command line parameter, and in a PowerShell cmdlet logs.
• Server and IIS version information is included the response header of the Veeam Backup Enterprise Manager web app.
• Security of Agent Management architecture has been improved.
• Security of Cloud Connect architecture has been improved.
• libcurl version has been updated to 7.70

• A custom security descriptor was provided for the driver's control device (vulnerability reported by Mile Karry).
• Deserialization issues were fixed (vulnerability reported by Harrison Neal).
• A user authorization issue was fixed (vulnerability reported by Harrison Neal).
• Security of the Enterprise Manager Web App configuration was improved.
• Support for weak SSH ciphers was disabled in the recovery media ISOs.
• OpenSSL library was updated to version 1.0.2t
• Putty was updated to version 0.73
• Weak TLS ciphers were disabled in Surebackup Linux appliance.

More Information

Challenge

Cause

Solution

• Sensitive information used by managed Linux agent may get logged in the Linux operating system logs.
• Creating an SMB repository using CLI command causes plain text password to be logged in the Veeam debug log.

• An issue of insecure file permissions was addressed (vulnerability reported by RACK911 Labs).
• OpenSSL was updated to version 1.0.2t.

More Information

Challenge

Cause

Solution

• No security related changes.

• A custom security descriptor was provided for the driver's control device (vulnerability reported by Mile Karry).
• Deserialization issues were fixed (vulnerability reported by Harrison Neal).
• A user authorization issue was fixed (vulnerability reported by Harrison Neal).
• OpenSSL was updated to version 1.0.2t

More Information

Challenge

Cause

Solution

1. Make sure no jobs/tasks are running and stop all Veeam Backup services on the Veeam server
2. Navigate to Veeam installation directory (by default C:\Program files\Veeam\Backup and Replication\Backup)
3. Make a backup of Veeam.Backup.SanPlugin.IbmLib.dll and Veeam.Backup.PublicStoragePlugin.Ibm.dll files
4. Replace Veeam.Backup.SanPlugin.IbmLib.dll and Veeam.Backup.PublicStoragePlugin.Ibm.dll with those from hotfix package
5. Start all Veeam Backup services

More Information

Challenge

Cause

Solution

Challenge

Solution

1. Create the following registry keys:

   Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
   Type: DWORD
   Name: AzureManagedDisksCheckMaxDiskSize
   Data: 0 (zero) 
   
   Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication 
   Type: String 
   Name: AzureManagedDisksForcedAPI  
   Data: 2017-03-30

2. Restart the Veeam Backup Service. 

Challenge

Cause

Solution

Challenge

Cause

Solution

1. Navigate to \Addons\Extension Modules on the Veeam ONE ISO.
2. Copy all the files from the Extension Modules folder to C:\Program Files\Veeam\Veeam ONE\Veeam ONE Reporter Server.
3. Restart Veeam ONE Reporter server service.
4. Make sure that all the files you copied to the Veeam ONE installation folder have been automatically removed after the service restart.
5. Check whether the reports you need are available.
6. Install the latest Update for your Veeam ONE version. If the update has already been installed, repeat the installation.

More Information

Challenge

Product Information:

Company name: MinIO
Product Family: MinIO
Status: Veeam Ready – Object with Immutability
Classification Description: Verified object storage solutions that has been tested with Veeam Backup & Replication Cloud Tier object storage immutability features.

Solution

Product Details:

Model number: RELEASE.2020-07-12T19-14-17Z *
Storage Category: S3 Compatible Object Storage
Drive quantity, size, type: 8 – 1.8TB HDD
Storage configuration: JBOD with default n/2 erasure coding enabled
Connection protocol and speed: 2 x 10Gbps links (LACP)
Additional support: All models and configurations of MinIO for S3 compatible storage with specifications equivalent or greater than the above. *Please use MinIO RELEASE.2020-08-18T19-41-00Z or higher.

General product family overview:
MinIO is a high-performance, software defined, S3 compatible object store. MinIO was purpose-built to serve only objects and its single-layer architecture can run in user space and is easily containerized and can be orchestrated using Kubernetes. The software is scalable and offers resilience through inline erasure coding and bitrot protection. The software also contains a number of important security features including encryption for data in flight and at rest, object locking, auditing and identity management.

Veeam testing configuration:

Veeam Build Number: 10.0.0.4461

Challenge

Cause

What’s New:

• Alarm for detecting unsupported Veeam Backup & Replication server configurations.

Enhancements:

Usage Reporting

• Usage reporting data for Veeam Backup & Replication servers v9.5 U4 located in time zones with more than 8 hours difference is now included in the report with the auto-approve function. 

Backup reporting

• Reports generation performance has been improved. 

Monitoring

• “Management agent connection” alarm is not triggered when a Cloud Connect server is rebooted. 

Management Agent

• Authentication between management agents and Veeam Service Provider Console has been improved. 

RESTful APIs v3

• Date format in "/licensing/reports/" GET request has been changed to “yyyy-MM”.  
• Report generation parameters have been simplified for licensing/reports/ methods: report interval is used instead of report generation date.

 
Resolved issues:

• "Protected computers" and "Protected VMs" widgets in the Overview tab do not account for the default RPO period of 30 days.  
• Filtering by company/reseller name does not work on the Failover Plans tab. 
• When using “Select All” checkbox on the Backup Jobs -> Computers tab, job actions are applied to all managed Agents regardless of selected filters. 
• Under certain conditions, the "Unrecognized Guid format" error message is displayed on the Companies tab.
• Under certain conditions, “Object reference not set to an instance of an object” error message is displayed on the Backup Jobs tab. 

• Under certain conditions, the Veeam Service Provider Console service fails to start. 

• If a company is removed from the configuration, error message “Usage data for sites <Site name> was not included in this report due to a data collection failure. Please collect usage details and report correct numbers manually in order to stay compliant with licensing rules.” is displayed in the usage report. 
• Usage Report generation fails if there are several Veeam Backup & Replication servers with the same installation ID. 
• Usage Report shows incorrect data if there are several Veeam Backup & Replication servers with the same database instance ID. 
• Usage Report always shows “Standard” license edition when using Veeam Backup Enterprise Manager.

• “Immediate backup copy job state” alarm is triggered for deleted VMs.
• “Workstation/Server agent backups stored in cloud repository” alarms may be triggered incorrectly. Veeam Backup & Replication v10 CP2 installation is required.

• Manually created invoices are sent to managed companies automatically, even if scheduling is disabled. 

• Under certain conditions, /v2/backupRepositories get method returns incorrect "freeSpace" data.
• /v2/backupRepositories get method can take a considerable amount of time for execution.
• /v2/failoverPlans get method fails with “Object reference not set to an instance of an object” error message.
• /v2/tenants/id/backupServers now returns GUID information for Veeam Backup Enteprise Manager in the query.

• Limit/offset parameters are not working correctly for /protectedWorkloads/virtualMachines get request.

• Description for retention and database clean-up events is missing.

• When using Custom Fields in ConnectWise Manage, billing synchronization fails with “Addition object is invalid” error message. 

• Under certain conditions, ConnectWise Automate plugin status is switching from  “Healthy” to “Error” state.

Solution

1. Back up the VSPC database.
2. Execute VSPC.ApplicationServer.x64_4.0.0.4911.msp as administrator on the VSPC server, or run this cmdlet as administrator:

   msiexec /update c:\VSPC.ApplicationServer.x64_4.0.0.4911.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VSPCApplicationServerSetup.txt

3. Execute VSPC.WebUI.x64_4.0.0.4911.msp as administrator on the VSPC UI (IIS) server, or run this cmdlet as administrator:

   msiexec /update c:\VSPC.WebUI.x64_4.0.0.4911.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VSPCWebUI.txt

4. Execute VSPC.ConnectorService.x64_4.0.0.4911.msp  as administrator on the VSPC server, or run this cmdlet as administrator:

   msiexec /update c:\VSPC.ConnectorService.x64_4.0.0.4911.msp /l*v C:\ProgramData\Veeam\Setup\Temp\ConnectorService.txt

More Information

Challenge

Configuring the required granular permissions for Veeam Backup for Microsoft Office 365.

Cause

Veeam Backup for Microsoft Office version 4c supports two different modern authentication methods and a basic authentication method for working with Office 365 organizations.

Depending on your Office 365 tenant configuration and the restrictions on using legacy authentication protocols, you can use one of the following authentication methods: 

• Modern authentication  
• Modern authentication with legacy authentication protocols 
• Basic authentication 

These authentication types require different sets of permissions to be configured.

Solution

Configure permissions required for organizations with modern authentication

Veeam service account permissions

Veeam service account you are going to use should have a Global Administrator role.
 

Azure AD Application permissions

You can either allow Veeam Backup for Office 365 to create the Azure AD application and all the required application permissions will be granted automatically:

https://helpcenter.veeam.com/docs/vbo365/guide/register_ad_application.html?ver=40#cna

Or you can create an Azure AD application manually: https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app

And then configure the permissions in accordance with "Veeam Backup for Microsoft Office 365 Version 4c" section of this User Guide page: https://helpcenter.veeam.com/docs/vbo365/guide/azure_ad_applications.html?ver=40 

Both Application permissions and Delegated permissions should be configured in the Azure AD application.
For more information about permission types please visit https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent

Configure permissions required for organizations with modern authentication and legacy authentication protocols

Before configuring permissions below, make sure that Security Defaults are disabled in your Office 365 tenant:

https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults

And Conditional Access policies are not blocking legacy authentication protocols for the Veeam service account:

https://docs.microsoft.com/en-us/azure/active-directory/conditional-access/

Veeam service account permissions

Exchange

 Configuring permissions for Exchange Online.

• Create a role group in the Exchange Admin Center: https://outlook.office365.com/ecp/  as explained here: https://docs.microsoft.com/en-us/exchange/permissions/role-groups?view=exchserver-2019#use-the-eac-to-create-a-role-group
  • Add Roles specified in this User Guide page: https://helpcenter.veeam.com/docs/vbo365/guide/permisisons_exchange.html?ver=40
  • Add the Veeam Service account to role group members and save the role group.
• Connect to Exchange Online PowerShell module: https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell?view=exchange-p and run the "Get-AuthenticationPolicy" command.
  • If there is a blank return then there is no policy in place and you need to configure one.
  • If the current policy does not have "true" for AllowBasicAuthPowershell and AllowBasicAuthWebService contact your Domain admins to update the policy with the following Microsoft document: https://docs.microsoft.com/en-us/powershell/module/exchange/set-authenticationpolicy?view=exchange-ps

Below you may see the example of PowerShell cmdlets you could use to configure a new authentication policy with enabled AllowBasicAuthPowershell and AllowBasicAuthWebService for the Veeam service account.

To create a new authentication policy named "Allow Basic Auth":

New-AuthenticationPolicy -Name "Allow Basic Auth"

To enable AllowBasicAuthPowershell and AllowBasicAuthWebService in the new authentication policy:

Set-AuthenticationPolicy -Identity "Allow Basic Auth" -AllowBasicAuthPowershell
Set-AuthenticationPolicy -Identity "Allow Basic Auth" -AllowBasicAuthWebService

To assign this policy to the Veeam service account:

Set-User -Identity <UserIdentity> -AuthenticationPolicy "Allow Basic Auth"

Where <UserIdentity> is the Veeam service account.

SharePoint

Configuring Permissions for SharePoint Online.

• Grant Veeam service account with the SharePoint Administrator Role in Azure Admin Center: https://admin.microsoft.com/
• In the SharePoint Admin Center open [Policies] > [Access control]  and set "Apps that don't use modern authentication" to "Allow Access".
  
  
  

App password

Configuring user App password.

• Set up Veeam service account user to leverage Multi-Factor Authentication: https://account.activedirectory.windowsazure.com/UserManagement/MultifactorVerification.aspx?
• Make sure users are allowed to create app passwords: https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-app-passwords#allow-users-to-create-app-passwords
• Sign-in to https://portal.office.com using Veeam service account credentials and create app password as explained here: https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-app-passwords#create-and-delete-app-passwords-using-the-office-365-portal

Azure AD Application permissions

Configuring Azure AD Application.

• Register a new application in Azure AD as explained here: https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app
• Go to API permissions and click [Add a permission].
  
• Configure the permissions in accordance with "Veeam Backup for Microsoft Office 365 Version 4" section of this User Guide page: https://helpcenter.veeam.com/docs/vbo365/guide/azure_ad_applications.html?ver=40

  All permissions must have Application type.

  For more information about permission types please visit https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent

• After all API permissions have been added you will need to grant admin consent.
  
• Select [Certificates & secrets] and [+New client secret].
  
• Add a Description> Choose Expiration> Add.
  
• Copy the Client's secret value.
• Go to Overview and copy the Application (client) ID.
  

Configure permissions required for organizations with basic authentication

Exchange

Exchange permissions should be configured the same way as in the "Exchange" section of this article. 

SharePoint

SharePoint permissions should be configured the same way as in "SharePoint" section of this article.

Challenge

Cause

Solution

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

Challenge

1. The absolute log file directory from which all logs will be stored.
2. Maximum size each log is allowed to reach before beginning a new log file.
3. Number of archived logs to be kept before removal.
4. The total size for a job’s logs to reach before Veeam will begin archiving (gzip) inactive logs to save space.
5. Amount of logs that are archived per job.
6. The size of an agent log file on sever with a proxy, repository or mount server role.
7. Amount of the agent rollback files on sever with a proxy, repository or mount server role.

Solution

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the server where the Veeam Backup & Replication console is installed
Type: String
Name: LogDirectory
Data: <Path>
Default: C:\ProgramData\Veeam\Backup

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the server where the Veeam Backup & Replication console is installed
Type: DWORD
Name: MaxLogSize
Data: <value in KiB>
Default: 10240 (binary)

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the server where the Veeam Backup & Replication console is installed
Type: DWORD
Name: MaxLogCount
Data: <numeric value>
Default: 10

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the server where the Veeam Backup & Replication console is installed
Type: DWORD
Name: UncompressedLogsMaxTotalSize
Data: <value in KiB>
Default: 512000

Type: DWORD
Name: LogsArchivesMaxCount
Data: <numeric value>
Default: 10

For Windows:

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication  on the proxy in question.  
Type: DWORD
Name: AgentMaxLogSize
Data: in BYTES

Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the proxy in question.   
Type: DWORD
Name: AgentMaxLogCount
Data: <numeric value>

For Linux:

*6. The size of an agent log file.

Location: /etc/VeeamAgentConfig 
Name: AgentMaxLogSize
Data: in BYTES
Default: 15728640

*7. Amount of the agent rollback files

Location: /etc/VeeamAgentConfig 
Name: AgentMaxLogCount
Data: <numeric value>
Default: 10

Challenge

If a Virtual Machine's MoRef is changed due to migration to new vCenter / inventory change in vCenter, you would receive the error "Object "vmname" is not found". Please see KB1299 for more details.

This utility is designed to assist in re-mapping the MoRef ID's of the Virtual Machines you are backing up and/or replicating using Veeam Backup & Replication, so that backup incremental chains remain intact, after an inventory change in VMware vSphere vCenter. Do not run any jobs before using the migration tool otherwise, the migration process won't be possible.

Please note that this utility is provided as a courtesy, and is not supported by Veeam Technical Support. Use at your own risk.

This tool cannot fix jobs targeting a Cloud Connect repository. You must perform an Active Full backup.

Important! If you are migrating to a 6.5U2 VCSA, VMware has a new migration utility, which could preserve existing VM MoRef IDs:
https://blogs.vmware.com/vsphere/2016/09/vcenter-server-migration-tool-vsphere-6-0-update-2m.html
https://blogs.vmware.com/vsphere/2017/01/vcenter-server-appliance-6-5-migration-walkthrough.html
This VMware utility only works for a 5.5 VC to a 6.5U2 VCSA!

Cause

Veeam Backup & Replication tracks Virtual Machines in its jobs by a Managed Object Reference ID (MoRef), which will change when moved to another vCenter, or removed and re-added to inventory.

Solution

Please download this bundle: tool and instructions.
Before using this tool, it is highly recommend to take a configuration backup of your VeeamBackup database.

Note:

• The utility will work only with a source vCenter migration. If Replication target vCenter is changed, you will need to manually map source VMs to their replicas.
• If for some reason you have already added the new vCenter or ran any jobs which will have added new Ref-IDs to the database, you can roll the Veeam Backup & Replication database back using the Configuration Restore function.

If you use the VMware provided VCSA Migration tool, you will need to remove and re-add VMs to jobs manually through the new VCSA after adding it to Veeam.

VMware highly recommends migrating to a VCSA, as they are phasing out the Windows based vCenter.
https://blogs.vmware.com/vsphere/2017/08/farewell-vcenter-server-windows.html

More Information