Challenge

This article explains how to collect the diagnostic information required for a Veeam Agent for Microsoft Windows support case.

Solution

To encourage quicker support case processing, attach the following logs to the support case:

• Veeam Agent logs
• VSS diagnostics
• Event logs
• File system minifilter diagnostics

You can collect logs in the following ways: 

• Automatically using the script that is attached to this article.
• Manually in case the script fails to retrieve required logs.

Collect Logs Automatically

The log collection script grabs the following information:

• Veeam Agent for Microsoft Windows log files located in C:\ProgramData\Veeam\Endpoint
• Information provided by the systeminfo command
• Information provided by the vssadmin command
• Information provided by the fltmc command
• Windows Application, System, Security and Veeam Agent events
• The HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Endpoint Backup registry key.
• Computer UUID

1. Download the script from the link above.
2. Extract the script to any folder on the computer where Veeam Agent for Microsoft Windows is installed (for example, C:\temp\).
3. Open an Elevated Command Prompt or PowerShell window and run the command:

PowerShell.exe -ExecutionPolicy ByPass -File <Path to the script>

4. Run the script with PowerShell. The script executes simple select queries, no system changes are made.

If the script runs successfully, the warning message will appear. 

After the script execution, the message about the successful execution will appear. The script creates a new folder with the VeeamAgentLogs_<DDMMYYYY> name, where <DDMMYYYY> is the current date, and stores the collected information into this folder. 

5. Attach the folder with all the contents to the support case.

If the script fails to collect the logs, proceed to collect the information manually as outlined below.




 

Collect Logs Manually

The following steps are to be done on the computer where Veeam Agent for Microsoft Windows is installed.

1. Collect Veeam Agent Logs

1. Navigate to the following folder: C:\%Programdata%\Veeam\Endpoint (this folder is hidden by default, copy and paste the provided path). 
2. Archive the content of this folder.

2. Collect VSS Diagnostics

1. Run the following commands using the administrative command prompt:

• vssadmin list writers > C:\vss_writers.log
• vssadmin list providers > C:\vss_providers.log
• vssadmin list shadows > C:\vss_shadows.log
• vssadmin list shadowstorage > C:\vss_shadow_storage.log

2. Archive the resulting files.

3. Collect Event Logs

1. Run the Event Viewer (eventvwr.msc).
2. In the left menu, find the Application logs.
3. Right-click the Application logs and select Save All Events As.
4. Enter the file name that includes the log type and the computer name the logs were exported from (for example, when exporting Application logs from a computer named HV01, enter Application_HV01).
5. In Save as type, select Event Files (*.evtx).
6. Include the display information.

7. Repeat steps 3—6 for the System logs.
8. Archive the logs. Include the LocaleMetaData folder in the archive as well.

4. Collect File System Minifilter Diagnostics

1. Run the fltmc instances > C:\filter.log command using the administrative command prompt.

Send Logs to the Veeam Support

1. Create a support case.
2. U​​pload the logs to the FTP server provided by the Veeam Support engineer. If you need a Secure FTP, see this article.
3. After the upload, notify the Veeam Support engineer.

More Information

Challenge

Product Information:

Company name: FAST LTA, AG
Product Family: Silent Brick System
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: FAST LTA - Silent Brick FLASH
Storage Category: Hybrid Storage
Drive quantity, size, type: 12 - SATA FLASH discs via SAS Bus
Storage configuration: SecureNAS - Triple Parity
Firmware version: 2.11
Connection protocol and speed: 10GbE
Additional support: All models and configurations of Silent Brick System with specifications equivalent or greater than the above.

General product family overview: Simplify your backup with Silent Bricks. On-premise storage appliances with integrated data protection for all NAS and VTL requirements. Media replication and rotation. Online, offline and off-site, with support for air gap environments. All in one system. The Silent Brick System is based on modular and transportable storage units called 'Silent Bricks' where each Silent Brick consists of 12 Disks. Depending on the Silent Brick Type these disks may be HDD (Silent Brick ) or SSD ( Silent Brick FLASH ).
The assignment is done on a per Brick Basis where each Brick can act as a Backup to Disk Storage, a virtual Tape Library or an Archival Storage. The redundancy when set up as Backup to Disk Storage is configurable with 2 or 3 Parities per Silent Brick. A SecureNAS Volume can then be spanned over 1 to many Silent Bricks. Extending a SecureNAS Volume is possible on the fly by assigning additional Bricks. The parities will still be on a per Silent Brick Basis. For optimal performance the Silent Brick System offers dual 10GB Ethernet connection.

Veeam testing configuration:

Note: The following settings were used by the vendor to meet Veeam Ready testing requirements and should not be considered best practices. Additional changes or settings may be needed to meet the storage efficiency or performance needs for each environment. For each setting, reference links are provided for further clarification.

Veeam Build Number: 9.5.0.1922

Job Settings:

• Deduplication: Enabled (Default)
• Compression: Optimal (Default)
• Storage Optimization: Local Target (Default)

Repository Settings:

• Repository Type: Shared Folder
• Align backup file blocks: Disabled (Default)
• Decompress before storing: Disabled (Default)
• This repository is backed by rotated hard drives: Enabled
• Per-VM Backup Files: Disabled (Default)

Vendor recommended configuration:

Hardware Settings:

• Jumbo frames used in testing environment
• SMB storage protocol used during testing. Vendor also supports NFS, and ISCSI or Fibre Channel for virtual tape library support.
• Dedicated SMB user for storage share, used default settings for share configuration
• 3 parities used in RAID Z configuration

More Information

Challenge

Cause

• When adding an Organization there are 11 Verification checks that happen. Below lists each of the verification and what it means if it fails. In addition any permission assigned in Office 365 Online can take 15-60 minutes to apply on Microsoft's end.
   

Connection to Microsoft Graph: This meaning can change based on if you are using Modern authentication or Basic authentication.

Modern authentication: This would mean that the Application ID and secret failed to authenticate.

• Make sure the Application ID and Secret were properly entered
• Check the Application APIs from the "Azure AD Application" section below
• Try generating a new Application Secret from Azure AD

Basic authentication: This would mean that your username does not have permissions to authenticate with the Microsoft Graph Online.

• If your Organization is Federated try creating a new cloud user account from Microsoft Azure AD for authentication.
• Make sure that the user belongs to the Organization.
• Check that user has all permissions assigned from the above SharePoint and Exchange section

Connect to EWS: This is a connection to the Exchange Web Service.

Modern authentication: This uses a duel authentication leveraging the Application ID and Username.

• For the Application ID check in Microsoft Azure AD that the correct APIs were assigned as Application and not Delegated.
• For the Username check Microsoft Exchange Admin center that all permissions have been assigned to the user as documented below.

Basic authentication:

• For the Username check Microsoft Exchange Admin center that all permissions have been assigned to the user as documented below.

Connection to PowerShell: This step checks that we can connect to Exchange Online PowerShell. Only the username is used for this verification. Require AllowBasicAuthPowershell and AllowBasicAuthWebServices to be allowed in the group policy this user is added to.

• In Azure AD check Conditional Access- Polices for anything blocking legacy authentication and make sure the service account is being excluded from these policies: https://aad.portal.azure.com/#blade/Microsoft_AAD_IAM/ConditionalAccessBlade/Policies
• Check through PowerShell to see if the policies are allowed for the user:
  1. Connect to Exchange Online PowerShell module: https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell?view=exchange-ps
  2. Next you will run the "Get-AuthenticationPolicy" command. If there is a blank return then there is no policy in place.
  3. If the current policy does not have true for AllowBasicAuthPowershell and AllowBasicAuthWebService contact your Domain admins to update the policy with the following Microsoft Document
  4. https://docs.microsoft.com/en-us/powershell/module/exchange/organization/set-authenticationpolicy?view=exchange-ps

Check Exchange plan and SharePoint plan: If either of these fail then your plan cannot automatically identified as Valid. Use the below KB to add the proper plans to your configuration

Service plan is not found when adding Office 365 organization: https://www.veeam.com/kb2946

Check Required cmdlets access, Mailbox Search role, and Applicationimpersonation role: The roles are all related to Exchange Online.

• Check that the user has been properly added to the role group with all necessary permissions as shown above in the Exchange section

Check SharePoint Online Administrators role: This refer the SharePoint Administrator role that you assign to the user in the Admin Center.

• Refer to the SharePoint section above and make sure that the service has the SharePoint Administrator role assigned

Check LegacyAuthProtocolsEnabled: This is a setting in SharePoint Admin center to enable legacy Authentication. 

• Refer to the SharePoint section above to Allow Apps that don't use modern authentication.
• “Unauthorized” error is thrown by SharePoint Online and/or OneDrive for business backup jobs: https://www.veeam.com/kb2714

Solution

Exchange
SharePoint
User App Password
Azure AD Application

Exchange

• First log into the Exchange Admin Center: https://outlook.office365.com/ecp/
• Select Permissions and add a New Role Group

• Create your role Group: -Name your role group appropriately (Example: VBO Permissions) -Add Roles: ApplicationImpersonation, View-Only Configuration, View-Only Recipients, Mailbox Search, and Mail Recipients -Add the Veeam Service account under Members and Save

SharePoint

• Add the SharePoint Administrator Role to user In Azure Admin Center: https://admin.microsoft.com/ Select: Users > Active Users >  Select Backup Service account

• Allow Apps that don't use modern authentication In the SharePoint Admin Center: https://[ORGANIZATION NAME]-admin.sharepoint.com/ Select: Access control > Apps that don't use modern authentication > Allow Access

User App Password

• Set up user to leverage Multi Factor Authentication: https://account.activedirectory.windowsazure.com/UserManagement/MultifactorVerification.aspx? Select: Service account > Enable.

• Once the account have been enabled for Multi-factor authorization you will need to sign-in with the service account and follow the steps for configuring Multi Factor authentication.
• Generate an application password for the user: https://portal.office.com/account/#security Select: Additional security verification> Create and manage app password:

Azure AD Application

• Register a new App in Azure AD: https://aad.portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade Can also be found under: All services> App Registrations-

• Name the Application, add Redirect URL (URL does not need to be real. Example: "http://localhost/"), and Click Register

• Select API permissions and Add permission. Please keep in mind that all permissions must be added as Application permissions and not Delegated permissions

• First API will be at the top of the page: Microsoft Graph> Application Permissions

• Check Directory.Read.All and Group.Read.All then select Add permissions.
  Directory.Read.All permission is required to read organization and users’ properties. Group.Read.All is required to read groups properties and membership.

• If you’re going to use application certificates, second API will be halfway down the page: SharePoint > Application Permissions

• Check Sites.Fullcontrol.All and User.Read.All. Then select Add permission.
  Sites.Fullcontrol.All is required to read sites content. User.Read.All is required to read user profiles.

• Last API is also required only if using application certificates. It will be at the bottom of the page Under Supported legacy APIs: Exchange> Application Permissions.

• Check full_access_as_app. Then select Add Permission.
  full_access_as_app is required to read mailboxes content.

• After all APIs have been added you will need to Grant consent:

• Select Certificates & secrets and New client Secret

• Add a Description> Choose Expiration> Add

• Copy Secret Value because it will no longer be available once you close the window:

• Locate Application ID: Overview> Application (client) ID

Challenge

Cause

Solution

• Defragment and compact functionality performs multiple passes on the same full backup for backup jobs that do not use per-VM backup file chains.
• Querying WMI API class "Backup" fails with error code 0x80131500, which affects integration such as Veeam Service Provider Console (formerly Veeam Availability Console).
• Start-VBRInstantRecovery PowerShell cmdlet fails to process restore points created by Backup Copy jobs.
• VCSP rental license point weights were updated according to the March 1st 2020 changes.

• Copying NAS backups to secondary target erroneously consumes licenses (in addition to licenses consumed by the primary file backup job).

• Capacity Tier copy policy may erroneously upload full backups to object storage periodically when source backup chains do not have periodic fulls. 

• Resetting CBT on Active Full backup fails if the database contains orphaned ESXi hosts.
• Deleting a vApp from vCloud Director backup may delete the entire backup if backup chain type was recently changed to per-VM backup.

• Hyper-V backup jobs fail the "Object reference not set to an instance of an object" error in environments where at least one virtual machine is stored in the SCVMM Library.
• Replica VMs disk space estimation logic may return incorrect value due to case sensitivity with virtual disk paths, resulting in some disks contributing to the total size twice.

• SureBackup jobs using advanced multi-host virtual labs fail with the "Invalid configuration for device " error.
• SureBackup log export functionality fails with the "Object was not found" error for SureBackup jobs that include an Application Group.

• Backup to Tape job skips processing synthetic full backups with forever forward incremental source backup chains on days when full backup file merge happens.

More Information

If you have v10 installed (GA build 10.0.0.4461), you can apply Cumulative Patch 1 using the following update package:

[[DOWNLOAD|DOWNLOAD CUMULATIVE PATCH|https://www.veeam.com/download_add_packs/vmware-esx-backup/kb3127/]]
MD5: 7826927eae9b05f963ef04e07d15f039
SHA1: 5749fc2e18e45dd9a44f77eb65269e85bf342134

For new installations and upgrades from the previous versions, use the updated v10 ISO image version 20200401, which has the Cumulative Patch 1 included:

[[DOWNLOAD|DOWNLOAD ISO|https://www.veeam.com/send_license/vmware-esx-backup/]]
MD5: f42cbeab6000c302bcb7518f70beb31f
SHA1: 5fcc743f0e26d6d967cb1a7af5267a41af480ae5

Challenge

Cause

Solution

Prerequisites:

• You need access to both Service Provider (SP) and tenant Veeam backup consoles.
• You need a second SP Veeam backup server (in addition to the original SP Veeam backup server and tenant Veeam backup server) where the metadata file of the affected backup will be imported. The second Veeam backup server can be used on the tenant side instead of the SP side, but in this case, SP has to provide the tenant with the original .vbm file and the list of storage files on a repository.
• You need one local repository connected to the second Veeam backup server.
• You need to back up original .vbm file by copying it to another folder outside the tenant quota.
• If the backup is encrypted, you need to know the password of the backup.
• To avoid any possible issues at steps 4–5, we recommend to use Veeam Backup & Replication version 9.5.4.2753 or higher in both tenant and SP installations.

Steps:

1. Connect to the tenant Veeam backup console.
2. Open Home – > Jobs.
3. Find your cloud job that processes the backup you want to modify, right-click the job and select Disable. The option can be greyed-out if the job is not scheduled to run at some particular time.
4. Open the Home –> Backups –> Cloud view.
5. Press and hold the [CTRL] key and right-click the backup you want to modify.
6. Select the Remove from configuration option.
7. Connect to the SP infrastructure.

Perform the following steps on the SP side until you see a notice to connect to the tenant infrastructure.

8.  Locate tenant quota in the SP repository used for storing tenant backups.

           Example:

The path to repository root is C:\VeeamBackup\. You can check this path in Backup Infrastructure –> Backup Repositories in the SP Veeam Backup & Replication console.
The tenant name is “Tenant1”.
The cloud job name is “CloudBackup”.
The path to tenant backup in the quota will be: C:\VeeamBackup\Tenant1\CloudBackup.

You can perform the following steps on the tenant side if the second Veeam backup server is deployed there and the tenant gets the required .vbm file from its SP.

9. Copy the cloud backup metadata file (.vbm file) from the tenant quota to the second Veeam backup server.
10. Put the .vbm file in the root folder of the local repository connected to the second Veeam backup server.
11. Open the Backup Infrastructure –> Backup Repositories view in the Veeam backup console of the second Veeam backup server.
12.  Find the local repository in the list, right-click it and select Rescan.
13. After the rescan process completes, you will see the imported backup in the Home –> Backups –> Disk (Imported) node. 
    If the backup is encrypted, you will see the backup in the Home –> Backups –> Disk (Encrypted) node. Find the backup in the list, right-click it, select Specify the password and use the latest password specified in the cloud job settings.
14. Right-click the imported backup and select Properties.
    All unavailable restore points will have red marks. Your goal is to remove the restore points that are missing on the cloud repository. Compare the list of points in the backup chain on the original repository and find the missing restore points. If there are some points dependent on the missing point, they must also be deleted from .vbm.
15. In the Backup Properties window, find the point you want to remove from .vbm.
16. Right-click the point and select the Forget option.
    If there are any points that depend on the missing point, select the first missing point and choose Forget –> This and dependent backups.
17. After the operation completes, copy the .vbm file from the local repository on the second Veeam backup server back to the tenant job folder in the tenant quota on the cloud repository.
    If the modification was made on the second Veeam backup server on the tenant side, pass the modified .vbm file to the SP.
    If the cloud quota for the tenant is created on a scale-out backup repository (SOBR), the metadata file should be copied to tenant job folders on every extent of SOBR.
18. Remove the backup from configuration on the second Veeam backup server on the SP side and do not forget to delete the .vbm file from the local repository.
19. Connect to the tenant Veeam backup console.
20. Open the Backup Infrastructure -> Backup Repositories view.
21. In the inventory pane, find the cloud repository where the affected backup is stored, right-click it and select Rescan.
22. During the rescan operation, in the statistics window, you will see that the backup was imported.
    The modified backup should appear in the Home –> Backups –> Cloud node in the tenant Veeam backup console.
    If you open the backup properties, you will see that the points removed at previous steps are no longer in the backup file list.
23. Open Home –> Jobs and find the cloud job associated with this backup.
24. Right-click the job, select Edit, proceed to the Storage step of the wizard and select Map backup.
25. Select the modified cloud backup from the cloud repository. Then, click OK to finish mapping.
26. Save the settings and enable the job. Then run it or wait for the job to run upon schedule.
    When the job runs after modifications for the first time, it is supposed to create an incremental restore point and back up data that has changed since the last restore point in the cloud backup chain was created.
27. Check the list of restore points on the repository and in the backup properties. If some points are absent in properties but exist on the repository, you can remove these files from the repository to free up the quota space.

Challenge

Cause

What’s New 

• Support for Veeam Backup & Replication v10 compatibility.
• Support for Veeam Cloud Connect v10 compatibility. 

• Veeam Backup for Microsoft Office 365 version 4.0.0.2516 is applicable to:
  • standalone installations of Veeam Backup for Microsoft Office 365
    • version 2.0 (builds 2.0.0.567, 2.0.0.594 and 2.0.0.814)
    • version 3.0 (builds 3.0.0.422 — 3.0.0.480)
    • version 4.0 (build 4.0.0.1345)
    • version 4a (build 4.0.0.1553; see KB3035 for more details)
  • installations on top of Veeam Backup & Replication v10 and Veeam Cloud Connect v10.
• Version 4.0.0.2516 is NOT applicable to installations on top of
  • Veeam Backup & Replication 9.5 Update 4, 4a or 4b
  • Veeam Cloud Connect 9.5 Update 4, 4a or 4b
• Upgrading backup repositories from version 3.0 to 4b may take significant time to complete and the upgrade duration highly depends on the amount and type of backup data in a repository. According to the test results, it may take up to 8 hours in the setup with 1 TB of data in a backup repository. Plan for an appropriate maintenance window when your backup repository that is been upgraded is not available for use. Note that upgrading version 2.0 repositories to 4b requires a bigger maintenance window due to many underlying changes to backup repositories. The upgrade process requires an additional 15% of free space of the current 2.0 repository size and may take up to 17 hours per 1 TB of data in a backup repository. 

Resolved Issues

• Retention jobs do not fully purge the outdated service data, which causes product configuration database growth and impacts product upgrade duration. 

• Adding users with empty Display Names to a backup job fails with the “Value cannot be null” error. 
• Backup of user’s OneDrive fails if the user name has been changed.  
• Backup job containing excluded SharePoint sites with changed URLs fails with the following warning: “Failed to get site collection: {url}. Access denied. You do not have permission to perform this action or access this resource.” 

• Backing up public folders to a repository extended with object storage may fail with the following error: “This id is of the wrong type for this conversion operation (mailbox id vs. public folder id).”
• Adding OpenStack Swift (Stein) S3 Compatible object storage repository with the s3api middleware for S3 REST fails with the “Unable to create folder (name: {0})” error.
• Backup and retention jobs targeted at repositories extended with Amazon S3 object storage may fail with the “Specified argument was out of the range of valid values. Parameter name: count” error.
• If an Azure storage container that is in use with an object storage repository has been deleted, the corresponding backup repository is put into the Invalid state and the backup proxy disk usage is at 100%.
• A repository extended with Azure Blob Storage is still available for use (i.e. it’s not put to an Invalid state) when Azure storage account credentials become invalid. 
• After changing retention period for a repository extended with object storage, backup jobs targeted to this repository stop processing.

Solution

1. Download the ZIP archive: VeeamBackupOffice365_4.0.0.2516.zip.
2. Run the Veeam.Backup365_4.0.0.2516.msi from the downloaded ZIP archive and follow the wizard steps to install Veeam Backup for Microsoft Office 365.
3. Run the VeeamExplorerForExchange_10.0.0.443.msi from the downloaded ZIP archive and follow the wizard steps to install Veeam Explorer for Microsoft Exchange. 
4. Run the VeeamExplorerForSharePoint_10.0.0.443.msi from the downloaded ZIP archive and follow the wizard steps to install Veeam Explorer for Microsoft SharePoint. Note that Veeam Explorer for Microsoft OneDrive for Business is distributed in one package with Veeam Explorer for Microsoft SharePoint and will be installed automatically.   

More Information

Challenge

Cause

Enhancements

• The amount of RAM reserved for object storage metadata cache is extended to 512 MB to reduce egress costs associated with retrieving data from an object storage repository. 
• The limitation on the minimum Access Key length required for adding S3 compatible object storage removed. 
• The default setting for applying retention policy to object storage repositories is changed to monthly in order to reduce egress costs. 

Resolved Issues 

• Adding Office 365 organizations using U.S. Government plans fails with the “Confidential Client is not supported in Cross Cloud request” error. 
• Working with Office 365 organizations using U.S. Government GCC High plans fails with the “ProtocolError - The remote server returned an error: (403) Forbidden” error. 
• Incremental backup jobs processing OneNote data fails with the “File not found in the database” error.  
• Email notifications on backup job completion results do not include information about users that exceeded the license count.  
• The Veeam.Archiver.Proxy service fails with the “System.TimeoutException: This request operation sent to net.tcp://%IP%port% did not receive a reply within the configured timeout.” error if the Veeam Backup Proxy server is being backed up using a VSS writer.

• Moving existing backup data from a local to an object storage repository fails with the “JetError -1011, JET_errOutOfMemory, Out of Memory” error. 
• Moving existing backup data from a local to an object storage repository fails with the “JetError -1101, JET_errOutOfSessions, Out of sessions” error. 
• Under certain conditions, moving existing backup data from a local to an Amazon S3 or an S3 compatible object storage repository fails with the “JetError -1605, JET_errKeyDuplicate, Illegal duplicate key” error. 
• Configuring object storage consumption limit to more than 1TB in the product UI doesn’t apply these changes to the configuration database.  
• Adding an S3 compatible object storage repository fails with the “Failed to request region for bucket %Bucket Name%, bucket skipped” error. 
• Adding an Amazon S3 object storage repository with a bucket in the AWS GovCloud (US) region fails with the “The AWS Access Key Id you provided does not exist in our records” error. 
• The amount of RAM reserved for metadata cache locally when retrieving data from an object storage repository exceeds the expected default size.  

• OneDrive data backup to a repository extended with object storage fails with the “Unspecified Error”. 
• Exchange data backup to a repository extended with object storage fails with the “Specified argument was out of the range of valid values. Parameter name: count” error. 
• Backup to object storage repository fails with the “JetError -1069, JET_errVersionStoreOutOfMemory, Version store out of memory” error. 
• Backup to a repository extended with object storage fails with the “JetError -1311, JET_errTooManyOpenTables” error.
• Backup of OneNote files retained in the Preservation Hold Library fails with the “Block request failed. (403) Forbidden” error. 

• When exploring backup jobs configured with exclusions, Veeam Explorer for Microsoft SharePoint fails with the “Some backup data is not available” error. 
• Restore of OneDrive accounts which URLs have been changed after the backup job run fails with the “A file or folder with the name %path% already exists” error. 
• Export to PST fails with "Unable to read beyond the end of the stream".  

• The “GET v4/Organizations/{organizationId}/Groups/{groupId}” resource doesn’t return the “managedBy” parameter. 

• Under certain conditions, execution of the Get-VBOUsageData PowerShell cmdlet fails with the “Specified cast is not valid” error.  

Solution

1. Download VBO4b-KB3119.msp.
2. Execute VBO4b-KB3119 as administrator on the Veeam Backup for Microsoft Office 365 server. 

More Information

Challenge

Cause

Resolved Issues 

Object storage repositories

• Configuring object storage consumption limit to more than 1TB in the product UI doesn’t apply these changes to the configuration database.  
• Adding an S3 compatible object storage repository fails with the “Failed to request region for bucket %Bucket Name%, bucket skipped” error. 
• Adding an Amazon S3 object storage repository with a bucket in the AWS GovCloud (US) region fails with the “The AWS Access Key Id you provided does not exist in our records” error. 
• Object storage repository synchronization fails with the “JetError -1069, JET_errVersionStoreOutOfMemory, Version store out of memory” error. 
• Repository upgrade fails with the “JetError -1311, JET_errTooManyOpenTables” error.  
• Backup job fails with the “JetError -1311, JET_errTooManyOpenTables” error. 
• Under certain conditions, moving existing SharePoint or OneDrive for Business backup data from a local to an object storage repository may fail with the “Unspecified Error”. 
• Moving existing backup data from a local to an object storage repository fails with the “JetError -1101, JET_errOutOfSessions, Out of sessions” error. 
• Adding or editing an object storage repository and selecting (or creating) a folder in the Amazon S3 bucket fails with the “Could not establish trust relationship for the SSL/TLS secure channel” error. For more details, see KB3095. 
• Creating a folder in an S3 compatible object storage bucket fails with the “Unable to create folder (name: <folder_name>)” error. For more details, see KB3097. 

General

• Working with Office 365 organizations using U.S. Government GCC High plans fails with the “ProtocolError - The remote server returned an error: (403) Forbidden” error. 
• Adding Office 365 organizations using U.S. Government plans fails with the “Confidential Client is not supported in Cross Cloud request” error. 
• Incremental backup jobs processing OneNote data fails with the “File not found in the database” error.  

Backup

• Under certain conditions, backup to a repository extended with Amazon S3 or S3 compatible object storage or retention jobs applied to such repositories may fail with the “Specified argument was out of the range of valid values”. 
• Backup to object storage repository fails with the “JetError -1069, JET_errVersionStoreOutOfMemory, Version store out of memory” error. 
• Under certain conditions, Office 365 groups backup to a repository extended with object storage fails with the “This id is of the wrong type for this conversion operation” error. 
• Adding users without a Display Name to a backup job fails with the “Value cannot be null. Parameter name: key” error. 
• Backup job configured to exclude SharePoint sites with changed URLs completes with a warning. 
• Incremental backup job ignores changes in OneDrive accounts URLs that may have occurred after the full backup job run. 
• When editing a backup job, the “Personal sites” node remains available for selecting even if it has already been added to the same backup job. Likewise, individual personal sites remain available for selecting, even if the entire “Personal sites” node has already been added to the same backup job. 

Restore 

• Export to PST fails with "Unable to read beyond the end of the stream"
• When exploring backup jobs configured with exclusions, Veeam Explorer for Microsoft SharePoint fails with the “Some backup data is not available” error. 

PowerShell 

• Under certain conditions, execution of the Get-VBOUsageData PowerShell cmdlet fails with the “Specified cast is not valid” error.  
• An output of OneDrive data requested with the Get-VBOEntityData PowerShell cmdlet may be incorrect if the OneDrive account URL has been changed after the full backup job run. 

Solution

1. Download VBO4a-KB3089.msp 
2. Execute VBO4a-KB3089.msp as administrator on the Veeam Backup for Microsoft Office 365 server. 

More Information

Challenge

Cause

Solution

• Managed by Agent backup policies do not respect the retention policy setting for Microsoft Windows machines, always forcing it to 7 regardless of the value specified.

• Added support for Cape Town and Milan data center regions for Capacity Tier, Direct Restore to EC2 and External Repository functionality.

• Failed GFS full backup transform may result in the job to start failing with the “Failed to process method {Transform.Patch}: There is an item with the specified name and another type” error.

• Under certain circumstances, backup jobs may fail with the “Cannot retrieve physical extents for connecting iSCSI target IQN {0} on proxy IQN {1}” error.

• In the environments where Network Extension Appliance is named the same as the corresponding Tenant’s name, then deleting the appliance will also remove tenant’s replicas.
• Under certain circumstances, task slots may not be released at the end of the job session, resulting in various issues due to lack of available task slots.
• Cloud Connect Replication fails with the “Cannot find an available target” error in vCloud Director environments with multiple clusters due to not searching all available vDC pools for the required vOrg pool.
• Using the UseCachedSshConnections registry value may result in jobs to stop running due to already closed connection erroneously cached. This issue may result in various errors, typically “The session is not open” error.
• Opening the Tenants view takes a long time.

• High CPU consumption on the SQL Server hosting the configuration database caused by the AggregateSqlPointsInfo stored procedure during transaction log backup.

• Mounting a restore point with the dynamic disk fails with the “Invalid secondary GPT partition entry array checksum” error.
• In some environments, mounting a restore point for file-level recovery may fail with the “Invalid secondary GPT header signature” error.

• VSS snapshot fails to be created with the “Access is denied” error for SMB file shares hosted on some NAS devices due to lack of impersonation.
• Backup of different files shares from the same Windows Server fails with the “Failed to create a VSS snapshot” whenever Microsoft VSS snapshot creation calls overlap.

• Resource scheduler may experience significant delays issuing the backup infrastructure resources in environments with a large number of agent-based, application plug-in based or host-based backups (with per-VM backup file chains disabled) are stored in a scale-out backup repository.

• High backup server CPU consumption by the Veeam Backup Service in large environments while the user interface is open and no jobs are active.
• High backup server RAM consumption by the UI Service process and slow performance of historical sessions view in environments with very large number of sessions.
• Opening an existing VMware backup job’s settings takes a long time for jobs using a scale-out backup repository with a large number of extents as the target.

• Invalid tenant workloads count is returned by the backup server’s WMI API, resulting in Veeam Service Provider Console to issue false warnings.

• WAN accelerator may deadlock during while transferring an increment with large contiguous set of zeroed segments, making the data transfer appear to slow down dramatically

More Information

Challenge

Cause

Solution

More Information

Challenge

Cause

Such behavior occurs in case there is a Microsoft account added in Settings > Accounts > Access work or school.

Solution

There are 2 options to use Microsoft OneDrive as a backup target when configuring a Veeam Agent backup job:

1. Use the account that is already set in Settings > Accounts > Access work or school to store backups on Microsoft OneDrive.
2. Disconnect the account that is set in Settings > Accounts > Access work or school. Veeam Agent allows to change accounts if the computer is disconnected from the account. Configure the Veeam Agent backup job, then re-connect the account.

Challenge

Cause

Solution

More Information

Challenge

This article describes list of VMware vSphere versions currently supported by Veeam Backup & Replication.

IMPORTANT: It is recommended to always use the latest Veeam version to ensure modern platforms support and security compliance.

Solution

More Information

Challenge

Cause

Solution

You can edit or delete subkeys of the FilesNotToSnapshot registry key to include automatically excluded files in the backup. For example, to include OST files, do the following:

1. Open Registry Editor (click the Start button, select Run, enter regedit in the Open: box and select OK).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToSnapshot registry key.
3. Delete the OutlookOST subkey.

Keep in mind that the OST files exclusion is recommended by Microsoft. To learn more about the recommendation, see this Microsoft Tech Community article.

Challenge

This article describes list of VMware vSphere versions currently supported by Veeam Backup & Replication.

IMPORTANT: It is recommended to always use the latest Veeam version to ensure modern platforms support and security compliance.

Solution

More Information

Challenge

Cause

Solution

You can edit or delete subkeys of the FilesNotToSnapshot registry key to include automatically excluded files in the backup. For example, to include OST files, do the following:

1. Open Registry Editor (click the Start button, select Run, enter regedit in the Open: box and select OK).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToSnapshot registry key.
3. Delete the OutlookOST subkey.

Keep in mind that the OST files exclusion is recommended by Microsoft. To learn more about the recommendation, see this Microsoft Tech Community article.

Challenge

This article describes list of VMware vSphere versions currently supported by Veeam Backup & Replication.

IMPORTANT: It is recommended to always use the latest Veeam version to ensure modern platforms support and security compliance.

Solution

More Information

Challenge

Cause

Solution

To fix the issue, do the following:

1. From the main menu of the Veeam Backup & Replication console, select Manage Credentials.
2. In the Manage Credentials window, select the record with empty Last edited field and click Edit.
3. In the opened Credentials window, click OK (no need to do any modifications).
4. Repeat steps 1 and 2 for all credentials records whose Last edited field is empty.
5. After editing all records, click OK in the Manage Credentials window.

This will update empty Last edited fields and fix the issue. You will be able to add a new AHV cluster.

Challenge

Cause

The following error message can be seen in the Svc.VeeamCatalog.log:

[DD.MM.YYYY HH:MM:SS] < <04> Info Connecting to the enterprise DB.
[DD.MM.YYYY HH:MM:SS] < <04> Error Login failed for user 'DOMAIN\Account'. (System.Data.SqlClient.SqlException)
[DD.MM.YYYY HH:MM:SS] < <04> Info Connection state: [Closed].

Solution

More Information

Challenge

Solution

Implementation step 1 - Backup & Replication 

1. Use a new Windows Server and install Veeam Backup & Replication v10 or newer if you do not have a Veeam Backup Server. The Server can run within any VMware cloud on AWS/DellEMC SDDC, the ENI connected AWS environment or on-premises datacenter environments, if the network connection to the VMC vCenter/Veeam Servers and potentially the VMs (for Guest processing) is possible.
2. Add DNS network settings so that this Server can resolve Internet DNS names to be able to resolve the fully qualified domain name of the VMC vCenter server
3. Check the below information carefully for any known limitations and configuration steps before you proceed.

Implementation step 2 - VMware Cloud

Firewall Configuration for vCenter connection

1. Usage of the local connection for customers with VMware NSX-t (default) 
   NSX-t allows VMC customers to directly access the management network over the built-in firewall. TCP Port 443 needs to be opened from all Veeam Backup and Veeam Proxy Servers as a Source with the vCenter internal IP as a target.
   a) Configure DNS entry of the vCenter for local IP address usage.

   Go to your SDDC Management – Settings – vCenter FQDN and select the Private vCenter IP address.
   Hint: If you configure the vCenter DNS record for the internal IP address, you will lose VMC connection from Backup and Replication Server outside of VMC. You can use the local hosts file or any other DNS method to resolve the vCenter FQDN with the public IP address on the Veeam Server outside of VMC. Optionally, use the Public IP address for the VMC internal and external Veeam Server.

   b) Open firewall ports for vCenter Server access

   On the Management Network
   
   On the Compute Gateway
   

2. Usage of the vCenter public IP for customers with NSX-v
   • Open Port TCP 443 from Backup Server and Proxy Server to the predefined vCenter object on the Compute Network.
   • Allow the Compute Gateway Public IP to communicate over TCP 443 with the predefined vCenter object on the Management Network.
     
3. Usage of a VPN tunnel for customers with VMware NSX-v 
   To be able to directly access the vСenter within VMC, please follow the VMC internal guidelines to create a VPN tunnel from the compute network to the management network: https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.getting-started/GUID-30BED7B3-D312-4DF3-BD7A-66F8D1C619DC.html
   Please update your DNS Servers to resolve the FQDN of the vСenter to its private IP address. If you want to use hosts entries on the Veeam Server for it, add them on all Veeam Backup and Proxy Servers.
   If your Backup & Replication (Management) Server is outside of the VMC cluster, please implement the same VPN connection for it.

Implementation step 3 - add vCenter

Add vCenter to the Veeam console as described here: 
https://helpcenter.veeam.com/docs/backup/vsphere/add_vmware_server.html?ver=95

1. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.

2. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

Implementation step 4 - add Veeam Proxy

For any VMware Cloud on AWS/DellEMC SDDC Cluster, roll out at least one Veeam Proxy Server to be able to process HotAdd / Virtual Appliance Backup Mode. The Backup & Replication itself can be used when installed at the SDDC Cluster (Proxy preinstalled). Please look at the Veeam documentation for details: https://helpcenter.veeam.com/docs/backup/vsphere/add_vmware_proxy.html?ver=95

Veeam Linux based Proxy Server can not be used with VMware Cloud on AWS/DellEMC as they do not enable specific mandatory VMware VDDK settings.

Implementation step 5 - add Veeam Repository

VMware Cloud on AWS/DellEMC has only one accessible vSAN disk. It would not make sense to use that disk for production workloads and backups. An external Backup device needs to be added. Depending on the use case there are several ways to achieve this with different economic factors. For VMware Cloud on DellEMC a Windows or Linux Server plus storage on a separate on-premises environment can be used. Please find below an example of an Amazon S3 EC2 Linux Server (e.g. EC2 C4 Server with EBS ST1 storage) used as a backup target over the VMware Cloud on AWS integrated ENI network connection.



To connect the EC2 Server(s) used as Veeam Repositories the following Firewall configuration is needed:

1. On the Compute Network:

   1. Open TCP 22 (SSH) port from Veeam Backup server and Veeam proxy server to the Amazon VPC where the EC2 Server was installed. You can as well define the exact IP addresses of the repository server as Destination.

   2. Open TCP 2500-5000 ports for Veeam Data Transport in both directions for same servers. It is recommended to use the VMware Cloud on AWS/DellEMC integrated high throughput/low latency ENI network connection to avoid any traffic costs.

    

2. Open the same ports on the Inbound Firewall of the Amazon EC2 server used as a repository server. As the Firewall Rule Source you should add all Veeam Backup Servers (including Proxy/Repository/MountServer/Console/…) instead of 0.0.0.0/0                                                                                                                                                                 

Implementation step 6 - add secondary backup target

1. Veeam Scale out Backup Repository – Archive Tier usage. In copy mode this feature can be used to create additional backup copies at Amazon AWS S3. Please use the internal interface of the S3 target to avoid internet traffic costs.
2. For VMware Cloud on AWS a Veeam Backup Copy Job to a second EC2 Server can be used as an additional Repository. The second EC2 Server can be placed on another AWS Availability Zone or AWS Geo Location.
3. AWS Storage Gateway Software in VTL mode can be used to emulate a Tape Library to write data to S3. Veeam Backup to Tape Jobs can be used with it. For details see: https://www.veeam.com/wp-using-aws-vtl-gateway-deployment-guide.html
4. Veeam Backup Copy Job to on premises or Veeam Cloud Connect (Enterprise). There is no special configuration needed for this use case beside network and firewall connections. For standard Repository usage on premises it is recommended to create a VPN tunnel from VMware Cloud on AWS/DellEMC to the on premises datacenter. This can be done by the VMC integrated VPN functionality, by Veeam PN or Third Party.

1. VMware Cloud on AWS/DellEMC used as Restore target. 
   1. Implementation steps 1-4 are needed.
2. Veeam VM Replication.
   1.  Implementation steps 1-5 are needed. The Repository Server (when NOT used for Backups, can run within the VMware Cloud on AWS/DellEMC SDDC to store the Veeam Replication data. On premises standard datacenter to VMC, VMC to VMC and VMC to on premises standard datacenter is possible. Usage of Veeam Availability Orchestrator is possible in specific scenarios, see VAO deployment guide: https://helpcenter.veeam.com/docs/vao/deployment/welcome.html?ver=10

VMware Cloud (VMC) specific problems and solutions:
 

Problem

Solution

Problem

Solution

1. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
2. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

Problem

Solution

• vsanDatastore datastore
• Management VMs folder
• Mgmt-ResourcePool resource pool

Problem

Solution

1. Please check this article for minimum required Veeam Backup & Replication version or Patches.
2. For customers with Socket based licensing, make sure that the from VMware automatically newly generated ESXi hosts can receive Veeam licenses. Potentially older ESXi hosts need to be revoked from the license. We recommend to use Veeam Universal Licensing to avoid any license specific issues with VMware Cloud on AWS or VMware Cloud on DellEMC. 

Problem

Solution