Quantcast
Channel: Veeam Support Knowledge Base
Viewing all 4477 articles
Browse latest View live

Support statement for Tiger Bridge

$
0
0

Challenge

Using cloud storage gateway software like Tiger Bridge to copy backup data to storage providers like Backblaze and/or reclaim space belonging to copied backups from the primary backup target based on certain thresholds.

Cause

Veeam Backup & Replication architecture and interaction with its backup files requires that all backup file's data remains in the backup repository, and does not support external data movers that offload some or all of backup file's data to another storage.

Veeam Backup & Replication recommends using native mechanism for offloading backup files to object storage, which operates in a significantly different manner:

1. It preserves certain parts of offloaded backup files' data in the backup repository permanently.

2. It passes the awareness of offloaded data blocks to various product logic, which then treats such data blocks in a special manner to ensure reliable operations, acceptable levels of performance, and reduced storage/ingress/egress costs.

Solution

Veeam Customer Support will not support environments where some or all of backup files' data is being offloaded to another storage with 3rd party solutions. For additional information, please refer to the User Guide.

Veeam Customer Support is unable to assist with failed restores from copies of backups made by 3rd party solutions, because Veeam cannot guarantee that the backup files' data was correctly managed by the 3rd party application, and is consistent. This is especially true with object storage, which often provides "eventual consistency" only.

How to configure Veeam B&R to work with replicated volume snapshots on the target Pure Storage array

$
0
0

Challenge

Replicated volume snapshots on the target Pure Storage array are not displayed in the storage system hierarchy in the Veeam Backup & Replication console.

Cause

The Pure Storage Plug-In for Veeam Backup & Replication version is earlier than 1.2.45 or the Veeam.Backup.PublicStoragePlugin.PureStorage.dll.config file is missing or improperly configured.

Solution

Starting from the Pure Storage Plug-In for Veeam Backup & Replication version 1.2.45, you can use replicated volume snapshots on the target Pure Storage array to browse or restore VM data.
To display replicated volume snapshots in the storage system hierarchy:

1. Make sure that the Veeam.Backup.PublicStoragePlugin.PureStorage.dll.config file exists in the plug-in installation folder.

NOTE: By default, the plug-in is installed to the Plugins\Storage\Pure Storage Inc subfolder of the Veeam Backup & Replication installation folder. If the configuration file does not exist, reinstall the plug-in.

2. Make sure that the ExposeShadowVolumes key in the configuration file has the True value:
<configuration>
  <appSettings>
    <add key="ExposeShadowVolumes" value="True"/>
  </appSettings>
</configuration>
If the configuration file exists and the key has the True value, Veeam Backup & Replication will create auxiliary volumes with the VEEAM-ShadowReplica prefix on the target Pure Storage array during the storage system rescan. After that, Veeam Backup & Replication will mount to the auxiliary volumes the replicated volume snapshots and display the snapshots in the storage system hierarchy.

More Information

If you want to hide replicated volume snapshots from the storage system hierarchy, set the ExposeShadowVolumes key in the configuration file to False. Mind that after you change the key value to False, the auxiliary VEEAM-ShadowReplica volumes will actually remain on the target Pure Storage array. Veeam Backup & Replication deletes the auxiliary volumes only when the Pure Storage array is removed from the Veeam backup infrastructure. If you do not plan to remove the Pure Storage array from the Veeam backup infrastructure, you can delete the auxiliary volumes manually in the Pure Storage console. 

“The default account is not initialized” error is displayed when adding an IAM role to Veeam Backup for AWS

$
0
0

Challenge

The Default Backup Restore IAM role is not created upon the Veeam Backup for AWS installation. Attempts to add IAM roles to Veeam Backup for AWS in the Add Account wizard fail with "The default account is not initialized" error.
User-added image

Cause

The backup server cannot access the required AWS services over the internet. Required AWS services are listed in the Requirements section of the Veeam Backup for AWS User Guide.

Solution

1. Make sure the backup server has internet access to the required AWS services. Consult this article for more information. Make sure all the endpoints for the required AWS services specified in this article are reachable from the backup server.
2. After granting the access, log in to the backup server (EC2 instance where Veeam Backup for AWS is installed). Using Git for Windows or similar software, connect to the backup server by running the following command:
ssh -i "<backup server's .pem key>" ubuntu@<Public DNS (IPv4) parameter of the backup server>
3. After establishing the access please run the following commands in order to restart the service:
sudo systemctl restart veeamawsbackup.service
4. The default role should be correctly added automatically after these modifications.
 

Veeam Management Pack 8.0 Update 6 cumulative patch for Veeam Backup & Replication monitoring

$
0
0

Challenge

Veeam Management Pack 8.0 Update 6 cumulative patch for Veeam Backup & Replication monitoring.

Cause

Veeam Support has received quite enough customer feedback on issues related to MP for Veeam Backup & Replication provided as part of Veeam Management Pack for System Center. For example, users frequently encounter the following issues:
  • After upgrade from Veeam Management Pack 8.0 Update 5 to Update 6, some Veeam Backup Reports start displaying incorrect data or the “No relevant data found” message in the output. For example, the ‘Protected VMs’ report considers a number of analyzed VMs to be Unprotected.
  • Some Veeam Backup & Replication views in the Ops Mgr console show incorrect data for a number of Veeam Backup & Replication servers.
  • The ‘Veeam Backup Repository Maintenance Mode Renewal’ monitor tends to change its state from Healthy to Warning once in a while (every 1 hour by default).

Solution

To resolve the listed issues, Veeam R&D released a Veeam Management Pack 8.0 Update 6 cumulative patch for Veeam Backup & Replication monitoring.

To install the cumulative patch:
  1. Check that you have Veeam Management Packs version 8.0.0.2467 installed. To do that, navigate to Administration > Management Packs in the Ops Mgr console.
  2. Download the archive file using the link provided in the ‘More Information’ section.
  3. Unpack the archive.
  4. Import Veeam management packs included in the patch to the Ops Mgr console. To do that, navigate to Administration > Management Packs > Import MPs. For detailed instructions about importing management packs, see this Microsoft KB article.
  5. Ensure that the Veeam Backup Monitoring and Veeam Backup Discovery management packs now have the following build numbers: Veeam Backup Monitoring MP — 8.0.0.2480, Veeam Backup Discovery MP — 8.0.0.2474.

More Information

[[DOWNLOAD|DOWNLOAD CUMULATIVE PATCH|https://www.veeam.com/download_add_packs/vmware-microsoft-esx-monitoring/kb2940/]]

MD5: 142d82fe6921870db3d9f9cb5f3228ab
SHA-1: 1a14ceb5e3cab0e48f4f3e400c579e4af23a9581
SHA-256: c28f45757a30adb520c7bc88a5a74c03d26c2849bcddc1338d934027695036d2


Should you have any questions, contact Veeam Support.

Veeam Backup & Replication - Steps to Compile and Upload Logs

$
0
0

Challenge

If you are a Cloud Provider gathering logs from your infrastructure please see:  https://www.veeam.com/kb2096

This article covers how to properly compile your logs in Veeam Backup & Replication v8.x and v9.x, as well as what to include when submitting a Support case for technical assistance.
 

Solution

The following animation demonstrates how to compile logs in Veeam Backup & Replication.

User-added image

  1. To compile your logs via the log compilation wizard, click the Menu button in the top left-hand corner of the gui and go to Help -> Support Information in the Veeam Backup & Replication console. 
  2. This wizard will allow you various methods to compile logging. For Backup, Replication, and other jobs, select “export logs for this job.” If multiple jobs are affected, you can multiple-select. For problems with restore or replica failover, you should select “Export logs for this VM” and choose the  production VM  that was backed up or replicated. For issues with the GUI or problems that don’t fall into any of the above categories, select “Export all logs for selected components” and choose “This server.”
  3. If possible, compile the logging for each job with issues for at least the last week.
  4. You may save these logs to any location. Ensure that the location referenced has free space for compilation.
  5. When selecting "Next", the wizard will notify of its progress in the compilation process. Please attach these to your case, or in the event that the total attachment size is under 3.5MB, you may attach via email reply. If none of these options for upload are working, please notify Veeam Support and an engineer will provide FTP credentials for the case.


Below is an animation demonstrating how to upload logs to the Veeam Support FTP.

Note: Please use the FTP account provided to you by the engineer for your case. When submitting logging data for analysis by request of an engineer, ensure that the case is replied to when the log submission has completed. In this reply, please include the names of any tasks and associated VMs with issues.

User-added image

Note: SFTP is now supported when uploading to supportftp2, please place files in the \upload\ folder.

More Information

Below are the relative log path lists for Veeam Backup & Replication. If there are difficulties compiling the logging requested via the wizard-based instructions above, please obtain the files from the appropriate directory below and provide Veeam Support a .zip file copy. 

  • Windows 2003, XP – C:\Documents and Settings\All Users\Application Data\Veeam\Backup\
  • Windows Vista, 7 - C:\Users\All Users\Veeam\Backup\
  • Windows 2008/2008 R2/2012 - C:\ProgramData\Veeam\Backup\
  • Linux - /var/log/VeeamBackup/

Typical log files requested for a manual log compilation will be:

  • The task's folder located in the parent directory, zipped. This will contain all relevant job/task/agent data.

  • All logs from the parent directory named svc.*.log, util.*.log, as well as the VeeamBackupManager.log and VeeamShell.log
 

Please note that information provided to Veeam for purposes of customer support may be considered protected under certain data protection regulations and/or internal company policies. A list of potentially sensitive data types is available in the following KB article. The Support team may provide with a log scrubbing utility upon request. Veeam will only utilize this information for the specific purpose of providing customer support to you and will protect the information in accordance with its privacy policy. In any event, the information provided will be securely deleted after 45 inactive days as of the last upload to the FTP. If you would like it deleted sooner, please express this request directly to your support contact once the issue has been resolved. For more information on how we process the collected data, please visit: https://vee.am/processing_of_personal_data

Veeam ONE Database Partitioning

$
0
0

Challenge

Improving Veeam ONE performance in large infrastructures.

Cause

Fine tuning database structure for large infrastructure deployments.

Solution

To improve Veeam ONE SQL database performance, partitioning can be used. Note, that SQL Server Enterprise Edition is required to use partitioning.

NOTE: Do not deploy database partitioning if you use Always On Availability Groups on the SQL server.

Partitioned tables and indexes are created by two scripts. Before running the scripts, it is recommended to create a backup of Veeam ONE database. The scripts can be executed in any order. Before applying them, you will need to specify the following values:

1. Locate %UserName% parameter in the script and specify DB admin (sa or any domain account that has full control permissions to Veeam ONE DB).
2. Locate %Path% parameter and specify directory where database partitions will be created/stored.
3. If Veeam ONE database name is different from default, then change the name of the database (VeeamOne) in both scripts manually
4. Make sure SQL Agents are running before executing the scripts

After successful script execution you should see new functions and partition schemes: 

User-added image

SQL Server agent will have two maintenance jobs created:

User-added image

The directory specified for database partitions will have files like this:

VeeamOneLow_FG1.ndf
...
VeeamOneLow_FG13.ndf

VeeamOneMed_FG1.ndf
...
VeeamOneMed_FG15.ndf

More Information

[[DOWNLOAD|DOWNLOAD CUMULATIVE PATCH|https://www.veeam.com/download_add_packs/virtualization-management-one-solution/kb1954/]]

MD5: 9927E48A91A9EFC74AA1B5BE92F0AFEE
SHA-1: AEB821296438F2A16CFE4C00068C17C07244D2AD
SHA-256: 48CAFE38B60EEB1A71A451564ABB68CCDCA6DA34988B998C7BA48ED14CB11561

Should you have any questions, contact Veeam Support.

Microsoft Windows OS is stuck on boot screen after Veeam CBT driver is installed

$
0
0

Challenge

The Microsoft Windows OS is stuck on the boot screen after the Veeam CBT driver is installed.

Cause

The issue occurs if you have the Veritas System Recovery software installed on the Veeam Agent machine. The Veeam CBT driver is incompatible with the VTrack driver provided by Veritas. 

Solution

Resolve the issue in one of the following ways:
  • When your machine boots, press the F8 key before the Windows logo appears. Select the Last Known Good Configuration option.
  • Remove the Veeam CBT driver with Veeam Recovery Media. For details, see Removing CBT Driver with Veeam Recovery Media.
After the Veeam Agent machine is rebooted, remove the VTrack driver and install the Veeam CBT driver again. For details, see Installing Veeam CBT Driver.

Veeam Agent backup job finished with "Collecting recovery media fails Details: The system cannot find the file specified" warning

$
0
0

Challenge

A Veeam Agent backup job managed by the backup server finishes with a warning:
Collecting recovery media fails Details: The system cannot find the file specified
User-added image
 

Cause

Veeam Agent for Windows runs the Recovery Media collection process at the end of the backup job. The process enables Veeam Agent for Windows to generate the Veeam Recovery Media for the machine.

The Recovery Media collection process is based on the Windows RE/PE components that may be missing. For example: 
  • Starting from Windows 10 version 1809, the WinPE component is an add-on to the Windows Assessment and Deployment Kit (ADK).
  • Veeam Agent machine has a hardware recovery partition instead of the system partition.
As a result, the backup job fails to process the WinRE image and displays the warning.

Solution

Check the status of the Windows RE components with the following command:
  1. Start the command prompt.
  2. Run the command: reagentc /info.
    • If the command returns Enabled, there is a possibility that system might be corrupted. It is worth checking system integrity. Use the dism.log command that may point to a possible cause.
    • If the command returns Disabled, it is most likely that the Windows RE/PE components are not installed or configured on the Veeam Agent machine.
    User-added image
If Windows RE/PE components are not installed or configured on the Veeam Agent machine, fix the Recovery Media collection process as shown here.

If you want to eliminate the warning, exclude the Veeam Agent machine from the Recovery Media collection process. Add the following registry key:
  1. Run regedit.exe.
  2. Locate the HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam EndPoint Backup key.
  3. Add the following value under the key:
    • Name: DisableRECollection
    • Type: REG_DWORD
    • Value: 1
  4. Go to Control Panel > System and Security > Administrative Tools > Services.
  5. Restart the Veeam Agent for Microsoft Windows service. 

After the registry key is created, Veeam Agent for Windows does not start the the Recovery Media collection process at the end of the backup job.


How to Configure Antivirus Exclusions for Veeam Backup for Microsoft Office 365

$
0
0

Challenge

Deployment of required Veeam Backup for Microsoft Office 365 services and executable files fails because of the antivirus software.

Solution

Add the following paths to the exclusion list of your antivirus software on the Veeam controller server:
  • C:\Program Files\Veeam
  • C:\Program Files (x86)\Veeam
  • C:\Program Files\Common Files\Veeam
  • C:\Program Files (x86)\Common Files\Veeam
  • C:\Windows\Veeam
  • %programdata%\Veeam
  • The folder that contains repository databases
Add the following paths to the exclusion list of your antivirus software on the remote backup proxy server:
  • C:\Windows\Veeam
  • %programdata%\Veeam
  • The folder that contains repository databases.

More Information

Should you have any questions, contact Veeam Support.

Deduplication Appliance Best Practices

$
0
0

Challenge

When backing up to a deduplication storage appliance from Veeam Backup & Replication, performance or deduplication ratio is low without using best practice settings. 

Solution

The default options of Veeam Backup & Replication are intended for non-deduplication storage.

If you use deduplication storage systems, you need to additionally configure the backup architecture and Veeam settings to achieve best performance. Below are some hardware and software configuration options that can be used with deduplication storages. In some cases, you may need to combine more than one option, but in all cases, it is strongly encouraged that you consult your deduplication appliance vendor and/or a value added reseller to discuss individual needs.

For the general architecture planning, Veeam’s best practices is to follow the industry 3-2-1 backup rule:

3: Maintain at least three copies of your data and applications. That’s the one copy you’re using and two backups. This way, if one of your backups is unavailable for any reason (for example, if you store the backup in the same locations with the copy you use and both go down), you can still recover what you need.

2: Store your backups on at least two different types of media or storage controller logics. One reason for this is that each type of media has its own vulnerabilities, and you don’t want both of your backups susceptible to the same problem. 

For example: run your business on a general purpose fast storage system and store your backups on a storage system that is specialized for this purpose, or a separate server with local disks.

1: Keep one of the backups in a different location. Consider a crisis in your primary data center, such as a fire or power failure. If all your copies are collocated, they can all be affected, taking down your organization.

To achieve this, use the Veeam best practices. However, each deduplication storage vendor has specific unique feature sets that allow the usage of their storages in different ways. Please check the following links for vendor specific guidelines:

 

General Deduplication Storage Best Practices

 

Step 1:  Creating backups on the first site with long term retention.

Use a backup target storage system (general purpose storage system) for short-term primary backups and instruct Veeam to copy the backups to a deduplication storage system for long-term retention.
User-added image
A variant of this approach is to use a standard server with battery backed RAID controller and disks to store the primary backups (cache approach) and use backup copy to deduplication storage systems for long-term etention.

User-added image

Step 2: Add offsite target.

In addition to the above scenarios, you can use the following options to create an offsite copy of your data.

 
1. Place a deduplication storage at the second site. Use Veeam backup copy job to create the secondary offsite backup from the primary backup.  
User-added image
2. Place a deduplication storage on both sides:  
User-added image
3. Place a deduplication storage on primary side and use object storage or tape on second site. Use Veeam backup to tape jobs or Veeam scale-out backup repository cloud tier (connection to object storages) to store data offsite.
User-added image

The scenario above reflects general best practices. Please contact your deduplication storage vendor for further guidance and check the vendor links provided above for additional usable scenarios with the specific storage.
 

Best Practices For Deduplication Storage Systems

 

Backup Chains


Veeam offers a wide range of backup chains but for deduplication storages the following are recommended:
  • General Deduplication Storages + Backup Copy Job That Reads From Source Backup Chain (Exagrid, Quantum DXi, Infinidat InfiniGuard and Fujitsu Eternus CS800):
To optimize the deduplication storage performance, select at least one GFS restore point (weekly backup is mandatory for deduplication storages). Enable the “Read the entire restore point from source backup instead of synthesizing it from increments” option to avoid creating synthetic restore points from the deduplication storage (data rehydration). 
User-added image 
  •  HPE StoreOnce Catalyst / Dell EMC DataDomain DDboost  + Copy Over WAN + Backup Copy Job:

The previous described option provides fast processing in situations that do not involve transport over WAN links. Also, optimized integration with Catalyst and Dell EMC DDBoost allows using synthetic processing. For these environments uncheck the “Read the entire restore point from source backup instead of synthesizing it from increments” but leave the weekly GFS backup enabled.
User-added image

  • Special Scenarios Based on Vendor Own Guidelines + Backup Job With Active Full or Synthetic Full Backup:
The above vendor best practices and design guides mention situations where a deduplication storage can be used as primary backup target. Depending on the guidelines, select the incremental backup mode with active or synthetic full backup.
User-added image
User-added image

 

Inline Deduplication

Disable inline deduplication setting when writing into deduplication storages.
User-added image
User-added image

However for deduplication devices that have a non deduplicated landing zone (Like ExaGrid) and backup only VMs, you can save some space: to do this, enable inline deduplication

User-added image
User-added image
 

Compression

By default, the Veeam source DataMover (Veeam Proxy or Agent) compression level is set to “optimal” to reduce data that needs to be transported over the network. The repository or the gateway server uncompress the data before storing the data to the deduplication device.

For backup copy jobs, set the comression level to “Auto” to leave the data in the way it was stored originally on the primary backup target.

For deduplication devices that have a non deduplicated landing zone (Like ExaGrid) and backup only VMs, you can save some space: to do this, set the compression level to “Dedup Friendly” and uncheck the “Decompres backup data blocks before storing” option.

The general best practices are: 
User-added image
User-added image
User-added image

 

Block Size

Veeam can use different block sizes to store backups on the target. The block size is defined in the backup job settings and will be kept across all copies. Non-VM backups settings are set to “Local Target” automatically, and the size can not be modified.

Block size changes are only activated by an active full and will only be used for the new backup chains (the data stored already are not touched).

Only 2 of the block size settings are recommended for deduplication storage usage at VM backups:
 
  • Storage optimization: Local target (large blocks)
Use this option for any general deduplication storage / HPE StoreOnce Catalyst / Dell EMC DataDomain DDboost / Quantum DXi. 
User-added image
  • Storage optimization: Local target 

Use this option for ExaGrid.
User-added image

 

Block Alignment

By default, this advanced repository setting is disabled. Enable it for deduplication storages that work with fixed block length deduplication (e.g. NetApp ONTAP deduplication) or in situations where you can use ReFS/XFS Fast Cloning. 
 

Encryption

Encryption will create random data at the backup targets; as a result, the deduplication storages will not work effectively. It is recommended not to use encryption with deduplication storages.
 

Health Check

Health check reads all data from the last restore point and across the backup chain. When used with deduplication storages, this data is rehydrated from the storage which may cause slow process. It is recommended to enable health check on the primary backup job before data is transported by backup copy job to a deduplication storage.

  
Veeam Defragmentation And Compact Settings

If you followed the above described guidelines, you do not need these settings. It is recommended that you disable them.

More Information


  

ESXi firewall changes required to process a Veeam backup from storage snapshot with Cisco HyperFlex IOvisor

$
0
0

Challenge

To process a Veeam backup from storage snapshot with Cisco HyperFlex IOvisor, you have to change the ESXi firewall settings. The process outlined below applies to all HyperFlex clusters running HX Data Platform 3.0 or later.

Solution

Before you add the HyperFlex Cluster into the Veeam Backup & Replication Console, perform the following steps to leverage the Cisco HyperFlex IOvisor for backup processing (recommended).
  1. Navigate to the ESXi host Firewall screen within the vSphere client under <hostname>/Configure/System/Firewall. Click Edit.

image-20191223123336-1.png

  1. Under Edit Security Profile, use the “NFS” filter. Select NFSAccess to modify settings.

1577100838604-913.png

  1. Enter IP addresses of all proxies that will be used to back up data from the HyperFlex Cluster. Once complete, click OK.

1577195830141-760.png

  1. Repeat steps 1 through 3 for all ESXi hosts within the HyperFlex cluster.

Warning: "User does not have a valid Microsoft Office 365 license with SharePoint plan enabled."

$
0
0

Challenge

Microsoft OneDrive for Business or Microsoft SharePoint backups may not be backed up, resulting in a warning: User does not have a valid Microsoft Office 365 license with SharePoint plan enabled.

Solution

To resolve this issue, add missing Plans IDs to the list according to the following steps:
  1. Get the current Office 365 plan IDs assigned for the organization. You can do this via ‘MSOnline’ AzureAD module and the following PowerShell One-Liners: *
    Note: This script is provided as-is as a courtesy for Gathering Plan IDs from MSOnline. Please be aware that the commands and parameters may be altered by Microsoft in the future, which may cause the script to function incorrectly. There is no support provided for this script, and should it fail, we ask you to contact Microsoft support. This script may be updated in the future to reflect changes to the Office 365 Powershell Environment
    1. To call "get-msolaccountsku" (use the CMDLET), call "Connect-MsolService" first. This requires the following modules to be installed and activated:
      • install-module AzureAD
      • install-module AzureADPreview
      • install-module MSOnline
    2. To get the list of Exchange Online service plans IDs:
      • (get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "Exchange"} | ft ServiceName,ServicePlanId
    3. To get the list of SharePoint Online service plans IDs:
      • (get-msolaccountsku).servicestatus.serviceplan | where {$_.ServiceType -match "SharePoint"} | ft ServiceName,ServicePlanId
  2. Make sure that no backup jobs or restore sessions are running.
  3. Stop the Veeam.Archiver.Service and Veeam.Archiver.Proxy services.
  4. Navigate to the C:\ProgramData\Veeam\Backup365 directory on the Veeam Backup for Microsoft Office 365 server, create a backup of the Congif.xml file by copying it to another location, then (only after making the pre-edit copy) edit the Config.xml file by adding the following lines between <Archiver> tags:
  5. Repeat Step 4 but for the Proxy.xml file instead.
<Options>
<O365Plans AdditionalExchangePlans="your-Exchange-plans-IDs" AdditionalSharePointPlans="your-SharePoint-plans-IDs" />
</Options>

If you do not have any SharePoint plans, do not add the AdditionalSharePointPlans to the line mentioned

NOTE: You will have to add the Plans as the examples shown in the following screenshot. The GUIDs are entered in Coma Separated Value within the Quotation after each of the key values in the <O365Plans line.

  1. Start the Veeam.Archiver.Service and the Veeam.Archiver.Proxy services.
  2. Open the Add organization wizard and retry the procedure.

Veeam Agent backup job fails with "Failed to call RPC function 'TestCompatible': Error code: 0x80070008. Cannot initialize COM runtime" error

$
0
0

Challenge

A Veeam Agent backup job managed by the backup server fails with the following error:

Processing Error: Failed to call RPC function 'TestCompatible': Error code: 0x80070008. Cannot initialize COM runtime.

User-added image

Cause

When a backup job starts, Veeam Backup & Replication connects to the Veeam Agent machine to initiate the backup process. The connection is based on Windows API and uses RPC and WMI queries. If Microsoft Windows is low on desktop heap memory for services running under LocalSystem account, Microsoft Windows can reject the incoming queries. As a result, the following message is displayed in System Event Log of the Veeam Agent machine:
 
Log Name:      System
Source:        Win32k
Event ID:      243
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:     
Description:
A desktop heap allocation failed.
DCOM errors (Event ID 10010) can be found in System Event Log of the Veeam Agent machine too.
 

Solution

To resolve this issue, update the registry key to increase the desktop heap memory size. Mind that this change affects the desktop heap of all services, so do not make the value larger than necessary. To update the registry key, do as follows:

  1. Run regedit.exe.
  2. Locate the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems key.
  3. Locate the Windows string.

  1. Edit the third "SharedSection" value in the Value data field as shown below:

       Before:

        After:

  1. Click OK.
  2. Reboot the Veeam Agent machine.

More Information

To learn more about desktop heap memory, see:

You can check the desktop heap memory usage on your own in the following way:

  1. Download Windows Software Development Kit (SDK).
  2. Run Sdksetup.exe and install Debugging Tools for Windows.
  3. Download LiveKd.
  4. Copy livekd.exe to C:\Program Files (x86)\Windows Kits\10\Debuggers\x64.
  5. Run Command Prompt.
  6. Type the following commands:
livekd -y srv*http://msdl.microsoft.com/download/symbols
!dskheap

Command Prompt returns the list of sessions. If the desktop heap is running out of memory, the Used Rate(%) value for the session is close to 100%.

0: kd> !dskheap

  Winstation\Desktop            Heap Size(KB)   Used Rate(%)
------------------------------------------------------------
  WinSta0\Default                  20480                 0%
  WinSta0\Disconnect                  96                 4%
  WinSta0\Winlogon                   192                 2%
  Service-0x0-3e7$\Default           768                 4%
  Service-0x0-3e4$\Default           768                 1%
  Service-0x0-3e5$\Default           768                 0%
  Service-0x0-17eac$\Default         768                 2%
  Service-0x0-afb53$\Default         768                 0%
  1. Take the last part of the session name and convert the HEX value into Decimal:
    1. Open PowerShell.
    2. Paste the last part of the session name (after the last dash).
    3. Add 0x before the last part of the session name.

  1. Return to Command Prompt.
  2. Type the following WMI query to retrieve all objects related to the session and the name of the account:
Get-WmiObject -Query "ASSOCIATORS OF {Win32_logonsession.LogonId=999}"

Command Prompt returns the list of processes running under that account and the name of the account:

Caption                    Name                       SID
-------                    ----                       ---
SUP-A2025\SYSTEM           SYSTEM                     S-1-5-18
LsaIso.exe                 LsaIso.exe
lsass.exe                  lsass.exe
svchost.exe                svchost.exe
…

If one of the processes running under Local System Account consumes too much memory, we recommend to kill non-Microsoft processes one by one and re-run the !dskheap command (Step 6) to find the leaking application.

Backing Up and Replicating the Veeam Server and Database: Preparation and Considerations for Recovery and Failover

$
0
0

Challenge

When Backing up or Replicating the Veeam Server, some considerations must be taken before beginning the operations.

Cause

Depending on the settings of the Backup and Replication jobs, the backups and/or replicas containing the Veeam Database may not be consistent with changes made to the production database since the last run of the jobs. Should recovery or failover be required, you may require Veeam Support's assistance before the servers can take over normal operations.

Solution

Preparation: The server hosting the Veeam Database should be processed in isolation from other jobs, ideally when no other data is processing. The server hosting the Veeam Database should have sufficient resources to ensure that VSS processing is fast and does not cause interruptions.

Please note that if the server hosting the Veeam Database is backed up or replicated as part of a job containing other machines, processing of these machines may be delayed until the server hosting the Veeam Database finishes processing. The server hosting the Veeam Database is not processed in parallel with other machines in order to ensure backup integrity.

On Restore or Failover: It is recommended that you rely primarily on the built in Configuration Backup as a means for backing up and recovering the Veeam Configuration. This will preserve the current state of the Veeam Database and all job and infrastructure related configurations within Veeam.

If the Configuration Backup was taken more recently than the last backup or replica, be sure to Restore from the Configuration Backup to ensure that the most recent state of the Veeam Database is used.
You may need to contact Veeam Support for assistance should you experience any difficulties when restoring from a backup or failing over to a replica of the Veeam Server/Database.
 

Sensitive data types in Veeam Backup & Replication log files

$
0
0

Challenge

When opening a support ticket, Veeam Support might request log files.

Cause

These log files might potentially contain information which might fall under the regulation of data protection laws or internal company policies.

Solution

The below are the types of sensitive information that can be found in the Veeam log files:
  • Usernames;
  • Customer specific infrastructure object names: hosts, datastores, VMs, clusters;
  • VM file names and paths;
  • FQDN/Hostname or NetBIOS names;
  • IP addresses: IPv4 and IPv6;
  • Customer specific paths to backup files;
  • Backup file names;
  • SharePoint information: DB names, paths to DBs, SP URL;
  • Exchange information:  DB names, paths to DBs;
  • SQL information: DB and Instance Name, paths to DBs;
  • Oracle information: DB name, Home Name, Oracle SID, DB global name, paths to DBs;
  • File Copy: Customer specific paths and names of files;
  • File to tape: Customer specific paths and names of files.
The Support team may provide a log scrubbing utility upon request.

More Information

For more information on how we process the collected data, please visit: https://vee.am/processing_of_personal_data

 

Exploring VBO365 backups: Understanding Different Restore Scopes

$
0
0

Challenge

You can explore backups in three different scopes: Backup JobOrganizationAll organizations. Consider the following organizations added to the Veeam Backup for Microsoft Office 365 backup infrastructure; each of these organizations uses its own backup repository to store data:
  • Organization A that has one active backup job (Backup Job 2), and a backup job that has been deleted at some point (Backup Job 1).
  • Organization B that has two active backup jobs — Backup Job 3 and Backup Job 4.

Each of these jobs has several restore points in total (deletion of a backup job does not delete the corresponding restore points from the database).

Solution

Exploring Backups: Job Scope

To explore backups created by the selected backup job, right-click a backup job and select any Explore option.This method finds the specified restore point and loads it into the Veeam Explorer scope.
Consider, however, that such a method cannot be used for deleted backup jobs since they are not present in the console; use PowerShell or REST API to load restore points of deleted backup jobs instead.
User-added image

Exploring Backups: Organization Scope

To explore backups created by all backup jobs of the selected organization, right-click an organization and select any Explore option.

When exploring backups of any of the organizations, Veeam Backup for Microsoft Office 365 works as follows:

  • Scans all backup repositories in the backup infrastructure and finds all restore points that have been created by both existing and deleted backup jobs.
  • Finds restore points that were created on the specified date or on the closest earlier date.
  • Merges found restore points into a single one according to the following:
    • If there are two or more copies of the same item in several restore points, then only one copy will be loaded into the Veeam Explorer scope.
    • If there are two different versions of the same item, then only the latest one will be loaded into the Veeam Explorer scope.
    • Otherwise, all unique versions of all items will be loaded into the Veeam Explorer scope.

Note: The same method is used during Veeam Cloud Connect (BAAS) restores.

Exploring Backups: All Organizations Scope

To explore backups created for each of the organizations ever added to the backup infrastructure, right-click the root Organizations node and select any Explore option.

This method works in the same way as the previous one, except for it applies to each of the organizations added to the application scope.

Script to create active full backups on Veeam Availablity for Nutanix AHV 1.0

$
0
0

Challenge

Veeam Availability for Nutanix AHV 1.0 does not support scheduled active full backups. The feature will be implemented in the upcoming release. Until then, you can use the script described below.

Solution

To run the script that creates an active full backup for Veeam Availability for Nutanix AHV, do the following:
  1. Click Download Script at the bottom of this KB article to download the .zip archive with the script.
  2. Transfer the file to the Veeam Availability for Nutanix AHV (proxy appliance) server. Note that SSH is disabled on the proxy appliance server by default. To see how to enable SSH/SFTP, see the Security Considerations section of the Veeam Availability for Nutanix AHV User Guide.
  3. Log in to the shell terminal of the proxy appliance server.
  4. Install the script dependencies (make sure the backup proxy has an internet connection):
    $ sudo apt-get install python-requests
  5. Unpack the .zip archive using the unzip command:
    	$ unzip <the_zip_file_with-script>
  6. Make the script file executable:
    	$ chmod +x run_active_full.py
  7. Run the script:
$ ./run_active_full.py -j "backup_policy_name" -a backup_appliance_ip_address_or_hostname -u admin -p "password_of_admin_user_for_web_interface"

Where:

  • "backup_policy_name" is a backup job name in quotes.
  • backup_appliance_ip_address_or_hostname - the proxy appliance IP address or hostname. Make sure the hostname is resolved to an external IP address.
  • "password_of_admin_user_for_web_interface" - the password for admin user in quotes.

For example:

$ ./run_active_full.py -j "Backup policy" -a ahvproxy.local -u admin -p "W3bP@ssw0rd"

Use crontab to run the script on a defined schedule.

More Information

[[DOWNLOAD|DOWNLOAD SCRIPT|https://storage.veeam.com/tmp/File_03079_5cec1f3554.zip]]

MD5SUM - 6015e776ed316066bc6ea357ea5e62f5

Should you have any questions, contact Veeam Support.

HCL - Datera Data Services Platform

$
0
0

Challenge

VeeamReadyRepo

Product Information:

Company name: Datera Inc.
Product Family: Datera Data Services PlatformDCP (software platform)
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

*This Veeam Ready test was performed with storage having a configuration that includes SSD/Flash. As such, this classification only applies to similar configurations in which the same amount or more SSD drives are used. The use of non-SSD drives may negatively impact performance.

Solution

Product Details:

Model number: DCP (software platform)
Storage Category: Software Defined Storage
Drive quantity, size, type: 12 – 7TB HDD, 11 – 1.5TB NVMe across three nodes
Storage configuration: 4 node Datera cluster with 1 All Flash Node (SSD) and 3 Hybrid nodes, caching using NV-DIMM
Firmware version: V3.3.4
Connection protocol and speed: 10 GbE iSCSI
Additional support: All models and configurations of Datera Data Services Platform with specifications equivalent or greater than the above

General product family overview:

Datera enables commodity servers from many different vendors, to form software defined storage clusters, which deliver Tier 1 block (iSCSI) and Object Storage. The Datera platform enables bare metal, VM and container workloads to operate concurrently, within a single cluster. The clusters can scale to 64 nodes and may be comprised of a heterogeneous mix of Persistent Storage Class media, such as Intel Optane or Samsung’s Z-SSD, as well as NVMe, SAS and STA Flash. The platform supports rotational media as well, in hybrid nodes.

 

Veeam testing configuration:

Note: The following settings were used by the vendor to meet Veeam Ready testing requirements and should not be considered best practices. Additional changes or settings may be needed to meet the storage efficiency or performance needs for each environment. For each setting, reference links are provided for further clarification.
 

Veeam Build Number: 9.5.4.2866

Job Settings:

Deduplication: Disabled
Compression: Optimal (Default)
Storage Optimization: Local Target (Default)
 

Repository Settings:

Repository Type: Windows
Align backup file blocks: Disabled (Default)
Decompress before storing: Enabled
Per-VM Backup Files: Enabled
 

Vendor recommended configuration:

Hardware Settings:

  • array deduplication used in testing
  • array compression used in testing
  • Jumbo frames used in testing (MTU=9000)
  • Datera default storage settings were used for testing
  • Multi-pathing used during testing (MPIO)

Backup Policy IAM Role

$
0
0

Challenge

Backup Policy IAM role is used to run backup operations which include:
  • enumerating of the resources 
  • taking EBS snapshots of selected EC2 instances volumes
  • rehydrating of snapshots back to volumes
  • attaching volumes to the worker instance
  • and so on - see permissions below

Solution

There are two types of deployments where this role is required to perform the backup: backup inside production or isolated from production backup.
The isolated from production deployment is delivering exceptional level of protection when the whole production environment is lost. Veeam is recommending this type of the deployment as the most reliable one.


Isolated from production backup deployment


For this type of the deployment a second "Backup account" is required. Please create one using AWS web site. Veeam Backup for AWS Management appliance should be deployed in the Backup Account.
To allow Veeam Backup appliance to perform all required for backup operations inside of the Production Account, a Backup Policy Role should be created in the Production Account. This role should be made available for usage from the Backup Account. Please proceed through the following steps to create such role:

1. Please use the following JSON text to create IAM Policy by following instructions from How to create IAM Policy article 
    These permissions will allow backup service to perform required for backup operations:
{
	"Version": "2012-10-17",
		"Statement": [
		{
			"Action": [
				"ec2:CopySnapshot",
				"ec2:CreateSnapshot",
				"ec2:CreateSnapshots",
				"ec2:CreateTags",
				"ec2:DeleteSnapshot",
				"ec2:DescribeAvailabilityZones",
				"ec2:DescribeImages",
				"ec2:DescribeInstances",
				"ec2:DescribeSnapshots",
				"ec2:DescribeSubnets",
				"ec2:DescribeTags",
				"ec2:DescribeVolumes",
				"ec2:ModifySnapshotAttribute",
				"events:DeleteRule",
				"events:DescribeRule",
				"events:ListTargetsByRule",
				"events:PutRule",
				"events:PutTargets",
				"events:RemoveTargets",
				"iam:GetRole",
				"iam:ListAttachedRolePolicies",
				"iam:ListInstanceProfilesForRole",
				"iam:ListRolePolicies",
				"iam:ListRoleTags",
				"iam:UpdateAssumeRolePolicy",
				"s3:GetBucketLocation",
				"sns:CreateTopic",
				"sns:DeleteTopic",
				"sns:ListSubscriptionsByTopic",
				"sns:ListTopics",
				"sns:SetTopicAttributes",
				"sns:Subscribe",
				"sns:Unsubscribe",
				"sqs:CreateQueue",
				"sqs:DeleteQueue",
				"sqs:SetQueueAttributes",
				"sts:AssumeRole",
				"sts:GetCallerIdentity"
				],
				"Resource": "*",
				"Effect": "Allow"
		}
	]
}

2. Navigate to Roles.
3. Choose "Create role".
4. Select type of trusted entity "Another AWS Service".
5. In the "Account ID" enter the ID of your second Backup Account (you can obtain this number if you login in AWS console of the Backup Account and select "My account" in top right menu).
6. Select the checkbox "Require external ID" and enter a pass phrase to raise level of security for the role.
7. Press "Next: Permissions" button.
8. Enter the name of the policy selected on step 1 in "filter policies" edit box.
9. Select the policy with ticking on the checkbox in the first column.
10. Press "Next: Tags" button.
11. Enter tagging info if desired and press "Next: Review" button.
12. Give a name to the IAM Role - you will be using this name in the Veeam Backup for AWS UI (e.g. vb4aws_pol2role).
13. Press "Create role" button.
14. After successful creation of the role you will be able to see it in the list of all available roles.


Backup inside production deployment 


NOTE: this type of deployment (since it is not very secure) is recommended for test, demo and small environments only!

1. Please use the following JSON text to create IAM Policy by following instructions from How to create IAM Policy article 
    These permissions will allow backup service to perform required for backup operations:
{
	"Version": "2012-10-17",
		"Statement": [
		{
			"Action": [
				"ec2:CopySnapshot",
				"ec2:CreateSnapshot",
				"ec2:CreateSnapshots",
				"ec2:CreateTags",
				"ec2:DeleteSnapshot",
				"ec2:DescribeAvailabilityZones",
				"ec2:DescribeImages",
				"ec2:DescribeInstances",
				"ec2:DescribeSnapshots",
				"ec2:DescribeSubnets",
				"ec2:DescribeTags",
				"ec2:DescribeVolumes",
				"ec2:ModifySnapshotAttribute",
				"events:DeleteRule",
				"events:DescribeRule",
				"events:ListTargetsByRule",
				"events:PutRule",
				"events:PutTargets",
				"events:RemoveTargets",
				"iam:GetRole",
				"iam:ListAttachedRolePolicies",
				"iam:ListInstanceProfilesForRole",
				"iam:ListRolePolicies",
				"iam:ListRoleTags",
				"iam:UpdateAssumeRolePolicy",
				"s3:GetBucketLocation",
				"sns:CreateTopic",
				"sns:DeleteTopic",
				"sns:ListSubscriptionsByTopic",
				"sns:ListTopics",
				"sns:SetTopicAttributes",
				"sns:Subscribe",
				"sns:Unsubscribe",
				"sqs:CreateQueue",
				"sqs:DeleteQueue",
				"sqs:SetQueueAttributes",
				"sts:AssumeRole",
				"sts:GetCallerIdentity"
				],
				"Resource": "*",
				"Effect": "Allow"
		}
	]
}
2. Navigate to Roles.
3. Choose "Create role".
4. Select type of trusted entity "AWS Service".
5. Choose "EC2" in "Choose the service that will use this role".
6. Press "Next: Permissions" button.
7. Enter the name of the policy selected on step 1 in "filter policies" edit box.
8. Select the policy with ticking on the checkbox in the first column.
9. Press "Next: Tags" button.
10. Enter tagging info if desired and press "Next: Review" button.
11. Give a name to the IAM Role - you will be using this name in the Veeam Backup for AWS UI (e.g. vb4aws_pol2role).
12. Press "Create role" button.
13. After successful creation of the role you will be able to see it in the list of all available roles.

Release Notes for Veeam Backup & Replication 9.5 Update 4b

$
0
0

Challenge

Release Notes for Veeam Backup & Replication 9.5 Update 4b

Cause

Please confirm that you are running version 9.0 Update 2 (build 9.0.0.1715) or later prior to installing this update. You can check this under Help | About in Veeam Backup & Replication console. After the successful upgrade, your build number will be 9.5.4.2866.

When upgrading from versions earlier than Update 4a, you will also see the vendor name changed from Veeam Software AG to Veeam Software Group GmbH, reflecting the legal company name change. Please note that the name change has also resulted in a change of the code signing certificate.

Solution

Veeam Backup & Replication 9.5 Update 4b is a cumulative hotfix rollup that addresses issues reported by customers on the original build of Update 4 and Update 4a, and adds the following new features and enhancements:

Platform support
  • Microsoft Windows 10 May 2019 Update and Microsoft Windows Server version 1903 support as guest VMs, and for the installation of Veeam Backup & Replication and its components.
  • Microsoft Windows 10 May 2019 Update and Microsoft Windows Server version 1903 support by the Veeam Agent for Windows 3.0.2 (included in the update).
  • Linux Kernel version 5.0 support by the updated Veeam Agent for Linux 3.0.2 (included in the update).
  • Ubuntu 19.04, SLES 15 SP1, RHEL 8 and openSUSE Leap 15.1 distributions support in the Veeam Backup & Replication agent management functionality.
  • VMware vCloud Director 9.7 compatibility at the existing Update 4 feature level.
  • VMware vSphere 6.5 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. UPDATE: vSphere 6.5 U3 GA is officially supported with Update 4b.
  • VMware vSphere 6.7 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. UPDATE: vSphere 6.7 U3 GA is officially supported with Update 4b.
Application support
  • Oracle 19c on Linux support for RMAN-based and OCI-based application-aware processing, redo log backup and Veeam Explorer for Oracle in most configurations. Due to the recent availability of Oracle 19c, testing of DataGuard and RAC configurations is still being performed. UPDATE: Oracle 19c on Linux is officially supported with Update 4b.
  • Oracle 19c on Windows support for RMAN-based and OCI-based application-aware processing, redo log backup and Veeam Explorer for Oracle in basic configurations only. Due to the most recent availability of Oracle 19c for Windows, testing of advanced configurations (DataGuard, RAC, ASM, Pluggable DB, flashback, virtual accounts) is still to be performed. UPDATE: Oracle 19c on Windows is officially supported with Update 4b.
Cloud integrations
  • Veeam Backup & Replication will now periodically update the list of supported Amazon AWS regions from the Veeam update server. The list will be updated as Veeam QC validates compatibility with the newly added Amazon region. This functionality requires that you opt in for automatic product update check via the existing Check for product and hypervisor updates periodically check box in the Notification settings.
Storage integrations
  • Dell EMC Unity OE version 4.5 support.
  • The proprietary NFS client has been updated for compatibility with the Western Digital IntelliFlash (formerly Tegile) Universal Storage API plug-in. The plug-in will be made generally available along with the Update 4b release.
In addition, Update 4b addresses over 100 bugs reported by customers and found during the internal testing, including the following common support issues:
  • Tape virtual full backup reliability and performance, tape drive detection logic, WORM media support and NDMP backup reliability.
  • Agent-based backup jobs may create unnecessary full backups under certain circumstances, or fail in the environments with very long backup repository paths.
  • Scale-out backup repository race condition causes synthetic backup operations to dead lock when a very powerful backup proxy server is used.
  • PowerShell Direct calls may hang under certain circumstances in the application-aware processing logic of Hyper-V backup.
  • Veeam Agent for Windows: CBT driver block map reset issue; inability to failover to backup cache when it is hosted on a USB storage; inability to open a support case from the product UI under a restricted user account; improved command line tool for the Recovery Media creation.
  • Veeam Agent for Linux: Dell servers with NUMA enabled may crash during the snapshot creation; intermittent backup jobs failures when processing XFS volumes; entire computer backup failure on Ubuntu 18.04 with Intel Software RAID; enhanced encrypted devices compatibility through redesigned device enumerator; CLI improvements for backup scope specification.
  • NetApp ONTAP Snapvault retention processing in 7-mode clusters.
  • Performance and resource usage optimizations in a number of backup infrastructure components.
Update 4b is a cumulative update, meaning it includes all enhancements delivered as a part of Update 4a. For the detailed information on new features and enhancements included in the original release of Update 4, please refer to this What’s New document.

Veeam Cloud & Service Provider Partners

Veeam Backup & Replication 9.5 Update 4b also includes bug fixes for service providers offering Veeam Cloud Connect services, and managing backup servers with Veeam Availability Console. For the full change log, please see this topic on the private Veeam Cloud & Service Provider (VCSP) forum. Not a member of the VCSP forum? Click here to join. Had access before, but lost it? Click here to more info.

Veeam Cloud Connect service providers: please note that Update 4b is not a "breaking" update, meaning you don't have to upgrade to one before your tenants do.

More Information

If you have 9.5 Update 4 installed (RTM build 9.5.4.2399, GA build 9.5.4.2615, or Update 4a build 9.5.4.2753), you can upgrade to Update 4b using the following update package:

[[DOWNLOAD|DOWNLOAD UPDATE|https://www.veeam.com/download_add_packs/vmware-esx-backup/update4b_exe/]]
MD5: 34d20a0f21bdee7a2e639e7bf332e993
SHA1: 5254b2130bc76aebb2e74e926a73a4bee36058cd


For new installations and upgrades from the previous versions, use the 9.5 Update 4b ISO image:

[[DOWNLOAD|DOWNLOAD ISO|https://www.veeam.com/download_add_packs/vmware-esx-backup/update4b_iso/]]
MD5: 302e41d90862728efb372518a053807d
SHA1: dd7bb3a519ea886bea280b591f4fe49eeaed5e20

Should you have any questions at all, please contact Veeam Support and we'll be happy to assist you with the upgrade.

Viewing all 4477 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>