Challenge

Product Information:

Company name: Fujitsu Limited
Product Family: ETERNUS LT
Status: Veeam Ready - Tape
Classification Description: Tape solution where available hardware features have been tested to work with Veeam.

Solution

Product Details:

Model number: LT140
Library Firmware Version 1.0.0-D000
Drive firmware version: HB83
Driver for tape drive: IBM Ultrium 8 HH Tape Drive
Driver for media changer: Fujitsu ETERNUS media changer
Media type tested: LTO8

General product family overview:
The Fujitsu Storage ETERNUS LT140 is a cost-effective tape library providing extreme expansion flexibility up to 8400 TB. Starts with a 3U base unit and 20 slots while the pay-as-you-grow concept provides high investment protection that enables to scale and address long-term data storage requirements. Highly automated, simple and remote operation allows ease of use even at remote sites. The standardized LTO technology features high capacity, high speed and low cost that provide hardware-based data encryption offering enhanced security and compliance.

Veeam testing configuration:

Veeam Build Number: 9.5.4.2615

Challenge

You need to remove an existing Veeam Backup and Replication (VBR), Veeam Agent for Windows (VAW) or Veeam Agent for Linux (VAL) license from Veeam Backup and Replication (VBR) or Enterprise Manager (EM).

Cause

A license has to be removed from VBR/EM.

Solution

NOTE: Starting from Veeam Backup & Replication Update 4 it is no longer possible to remove a license from Enterprise Manager or delete individual product sections (vbr,vaw,val) from the license.


In order to determine which section of this article to use, you must know if Enterprise Manager is controlling the licenses within Veeam Backup & Replication. If you are unsure, attempting to change the license file within Veeam Backup & Replication will tell you. If it allows you to click the Install button within the license section without a message about Enterprise Manager then Enterprise Manager is not in use.

If licenses are managed by Enterprise Manager:

1. Open an administrative command prompt.
2. Navigate to the EM installation folder (default path in the example below):

   cd C:\Program Files\Veeam\Backup and Replication\Enterprise Manager

3. Remove the license by running

   Veeam.Backup.EnterpriseService.exe -removelicense [vaw, val]

4. After running this command, restart Veeam Backup Enterprise Manager service so that the license information is updated in the UI.

NOTE: Only for versions 9.5 Update 4 and newer

There is no way to remove a license from Enterprise Manager, because Enterprise Manager is not operational without a license. If Enterprise Manager is installed on the same machine next to Veeam Backup & Replication, please contact Technical Support for assistance.

If licenses are managed by Veeam Backup & Replication

1. Open an administrative command prompt.
2. Navigate to the VBR installation folder (default path in the example below):

   cd C:\Program Files\Veeam\Backup and Replication\Backup

3. Remove the license by running:
   For versions prior 9.5 Update 4:

   Veeam.Backup.Manager.exe -removelicense [all, vbr, vaw, val]

   For version 9.5 Update 4 and newer:

   Veeam.Backup.Manager.exe -removelicense

More Information

Note: These commands can also be run from PowerShell. Specify the desired parameter without brackets.

Example commands:

• To remove the Veeam Agent for Windows license from Veeam Enterprise Manager:

  Veeam.Backup.EnterpriseService.exe -removelicense vaw

• To remove Veeam Agent for Linux from Veeam Backup & Replication:

  Veeam.Backup.Manager.exe -removelicense val

If no parameter is specified, all existing licenses will be removed.

Note for Veeam Backup & Replication Update 4:
There is no way to re-install a legacy license after it has been removed. Please consider obtaining a new Veeam Instance Based license.

Challenge

Job may report warning: "Failed to finalize guest processing. Details: Failed to process 'TruncateSQLLog' command. Failed to truncate transaction logs for SQL instances: <instance name>. Possible reasons: lack of permissions, or transaction log corruption."

Solution

SQL logs truncation is done under user account specified in AAIP in Job settings, if it fails then GuestHelper tries to truncate transaction logs under LocalSystem account.

In order to understand why SQL logs truncation failed you will need to open the GuestHelper log on the Guest VM, and search for "Truncation Statistics".

• Windows 2008 or higher
  \\GUESTVM\c$\ProgramData\Veeam\Backup\VeeamGuestHelper_%date%.log
• Windows 2003
  \\GUESTVM\c$\Documents and Settings\All Users\Application Data\Veeam\Backup\VeeamGuestHelper_%date%.log

Known Errors and Solutions

1. Error: OpenFromInitializationString failed. [Login failed for 'DOMAIN\user'.]
   Solution: give DOMAIN\user permissions on SQL instance and add db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.
    
2. OLEDB Error: 'The server principal "DOMAIN\user" is not able to access the database "DATABASE" under the current security context.', HelpCtx: '0'
   Solution: give DOMAIN\user db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.
    
3. OLEDB Error: 'BACKUP detected corruption in the database log. Check the error log for more information.', HelpCtx: '0'
   Solution: error points to possible corruption and issues with SQL server
    
4. OLEDB Error: 'BACKUP LOG cannot be performed because there is no current database backup.'
   
   As a rule this is an issue with the secondary node of the SQL always on cluster. You can solve this by making a backup of the DB in question via SQL Management Studio. Otherwise, you can set the secondary node as primary for just one run of your backup job. As a result all its DBs will be backed up without "copy only" flag and the error will disappear.
   
   The issue occurs when the secondary node has always been backed up with "copy only" flag and its standalone DBs do not have any full backup. Thus during the truncation of the standalone DB logs we get the above-mentioned message.
   
   The same solution applies if you get this message with regard to the excluded vCenter database / Veeam database.
    
5. "Query timeout expired" If you see this entry in VeeamGuestHelper log, it usually means that we couldn't truncate SQL logs in allotted time (by default timeout is only 60 seconds). Usually you might experience such issues with rather large databases, and with large amount of transaction logs
   Solution: Implement the following registry value in affected VMs in [KLM\SOFTWARE\Veeam\Veeam Backup and Replication\] and [ HKLM\SOFTWARE\Wow6432Node\VeeaM\Veeam Backup and Replication] (if either location does not exist, create it):
   • SqlExecTimeout
   • Type: REG_DWORD
   • Default value: 60 (in seconds, decimal)
   Try to expand that value and run a backup afterwards, safe guess here is to set it for 600 seconds.

More Information

If you observe the following warning "Failed to truncate transaction logs for SQL instances: MICROSOFT WID" on Veeam B&R version 8.0.0.2084, please contact Veeam Support for the hot-fix.

Challenge

Product Information:

Company name: NetApp
Product Family: FAS
Status: Veeam Ready - Integrated
Classification Description: Integrated storage where joint development activities between the manufacturer and Veeam have occurred to create advanced backup or restore functionalities.

Solution

Product Details:

Model number: FAS2520
Drive quantity, size, type: 12 - Mix of Flash, SAS, NL-SAS
Firmware version: 8.1.0 - 8.3.2
Connection protocol and speed: 10Gb
Additional support: Any array running Data ONTAP with firmware versions above.

Whatever the size of your enterprise, you need powerful, flexible storage that manages growth and maximizes the long-term value of your IT investment. The NetApp® FAS series running ONTAP®software simplifies storage operations and builds a foundation for a data fabric that flows data seamlessly across flash, disk, and cloud. Its unified architecture supports a broader set of SAN and NAS workloads—with faster setup than other entry-class hybrid arrays.

Veeam testing configuration:

Veeam Build Number: 9.0.0.1715

Challenge

One or more bad blocks were detected and skipped

Cause

Solution

chkdsk volume_letter /b 

Challenge

Cause

Solution

Challenge

Cause

• Backup of failover clusters is supported in Veeam Agent for Microsoft Windows managed by Veeam Backup & Replication only. You cannot process a failover cluster by Veeam Agent for Microsoft Windows operating in the standalone mode.
• Agent license with a server counter must be installed in Veeam Backup & Replication to enable Agent Management features including failover cluster support.
• Veeam Agent for Microsoft Windows supports Windows Server Failover Clusters running Microsoft Windows Server 2008 R2 and later.
• Workgroup clusters, multi-domain clusters, and mixed OS version clusters are not supported.
• Cluster Shared Volumes (CSV) are skipped automatically during the backup.
• The following Microsoft Failover Cluster applications are supported: Microsoft SQL Server Failover Cluster Instances (Microsoft SQL Server 2008 SP4 or newer), Microsoft SQL AlwaysOn Availability Groups, Microsoft Exchange Database Availability Groups (Microsoft Exchange Server 2010 or newer).
• When using Basic Availability Groups on Microsoft SQL Server 2016 Standard Edition, consider that secondary replica node cannot be backed up because the secondary node is not readable. See the following article for more information.
• AlwaysOn Clusterless Availability Groups are not supported.
• Managed by backup server job with failover cluster type does not support the file-level backup mode.
• Recovery must be performed via Veeam Backup & Replication console.

Solution

Microsoft Windows Server Failover Clusters, Microsoft SQL Server Failover Cluster Instances, SQL Always On Availability Groups

Microsoft Exchange Database Availability Group

Starting Veeam Backup & Replication Update 4 Microsoft Exchange database availability groups (DAG) nodes are now automatically processed in a sequential manner, so it is possible to use Veeam Agent for Microsoft Windows to process Microsoft Exchange Database Availability Groups (DAGs).

The procedure of adding a DAG to a Veeam Agent backup job differs depending on the type of the DAG that you want to process:

• For a regular DAG, the backup job configuration procedure is the same as for any failover cluster mentioned above, so all the steps above are relevant
• For an IP Less DAG (a DAG without an Administrative Access Point), the backup job configuration procedure is similar to the same procedure for standalone servers (the steps below show it in detail)

Then just add servers that act as DAG nodes to a Veeam Agent backup job for servers configured in Veeam Backup & Replication.

During the backup process, Veeam Backup & Replication detects that servers in the backup job are part of a DAG, and creates a transactionally consistent backup of Microsoft Exchange databases running on these servers (this by design processes only passive database copies, active ones are excluded from VSS freeze operations).

Application-aware processing should be enabled for a consistent backup, log truncation and application-item restore possibility.

Challenge

Cause

Solution

Challenge

Cause

Solution

More Information

Challenge

Solution

Challenge

Solution

• <mydataboxno>.microsoftdatabox.com;
• blob.<mydataboxno>.microsoftdatabox.com;
• <storageaccountname>.blob.<mydataboxno>.microsoftdatabox.com

Challenge

Cause

vprob.storage.connectivity.lost
esx.problem.storage.connectivity.lost

Solution

More Information

Challenge

This article is intended to document how Forward Incremental works, and how its retention is enforced.

Solution

Forward Incremental Forever Backup Method – New in V8

Forward incremental-forever backup method is a default method for backup chain creation. To use this backup method, you must specify the following options in the backup job settings:

1. Select the Incremental backup mode.
2. Do not enable synthetic full backups and/or active full backups. If you enable synthetic and/or active full backups, Veeam Backup & Replication will produce a forward incremental backup chain (documented below).

The forward incremental-forever backup method produces a backup chain that consists of the first full backup and a set of forward incremental backups following it.

Veeam Backup & Replication creates a forward incremental-forever backup chain in the following way:

1. During the first run of a backup job, Veeam Backup & Replication creates a full backup file (VBK) on the backup repository.
2. During subsequent backup job sessions, Veeam Backup & Replication copies only VM data blocks that have changed since the last performed backup (full or incremental) and saves these blocks as an incremental backup file (VIB) in the backup chain.

Within this animation the lettered squares represent blocks on a disk.
 
Information regarding how retention works for this backup method can be found here:
https://helpcenter.veeam.com/docs/backup/vsphere/retention_forever_incremental.html?ver=95u4

Forward Incremental Backup Method

During the first run of a forward incremental backup, or simply incremental backup, Veeam Backup & Replication creates a full backup file (.vbk). During subsequent runs of the backup job, it will only retrieve changes that have taken place since the last run of the job (whether full or incremental) and saves this information as an incremental backup file (.vib). Each full backup files (.vbk) and its incremental files (.vib) that depend upon it are treated as a chain. Meaning that each incremental restore point depends on the one that came before it, the full restore point has no dependencies.




Within this animation the lettered squares represent blocks on a disk.

Incremental backup is the best choice if company regulation and policies require you to regularly move a created backup file to tape or a remote site. With incremental backup, you move only incremental changes, not the full backup file, which takes less time and requires less tape. You can initiate writing backups to tape or a remote site in Veeam Backup & Replication itself, by configuring Backup to Tape jobs.

If you decide to use the forward incremental backup method, it is necessary to schedule the creation of periodic active full or synthetic full backups. This will help you avoid long chains of increments, ensure safety of backup data, and allow you to meet the requirements of your retention policy. Below are animated examples of these things.










Within this animation the lettered squares represent blocks on a disk.

For more information please review the follow portions of the Help Center:
https://helpcenter.veeam.com/docs/backup/vsphere/incremental_forever_backup.html?ver=95u4
https://helpcenter.veeam.com/docs/backup/vsphere/retention_forever_incremental.html?ver=95u4
https://helpcenter.veeam.com/docs/backup/vsphere/forward_incremental_backup.html?ver=95u4
https://helpcenter.veeam.com/docs/backup/vsphere/retention_incremental.html?ver=95u4
https://helpcenter.veeam.com/docs/backup/vsphere/active_full_backup.html?ver=95u4
https://helpcenter.veeam.com/docs/backup/vsphere/backup_copy_remove_missing_point.html?ver=95u4

Challenge

Solution

Challenge

Solution

• <mydataboxno>.microsoftdatabox.com;
• blob.<mydataboxno>.microsoftdatabox.com;
• <storageaccountname>.blob.<mydataboxno>.microsoftdatabox.com

Challenge

Solution

VSSADMIN LIST PROVIDERS

If more than the default software provider is installed, another third party software suite may have enabled its provider on the guest.  When this provider remains on the system when backup has ceased, it may cause problems for native backup and other tasks that attempt to initiate a VSS backup. This component of the third party software will need to be removed if Veeam Backup & Replication is to process this source guest consistently.

For more information on removing specific providers, please refer to vendor documentation regarding VSS Provider and Agent installation. In some instances, the VSS Provider component can be removed while keeping the rest of the software functionally intact.

More Information

Challenge

Cause

Solution

Challenge

Cause

Solution

More Information

Veeam Backup & Replication support for VMware vSphere: https://www.veeam.com/kb2443

Challenge

Task fails with the error "root element is missing."
You can find such entries in the log file:

[timestamp] <01> Error Read summary xml error at Veeam.Backup.Core.CRawSummary.CreateOibSummary()
[timestamp] <01> Error at Veeam.Backup.Core.CSynOibAlg.CreateOib(CPoint fullPoint, CPoint rollPoint, CStorage fullStorage, CStorage rollStorage, COib rollOib)
[timestamp] <01> Error at Veeam.Backup.Core.CBackupJobPerformer.ExecuteTask(IBackupTask task, ITarget target)
[timestamp] <01> Error  Root element is missing. at System.Xml.XmlTextReaderImpl.Throw(Exception e)

Cause

This usually occurs when part of Veeam data could not be written to the summary.xml file (due to space, connectivity issue or some other specific issue). After that failure, cleaning up space on a target or fixing other problems that caused this behavior initially will not let you proceed with the incremental because the summary.xml is missing critical information on the previous backup.

Solution

There are two methods to potentially resolve this:

• The first option is to run a full backup on the corresponding job by right-clicking on a job and selecting "perform full backup". This is the method that leaves the least possibility of later issues.
   
• The second option is more involved, has some caveats, and is not guaranteed to work but may save some time.
  1. Verify the job itself is not currently running.
  2. Once verified go into the backup repository and locate the backup files for the job having the issue.
  3. Move the VBM file to another location without deleting it.
  4. After the VBM file has been moved, return to Veeam B&R and run a normal backup operation.
      

     Note: This also works for PerVM and SOBR based backups.
     
     Note 2: When moving the VBM, do so to a path outside the backup repository path. For instance, if the repository path is C:\Backups, move the VBM to either the root of the drive or another path from the drive. If it is placed in another subfolder of C:\Backups, and the repository is re-scanned, Veeam will attempt to import the VBM as a backup.
     
     This will attempt to create a new VBM file for the chain itself using metadata mirrored in the full backup storages, without the need to create another active full backup, avoiding larger amounts of data being sent to the repository. If this fails, the first option will likely be necessary.
     
     If the chain contains multiple full backups, only the latest full and increments following it will be used to recreate the VBM file. Retention may need to be performed manually by deleting older backup files when they are no longer needed.

      

Challenge

You receive the following error within moments of a job starting:
 
Cannot connect to the host's administrative share. Host:  [VM hostname or IP]. Account: [domain\user].
Win32 error: The network path was not found. Code: 53

NOTE: This applies to both VMware and Hyper-V hypervisors.

Cause

This error means that the Veeam Backup & Replication server cannot connect to the Windows guest’s OS feature to deploy guest processing components. This path will be the ADMIN$ share (\\hostname\admin$) of the guest itself.
 
These components are deployed directly from the Veeam Backup & Replication server. If networking is inherently impossible to the guest from the B&R server (different networks, firewall, etc), processing protocol may require inversion (VMware only).

Solution

• Make sure the Windows time on the Veeam Backup server is the same as in the guest OS.
• Make sure you do not have a firewall in the way (you can try disabling it for testing).
• Make sure you can connect to the admin share (for example, \\Server Name\admin$) with the same credentials as provided to Veeam Backup & Replication for VSS.
• Make sure the VSS writers are set to automatic in the guest OS.
• Make sure you provide Local Administrator (MACHINE\Administrator) or Domain Administrator (DOMAIN\Administrator) account privileges.
• Make sure the guest OS is in the same network as the Veeam Backup server.
• Make sure File and Printer Sharing is enabled in the guest OS.
  • ​Once File and Printer Sharing is Enabled on the guest OS, ensure to provide permissions through Windows Firewall if applicable
• If the guest OS is Vista/2008 or later:
  • In the Network and Sharing Center, verify that it is not set for a public domain.
  • Verify that the Remote Registry Service is started.

Note: As of 9.5 and later we have guest interaction proxies that deploy components: https://helpcenter.veeam.com/docs/backup/vsphere/guest_interaction_proxy.html?ver=95

More Information