Challenge

Job with truncation enabled shows warning: 

Failed to truncate transaction logs for SQL instances: MICROSOFT##WID

Note: This KB article only applies if the specified instance is named MICROSFT##WID.

Cause

After applying update 3 for Veeam Backup & Replication v8, we attempt to connect to an internal Microsoft Database during log truncation.

The Task.<vmname>.<refid>.log file for the VM in the job will contain:  
 

Error VssControl: TruncateSqlLogs failed (System.Exception)
Error at Veeam.Backup.VssProvider.CVssControl.TruncateSqlLogs(Guid jobId, CCredentials creds)
Error at Veeam.Backup.Core.CViGuestVssFreezer.TruncateSqlLogs(Boolean bSafe)
Error Failed to process 'TruncateSQLLog' command.
Error Failed to truncate transaction logs for SQL instances: MICROSOFT##WID. Possible reasons: lack of permissions, or transaction log corruption. (System.Runtime.InteropServices.COMException)
Error at VeeamGuestHelperCtrlLib.IVeeamVssProvider.TruncateSqlLogs(Guid jobUid, String sqlLogin, String sqlPwd)
Error at Veeam.Backup.VssProvider.CVssControl.TruncateSqlLogs(Guid jobId, CCredentials creds)

Solution

This issue is fixed in Veeam Backup & Replication v9.

If you are unable to upgrade to v9, please contact Veeam Support and reference Bug# 59525.

More Information

This issue only applies to 8.0.0.2084, and for the MICROSOFT##WID instance.
For any other truncation issues, please see the following KB: https://www.veeam.com/kb2027

Challenge

Product Information: 

Product Family: QNAP ES Series 
Status: Veeam Ready – Repository 
Classification Description: Verified backup storage that supports all Veeam backup and restore features. 

Solution

Product Details:

Model number: ES1640dc v2 
Number of Drives: 6 
Drive type: SAS HDD 
Firmware version: QES 1.1.4 
Connection protocol and speed: 10 GbE 
General product family overview: The ES1640dc v2 is whole-new product line developed by QNAP for mission-critical tasks and intensive virtualization applications. With Intel® Xeon® E5 processors, dual active controllers, ZFS, and fully enabling virtualization environments, the ES1640dc v2 delivers “real business-class” cloud computing data storage. The ES1640dc v2 features the brand-new QES operating system that allows almost limitless snapshots, block-level data deduplication, and thin provisioning with reclaim, assisting businesses in building VDI platforms and storage for critical data. The ES1640dc v2 provides the same user experience as the current QNAP NAS and QTS system, allowing business users to enjoy the easy learning curve and quickly deploy the system, and making it ideal for small and medium businesses or large-scale IT divisions coping with mission-critical tasks. 

Veeam Details: 

Veeam Build Number: 9.5.0.1038 
Veeam Settings:  

• Repository Type: Shared folder 
• Deduplication: Yes 
• Compression: Optimal 
• Storage Optimization: LAN target 
• Per-VM Backup Files: Yes 
• Decompress before storing: No 
• Align backup file blocks: Yes 

More Information

Company Information: 

Company name: QNAP 
Company overview: QNAP Systems, Inc., as its brand promise "Quality Network Appliance Provider", aims to deliver comprehensive offerings of cutting edge network attached storage (NAS) and network video recorder (NVR) solutions featured with ease-of-use, robust operation, large storage capacity, and trustworthy reliability. QNAP integrates technologies and designs to bring forth quality products that effectively improve business efficiency on file sharing, virtualization applications, storage management and surveillance in the business environments, as well as enrich entertainment life for home users with the offering of a fun multimedia center experience. Headquartered in Taipei, QNAP delivers its solutions to the global market with nonstop innovation and passion. 

Challenge

Product Information: 

Product Family: XCubeSAN 
Status: Veeam Ready – Repository 
Classification Description: Verified backup storage that supports all Veeam backup and restore features. 

Solution

Product Details: 

Model number: XS5200 
Number of Drives: 26 
Drive type: SAS SSD 
Firmware version: 1.2.2 
Connection protocol and speed: iSCSI 10 GbE 
General product family overview: QSAN XS5200 Series is the high performance, simple, secure and scalable SAN storage systems for enterprise. It is an ideal solution to the applications of mission critical data center, high performance computing, virtualization integration, or media and entertainment environments. 

Veeam Details: 

Veeam Build Number: 9.5.0.1038 
Veeam Settings:  

• Repository Type: Windows 
• Deduplication: Yes 
• Compression: Optimal 
• Storage Optimization: Local target 
• Per-VM Backup Files: No 
• Decompress before storing: No 
• Align backup file blocks: No 

More Information

Company Information: 

Company name: QSAN Technology, Inc. 
Company overview: QSAN Technology, leading storage technology designer and manufacturer, was founded in July 2004. Building on our vast experience in the industry, QSAN strives to build enterprise class storage systems that pride ourselves with outstanding performance, secure data protection and comprehensive data management. QSAN endeavors to provide the industry with high quality data storage systems that are simple, secure, scalable and reliable. For more information, please visit www.QSAN.com 

Challenge

When trying to add a Linux server to the Veeam Backup & Replication console, you get the following error:

Failed to login to "xxx.xx.xx.x" by SSH, port 22, user "root"
elevateToRoot False, autoSudo False
User authentication failed

However, it is possible to log on to this server with the same credentials using an SSH client.

Cause

The configuration of your SSH server does not allow using tunnelled clear text passwords.

Solution

1. Open the SSHD config file (usually, /etc/ssh/sshd_config) on the Linux server.

2. Find the PasswordAuthentication parameter and set its value to “yes”.

3. Restart SSHD on the Linux server (usually, /etc/init.d/sshd restart).

4. Try adding the server to the console again.

Challenge

The Snapshot creation fails with the following error:

CreateSnapshot failed, vmRef "vm-xxx", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True" Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

Within the vSphere Environment the following error can be seen in the task list for the VM:

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Cause

This issue occurs when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created.

This issue also occurs if you try to create a quiesced snapshot on a virtual machine that does not have free space. That is, if the free space on the virtual machine is 0 bytes.

Solution

This error will only occur when the Veeam Backup & Replication job has been configured to enable VMware Tools Quiescence. If this option is enabled Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication. The option "Enable VMware tools quiescence" is disabled by default. It is strongly recommended to leave it disabled when processing Windows systems. For Windows based systems, it is recommended to use the option "Enable application-aware processing" which will utilize VSS within the guest.

This error can be reproduced by manually creating a snapshot on the VM that is failing to be processed with the option for "Snapshot the virtual machine's memory" unchecked and checking "Quiesce guest file system".

As this error is caused by an issue with the creation of the snapshot on the Virtual Machine by the vSphere Environment, please refer to this KB article from VMware:
https://kb.vmware.com/s/article/1018194

Challenge

Cause

Solution

Challenge

Solution

1. Close the Veeam Backup & Replication console.
2. Stop all services that begin with Veeam.
3. Open the Task Manager on the Veeam Server and kill all VeeamAgent.exe processes.
   Note: Some VeeamAgent.exe process will be located on Source Proxies and Windows Repositories that are not the Veeam Server.
4. Wait 5-10 minutes for the tasks to timeout and fail. 
5. Remove snapshots from VM(s) that are part of the stuck jobs. (If they have not been already removed).
6. Start the services that were stopped in step 2.

1. Back up the Veeam database. (http://www.veeam.com/kb1471)
2. Run the following query against the VeeamBackup database:

Challenge

Cause

• SQL Server 2005/2008 Express Edition – maximum database size 4 GiB;
• SQL Server 2008 R2/2012 Express Edition – maximum database size 10 GiB;

Solution

Challenge

Solution

Challenge

Solution

More Information

Challenge

Solution

Challenge

• Unexpected End of File
• Exchange Web Services error code: ErrorExceededConnectionCount
• Error: Failed to get folder properties
• Error: You have exceeded the available concurrent connections for your account.  Try again once your other requests have completed.
• Error: Unexpected end of file has occurred. The following elements are not closed: m:Data, m:ExportItemsResponseMessage, m:ResponseMessages, m:ExportItemsResponse, s:Body, s:Envelope. Line 1, position 21397505.

Cause

Solution

Challenge

Cause

Solution

Part 1: Remove the Replica from configuration.

1. Navigate to the replica in the “Ready” section under “Replicas”
2. Right click the replica and click “Remove from configuration”

Part 2: Map the replication job to use the existing replica.

1. Navigate to the replication job. Right click the job and click “Edit…”
2. In the “Name” section, check the "Low connection bandwidth" option
3. Click next to the “Virtual Machines” section, and add the source VM back into the replication job
4. Navigate to the “Seeding” section, and check the “Map replicas to existing VMs” check box
5. Select the original VM, and click “Edit…”, then select the existing replica
6. Click “Finish”

Challenge

Solution

• This type of restore can be effected adversely by the aforementioned limitations of a Data Domain appliance, and also the type of VM being restored. Highly transactional VMs will require more IOPS from the Data Domain during the Instant Recovery than others. With this in mind you can expect to only be capable of running only a few Instant Recoveries simultaneously. Instant Recovered VMs that are started from a backup file stored on a Data Domain may react or start slowly as the majority of their read operations will be hindered by the Data Domain.
• For VMware users it is highly advised when performing an IR that the user select to have virtual disk updates redirected to a high performance Datastore. This will improve performance by caching written blocks to low latency storage.
• It is advised that if the VM is intended to be made permanent that the VM be migrated to production storage as soon as possible after the Instant Recovery has begun.

• The backup browser may take longer than usual to open if an increment is selected and furthermore by that increments distance from the full restore point.
• Navigating between folders within the Backup Browser may take additional time as each folder’s content must retrieved from the backup file to display it.

• Reverse Incremental performance will be very poor due to its highly random I/O.
  Note: When the Backup Job is configured with a DDBoost repository, Veeam Backup & Replication will prevent Reversed Incremental from being selected by the user.
• Synthetic Full creation will be very slow to a Data Domain, unless using DDBoost.
• Synthetic Full with Transforms are not advised.

• A retention longer than 30 is not advisable as restore operations will diminish in performance.
• The Health Check option may take a very long time as it is performing a read operation.

• Using the Datadomain to store Replica metadata is not advisable.
  Note: Veeam Backup & Replication will prevent the user from selecting a DDBoost repository.

• For CIFS/NFS presented repositories Forward incremental mode with periodic Active full backups is recommended to avoid the rehydration penalty during synthetic operations.
• For DDboost enabled repositories Forward incremental mode with either Active full or Synthetic full backups is recommended. Synthetically produced full backups will generally have the best restore performance and reduce the time VM is run off of a snapshot during the backup job run. However in some environments an Active Full job may run faster.
• Transforming previous backup chains into rollbacks is not advisable for both repository types.
• For forever forward incremental backup and backup copy on DDboost enabled repositories, the option “Defragment and compact full backup file” should be enabled if available. In most cases a weekly schedule is appropriate. This helps to avoid excessive growth of pre-compression data size for the full backup file.

• Support EMC Data Domain Boost 3.0
• Reduced impact of storage fragmentation during restore operations even with enabled parallel processing. This feature allows Veeam to store the VM backup in the dedicated backup chain so that fragmentation ratio will be minimum.
• Reduce the impact of the block size so you may define any block size without impact on the restore process. Veeam will be able to read data granularly so amount of the redundant will be minimum.

1. Launch the creation of a new Repository, on the Type tab select Deduplication storage appliance.
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. Select the deduplication storage as EMC Data Domain.
3. On the next tab configure the information to for connecting to the Data Domain appliance.
   Very bottom on this page: http://helpcenter.veeam.com/backup/80/vsphere/repository_server.html
4. On the Repository tab click Browse and select the necessary location from the list of available paths. http://helpcenter.veeam.com/backup/80/vsphere/repository_repository.html
5. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

1. Launch the creation of a new Repository, on the Type tab select CIFS
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. On the next tab configure the path to which the Repository will write to, and set credentials to access that share.
   Second section named Shared Folder on this page: http://helpcenter.veeam.com/backup/80/vsphere/repository_server.html
3. On the Repository tab within the advanced section, enable “Decompress backup data blocks before storing”
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

1. Launch the creation of a new Repository, on the Type tab select Linux
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. On the next tab select the Linux server that we be connected to. If it is not present in the list select “Add New…”
3. On the Repository tab specify the path on the Linux server that leads to where you mounted the Data Domain via NFS. On this tab in the advanced section enabled “Decompress backup data blocks before storing”.
   http://helpcenter.veeam.com/backup/80/vsphere/repository_repository.html
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Challenge

Cause

Solution

Challenge

Solution

• This type of restore can be effected adversely by the aforementioned limitations of a Data Domain appliance, and also the type of VM being restored. Highly transactional VMs will require more IOPS from the Data Domain during the Instant Recovery than others. With this in mind you can expect to only be capable of running only a few Instant Recoveries simultaneously. Instant Recovered VMs that are started from a backup file stored on a Data Domain may react or start slowly as the majority of their read operations will be hindered by the Data Domain.
• For VMware users it is highly advised when performing an IR that the user select to have virtual disk updates redirected to a high performance Datastore. This will improve performance by caching written blocks to low latency storage.
• It is advised that if the VM is intended to be made permanent that the VM be migrated to production storage as soon as possible after the Instant Recovery has begun.

• The backup browser may take longer than usual to open if an increment is selected and furthermore by that increments distance from the full restore point.
• Navigating between folders within the Backup Browser may take additional time as each folder’s content must retrieved from the backup file to display it.

• Reverse Incremental performance will be very poor due to its highly random I/O.
  Note: When the Backup Job is configured with a DDBoost repository, Veeam Backup & Replication will prevent Reversed Incremental from being selected by the user.
• Synthetic Full creation will be very slow to a Data Domain, unless using DDBoost.
• Synthetic Full with Transforms are not advised.

• A retention longer than 30 is not advisable as restore operations will diminish in performance.
• The Health Check option may take a very long time as it is performing a read operation.

• Using the Datadomain to store Replica metadata is not advisable.
  Note: Veeam Backup & Replication will prevent the user from selecting a DDBoost repository.

• For CIFS/NFS presented repositories Forward incremental mode with periodic Active full backups is recommended to avoid the rehydration penalty during synthetic operations.
• For DDboost enabled repositories Forward incremental mode with either Active full or Synthetic full backups is recommended. Synthetically produced full backups will generally have the best restore performance and reduce the time VM is run off of a snapshot during the backup job run. However in some environments an Active Full job may run faster.
• Transforming previous backup chains into rollbacks is not advisable for both repository types.
• For forever forward incremental backup and backup copy on DDboost enabled repositories, the option “Defragment and compact full backup file” should be enabled if available. In most cases a weekly schedule is appropriate. This helps to avoid excessive growth of pre-compression data size for the full backup file.

• Support EMC Data Domain Boost 3.0
• Reduced impact of storage fragmentation during restore operations even with enabled parallel processing. This feature allows Veeam to store the VM backup in the dedicated backup chain so that fragmentation ratio will be minimum.
• Reduce the impact of the block size so you may define any block size without impact on the restore process. Veeam will be able to read data granularly so amount of the redundant will be minimum.

1. Launch the creation of a new Repository, on the Type tab select Deduplication storage appliance.
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. Select the deduplication storage as EMC Data Domain.
3. On the next tab configure the information to for connecting to the Data Domain appliance.
   Very bottom on this page: http://helpcenter.veeam.com/backup/80/vsphere/repository_server.html
4. On the Repository tab click Browse and select the necessary location from the list of available paths. http://helpcenter.veeam.com/backup/80/vsphere/repository_repository.html
5. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

1. Launch the creation of a new Repository, on the Type tab select CIFS
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. On the next tab configure the path to which the Repository will write to, and set credentials to access that share.
   Second section named Shared Folder on this page: http://helpcenter.veeam.com/backup/80/vsphere/repository_server.html
3. On the Repository tab within the advanced section, enable “Decompress backup data blocks before storing”
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

1. Launch the creation of a new Repository, on the Type tab select Linux
   http://helpcenter.veeam.com/backup/80/vsphere/repository_launch.html
   http://helpcenter.veeam.com/backup/80/vsphere/repository_type.html
2. On the next tab select the Linux server that we be connected to. If it is not present in the list select “Add New…”
3. On the Repository tab specify the path on the Linux server that leads to where you mounted the Data Domain via NFS. On this tab in the advanced section enabled “Decompress backup data blocks before storing”.
   http://helpcenter.veeam.com/backup/80/vsphere/repository_repository.html
4. The default settings can be taken for the last steps in repository configuration.
   Unless your environment requires you to specify a different vPower NFS Server.

Challenge

Cause

Solution

msiexec /update c:\VAC.ApplicationServer.x64_2.0.1.1343.msp

3. To install the Web UI update, on the VAC Web UI server execute VAC.WebUI.x64_2.0.1.1343.msp as administrator, or run this cmdlet as administrator:

msiexec /update c:\VAC.WebUI.x64_2.0.1.1343.msp

4. Log in to VAC Web UI.

More Information

Challenge

This KB article documents the procedure for mapping a replication job to a replica that was not created by a tenant's replication job.

Use case examples:

• The tenant has sent backup files to the Service Provider, and the Service Provider restores the VMs from the backup files to create replication mapping targets.
• The Service Providers environment has been impacted in such a way that the hypervisor reference IDs have changed and the tenant's job can no longer find the previously existing replicas.

When creating a replication job, seeding or mapping can be used to minimize the amount of traffic sent to a Cloud Service Provider. In most cases, seeding is the preferred method because it is not possible to map a Cloud Connect replication job to a replica that was not created with Cloud Connect. However, it is possible for a Cloud Service Provider to work around this limitation by replicating the existing replica.

Note: It is not possible to use a backup of a replica as a seed, because Veeam searches the backup file for the ID of the source VM, which is different from the ID of the replica.

Solution

Tenants who are unable to map replicas should ask their service provider whether this workaround is available.

Steps to be performed by the Cloud Connect Service Provider.

1. Connect a console to a Veeam backup server with access to the infrastructure containing the original replica. Typically this will be a backup server with a per-socket license, and not the SP Veeam backup server used to provide cloud resources to tenants.
2. Use the tenant’s credentials to connect to a cloud gateway .
3. Create and run a job to replicate the original replica into the cloud infrastructure. When configuring the job, consider that adding the default suffix will result in VMs named *_replica_replca.
4. Delete the replication job created in step 3; do not delete the replica.
5. Remove the service provider that was configured in step 2. Advise the tenant to proceed with the steps below.

Note: The console used for these steps may still display the replica after the service provider is removed in step 5; the replica should not appear the next time the console is opened.

Steps to be performed by the tenant.

1. In the Backup Infrastructure view, add or rescan the cloud service provider.
2. Verify that the replica is visible in the Replicas node of the Backup & Replication view. Press F5 to refresh the view if needed.
3. Map the replica to a replication job. This must be done manually; the Detect option is not available.
4. Run the mapped job and test the replica. Let the service provider know that they may proceed with the final steps.

Steps to be performed by the Cloud Connect Service Provider.

1. Both SP and tenant will see duplicate replicas (with different source VMs) in the console. To resolve this, the SP must remove the older replica from configuration .
2. (Optional) When the new replica has been tested successfully, the Service Provider may delete the original replica that is no longer needed.

Challenge

The Service Provider wizard on the Tenant Veeam Server fails with errors.

Two examples:

Certificate validation failed. Unable to connect to the service provider.

Certificate validation failed. Authentication failed because the remote party has closed the transport stream.

Solution

Veeam Support engineers are only able to assist with isolation of certificate problems. Veeam Support is unable to assist in generating, altering, importing, exporting, or installing SSL certificates. For more information on certificate processes, please refer to your SSL certificate provider.

The following is a list of common causes.

1. The connection to the Service Provider Gateway(s) cannot be made with the default port TCP 6180.
   
   

   Ensure TCP/UDP port 6180 is allowed outbound from the tenant environment (for stateful firewalls only). If the firewall is stateless, a static rule will need added for the return traffic. Similar firewall exceptions with TCP/UDP 6180 need to be applied in the provider's firewall for traffic that is destined for each Cloud Connect Gateway. Additionally, please note that tenant proxies or repositories will connect directly to the Cloud Connect Gateways during job runs.
   
   

2. The certificate is expired and needs renewed.
   
   
3. The certificate was incorrectly keyed during the CSR process and needs re-keyed or the private key is missing entirely.
   
   

   Ensure the certificate with the private key is installed in the Service Provider Cloud Connect server. It does not need to be installed in the Cloud Connect Gateways if they are separate servers. The issued certificate with the private key will be a file with a .pfx extension.
   
   If your SSL certificate provider asks you to generate the PFX file using a private key you have generated as opposed to one they provide, it will be considered a security risk and will not be a supported configuration.
   
   

4. The certificate chain has not been fully installed in the Service Provider Cloud Connect server and the chain of trust cannot be found. The connection to the Service Provider Cloud Connect server will not be authenticated unless the Tenant Veeam server can validate a certificate ending in a Root CA certificate.
   
   

   Ensure the certificate chain is installed in the Service Provider Cloud Connect server, which includes subordinate (intermediate) and root CA certificates. Often the SSL certificate provider will include the chain in a separate file with a p7b extension.
   
   

5. The Cloud Connect Gateway(s) cannot resolve (with DNS) the Cloud Connect Server or the Cloud Connect Gateways cannot communicate internally or via the DMZ to the Cloud Connect Server.
   
   

   Ensure DNS can be resolved for the Cloud Connect Server from all Cloud Connect Gateways. Disable any gateways that are not going to be used.
   
   

6. In some cases, a firewall will have a type of adaptive security that filters SSL\TLS traffic. For example, some names for this are “deep packet inspection” (dpi), packet inspection, or SSL\TLS inspection. The usage of these features creates a Man-in-the-Middle scenario with the firewall and can cause issues when the certificate is exchanged to the Tenant Veeam Server.
   
   
7. Some newer certificate formats are only supported in 9.5 update 2 or later, for example, Microsoft Software Key Storage Provider.
   
   

More Information

Challenge

• The jobs are using Network transport mode
• Source Proxy is Windows 2008SP1/Windows 2008R2

Solution

More Information