Challenge

Failed to create snapshot. Error code -2147212300. 'Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{1d68f503-11fd-4591-9cf4-90e034386ae9}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].'
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{1d68f503-11fd-4591-9cf4-90e034386ae9}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4]. (System.Exception)

Log Name:      Application
Source:        SQLWRITER
Event ID:      24582
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TEST
Description:
Sqllib error: Failed to create VDS object. hr = 0x80770005.

Log Name:      Application
Source:        SQLVDI
Event ID:      1
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TEST
Description:
SQLVDI: Loc=IdentifySQLServer. Desc=Initialize. ErrorCode=(1332)No mapping between account names and security IDs was done.
. Process=5728. Thread=70504. Client. Instance=ACT7. VD=.

Solution

1. Open SQL Server Configuration Manager.
2. Navigate to  SQL Server Services.
3. Select an appropriate instance (e.g., SQL Server(ACT7)).
4. Change Log On  from  Built-in account: Local System   to   Built-in account: Local Service

Challenge

Note: This is a temporary workaround against this specific error. The release vehicle for the official out-of-the-box support for vSphere 6.7 U1 will be Veeam Backup & Replication v9.5 Update 4. 
For more information regarding Veeam Backup & Replication and vSphere compatibility please refer to https://www.veeam.com/kb2443

Cause

Solution

More Information

Challenge

Product Information:

Product Family: QNAP TS-1685
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: TS-1685
Storage Category: RAID Storage Array
Drive quantity, size, type: 12 – 4TB SATA HDD
Storage configuration: RAID 6
Firmware version: 4.3.4.0597
Connection protocol and speed: 10GbE
Additional support: All models and configurations of QNAP Enterprise SAS-NAS Series with specifications equivalent or greater than the above.

Veeam testing configuration:

Veeam Build Number: 9.5.0.1536

Job Settings:

• Deduplication: Enabled (Default)
• Compression: Optimal (Default)
• Storage Optimization: Local Target (Default)

• Repository Type: Shared Folder
• Per-VM Backup Files: Disabled (Default)
• Decompress before storing: Disabled (Default)
• Align backup file blocks: Disabled (Default)

Vendor recommended configuration:

Hardware Settings:

• Jumbo frames used during testing (9000 MTU)

More Information

Company Information:

Company name: QNAP Systems, Inc.
Company overview: QNAP Systems, Inc., as its brand promise "Quality Network Appliance Provider", aims to deliver comprehensive offerings of cutting edge network attached storage (NAS) and network video recorder (NVR) solutions featured with ease-of-use, robust operation, large storage capacity, and trustworthy reliability. QNAP integrates technologies and designs to bring forth quality products that effectively improve business efficiency on file sharing, virtualization applications, storage management and surveillance in the business environments, as well as enrich entertainment life for home users with the offering of a fun multimedia center experience. Headquartered in Taipei, QNAP delivers its solutions to the global market with nonstop innovation and passion.

Challenge

Cause

Solution

More Information

More information regarding this workaround can be found here: https://forums.veeam.com/vmware-vsphere-f24/vsphere-6-7-u1-support-t54673.html

This is a temporary workaround against the specific error, which will allow the jobs to complete successfully. Overriding VMware API version may potentially cause issues with other Veeam functionality, because we don't know the specific API changes that made VMware increment the API version. We're working with VMware to obtain these details while continuing to test vSphere 6.7 U1 with the workaround applied.

The release vehicle for the official out-of-the-box support for vSphere 6.7 U1 will be our next release (Update 4). We'll be able to determine supportability of vSphere 6.7 U1 with the workaround applied after full regression testing cycle is done against the U1 GA code (usually takes 3-4 weeks).

All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. The forum post above will be updated once the full regression testing cycle completes.

Challenge

Product Information:

Company name: FAST LTA, AG
Product Family: Silent Brick System
Status: Veeam Ready – Archive
Classification Description: Verified disk archive storage that can be used as a Backup Copy target. Synthetic full backups, granular restores, and vPower features may not provide sufficient performance or be supported.

Solution

Product Details:

Model number: G5000
Storage Category: Hybrid Storage
Drive quantity, size, type: 12 - SATA HDDs via SAS Bus
Storage configuration: RaidZ on 12 HDDs with 3 parities for storage
Firmware version: 2.11
Connection protocol and speed: 10GbE
Additional support: All models and configurations of Silent Brick System with specifications equivalent or greater than the above.

General product family overview: Simplify your backup with Silent Bricks. On-premise storage appliances with integrated data protection for all NAS and VTL requirements. Media replication and rotation. Online, offline and off-site, with support for air gap environments. All in one system. The Silent Brick System is based on modular and transportable storage units called 'Silent Bricks' where each Silent Brick consists of 12 Disks. Depending on the Silent Brick Type these disks may be HDD (Silent Brick ) or SSD ( Silent Brick FLASH ).
The assignment is done on a per Brick Basis where each Brick can act as a Backup to Disk Storage, a virtual Tape Library or an Archival Storage. The redundancy when set up as Backup to Disk Storage is configurable with 2 or 3 Parities per Silent Brick. A SecureNAS Volume can then be spanned over 1 to many Silent Bricks. Extending a SecureNAS Volume is possible on the fly by assigning additional Bricks. The parities will still be on a per Silent Brick Basis. For optimal performance the Silent Brick System offers dual 10GB Ethernet connection.

Veeam testing configuration:

Note: The following settings were used by the vendor to meet Veeam Ready testing requirements and should not be considered best practices. Additional changes or settings may be needed to meet the storage efficiency or performance needs for each environment. For each setting, reference links are provided for further clarification.

Veeam Build Number: 9.5.0.1922

Job Settings:

• Deduplication: Enabled (Default)
• Compression: Optimal (Default)
• Storage Optimization: Local Target (Default)

Repository Settings:

• Repository Type: Shared Folder
• Align backup file blocks: Disabled (Default)
• Decompress before storing: Disabled (Default)
• This repository is backed by rotated hard drives: Enabled
• Per-VM Backup Files: Disabled (Default)

Vendor recommended configuration:

Hardware Settings:

• Jumbo frames used in testing environment
• SMB storage protocol used during testing. Vendor also supports NFS, and ISCSI or Fibre Channel for virtual tape library support.
• Dedicated SMB user for storage share, used default settings for share configuration
• 3 parities used in RAID Z configuration

Challenge

Product Information:

Company name: FAST LTA, AG
Product Family: Silent Brick System
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: G5000
Storage Category: Hybrid Storage
Drive quantity, size, type: 12 - SATA FLASH discs via SAS Bus
Storage configuration: RaidZ on 12 SSDs with 3 parities for storage
Firmware version: 2.11
Connection protocol and speed: 10GbE
Additional support: All models and configurations of Silent Brick System with specifications equivalent or greater than the above.

General product family overview: Simplify your backup with Silent Bricks. On-premise storage appliances with integrated data protection for all NAS and VTL requirements. Media replication and rotation. Online, offline and off-site, with support for air gap environments. All in one system. The Silent Brick System is based on modular and transportable storage units called 'Silent Bricks' where each Silent Brick consists of 12 Disks. Depending on the Silent Brick Type these disks may be HDD (Silent Brick ) or SSD ( Silent Brick FLASH ).
The assignment is done on a per Brick Basis where each Brick can act as a Backup to Disk Storage, a virtual Tape Library or an Archival Storage. The redundancy when set up as Backup to Disk Storage is configurable with 2 or 3 Parities per Silent Brick. A SecureNAS Volume can then be spanned over 1 to many Silent Bricks. Extending a SecureNAS Volume is possible on the fly by assigning additional Bricks. The parities will still be on a per Silent Brick Basis. For optimal performance the Silent Brick System offers dual 10GB Ethernet connection.

Veeam testing configuration:

Note: The following settings were used by the vendor to meet Veeam Ready testing requirements and should not be considered best practices. Additional changes or settings may be needed to meet the storage efficiency or performance needs for each environment. For each setting, reference links are provided for further clarification.

Veeam Build Number: 9.5.0.1922

Job Settings:

• Deduplication: Enabled (Default)
• Compression: Optimal (Default)
• Storage Optimization: Local Target (Default)

Repository Settings:

• Repository Type: Shared Folder
• Align backup file blocks: Disabled (Default)
• Decompress before storing: Disabled (Default)
• This repository is backed by rotated hard drives: Enabled
• Per-VM Backup Files: Disabled (Default)

Vendor recommended configuration:

Hardware Settings:

• Jumbo frames used in testing environment
• SMB storage protocol used during testing. Vendor also supports NFS, and ISCSI or Fibre Channel for virtual tape library support.
• Dedicated SMB user for storage share, used default settings for share configuration
• 3 parities used in RAID Z configuration

Challenge

This article provides a method to change the repository where backup files are stored.
 

Solution

Important! If you have attempted to manually import the backup files using the Import button in the ribbon prior to performing the steps below you must first "Remove from configuration" any backups that were manually imported. These will identifiable in the Backups > Disk (Imported) section, they will be listed with the suffix _imported and will have no repository listed in the Repository column.

There are two primary situations and respective methods that this KB article documents.
Below is a summary of both methods.

Method A:
You have physically moved the backup files to new storage and want to map the old jobs to the backup chain in the new location.

Method B:
You are seeding a job, or you want to map a job to a backup chain in a new location. This method does not require that the job doing the mapping be the same as the job which created the restore points initially.

Method A: Same Job, New Backup location

1. In [Backup Infrastructure] > [Backup Repositories], create a new repository for the new location where backup files will be located.
2. Manually move the backup files to the new repository path. You must include the metadata file (.vbm), full backup files (.vbk), and any needed incremental files (.vib or .vrb).
   
   Notes:
   • It is recommended to move all backup files, but you can omit incremental files if needed. If you decide to not move/copy all backup files to the new storage, you will need to use the “Forget Missing Restore Points” function to clear any references to backup files that were not moved/copied.
   • Also keep in mind that there will be folders created inside the directory/path for job names.
     If your repository path is set as E:\Backups, Job A will go to E:\Backups\Job A.
3. Rescan the new repository.

4. If the backup files are not encrypted, skip this step.
   If the backup files are encrypted, the encrypted backup will appear under the Backups > Disk (encrypted) node in the inventory pane. In the working area, select the imported backup and click Specify Password on the ribbon or right-click the backup and select Specify password. More Information This is also relevant for encrypted Veeam Agent backups.
5. Edit the Backup\Backup Copy Job. Go to the Storage\Target tab, and from the drop-down menu, select the new repository.

6. Press Finish to finalize the setting. If you no error occurs, and the job now lists the new repository in its Repository column, the mapping has been completed.
   Note: If you encounter the following error: “Move all backup files to the new backup repository first”, some restore points the software was expecting to find were not found in the new repository. Cancel out of the job configuration and check both the old repository location and the new location to verify that all restore points have been moved. Then return to step 3 again.

If the steps above do not work, and you still get stopped at the "Move all backups files to the new backup repository first" error, please contact Veeam Support. You may also attempt Method B below.

Method B: New Job, New Backup location

To map the existing backup files to a new or different job, follow these steps:

1. Under Backups > Disk, right click the job you are changing repository for, and do either Remove From Backups (v8) or Remove from Configuration (v9).

2. If you are mapping the backup set from one job to another, but the backup location will stay the same, skip this step.
   If you are moving the backup files to a new location create a new repository for the new location where backup files are located.
3. Rescan the repository where the backup files are located.

4. If the backup files are not encrypted, skip this step.
   If the backup files are encrypted, the encrypted backup will appear under the Backups > Disk (encrypted) node in the inventory pane. In the working area, select the imported backup and click Specify Password on the ribbon or right-click the backup and select Specify password. More Information This is also relevant for encrypted Veeam Agent backups.
5. Edit the Backup\Backup Copy Job. Go to the Storage\Target tab, and from the drop-down menu, select the repository where the backup files you wish to map are located.

6. Click Map Backup and select the existing backup chain

If you are attempting to seed failover cluster Veeam Agent for Microsoft Windows job’s restore points please follow the actions below (it has a bit different workflow due to the product design):

1. Clone the source cluster job
2. Remove the source job (it will also remove from configuration its points; not from the disk but from Veeam config only)
3. Move the source backup folder to a new backup folder;
4. Rescan the new repository to import the moved files
5. Change the cloned backup job to the new repository and map the imported points there

More Information

Challenge

Cause

Solution

Challenge

Cause

Solution

Challenge

When attempting to backup a SQL server that is hosting a significant amount of databases the following error is thrown:
 

Unable to release guest. Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{}]. Instance ID: [{}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{}]. Instance ID: [{}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Cause

In SQL Server, the snapshot backup of each database uses five threads in the Sqlservr.exe process. Additionally, other activities may also use threads in the Sqlservr.exe process. Depending on the configuration of SQL Server, the available threads may be used up if you create a snapshot backup of many databases at the same time.

It is possible to process the VM in a crash-consistent manner by disabling Application-Aware Image Processing within the job.

Solution

It is advised to create another SQL Server and move the databases that are causing failure to the other SQL server.
In certain scenarios changing the max worker threads Server Configuration Option may help negate the issue. However, it is advisable to contact Microsoft support for verifying this step.

Please refer to the following articles for details:

• https://support.microsoft.com/en-us/kb/943471
• http://blogs.msdn.com/b/psssql/archive/2009/03/03/how-it-works-sql-server-vdi-vss-backup-resources.aspx
• https://msdn.microsoft.com/en-us/library/ms190219.aspx?f=255&MSPPError=-2147217396
• https://support.microsoft.com/en-us/kb/2157129

More Information

Note: Veeam Support is not able to provide assistance with modifying the number of threads within SQL. Please contact Microsoft Support for assistance.

Challenge

Product Information:

Company name: Dell EMC
Product Family: ML3 / ML3E
Status: Veeam Ready - Tape
Classification Description: Tape solution where available hardware features have been tested to work with Veeam.

Solution

Product Details:

Model number: 3555L3A7800003
Library Firmware Version 1.1.1.2-A00
Drive firmware version: LTO8 Firmware HB83
Driver for tape drive: Windows IBM ULT3580-HH8
Driver for media changer: (Veeam Option: Use native SCSI commands instead of Windows driver)
Media type tested: LTO8
Additional Support:All models based on the ML3 Base Module and optional ML3E (expansion modules)

General product family overview:
Dell’s newest tape automation product. A scalable unit that consists of a base module ML3 and up to 6 expansion modules ML3E. Each module supports 40 slots except the bottom row is not accessible so the bottom module only has 32 available slots.

Veeam Details:

Veeam Build Number: 9.5.0.1922

Challenge

Info [UploadManager] Checking windows package 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' signature on host 'HOSTNAME'
Error Failed to call RPC function 'PckgCheckSignature': Signature of module 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' is invalid..
Error Signature of module 'C:\ProgramData\Veeam\Agents\VAW\Veeam_B&R_Endpoint_x64.msi' is invalid Error --tr:Failed to call DoRpc. CmdName: [PckgCheckSignature].

Cause

Solution

For SHA256

1. Download DigiCert Assured ID Root CA here (right click on Download button - save as)
2. Double click the downloaded .srt file
3. Click Install Certificate
4. Choose Local Machine and click on Next
5. Place the certificate into Trusted Root Certification Authorities by clicking the Browse button.

For SHA1

1. Download R1 GlobalSign Root Certificate (Thumbprint B1BC968BD4F49D622AA89A81F2150152A41D829C) from here
2. Double click the downloaded .srt file
3. Click Install Certificate
4. Choose Local Machine and click on Next
5. Place the certificate into Trusted Root Certification Authorities by clicking the Browse button.
6. Download Code Signing SHA-1 G3 Intermediate Certificate (Thumbprint F1E7B6C0C10DA9436ECC04FF5FC3B6916B46CF4C) from here
7. Double click the downloaded .srt file
8. Click Install Certificate
9. Choose Local Machine and click on Next
10. Place the certificate into Intermediate Certification Authorities by clicking the Browse button.

Challenge

Cause

Solution

1. Verify that the account used to connect to Veeam Backup and Replication OR Hyper-V server in Veeam ONE has correct permissions. To check the account name, go to Veeam ONE Monitor > Data Protection View > right-click on the Veeam Backup and Replication object. Note the specified account. You can check the same for the Hyper-V server: Veeam ONE Monitor > Infrastructure tree > right-click on the Hyper-V object.
2. On the Veeam Backup and Replication OR Hyper-V server, confirm that this account as well as the Veeam ONE service account are added to the Event Log Readers Group and Administrators group in Local Systems and Groups
3. Even if the Firewall is off, activate the following firewall rules on the Veeam Backup and Replication OR Hyper-V server side: Remote Event Log Management and COM+ Network Access (DCOM-In)
4. Check UAC settings.

1. Change the type of the all Public network connections to Private using the following Powershell cmdlet: 

Set-NetConnectionProfile -NetworkCategory Private

Or using the registry editor (machine restart will be required):

More Information

Challenge

The Veeam Backup & Replication console is slow to respond when switching between different views. The issue is more noticeable when attempting to search through the history of jobs.

Cause

In rare cases the VeeamBackup SQL DB can have many thousands of previous task sessions which causes queries run by the console to be delayed.

Solution

Note: If you are using Veeam Backup & Replication 9.5 U3 (9.5.0.1536), and are observing slow GUI performance within the console, this may be related to a known issue found only in 9.5.0.1536. This was corrected in the next update, please ensure you have the latest update installed, https://www.veeam.com/updates.html.

There are two parts to resolving this, one resolves the issue in the short term by removing old sessions from the SQL DB, the other is to limit how many sessions are stored in the SQL DB.

Clear old job sessions

Notes:

• SQL Management Studio is required to perform this task. All jobs must be stopped.
• This stored procedure removes sessions from the database, if the database contains many session entries this may take a very long time (hours). Please plan accordingly.

1. Stop all Veeam services on the Veeam Backup Server.
2. Connect to the SQL instance hosting the VeeamBackup database.
3. Create a backup of the VeeamBackup database. (http://www.veeam.com/kb1471)
4. Expand the DB to Programmability > Stored Procedures > dbo.DropStoppedOldJobSessions
5. Right click > Execute... > in the window that appears enter how many days you want to keep
6. Press OK to execute, it will report “Query executed successfully.” when it completes.
7. Start all Veeam services.

Changing Session history retention

The sessions to be stored in the configuration database are controlled by retention defined in number of weeks. If you have many jobs running often the default setting may cause the database to retain many entries. You can specify session history settings within the Veeam Backup & Replication console.

1. From the main menu, select General Options.
2. Click the History tab.
3. In the Sessions section, specify the number of sessions to display in the Sessions list of the History view.
4. In the Session history retention section, specify the number of weeks for which Veeam Backup & Replication must keep session information in the configuration database.

Challenge

Solution

Product Details

Special Considerations

More Information

Challenge

This article documents how change the temporary staging location where SQL transaction log backups are stored during a Veeam Agent for Windows backup operation when Transaction Log Backup is enabled.

For more information on the Transaction Log Backup process for Veeam Agent for Windows please click here.

Cause

When SQL Transaction Log Backup is enabled, Veeam Agent for Windows copies transaction log files from the log archive destination (set by the Microsoft SQL Server administrator) to a temporary folder on the server. By default the path is: C:\ProgramData\Veeam\Endpoint\SqlLogBackup.

Solution

To change the location where the SQL Transaction Logs are temporarily stored, create the following registry entry on the server.

Note: This registry change is to be made on the machine where Veeam Agent for Windows is installed, and is running the backup with SQL Transaction Log backup enabled.

Key: HKLM\SOFTWARE\Veeam\Veeam Endpoint Backup
Type: REG_SZ (String Value)
Name: SqlTempLogPath
Value: Path to desired location (without trailing backslash)

Note: if the backup job is managed by Veeam Backup & Replication, then the path for the registry should be:

Key: HKLM\SOFTWARE\Veeam\Backup and Replication

Example

More Information

Challenge

Cause

Solution

• Restore to a different Hyper V host.
• Do not select power on when restoring and manually set the NIC to dynamic in virtual machine settings in Hyper V manager.
• Do not map the Instant Recovery to a Network and add a NIC manually afterward.

More Information

Challenge

Cause

• The Default credentials are:
  • Username – veeam
  • Password – veeam

Solution

1.    Log into Prism and launch your Veeam Appliance VM to gain access to the terminal interface

2.    Log in using the default credentials (veeam\veeam)
3.    Type the following (without quotes) “sudo passwd veeam” and hit enter
4.    Enter the current password and hit enter
5.    Type in your new password and hit enter
6.    Type in your new password to confirm and hit enter

7.    Type logout and hit enter

8.    Log in using the new credentials you just created

Challenge

We receive the error Failed to connect to Veeam Backup & Replication server: No authority could be contacted for authentication. This can manifest as a failure to connect to the Veeam Backup & Replication console, on restore, or on a VM running in a surebackup virtual lab.

Cause

The server has lost trust with the Active Directory domain.

Solution

Remove the Veeam Backup & Replication server, or effected restore, or surebackup VM from the domain, reboot, and re-add Veeam Backup & Replication server to the domain.

More Information

Please reference the Microsoft KB relative to your operating system.

http://technet.microsoft.com/en-us/library/cc770919(WS.10).aspx

Challenge

Solution

More Information