Quantcast
Channel: Veeam Support Knowledge Base
Viewing all 4474 articles
Browse latest View live

NetApp hardware VSS provider limitation

$
0
0

Challenge

Hyper-V Backup/Replication Job utilizing NetApp Data ONTAP VSS Hardware Provider reports warning or error message:
"Another shadow copy creation is already in progress. Please wait a few moments and try again"
 

Cause

NetApp hardware VSS provider doesn't allow concurrent snapshot creation within cluster, even when snapshots are initiated on different hosts and for different CSVs.
With NetApp hardware VSS provider you can have only one snapshot creation running within cluster at once - there is no concurrency.

Microsoft software VSS provider (Microsoft CSV shadow copy provider) has no such limitation.

Solution

You can switch to Microsoft software VSS provider:
https://helpcenter.veeam.com/backup/hyperv/hv_server_volumes.html#volume

Or disable concurrent shadow copy snapshot creation within cluster for all providers:
Create HyperVRestrictConcurrentSnapshotCreation (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the Veeam backup server.
And set it to 1.
Then stop all Jobs and restart Veeam Backup service.

 

Windows File Level Restore hangs on "Starting restore session"

$
0
0

Challenge

Windows File Level Restore hangs on "Starting restore session" for abnormally long period of time while free disk space on drive C:\ keeps decreasing dramatically.
You may find following path with cache file constantly growing:
C:\Windows\system32\config\systemprofile\AppData\Local\mount_cache{GUID} or C:\Windows\SysWOW64\config\systemprofile\AppData\Local\mount_cache{GUID}.
 

Cause

The issue occurs when guest Windows has Transactional NTFS (TxF) turned on. When mounting to Veeam server it can cause NTFS driver to enter infinite loop trying to re-apply transaction log.

Solution

To resolve:

1. Login to guest OS as administrator. Run cmd with elevated privileges. Execute 2 following commands for each logical disk in VM. Example for disk C:
fsutil resource setautoreset true C:\
fsutil resource setavailable C:\


2. Make a new backup point of that VM. verify that File Level Restore starts fine from new backup point

More Information

For more information about Transactional NTFS (TxF)
https://msdn.microsoft.com/ru-ru/library/windows/desktop/bb968806(v=vs.85).aspx

Leap Second Compatibility

$
0
0

Challenge

A leap second is a one-second adjustment that is occasionally applied to Coordinated Universal Time (UTC) in order to keep its time of day close to the mean solar time. Some software experiences problems when leap seconds are inserted (e.g. inserting it at an earlier date).
 

Solution

The previous leap second insertion, which occurred on June 30, 2015, revealed no issues with any Veeam products, and it is expected that this will be the case for the next leap second insertion.

However, this may not be true for any other infrastructure components and/or software, and it is recommended to check with third-party vendors which your company's business relies upon.

JetError -1404, JET_errIndexNotFound, No such index

$
0
0

Challenge

When trying to open a database in VEAD, you receive the following error message:

JetError -1404, JET_errIndexNotFound, No such index
 

Solution

Log in to the domain controller from which you are trying to restore the database, run cmd as Administrator and run the following:
 
  1. Type regsvr32 schmmgmt.dll

    This command will register Schmmgmt.dll on your computer.
  2. Click Start, click Run, type mmc /a, and then click OK.
     
  3. On the File menu, click Add/Remove Snap-in, and then click Add.
     
  4. Under Available Standalone Snap-ins, double-click Active Directory Schema, click Close, and then click OK.

Next, open Active Directory Schema, find the attribute ObjectClass, select Properties and check the Index this attribute box.

User-added image

Apply the changes and back up the domain controller again.

More Information

The ObjectClass attribute was not automatically indexed in Windows 2003. In later versions of Windows, indexing is enabled automatically and the issue does not occur.

VMware vCenter server experiences performance problems while waiting for tasks to complete

$
0
0

Challenge

Hung tasks, slow response time or disconnects from VMware vCenter server in VMware vSphere client or in any monitoring product which uses VMware vCenter server for gathering data.

Regular disconnects from VMware vCenter server.

Cause

If you consistently have slow response times in the VMware vSphere client while being connected to VMware vCenter server, please check your ESX hosts for resource problems.

A possible scenario is that an ESX host has several third party agents installed that are consuming Service Console memory. Backup and enterprise monitoring agents are common examples. It is also possible that the resource needs of the virtual machines have exceeded the host’s physical capabilities. Although cpu and memory over-allocation is possible with ESX, there is always a limit to the number of guests that can coexist together peacefully. That number depends on guest workload types and individual VM virtual hardware configuration. Multiple VMs with applications that heavily use memory or that require multiple virtual cpus usually do not perform well if contained on the same ESX host.

Performance troubleshooting will ultimately pinpoint the exact cause of the problem, but there are some general changes that can be made first.

Solution

Increasing the amount of RAM assigned to the ESX Server service console is the VMware Knowledge Base article that explains how to increase the Service Console allocated memory (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003501&sliceId=2&docTypeID=DT_KB_1_1&dialogID=64395968&stateId=0%200%2064393864).

From the KB article:
“By default the service console on an ESX Server host is usually not configured to utilize the maximum amount of RAM.” [ ... ] On ESX Server 3.x the amount of RAM is set to 272MB.

The default values are usually sufficient, however there are several instances where VMware recommends to increase the RAM assigned:

When third-party system management agents are installed or crashing on the service console.

When a backup agent is installed on the service console.

When heavy swap file utilization is noticed on the service console

Hopefully you already have a swap partition that will support bumping your Service Console ram up to the 800 mb maximum. If not, try increasing the memory to at least 512 mb and see if you notice any improvement.

Also from the same KB article, Here are the step by step instructions for increasing the Service Console memory:

“To change the amount of RAM assigned to the service console:

  1. Log in to VirtualCenter from the Virtual Infrastructure Client with a user that has administrative rights.(Note: If you do not have VirtualCenter, log in directly to the ESX Server host as root.)
  2. From the Inventory select the ESX Server host.
  3. Click the Configuration tab.
  4. Click Memory.
  5. Click Properties.
  6. On the Memory window enter a value between 256MB and 800MB for the service console parameter.(Note: For troubleshooting purposes, VMware recommends that you increase the service console RAM to 800MB.)
  7. Click OK.


Changes do not take effect until the ESX Server is rebooted. For more information on rebooting an ESX Server host, see Rebooting an ESX Server (1003530). http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003530

Checking for resource starvation of the ESX service console is a VMware Knowledge Base article that explains how to use top utility to troubleshoot for (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003496&sliceId=2&docTypeID=DT_KB_1_1&dialogID=64395968&stateId=0%200%2064393864):

High CPU utilization on an ESX Server

High memory utilization on an ESX Server

Slow response when administering an ESX Server

Veeam Backup & Replication Upgrade Paths

$
0
0

Challenge

This article documents the path to be taken to get to the most recent version.

Solution

Important Note Regarding Performance
Veeam Backup & Replication provides an option during installation to create a local SQL Express instance. This option is often taken and as such older versions of Veeam that are then updated may be running on old SQL Express instances. This can affect performance. It may be advised to upgrade the local SQL instance to the most current supported version after first upgrading Veeam to the latest version and update. All versions of Veeam Backup & Replication after 8.0.0.917 (Patch 1) support SQL 2014.
 
Note: Versions 7 and higher require a 64 bit OS for installation.
 
Versions Prior to 6.0.0.x
If the server where the older version of Veeam Backup & Replication is currently installed meets the system requirements it is advised to uninstall the older version and install a newer version.
 
6.0.0.x\6.1.0.x
6.0.0.x\6.1.0.x -> 6.5.0.144 -> 8.x
Upgrade to 6.5.0.144 using https://www.veeam.com/kb1805
Upgrade to 8.x using the installer found on the download page. https://www.veeam.com/backup-replication-vcp-download.html
 
6.5.0.x
6.5.0.x -> 8.x
Upgrade to 8.x using the installer found on the download page. https://www.veeam.com/backup-replication-vcp-download.html
 
7.0.0.x
7.0.0.x -> 8.x
Upgrade to 8.x using the installer found on the download page. https://www.veeam.com/backup-replication-vcp-download.html
 

Creating Custom SureBackup Script

$
0
0

Challenge

If needed, you can create a custom SureBackup script to verify other ports or other tests.

Solution

To create a custom SureBackup script:

  1. Go into the SureBackup job and under Linked Jobs, select the VM you want to run the test script on and select Edit.
  2. Click Test Scripts
  3. Click Edit
  4. Name the script, click Browse.
  5. Pick c:\Program Files\Veeam\Backup and Replication(\Backup in version 8)\Veeam.Backup.ConnectionTester.exe
From here, you can type %vm_ip% to refer to the VM by IP address and type %vm_ip% <port_number> to test a VM on a specific port.

User-added image

User-added image




User-added image


 

SQL Transaction Log Backup - Failed to prepare guest for SQL transaction log backup Details: is responsible for SQL transaction logs backup

$
0
0

Challenge

Transaction Log Backup status on a job displays the following warning.
 
Failed to prepare guest for SQL transaction log backup Details: <Other Job> is responsible for SQL transaction logs backup

Cause

Veeam Backup & Replication is designed to only allow one Backup Job to be responsible for a SQL servers log backup. This is done to prevent possible restore issues. In order to prevent another job from attempting to take over the SQL log backup process there is a 7 day timeout in which the job currently responsible for the SQL log backup must not run.

Solution

If the job displaying this warning is not intended to be the primary SQL log backup job, disable the transaction log backup for that VM within that job.
 
If the job displaying this warning is intended to be backing up the SQL transaction logs, after disabling transaction log backup for the VM in the other job, the following three options are available:
 
1. Wait the 7 day timeout.
 
2. Change the TimeOut to a lower setting with the following registry key.


HKLM\SOFTWARE\Veeam\Veeam Backup & Replication\
DWORD: SqlBackupLogsAgeDaysToSkipLogBackup or SqlBackupLogsAgeDaysToSkipTruncate


Value: x
Where is x is the number of days.
 
Log truncate will be skipped for a VM if there are SQL logs of this VM collected last X day (defined in the registry as value
SqlBackupLogsAgeDaysToSkipTruncate/default is 7) by another existed job.
SQL log backup will be skipped for a VM if there are SQL logs of this VM collected last X day (defined in the registry as value
SqlBackupLogsAgeDaysToSkipLogBackup/default is 7) by another existed job.
 
3. Force Veeam to not consider the old job by expanding the old backup set and removing the transaction logs from the job. Doing so will remove all transaction logs for the SQL VMs in that backup set that were collected after the most recent restore point was created. You will still have point-in-time restores for all restore points leading up to the most recent backup. 

User-added image

 

Using VMware’s MOB (Managed Object Browser) to troubleshoot snapshot creation

$
0
0

Challenge

There presents a situation in which manual creation of a snapshot is needed.

Solution

First you must determine the Reference-ID for the VM to be tested. To discover this, navigate to the logging directory %programdata%\Veeam\Backup, within that directory find the folder that is named after the job that the VM is in. Open that folder and you will find a log named Task.<vmname>.<Ref-ID>.log
 
If the Ref-ID starts with “vm-“ the job is connecting to the vCenter server.
If the Ref-ID is just a number the job is connecting to a standalone host.
 
Now using an web browser navigate to: https://<vc/host ip>/mob/?moid=<Ref-ID>
 
Example for VC: https://abp-vc.abp.lab/mob/?moid=vm-688
Example for host: https://esxi1.abp.lab/mob/?moid=26
 
 
1.  Confirm that the name matches what is expected:

User-added image

2.  Click on “CreateSnapshot_Task”

User-added image

3.  In the window that pops up fill in the form.
(Note: If you are troubleshooting an issue where VMware Tools Quiescence was enable at the job level, enter a “yes” for the “quiesce” field. Otherwise leave both Boolean values set to no.)

User-added image

4.  Click “Invoke Method” to send the create snapshot task.
 
The snapshot creation task will then be processed. If the snapshot fails to create it is advised to contact VMware Support to investigate further.

User-added image
User-added image

If the snapshot creation is successful, remember to remove it.

 

Unable to retrieve next block transmission command when backing up to tape from a Linux repository

$
0
0

Challenge

When backing up to tape from a Linux repository, the following error may appear:

Unable to retrieve next block transmission command. Number of already processed blocks: [number of blocks].

The log file contains the following information:

cli| WARN|[Tape] Default EOM Warning (2147483648)! Initial free space: 5251268608, Written (this tape): 3104833536, Total written: 3104833536
     ERR |Asynchronous data reader has failed.
     ERR |End of file
     >>  |--tr:Cannot read data from the socket. Requested data size: [952531].
     >>  |--tr:Failed to unserialize buffer with data.
     >>  |--tr:Failed to unserialize data block.
     >>  |--tr:Failed to unserialize FIB block.
     >>  |Unable to retrieve next block transmission command. Number of already processed blocks: [4944].
     >>  |--tr:Next asynchronous read request cannot be processed.

Cause

By default, most SSH servers have set a timeout to disconnect idle clients. After idling for some time, your connection is dropped.
 

Solution

To make the keep alive settings available for all connecting clients, it is recommended to configure the settings in server side.

Modify the /etc/ssh/sshd_config and add the following lines :

ClientAliveInterval 30
TCPKeepAlive yes
ClientAliveCountMax 99999

After setting this, you have to restart the SSH server by issuing this command :

sudo service ssh restart

Note that restarting SSH server will not disconnect existing connecting clients via SSH.

More Information

Keep in mind that these changes will affect all SSH connections.

Error “Check share name failed” during install

$
0
0

Challenge

Veeam Backup & Replication installer halts with the error “Check share name failed.”

Cause

The “Server” service is either not started or non-existent.

Solution

If the “Server” service is present, start the service.
 
If the “Server” service is not present, follow these steps:
 
1.    Click Start, Run. Type ncpa.cpl and press [ENTER]
2.    Right-click on the LAN connection, and choose Properties
3.    Click Install
4.    Select Service, and click Add
5.    Select File and Printer Sharing for Microsoft Networks
6.    Click OK.

 

0x80042327 or Error: VSSControl: -2147212505

$
0
0

Challenge

You receive the following error on backup or replication processes during VSS guest processing:

0x80042327 or Error: VSSControl: -2147212505

Solution

0x80042327 translates to the VSS error “VSS_E_REBOOT_REQUIRED”.
 
This error is reflecting that the system is pending reboot due to an application installation or due to other processes that place the system into a reboot-pending state. It will be impossible to create a system state backup while this system is in a pending-reboot state, and only a crash-consistent backup may be achieved.

In some instances, you may be able to fool the system into thinking that this will be possible by restarting the Volume Shadow Copy Service. Eventually the system must be restarted, however.

Unable to access file [unspecified filename] since it is locked

$
0
0

Challenge

VMware holds a lock on the snapshot file while it is being consolidated. If there is an issue during this process, it is possible for further snapshots to be unable to consolidate, causing removal failures. You receive warnings that the snapshot removal process failed because '<unspecified filename>' is locked. Specifically:

RemoveSnapshot failed, snapshotRef "snapshot-xxx", timeout "3600000"
Unable to access file <unspecified filename> since it is locked

Cause

There are two possible causes:

  1. The VMware host held a lock on that snapshot, typically due to a communication error between Veeam and the host/VC.
  2. Virtual Appliance mode failed to unmount hot-added disks, which prevents the snapshot from being commited.

Solution

The solution varies based on the cause.
  1. If the VMware host held a lock on that snapshot
  2. Restart the management agents on that host
  3. Create a snapshot manually
  4. Perform a "Delete All" operation. Note: This may cause a temporary "stun" period, so take necessary maintenance precautions if necessary. IE http://www.veeam.com/kb1681
  5. This should clear the snapshot(s) present
If the Virtual Appliance mode failed to unmount hot-added disks:
  1. Remove all hot-added disks from the Veeam VM
  2. Create one snapshot manually
  3. Perform a "Delete All" operation

More Information

If the first problem occurs frequently, it is worth checking both the network connection between Veeam and the source host, as well as the overall load on the source host at the time the backup typically finishes.
 

SureBackup for a Virtual Machine with excluded disks

$
0
0

Challenge

How to run a VM directly from a backup file if some of its disks are excluded from backup.

Solution

To run a VM directly from a backup file, make sure you select ‘Remove excluded disks from VM configuration’ option when excluding your disks from the backup:

 

User-added image

More Information

If the ‘Remove excluded disks from VM configuration’ option is not enabled, the target VMX file needs to be edited manually and all excluded drives need to be removed from configuration of the Virtual Machine booted up in the isolated environment. 

Data error (cyclic redundancy check)

$
0
0

Challenge

The following entry can be seen in the job log:

[timestamp]  Error    Failed to delete oib '[xxx]' from ‘Filepath\file.vbk', redo. ‘Filepath\file.vrb'  at Veeam.Backup.Core.COibEx.Delete(COib currOib, COib prevOib, CBackupClient agent, IFileCommander fileComm)

[timestamp]  Error       at Veeam.Backup.Core.CSynOibAlg.Repair(COib currOib, CBackupSession session, CBackupClient agent, IFileCommander fileComm)

[timestamp] Error       at Veeam.Backup.Core.CVmBackupTarget.RepairOib(COib currOib, CDbBackupJobInfo job, CBackupSession session, CBackupClient agent, IFileCommander fileComm)

[timestamp] Error       at Veeam.Backup.Core.CBackupTarget.Repair()

[timestamp] Error       at Veeam.Backup.Core.CBackupTarget.Veeam.Backup.Core.ITarget.PreProcess()

[timestamp] Error       at Veeam.Backup.Core.CBackupJobPerformer.ExecuteJob()

[timestamp] Error       at Veeam.Backup.Core.CBackupJobPerformer.Execute()

[timestamp] Error    Undir failed   at Veeam.Backup.AgentProvider.CBackupClient.Undir(IDirVfsLink vbkDirLink, String rbkFilePath)

[timestamp] Error       at Veeam.Backup.Core.COibEx.Delete(COib currOib, COib prevOib, CBackupClient agent, IFileCommander fileComm)

[timestamp] Error    Client error: Data error (cyclic redundancy check)

[timestamp] Error    Failed to write data to the file [Filepath\file.vbk]. 

Cause

Cyclic Redundancy Check is an algorithm allowing Veeam Backup & Replication to check backup consistency. The error indicates that the backup chain did not pass this check, and the target files got corrupted.  This can also indicate problems on the physical storage where the backups are stored.

Solution

In the Veeam console, open the Backups section and locate the problem backup job. Right-click the job name and select "Delete from disk." After that is done, restart the job.


How to create a transactionally consistent backup of both Windows and Linux VMs

$
0
0

Challenge

A job backing up/replicating both Linux and Windows VMs needs to be configured.

Solution

Enable both the ‘Enable VMware tools quiescence’ option when specifying advanced settings at the ‘Backup Destination’ step of the wizard and the ‘Enable application aware processing’ option at the ‘Guest Processing’ step.

In the Guest OS credentials section, specify guest operating system credentials for target VMs that are required to deploy the agent. Please note that the user name must be supplied in the DOMAIN\USERNAME format.

Click Advanced… to specify advanced options for Veeam VSS. The Advanced Options window contains a list of VMs that will be processed with Veeam VSS and indexing tools. You can exclude specific VMs from processing or add them:

  • To add a VM, click Add VM… and select a VM you want to process. The Add objects list contains only those VMs that you added to the backup job. To display all VMs in the virtual infrastructure hierarchy, select the Show full hierarchy check box.
 
  • To exclude a VM, select it in the list and click Remove.

To provide granular quiesencing option for a VM, select it in the list and click Edit. In the Applications section on the Applications tab select the ‘Disable Application processing’ or ‘Ignore application processing failure’ option for all Linux machines.

More Information

When you select both the ‘Enable VMware tools quiescence’ option and the ‘Enable application aware processing’ option for a job at the same time, the VSS module will  be used for processing backed up and replicated VMs at first. In case VSS fails or is disabled for a specific machine (e.g. Linux VM), this machine will be processed with the VMware tools quiescence option instead.

Failed to start client agent on the host

$
0
0

Challenge

Backup job fails with lines in the logs like this:

[timestamp] <01> Error    Failed to start client agent on the host '***.***.*.***' 'à Veeam.Backup.AgentProvider.CSshAgentFactory.Veeam.Backup.AgentProvider.IAgentFactory.StartClient() 

 <…>

[timestamp] <01> Error    Timeout to start agent   à Veeam.Backup.AgentProvider.CClientAgentProtocol.WaitReadyState()

 

[timestamp] <01> Info     Task operation result: 'Failed to start client agent on the host '***.***.*.***'\nTimeout to start agent'

 

[timestamp] <01> Info     Job has been stopped successfully. Name: [Réplication ***].

Cause

Non-latin characters in the job name are not supported.

Solution

Rename the job. (example: "Réplication" - "Replication")

vCenter Migration Utility

$
0
0

Challenge

If a Virtual Machines MoRef is changed due to migration to new vCenter / inventory change in vCenter, you would receive the error "Object "vmname" is not found".  Please see KB1299 for more details.

This utility is designed to assist in re-mapping the MoRef ID's of the Virtual Machines you are backing up and/or replicating using Veeam Backup & Replication, so that backup incremental chains remain intact, after an inventory change in VMware vSphere vCenter. Do not run any jobs before using migration tool otherwise migration process won't be possible.

Please note that this utility is provided as a courtesy, and is not officially supported by Veeam Technical Support.  Use at your own risk.

VMware vSphere 6.5 and Veeam 9.5 is supported now!

Cause

Veeam Backup & Replication tracks Virtual Machines in its jobs by a Managed Object Reference ID (MoRef), which will change when moved to another vCenter, or removed and re-added to inventory.

Solution

Please download this bundle: tool and instructions.  Before using this tool, it is highly recommend to take a configuration backup of your VeeamBackup database
Please note that the utility will work only with a source vCenter migration. If Replication target vCenter is changed, you will need to manually map source VMs to their replicas

More Information

MD5 checksum for the archive: 75ad4597e8b21c78d72cff37859312e9

Version string portion was too short or too long

$
0
0

Challenge

When trying to configure Veeam ONE Reporter data collection tasks, the following error message pops up:
Version string portion was too short or too long

Cause

The issue is due to manual Veeam ONE database deployment that leaves blank fields in the common.serversettings table required for correct product functioning.

Solution

Before applying the instructions below, make sure you have a full backup of Veeam ONE SQL database.

To find out the Windows version, on Veeam ONE server open CMD and run ver command. Use the x.x.x.x format.

Run the following queries against Veeam ONE database, substituting the fields marked in red with appropriate values:

exec common.SetSetupDataReporterURI 'https:\\reporterfqdn:1239'
go

exec common.SetSetupDataBusinessViewURI 'https:\\businessviewfqdn:1340'
go
exec common.SetSetupDataWindowsVersion '%WINVERSION%'
go


 

More Information

If you have any questions, contact Veeam Support.

How to create Veeam ONE database manually

$
0
0

Challenge

  • You need to pre-create the Veeam ONE database on a Microsoft SQL Server before installing Veeam ONE.
  • You need to re-create the Veeam ONE database on a Microsoft SQL Server.

Solution

  1. Log on to a SQL instance intended to host the Veeam ONE database using SQL Management Studio.
  2. Right-click on the Databases node > New Database > specify the name. This will create the container for the new Veeam ONE database.
  3. Locate the Veeam ONE database creation script in Veeam ONE .iso file's Addins\SQLScript directory.
  4. Right-click on the newly created Veeam ONE database > New query and load the contents of the script and execute it. This will create all necessary Veeam ONE  tables, stored procedures, etc.
  5. Perform the steps outlined in the Solution section of this KB article.
  6. Configure the connection to the database following the steps from this KB article. This step is omitted if you are pre-creating the database before software installation.
  7. To install Extension Modules for Veeam Reporter, locate them in Veeam ONE .iso file's Addins\ExtensionModules directory, copy and place them into "%PROGRAMFILES%\Veeam\Veeam ONE\Veeam ONE Reporter Server" folder on Veeam ONE server > restart Veeam ONE Reporter Server service. Make sure these files disappear after the service restart.
NOTE: The database will be of a release version. If Veeam ONE installation is patched you must install the patch once again after configuring the SQL connection and installing Extension Modules. Failure to do so will result in database/software versions mismatch and product nonoperation.
 
 

More Information

If you have any questions, contact Veeam Support.
Viewing all 4474 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>