Quantcast
Channel: Veeam Support Knowledge Base
Viewing all 4506 articles
Browse latest View live

How to Escalate a Support Case or Talk to a Support Manager

$
0
0

Challenge

The Veeam Support Management Team welcomes all feedback, and can assist you with getting a case escalated or reassigned.

Instructions on creating a support case can be found here.

Solution

It is Veeam Support internal policy that all request for remote session or escalation are to be honored. This may be communicated to the Veeam Support engineer currently assigned to the case either in an e-mail message, or during a phone conversation. Please, keep in mind that to satisfy the request for escalation to a higher tier, the current engineer may have to collect more diagnostic information from you. If you have non-technical concerns with the handling of the case, you may also request to talk to a manager. The engineer will communicate your feedback to the Veeam Support Management Team, and manager will contact you.

If you need to get in touch with the Veeam Support Management Team directly, please use the Talk to a Manager section found in the Veeam Customer Portal. Along with your feedback, please provide the number of the case that you would like to discuss and the contacts we can use to reach out to you.
 

User-added image


Windows 10 VM Hyper-V Guest processing skipped on Host W2K12R2

$
0
0

Challenge

Guest Windows 10 VMs will fail to engage VSS when application-aware image processing is enabled generating an error:
Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
 

Cause

Integration services on Windows 10 VM are newer than the integration services on the host and not (or at least at this point) backwards compatible. You can verify this by opening Hyper-V host Event Viewer and go to:
 
Event Viewer -> Applications and Serivces Logs -> Microsoft -> Windows -> Hyper-V Integration -> Event ID 4000

User-added image

If you log into the VM and try to update the integration services, you will get a message:

User-added image
 

Solution

Disable application aware image processing for the Windows 10 VM and use "Hyper-V quiescence" instead or engage Microsoft for further information regarding integration services compatibility.

Note: In some cases this issue might occure when guest OS is Windows Server 2016. 
If this is the case ensure that all updates are installed on host and guest OS.
If this doesn't resolve the issue, you might want to uncheck 'guest services'.
 

More Information

For more information on "Hypver-V quiescence" please go to our Helpcenter.

Job reports warning "Failed to truncate transaction logs for SQL instances: Possible reasons: lack of permissions, or transaction log corruption."

$
0
0

Challenge

Job may report warning: "Failed to finalize guest processing. Details: Failed to process 'TruncateSQLLog' command. Failed to truncate transaction logs for SQL instances:  <instance name>. Possible reasons: lack of permissions, or transaction log corruption."

Solution

SQL logs truncation is done under user account specified in AAIP in Job settings, if it fails then GuestHelper tries to truncate transaction logs under LocalSystem account.
In order to understand why SQL logs truncation failed you will need to open the GuestHelper log in Guest VM:
\\GUESTVM\c$\ProgramData\Veeam\Backup\VeeamGuestHelper_%date%.log (for Windows 2008 or higher)
\\GUESTVM\c$\Documents and Settings\All Users\Application Data\Veeam\Backup\VeeamGuestHelper_%date%.log (for Windows 2003)
Then do search within the log using keyphrase "Truncation Statistics"

1) Error: OpenFromInitializationString failed. [Login failed for 'DOMAIN\user'.]  
Solution: give DOMAIN\user permissions  on SQL instance and add db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.

2) OLEDB Error: 'The server principal "DOMAIN\user" is not able to access the database "DATABASE" under the current security context.', HelpCtx: '0'
Solution: give DOMAIN\user db_backupoperator role for all FULL and BULK databases, or give it a sysadmin role.

3) 
OLEDB Error: 'BACKUP detected corruption in the database log. Check the error log for more information.', HelpCtx: '0'            
Solution: error points to possible corruption and issues with SQL server

4)  OLEDB Error: 'BACKUP LOG cannot be performed because there is no current database backup.' 
 
As a rule this is an issue with the secondary node of the SQL always on cluster. You can solve this by making a backup of the DB in question via SQL Management Studio. Otherwise, you can set the secondary node as primary for just one run of your backup job. As a result all its DBs will be backed up without "copy only" flag and the error will disappear.

The issue occurs when the secondary node has always been backed up with "copy only" flag and its standalone DBs do not have any full backup. Thus during the truncation of the standalone DB logs we get the above-mentioned message.

The same solution applies if you get this message with regard to the excluded vCenter database / Veeam database.
 
5) "Query timeout expired". If you see this entry in VeeamGuestHelper log, it usually means that we couldn't truncate SQL logs in allotted time (by default timeout is only 60 seconds). Usually you might experience such issues with rather large databases, and with large amount of transaction logs.

Solution: Implement the following registry value in affected VMs in [HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\] and [ HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\VeeaM\Veeam Backup and Replication] (if there is no latter, you will need to create it):

SqlExecTimeout

Type: REG_DWORD
Default value: 60 (in seconds, decimal)

Try to expand that value and run a backup afterwards, safe guess here is to set it for 600 seconds.

If you observe the following warning "Failed to truncate transaction logs for SQL instances: MICROSOFT WID" on Veeam B&R version 8.0.0.2084, please contact Veeam Support for the hot-fix.

SQL Log Truncation and Log Backup Silently Fails After Installing Veeam 9.5 Update 3

$
0
0

Challenge

After installing Veeam 9.5 Update 3, SQL logs are truncated during backup, but truncation task is still marked as success in job statistics. SQL Log Backup sessions complete with 0 KB transferred.

You can confirm if any SQL VMs are affected by this change by checking the Guest Helper log located on the Guest VM.

GuestHelper log location:  \\GuestVM\c$\ProgramData\Veeam\Backup\VeeamGuestHelper_%date%.log

The specific error you will find in the GuestHelper log if affected is the following:

Description = The UPDATE permission was denied on the object 'backupset', database 'msdb', schema 'dbo'.

 

Cause

This is due to a change in required SQL Permissions in 9.5 Update 3. These permissions requirements were changed to resolve a previous issue with how Veeam sends requests to SQL to finalize database backup and log truncation. The specific change in permissions is that the msdb system database now requires db_datawriter role, whereas before this was not needed.

Required SQL Permissions for SQL log backup and truncation can be found here:
https://helpcenter.veeam.com/docs/backup/explorers/vesql_permissions.html?ver=95

 

Solution

To resolve this issue, please give the following SQL permissions to the account credentials specified to use for Guest Processing:
  • Sysadmin role, or the following can be used if you need to provide only minimal permissions:
    • SQL Server instance-level roles: dbcreator and public
    • Database-level roles: db_backupoperator, db_denydatareader, public
    • For System Databases:
      • master - db_backupoperator, db_datareader, public;
      • msdb - db_backupoperator, db_datawriter, db_datareader, public
    • Securables: view any definition, view server state
    • For truncation of SQL Server 2012 or SQL Server 2014 database transaction logs, this account should have the db_backupoperator database role (minimal required) or the sysadmin server role.



 

VeeamZIP licensing VMs in Full version of Veeam

$
0
0

Challenge

VeeamZIP began using a license up in paid versions after 9.5.0.1536.

Cause

This was expected behavior, but the decision has been made to change this behavior in order to allow users to back up VMs with VeeamZIP without using a license.

Solution

This will be fully changed in future updates. There is a fix that will change the behavior in the current version of the software. Please contact support to retrieve this fix.

More Information

https://www.veeam.com/support.html

How to move the Veeam Backup & Replication software to another server

$
0
0

Challenge

You need to move Veeam Backup & Replication software and database to a new server.

Solution

Note: This section will refer to the server where Veeam Backup & Replication is installed prior to the move as the “old server”, and the location after the move as the “new server”.

Performing migration using Configuration Backup:
 
Perform the following steps on the old server:
1. Stop and disable all jobs.
2. Perform configuration backup:
http://helpcenter.veeam.com/backup/80/vsphere/index.html?vbr_config_manually.html?ver=95
3. Close Veeam GUI and stop all Veeam Services.


 
Perform the following steps on the new server:
 
1. Install Veeam Backup & Replication with a brand new DB.
2. Apply the latest patch to Veeam Backup & Replication: http://www.veeam.com/patches.html
3.  Make sure all local drives that were being used as repositories on the old server are now attached with the same drive letters on the new server.
4. Perform configuration backup restore:
http://helpcenter.veeam.com/backup/80/vsphere/index.html?vbr_config_restore.html?ver=95 
Note: It will be necessary to re-enter the password for every account during restore of non-encrypted configuration backups.
5. Run a test job to make sure everything moved correctly.
 

Performing migration if Configuration Backup is not available:
Note: Use this option only if Configuration Backup is not available. If encryption was used on the previous installation, a newly installed backup server will not be able to decrypt such information if attached to the existing database, so any encrypted information will have to be supplied manually – existing encrypted backups have to be re-imported to provide the password.
 
Perform the following steps on the old server:

1. Stop and disable all jobs.
2. Close Veeam GUI and stop all Veeam Services.
3. Create backup of the SQL database. (
http://www.veeam.com/kb1471)

To find out where the VeeamBackup database is located, open regedit on the old server and navigate to:
HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
 
There you will find 3 dword values:

- SqlServerName
- SqlInstanceName
- SqlDatabaseName

 
Perform the following steps on the new server:

1. Install Veeam Backup & Replication with a brand new DB.

2. Apply the latest patch to Veeam. http://www.veeam.com/patches.html
3. Make sure all local drives that were being used as repositories on the old server are now attached with the same drive letters on the new server.
4. Restore the Veeam database to the instance on the new server or on a SQL server which will host Veeam DB. (link)
5. To associate Veeam with the restored database, please use the DBConfig Utility to change the DB and settings that Veeam is connecting to.
Please refer to this user guide page on using the DBConfig Utility:
https://helpcenter.veeam.com/docs/backup/hyperv/using_dbconfig_utility.html?ver=95
6
. Restart Veeam Services to apply the changes.
7. Open the Veeam console and using the “Manage Credentials” button from the main menu and check the passwords for every account to confirm they are up to date.
8. Run a test job to make sure everything moved correctly.
 

More Information

In situations where physical local storage is used please see the following KB that details how to relocate Veeam Backup and Replication backup file locations: https://www.veeam.com/kb1729

For steps on migrating the Veeam database to another SQL Instance or Server, please refer to KB 1250:
https://www.veeam.com/kb1250

Veeam Backup for Office 365 Timeout Errors

$
0
0

Challenge

A Veeam Backup for Office 365 job fails with a timeout error.  

These commonly are reported as one of the the following errors:
  • Unexpected End of File
  • Exchange Web Services error code: ErrorExceededConnectionCount
  • Error: Failed to get folder properties
  • Error: You have exceeded the available concurrent connections for your account.  Try again once your other requests have completed.
  • Error: Unexpected end of file has occurred. The following elements are not closed: m:Data, m:ExportItemsResponseMessage, m:ResponseMessages, m:ExportItemsResponse, s:Body, s:Envelope. Line 1, position 21397505.
But are not limited to these errors.

 

Cause

There are multiple causes for these timeout errors.  The most common being a bad connection to the Exchange Web Service (EWS) from the Veeam Backup for Office 365 server or proxy.  This can be caused by a flooding of open worker threads to the Exchange Web Services server, network traffic saturating the throughput of the connection, a slow internet connection to the Office 365 EWS server amongst other issues.

Solution

In 1.5, performance settings to help with timeout errors can be adjusted through the proxy performance settings (Concurrent Threads, and limit network bandwidth).  Test modifications of the proxy performance settings FIRST, this will create the <source   / > entry in the Proxy.xml file for you.  If adjustment of these performance settings still cannot resolve the timeout errors, we can modify the Proxy.XML data (found by default at %programdata%\Veeam\Backup365\) by adding the following information into the XML document under <source>:

<Source BatchSize=”100” BatchPart= “100” BatchTimeout= “600” BatchMaxItemSize=”256” />

Keys explanation (default values):
•    BatchSize (100) - Maximum number of items requested from EWS (Exchange Web Services) per one query
•    BatchPart (100) – Should be set the same as the BatchSize at this moment
•    BatchTimeout (600) - Timeout for getting a reply with results from EWS in seconds
•    BatchMaxItemSize (256) - Maximum size of an item (in MB) requested from EWS per one query. If the item size exceeds BatchMaxItemSize, item will be requested separately
In 1.0, these performance settings would need to be configured in the Config.xml file (found by default at %Programdata%\Veeam\Backup365).

 

Backup job fails with “VIX Error Code: 13” or “Access is denied. Code: 5”

$
0
0

Challenge

The backup or replication job fails with the following error message:
 
VIX Error: You do not have access rights to this file Code: 13
 
OR
 
RPC function call failed. Function name: [IsSnapshotInProgress].RPC error: Access is denied. Code: 5
 

Cause

The issue occurs when Veeam is unable to start the agent that triggers Microsoft VSS on the guest machine because of insufficient privileges. It may also be related to VMware Tools malfunction.

Solution

To resolve:

1. Ensure that the account being used by Veeam is a member of the Local Administrators group on the VM that is to be backed up.
2. If the account being used is not named “Administrator”, you must disable UAC on the Guest OS of the VM to be backed up.

 
a.  For 2008/2008 R2, in the “Change User Account Control Settings”, move slider to Never Notify
b.  For 2012/2012 R2/2016, you must change the “EnableLUA” DWORD to 0 in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\policies\system

3.  Uninstall VMware Tools, and Reinstall VMware Tools. (Please see https://kb.vmware.com/kb/2010137 for more information). If you are unable to upgrade existing VMware tools see https://kb.vmware.com/kb/1001354

 

More Information

For more information regarding disabling UAC under 2012 and 2012 R2, please review the following: http://social.technet.microsoft.com/Forums/windowsserver/en-US/0aeac9d8-3591-4294-b13e-825705b27730/how-to-disable-uac?forum=winserversecurity

While most service accounts created have the ability to manage files, folders, and services, many may not have the rights to execute VSS tasks. This is why a user that is in the Administrator group should be specified for application-aware image processing and guest filesystem indexing.
 
By default in a VMware vSphere environment if Veeam Backup & Replication is not able to reach the <ip>\Admin$ share of the Guest VM, it will failover to a network-less protocol called VIX.
 
In situations where UAC must remain enabled, named Administrator accounts must be used for this process. Only administrative accounts with SID-500 access will be able to execute remote administration commands with this Windows feature enabled. These will be the local “Administrator” account made locally when installing windows, or the “Administrator” account used with the domain. Created domain administrator accounts have a default SID-512 and may not be sufficient for remote administration.

 

Veeam B&R limitations for VHDX Live resize usage

$
0
0

Challenge

Live resizing of VHDX files (available in Windows 2012 R2 Hyper-V and later) will be causing following scenarios:

a)  If you shrink the VM disk the backup task for this VM will fail and the following exception can be found in the Agent.<Job name>.Source.<VM name> log file:
[date time] <  6440> cli| VHDX.GetDiskInformation
[date time] <  6440> cli|   (EString) DiskPath = vhdx://<shadowSpec><extent path="full path to the disk" shadow="full path to the disk in the shadow copy" isAutoRecovery="False" /></shadowSpec>
[date time] <  6440> cli|   (EString) Option.Container = {disk}
[date time] <  6440> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[date time] <  6440> vhd| Opening VHDX by full path: [full path to the disk].
[date time] <  6440> cli| File 'HostFile{full path to the disk in the shadow copy}' opened (read_only: true, no_cache: false)
[date time] <  6440> dsk| Loading VHDX log.
[date time] <  6440> dsk| Loading VHDX log. Failed.
[date time] <  6440> cli| File 'HostFile{full path to the disk in the shadow copy}' closed
[date time] <  6440> cli| ERR |Failed to process method {VHDX.GetDiskInformation}
[date time] <  6440> cli| >>  |VHDX file was truncated. Current file size: [1503242747904]. Flushed file size: [1825365295104].
[date time] <  6440> cli| >>  |Failed to replay VHDX log.
[date time] <  6440> cli| >>  |--tr:Failed to load VHDX log.
[date time] <  6440> cli| >>  |An exception was thrown from thread [6440].
[date time] <  6440> cli| Trying to poll message from the notification channel (from server side).
[date time] <  6440> net| Triggering AUX channel synchronization. Request id: {guid}
[date time] < 10260> net| Received synchronization response {guid}
[date time] <  6440> cli| No server exception available.
[date time] <  6440> cli| ERR |Failed to process {Invoke} command
[date time] <  6440> cli| >>  |Details:
[date time] <  6440> cli| >>  |VHDX file was truncated. Current file size: [1503242747904]. Flushed file size: [1825365295104].
[date time] <  6440> cli| >>  |Failed to replay VHDX log.
[date time] <  6440> cli| >>  |--tr:Failed to load VHDX log.
[date time] <  6440> cli| >>  |Agent failed to process method {VHDX.GetDiskInformation}.


b) If you expand the VM disk the backup of such a VM may become inconsistent.
 

Cause

After Live resize VHDX metadata is not automatically flushed.

Solution

After performing Shrink of a running virtual machine you should shutdown the VM (not Reboot) and power it on again so that backup job would not fail. However after Expand of a running virtual machine additionally you have to perform new backup for this VM.
 

Migrating Veeam Availability Console to a New Server

$
0
0

Challenge

You need to migrate Veeam Availability Console (VAC) and its database to a new server. 

Cause

Follow the steps below to move VAC to a new server.

Solution

Before starting the migration procedure, make note of the existing VAC server version.
  1. Install a new instance of VAC on a new server. 
  2. If you are planning to use a fresh VAC database, create one during the installation on the SQL Server Instance wizard step.
  3. If you wish to use the existing database: 
      • Move the database beforehand to an existing SQL server instance using one of the ways outlined here.
      • On the SQL Server Instance wizard step specify the valid database name.
        • NOTE that in case the previous VAC installation was updated there will be a server-database mismatch. Create a new database during the installation on the SQL Server Instance wizard step. By default VAC service creates a database with the release version build that you need to update. Install the latest update, applied on the old VAC server, on a new VAC instance and proceed to the next step.
    • Access VAC UI and navigate to Configuration > Settings > SQL Server Connection and specify the valid database name and click Save.
    User-added image

     

    More Information

    The account used to connect to the Microsoft SQL Server hosting theVeeam Availability Console database must have the following permissions:
    • public role (default permissions)
    • dbcreator role (needed to create the Veeam Availability Console database during the Veeam Availability Console installation procedure)
    • db_owner and public roles on the Veeam Availability Console database
    • public permissions on the master database
    • public permissions on the msdb database
    • [For Always-On Availability Groups] VIEW SERVER STATE, VIEW ANY DEFINITION permissions

    An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

    $
    0
    0

    Challenge

    After installing Windows April update KB4015553 Veeam jobs and services fail at different operations with:
     
    "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full"
    
    

    Cause

    This particular update (and all consequent rollups that include it) causes the system to run out of free RPC ports when there are failed iSCSI connections.

    Solution

    Repairing or removing unsuccessful iSCSI connections, followed by the server reboot, was found to fix the issue. Additionally, remove any unused targets from the ‘Favorite Targets’ tab that are associated with these connections as they may be reconnected after reboot.

    This issue has been addressed by Microsoft. Apply the appropriate update below to resolve the issue

    For 2016:           https://support.microsoft.com/en-us/help/4025334
    For 2012 R2:      https://support.microsoft.com/en-us/help/4025335

    -------------------------------------------------------------------------------------------------------------------------------------------------

    The update that resolved the above Buffer Space iSCSI issue (KB4025335) has caused another iSCSI issue that only affects Backup from Storage Snapshots using an iSCSI connection.
     
    After installing Windows Update KB4025335, KB4034681, KB4025333, or KB4025336, Veeam jobs utilizing Backup from Storage Snapshots using an iSCSI connection fail with the following errors:
     
    “Error: Cannot retrieve physical extents for connecting iSCSI target IQN iqn.2007-11.com.nimblestorage:lun-nim-oracle-veeam-lun-nim-oracle-v069a713ff86811d1.000000d1.fef260ee.s069a713ff86811d1.000000de.0000b1cb on proxy IQN iqn.1991-05.com.microsoft:mtl-s-fs0005.solotech.local”
     
    In the logs, it can be seen with this error:
     
    Task Log:
    “[25.07.2017 09:44:54] <61> Error Failed to find physical drive for LUN with lun Id 'WwnOrIqn: 'iqn.1992-04.com.emc:cx.apm00140946425.a1', LunId: '10',
     
    [25.07.2017 09:44:54] <61> Error Unable to retrieve devices for iSCSI session. number of retry attempts: '20'. Interval: '100'. (Veeam.Backup.Common.CCppComponentException)
    [25.07.2017 09:44:54] <61> Error in c++: void __thiscall disklib::vmfs_san::iscsi_api::iscsi_discovery::GetDevicesForSessionWithRetry(struct _ISCSI_UNIQUE_SESSION_ID &,class std::vector<struct ISCSI_DEVICE_ON_SESSIONW,class std::allocator<struct ISCSI_DEVICE_ON_SESSIONW> > &,int,int) failed.
    [25.07.2017 09:44:54] <61> Error in c++: void __thiscall disklib::vmfs_san::iscsi_api::iscsi_discovery::LoginTarget(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,class std::vector<struct disklib::vmfs_san::iscsi_api::IscsiDriveInfo,class std::allocator<struct disklib::vmfs_san::iscsi_api::IscsiDriveInfo> > &) failed.”

     
     
    Proxy Resources Log:
     “[04.08.2017 13:50:55] < 4016> cli| VMFSSAN.ConnectISCSITarget
    [04.08.2017 13:50:55] < 4016> cli| (EString) TargetName = iqn.1992-04.com.emc:cx.apm00140946425.a1
    [04.08.2017 13:50:55] < 4016> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    [04.08.2017 13:50:55] < 4016> dsk| Retrieving devices for target 'iqn.1992-04.com.emc:cx.apm00140946425.a1'.
    [04.08.2017 13:50:55] < 4016> dsk| WARN|The data area passed to a system call is too small.”

     
    The above issue is resolved by installing Windows Update KB4034663, which can be found here:
     
    https://support.microsoft.com/en-us/help/4075212/windows-81-update-kb4075212

    More Information

    https://social.technet.microsoft.com/Forums/SharePoint/en-US/492f62ce-aec8-4777-98f0-0c9dbf302954/problem-with-kb4015553?forum=winserverwsus

    https://support.microsoft.com/en-us/help/4015550
    This security update introduced an issue in which, if an iSCSI target becomes unavailable, attempts to reconnect will cause a leak. Initiating a new connection to an available target will work as expected.

    If after performing the steps outlined above the issue still occurs, a support case with Microsoft may be needed.

    "The request was aborted: Could not create SSL/TLS secure channel" after installing Windows Updates

    $
    0
    0

    Challenge

    After installing the security Windows Updates released in September or October 2016, outgoing HTTPS connections from Veeam server to vCenter and NetApp storages may fail with

    The request was aborted: Could not create SSL/TLS secure channel.

    Any other HTTPS connections may be affected as well.

     

    Cause

    After installing one or more of the following security updates:

    2012 R2 and Windows 8
    KB3185331
    KB3188743
    KB3174644
     
    2008 R2 and Windows 7
    KB3185278
    KB3185330
    KB3192391
    KB3175024
    KB3172605


    Please note that Windows Monthly Quality Rollups may replace the aforementioned updates. For more information consult Microsoft Update Catalog.


    If the public key length is shorter than 2048 bytes (e.g. RSA 512), Windows may cancel the HTTPS connection. 

    Solution

    To fully resolve the problem, regenerate vCenter or NetApp certificates so that the public key length is at least 2048 bit.
    Please note that KB3172605 update causes issues with SHA-1 certificates even with the longer public key. For that one, you may want to generate the SHA-2 certificate.

    As a workaround, you can uninstall the updates if you have them installed.

    More Information

    Microsoft Security Bulletin Summary for September 2016  https://technet.microsoft.com/en-us/library/security/ms16-sep.aspx

    Vmware docs: Create Self-Signed Certificates
    NetApp docs: security certificate create​

     

    No disks have been mounted…Windows FLR

    $
    0
    0

    Challenge

    When attempting to perform a Windows FLR (Guest Files ‘Windows’) the restore fails with:

    User-added image

    No disks have been mounted. Either disks of this type aren’t supported or some errors occurred. See log file for details.

    Solution

    Confirm that the VM that the files are being restored from was using one the following filesystems.        
    • FAT
    • FAT32
    • NTFS
    • ReFS*
     ──────────────────────────────────────────────────────────
    Troubleshooting Steps
    ──────────────────────────────────────────────────────────
     
    1.  Clear all temporary files from “C:\Users\<serviceaccount>\AppData\Local\Temp\”

    Note: When the Veeam Backup Service account is ‘local system’, the temporary directory used is C:\Windows\Temp.

    2.  Open an Administrator Command Prompt and run the following commands:
    (The installation path for Veeam if it was upgraded from 6.5 is: C:\Program Files\Veeam\Backup and Replication\)
    (With Veeam v9 the operation must be performed on a Mount Server under the path: C:\Program Files\Common Files\Veeam\Backup and Replication\Mount Service)
    cd “C:\Program Files\Veeam\Backup and Replication\Backup”
    vdk remove
    vdk install

     
    3.  Confirm that the account being used for the Veeam Backup Service has the "Perform volume maintenance tasks" right assigned to it. This can be located within the group policy (gpedit.msc) under:
    Computer Configuration>Windows Settings>Security Settings>Local Policies>User Rights Assignment
    4.  Reboot the Veeam server

     

    More Information

    To restore files from a VM running Microsoft Windows ReFS 3.x, the console and mount server must be installed on machines running Microsoft Windows Server 2016.

    To support restore of files from a VM running Microsoft Windows ReFS 2.0 or earlier the console and mount server must be installed on machines running Server 2012/Server 2012 R2.


    If the server you are attempting to restore from has volumes with native 2012 data deduplication, the Veeam console and mount server must be installed on the same operating system and also have the “Data Deduplication” role enabled. The “Data Deduplication” role appears under "File and ISCSi Services" in server manager.

    If the Windows FLR continues to fail after attempting the steps outlined in this KB, please attempt and Other-OS FLR. For more information regarding how to do that please review the following:

    http://helpcenter.veeam.com/backup/70/vsphere/multios_restore_intro.html


     

    How to Install Veeam PowerShell Snapin

    $
    0
    0

    Challenge

    This KB documents how to install the optional PowerShell Snapin after installation of Veeam Backup & Replication has occurred.
     

    Solution

    Please find and execute BPS_x64.msi from the “Backup” folder found on the Veeam Backup & Replication installation media.

    User-added image

    More Information

    Read our blog for information on how to use Veeam PowerShell Snap-in for Hyper-V backup

    Starting in Version 9 of Veeam Backup and Replication, the Powershell Snap-In is included as a base install with the Console.

    VDDK error: 13 - Troubleshooting

    $
    0
    0

    Challenge

    Backup/Replication jobs fail with:
     
    VDDK error: 13.You do not have access rights to this file

    Solution

    Below is a list of possible solutions to this issue sorted by what transport mode was being using when this error occurred.
    Note: There are many causes for VDDK 13, this list is not intended to be all encompassing.

    Using Transport Mode: Network Mode
    • Solution 1:
      In some instances, you may receive this error when permissions are not correctly set. To confirm, please follow the steps outlined in the following KB article to test ports, permissions, and DNS resolution
      http://www.veeam.com/kb1198
    Using Transport Mode: Hotadd Mode
     
    • Solution 1:
      https://www.veeam.com/kb1054
      VMware Tools must be installed and up-to-date on the Veeam server and any Veeam proxies
       
    • Solution 2:
      https://www.veeam.com/kb1775
      Stuck Hotadded Disks – Review all virtual proxies for disks attached that do not belong to the VM in question, remove the disks not owned by this VM without deleting from the datastore. Once deleted, consolidate snapshots on the affected VMs.

       
    • Solution 3:
      kb.vmware.com/kb/2003638
      Orphaned Snapshots – Consolidate snapshots on affected VMs

       
    • Solution 4:
      http://www.veeam.com/kb1875
      Permissions - Please configure the account leveraged to access the vCenter as an Administrator within vCenter

       
    • Solution 5:
      http://www.veeam.com/kb1882
      Automount – In some cases, it may be necessary to perform an automount scrub to clear any existing mount points that are no longer in the system.

      Run the following via CMD Prompt on the proxy server:
      diskpart
      automount disable
      automount scrub
      exit

       
    • Solution 6:
      http://www.veeam.com/patches.html
      Patching – Please ensure that all VMware and Veeam components are fully up to date.

       
    • Solution 7:
      In some rare instances, performing a storage vMotion has been known to resolve the issue due to a locked VM file on the datastore where the VM files reside.

       
    • Solution 8:
      In some rare instances, restarting the vCenter appliance or alternatively restarting the ESXi management agents (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490) may yield positive results if all above testing does not resolve the issue.
       
    • Solution 9:
      Veeam Proxy has a Veeam replica with the same bios UUID within the vCenter Server. The following can be observed in 

      Job.Job_name.log:
      [time stamp] < threadID> Info VM VeeamProxy _replica(vm-100001) has the same bios uuid as proxy VeeamProxy.local in the DB

      Agent.Job_name.VM_name.Hotadd.Attacher.log:
      [time stamp] < threadID> vdl| [vddk] time stamp  error -[07132] [Originator@6876 sub=transport] Cannot use mode hotadd to access [datastore_name] VM_name/VM_name.vmdk: Cannot mount using this method. (Mounting VM vm-101 using transport hotadd failed : Cannot access datastore for one of the disks of Virtual Machine VM_name..)

      The behavior is observed when a Veeam Proxy is replicated with Veeam B&R, the replication process preserves the original VM bios UUID for restore purposes. To work around the issue setup a dedicated VM as a Veeam Proxy, replicating Veeam Proxies is not usually necessary due to ease of deployment.
    Using Transport Mode: Direct SAN
    Solution 1:
    When receiving VDDK Error 13 when leveraging DirectSAN access mode, please confirm that the LUNs are presented to the proxy correctly and have the proper permissions. For assistance deploying & troubleshooting DirectSAN please follow the below KB articles:
    http://www.veeam.com/kb1895

     

    More Information

    Additional Troubleshooting(All versions):
    • Manual Hotadd - http://www.veeam.com/kb1184
    • Hotadd Requirements - http://www.veeam.com/kb1054
    • Deploy an additional standalone proxy to isolate failures to a single specified proxy
    • To isolate a specific transport mode as the point of failure, you may also test the other transport modes to ensure that connectivity is correctly established and backup jobs work as intended via the other transport mode.
    Additional Troubleshooting (vSphere 5.1 and below):
    http://www.veeam.com/kb1777
    Backup via Standalone host – Configure backup job to backup via Standalone host rather than through vCenter; this will isolate vCenter as the point of failure. If backup via standalone host is successful, you can attempt restarting the vCenter as well as the management agents on the host(s) where the proxies/VM(s) reside.

    Information: Hotadd will only work when the proxy is on the same host as the object being backed up

    Additional Troubleshooting (vSphere 5.5):
    When attempting to hotadd leveraging a standalone host configuration (that is managed by a vCenter object) with vSphere 5.5 you will receive VDDK Error 13, as well as the error listed below:

    Error message: [00680 error 'transport'] Reconfigure attempt failed for VM "2" with Vmomi::MethodFault::Exception ("vim.fault.HostAccessRestrictedToManagementServer")\n

    To resolve this particular issue, please leverage Network Mode or Direct SAN transport mode as this configuration is not supported due to limitations of VMware

    Information: For more information regarding ‘HostAccessRestrictedToManagementServer’ please see below VMware documentation.

    http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.fault.HostAccessRestrictedToManagementServer.html

     

    How to Configure Email Notifications in Veeam Agent for Windows

    How to: Collect Information Necessary for Cloud Connect Technical Support Cases

    $
    0
    0

    Challenge

    Given the additional complexity of Cloud Connect compared with standalone installations of Backup & Replication, additional logging and information is needed to expedite cases with our Cloud Connect teams whenever feasible. All information should be obtained whether you are submitting an issue as a Veeam Cloud Connect Tenant or a Veeam Cloud Connect Provider .

    Solution

    Necessary Infrastructure Information

    Include the following information:

    • Provider Name:
    • Tenant Name:
    • Job Type that is having issues:
    • Job Name that is having issues:
    • Tenant Public IP:

    *Please document the public IP of the Tenant Veeam Server connected to the cloud gateway. This is used to identify the connection in the Cloud Gateway logs. You can identify this by using a utility such as http://www.whatsmyip.org/

    Tenant Logging

    Please reproduce the error (if applicable), then collect and submit tenant logs for the job as you would any standard Backup & Replication case as documented here: https://www.veeam.com/kb1832

    Collecting Logs from Tenant Environment

    • For failing jobs:
      Export logs for the job (backup, backup copy, or replication) that is having an issue.

      User-added image
    • For restores and replica failovers:
      • Export a log bundle for the backup or replication job (as above)
      • Manually collect the following directories from the Veeam Backup server.
        Starting in %ProgramData%\Veeam\Backup\:
        • Entire directory "CloudConnect" (if it exists)
        • Entire directory with the name of the VM that was backed up or replicated

    Service Provider Logging

    After reproducing the error (if applicable), collect and submit provider logs for the job using the instructions below:

    Collecting Logs from Service Provider Environment

    • For providers with newly-configured infrastructure, or a small number of clients, use the built-in log generator tool to gather logs from  all  managed servers related to the issue. It is important to gather logs from cloud gateways, repositories, and other relevant infrastructure in addition to the backup server.

      Specify the component logs for all servers and specify only the last 2-3 days. 
      User-added image
    • Providers with a larger number of clients can use this tool, but it will produce a large (multi-GB) log bundle. To avoid this, see “Advanced Service Provider Location”, below.

    Advanced Service Provider Location

    It’s best to save this information to a .zip file with an identifiable name such as "<tenant_name>-ProviderLogs.zip"

    Some files listed may not be present if they are not applicable to your issue, but make sure you are looking at the correct server for each different directory or filename. Some files will have several rotations, with the oldest files archived to .zip or .gz.

    Veeam Backup and Replication Server :

    Collect these files in the root of %ProgramData%\Veeam\Backup\:

    • Svc.VeeamBackup* (filenames beginning with “Svc.VeeamBackup”)
    • Zip files named "<date stamp>_Svc.VeeamBackup"
    • CloudAppliance*
    • In rare cases, other files in this directory may be relevant, so it is prudent to include anything with a recent modified date.

    Collect the contents of these folders :

    • %ProgramData%\Veeam\Backup\Console\ (if using remote console, collect from that machine as well)
    • %ProgramData%\Veeam\Backup\CloudConnect\
    • %ProgramData%\Veeam\Backup\CloudConnectService\<tenant_name>\
    • %ProgramData%\Veeam\Backup\CloudConnectService\ (excluding subfolders for other tenants)


    Cloud Gateway Servers :

    •  %ProgramData%\Veeam\Backup\Svc.VeeamGateSvc*

    WAN Accelerator (if applicable ):

    • %ProgramData%\Veeam\Backup\Svc.VeeamWANSvc*.log

    Repository Servers (for backup and backup copy jobs) :
    Check which server is listed as the Host for the Repository in the console: Backup Infrastructure > Backup Repositories. Collect the entire content of these folders:

    Windows:

    • %ProgramData%\Veeam\Backup\CloudConnectService\<tenant_name>\
    • %ProgramData%\Veeam\Backup\CloudConnectService\ (excluding subfolders for other tenants)

    Linux:

    • /var/log/VeeamBackup/CloudConnectService_<tenant_name>_* (may be multiple directories)

    More Information

    Logs from the Network Extension Appliances are automatically copied to their controlling Veeam Backup & Replication server, so in most cases it is not necessary to gather logging from these appliances manually.

    Be sure to alert your engineer if you have issues with being able to contact your provider or obtain this logging in any way so we can assist

    Using the Diskshadow Utility to manually test VSS operations

    $
    0
    0

    Challenge

    This article describes the steps needed to manually create a volume shadow copy using the Diskshadow command line utility in Windows.

    Cause

    Veeam products use the Microsoft Volume Shadow Copy Service (VSS) for a variety of tasks:
    • In VMware environments, VSS ensures transaction consistency when backing up Windows VMs.
    • In Hyper-V on Windows Server 2008R2, 2012, and 2012R2, VSS is the primary component of the backup API.
    • Veeam Agent for Windows (formerly Veeam Endpoint Backup) uses VSS to help make sure data is consistent.
    • VSS is used to trigger transaction log truncation for Microsoft Exchange.
    When troubleshooting, it is sometimes necessary to test if the Volume Shadow Copy Service is operating correctly. A quick test for a single volume can be performed using the Shadow Copies utility (see KB1794). Windows Server Backup is generally sufficient for isolation tests (and may be preferred by Microsoft Support), but it does not support all possible configurations.

    The Diskshadow command line utility provides a greater degree of support, and is a better match for how Veeam products use the VSS API. However, it is only available on Windows Server 2008 or newer; it is not available for client versions of Windows.

    Solution

    Test shadow copy creation, then review Interpreting the Results (below).  

    VMware Environments or Agent Backups

    Copy-Only Mode

    Use this method when transaction log truncation is not relevant. For best comparison, specify copy-only in the Veeam job settings.

    From an administrator command prompt run the following commands:

    ​diskshadow/l <output path for log file> (the /l parameter is optional)
    set verbose on
    set context volatile

    Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.
    Examples:

    add volume \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx}\
    add volume X:
    add volume C:\ClusterStorage\Volume1

    See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

    create

    Normally, shadow copies created this way will be automatically deleted when Diskshadow exits or is reset, but the delete shadows command is available if needed.

    Note: Some issues may be isolated to a single volume. To identify which volume is responsible for an error, add only one volume to the backup set, create the shadow copy, then exit or reset Diskshadow before starting over with the next volume.
     

    Full Mode

    Full mode is the default setting for Veeam jobs. This method will notify VSS writers that a backup has occurred. Some applications, including Microsoft Exchange, will truncate transaction logs automatically in response to this notification. Other applications, such as Microsoft SQL Server, record that a backup has been taken but do not truncate transaction logs.

    To test log truncation along with the creation of the shadow, run the following commands from an administrator command prompt:

    diskshadow /l <output path for log file> (the /l parameter is optional)
    set verbose on
    set context volatile

    Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.
    Examples:

    add volume \\?\Volume{3c8d1bcb-9d95-11e5-80b0-806e6f6e6963}\
    add volume X:
    add volume C:\ClusterStorage\Volume1

    See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

    begin backup
    create

    Once the shadow copy is created, inform the VSS writers that the backup has completed. For some writers, this will trigger transaction log truncation:

    end backup

    Normally, shadow copies created this way will be automatically deleted when the end backup command is completed, but the delete shadows command is available if needed.

    Hyper-V Environments

    Note: The following information is applicable to Server 2008R2, 2012, and 2012R2.

    Veeam Backup & Replication creates shadow copies of volumes containing virtual machines. For transactionally-consistent backups (Application-Aware Image Processing or Hyper-V Quiescence), the Hyper-V VSS Writer triggers the Hyper-V Volume Shadow Copy Requestor service in each VM to create shadow copies of volumes within the VM. Error messages generated by shadow copy creation failure generally do not indicate whether the problem is isolated to a hypervisor volume or an in-guest volume.

    Start by isolating whether the failure occurs only when using Application-Aware Image Processing or Hyper-V Quiescence. If the problem does not occur when both options are disabled, test shadow copy creation from within the virtual machine as described in the VMware Environments or Agent Backups section, above.

    If snapshot creation is failing for crash-consistent backups, or if in-guest shadow copy creation does not result in any errors, it may be necessary to test shadow copy creation on the Hyper-V host. Because Hyper-V backup can be complex, a scripted approach is recommended. Veeam Support can run a scripted test that also collects a significant amount of useful logging. Contact support to schedule a remote session, and ask about hv_mkshadow.

    Warning: Attempting to perform a shadow copy of a volume containing virtual machines will trigger in-guest shadow copy creation on all VMs on that volume unless specifically excluded. This may have negative effects on performance or responsiveness of these VMs and the Hyper-V host. Veeam Backup and the hv_mkshadow script exclude unnecessary VSS writer components from the shadow copy.

    Finding Volume GUIDs

    To add volumes that lack a mount point (such as the System Reserved Partition), use the volume GUID obtained through either the mountvol command-line tool, or the Shadow Copies utility.

    To access the Shadow Copies utility, right click any volume and choose Configure Shadow Copies. On older operating systems, this may instead be a tab in Properties. In the Shadow Copies utility:

    1. Select the volume displayed as a GUID;
    2. Click Settings…
    3. Copy the volume GUID from the settings window and paste it into DiskShadow.

    User-added image


    Alternatively, run “mountvol” from a command prompt; below the usage information will be a list of GUIDs and their associated mount points, if any.

    More Information

    Interpreting the Results

    Typically, if shadow copy creation fails via both Diskshadow and Veeam products, that indicates that the problem is isolated to shadow copy creation. Troubleshoot the errors reported by Diskshadow and any events appearing in Event Viewer.

    Troubleshooting shadow copy creation or transaction log truncation may require the assistance of Microsoft technical support.

    Two examples of failures reported directly by Diskshadow are as below. The first is an instance where the Volume Shadow Copy service fails, hangs, or is shut down midway through the creation process. The “RPC server” error can be a bit misleading; in this case a call was sent to the Volume Shadow Copy service and it was not there to receive it.

    COM call "m_pVssBackup->StartSnapshotSet" failed.
    
    The last operation failed.
    - Returned HRESULT: 800706ba
    - Error text: The RPC server is unavailable.

    Another example shows a more common issue, that of the volume having insufficient free storage for the shadow copy creation.

    The last operation failed.
    - Returned HRESULT: 8004231f
    - Error text: VSS_E_INSUFFICIENT_STORAGE

    These messages will usually immediately follow the list of included writers, and should be the last output returned in the Diskshadow process.

    In some cases, shadow copy creation will succeed via Diskshadow, but error or warning events will be generated in the event log. If similar errors are generated when Veeam backup or replication jobs fail to create a shadow copy, that usually indicates that the Veeam application is aborting the shadow copy in response to the errors. In this situation, the Veeam application can usually be assumed to be functioning correctly, so it is best to focus troubleshooting on any events appearing in Event Viewer.

    Troubleshooting shadow copy creation or transaction log truncation may require the assistance of Microsoft technical support.
    The error “COM call "lvssObject4->GetRootAndLogicalPrefixPaths" failed” may be ignored. For more information, see Microsoft KB3025158.

    How to configure antivirus exclusions to prevent interaction with Veeam Backup & Replication

    $
    0
    0

    Challenge

    This article documents antivirus exclusions that have been found to help prevent antivirus software from interrupting Veeam Backup & Replication communications with components such as proxies and repositories or with the guest OS of VMs that are being backed up or replicated.

    Cause

    In some rare instances antivirus can prevent certain executables, processes, or agents from being correctly deployed or leveraged from the Veeam server, proxy, repository, or even the guest OS.

    Solution

    Antivirus Exclusions:

    Folders on the Veeam Server:

    • C:\Program Files\Veeam
    • C:\Program Files (x86)\Veeam
    • C:\Program Files\Common Files\Veeam
    • C:\Program Files (x86)\Common Files\Veeam
    • VBRCatalog ( [HKLM\SOFTWARE\Veeam\Veeam Backup Catalog] CatalogPath value)
    • NFS (Configured in each repository, stored in [HKLM\SOFTWARE\Wow6432Node\Veeam\Veeam NFS] RootFolder value)
    • C:\VeeamFLR\*
    • C:\Windows\Veeam

    Folder on Guest OS for VSS:

    • C:\Windows\VeeamVssSupport
    • C:\Windows\VeeamLogShipper
    • \Device\HarddiskVolume*

    Folders on VMware Backup Proxies and CIFS Repository Gateway:

    • C:\Program Files(x86)\Veeam
    • C:\Windows\Veeam

    Folders on Windows Repositories:

    • C:\Program Files(x86)\Veeam
    • C:\Windows\Veeam
    • *Also include the path the backup files*

    Folders on WAN Accelerator:

    • C:\Program Files(x86)\Veeam
    • C:\Windows\Veeam
    • *Path of WAN cache on WAN accelerators*

    Files:

    • VeeamAgent.exe
    • VeeamAgent64.exe
    • *.vmdk
    • *.flat

    Due to the complex nature of antivirus software some additional exclusions may be needed.
    If your antivirus has a logging or history system please review this to see actions it may have taken that may affect Veeam Backup & Replication operations.

    HCL Tape - Quantum Scalar i6000

    $
    0
    0

    Challenge

    Product Information:

    Product Family: Scalar Tape Devices 
    Status: Veeam Ready - Tape 
    Classification Description: Tape device where available hardware features have been tested to work with Veeam. 

    Solution

    Product Details:

    Model number: i6000 
    Library firmware version: 760Q.GS25000
    Drive firmware version: H9EC
    Driver for tape drive: IBMTap 6.2.6.6 
    Driver for media changer: IBMTap 6.2.6.6 
    Additional support: M8 media, multipath (Up to LTO6)
    General product family overview: Scalar libraries are the intelligent choice for archiving applications and long-term data retention. Proactive and automated monitoring and management reduces administration time and service interruptions keeping IT staff focused on key business objectives. With ease of management, industry leading performance, reliability and scalability, it’s easy to see why Quantum is the market share leader in tape backup and automation. 


    Veeam Details

    Veeam Build Number: 9.5.0.1536

    More Information

    Company Information

    Company name: Quantum 
    Company overview: Quantum is a leading expert in scale-out storage, archive and data protection, providing intelligent solutions for capturing, sharing and preserving digital assets over the entire data lifecycle. We help customers maximize the value of these assets to achieve their goals, whether it’s top movie studios looking to create the next blockbuster, researchers working to accelerate scientific discovery, or small businesses trying to streamline their operations. With a comprehensive portfolio of best-in-class disk, tape and software solutions for physical, virtual and cloud environments, we enable customers to address their most demanding workflow challenges and opportunities.

    Viewing all 4506 articles
    Browse latest View live


    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>