Quantcast
Channel: Veeam Support Knowledge Base
Viewing all 4469 articles
Browse latest View live

Best Practices for Microsoft Deduplication

$
0
0

Challenge

You could experience significant slowdown in performance or failures(in rare cases) when performing Backup or Backup Copy tasks to volumes with Microsoft Windows deduplication enabled where resulted backup files are larger than 1TB in size.

Cause

The issue comes from Windows deduplication limitations, which could cause significant performance degradation while reading/appending data from/to very large files (more than 1TB). MS Requirements for Data Deduplication: https://msdn.microsoft.com/en-us/library/hh769303(v=vs.85).aspx

Solution

a) Avoid growth of backup files more than 1TB on deduplication storage, split backup jobs if possible(reducing amount of VMs per job).
b) If size of VMs does not allow splitting, try to avoid Windows deduplicated volumes usage and transfer backup files to another repository without MS Windows deduplication enabled.

More Information

Best practices for MS deduplicated volumes as backup repository:

1) It is important to format the volume with large NTFS File Record Segment (FRS) (4096 bytes instead of 1024 by default) as you could face NTFS limitation errors in future. You could verify FRS size with the following command:
fsutil fsinfo ntfsinfo <volume pathname>
Command to reformat NTFS volume with larger FRS (/L):
format <volume pathname> /L

2) Use maximum NTFS allocation unit size(cluster size) of 64Kb.
Considering option mentioned in pt. 1,  two commands should be united into one:
format <volume pathname> /A:64K /L

3) Avoid growth of files more than 1TB;

4) For Backup jobs preferred backup method is Forward Incremental with Active Full backups enabled.

Introduction to Data Deduplication in Windows Server 2012: http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx
Plan to Deploy Data Deduplication: https://technet.microsoft.com/en-us/library/hh831700.aspx

"The request was aborted: Could not create SSL/TLS secure channel" after installing Windows Updates

$
0
0

Challenge

After installing the security Windows Updates released in September or October 2016, outgoing HTTPS connections from Veeam server to vCenter and NetApp storages may fail with

The request was aborted: Could not create SSL/TLS secure channel.

Any other HTTPS connections may be affected as well.

 

Cause

After installing one or more of the following security updates:

2012 R2 and Windows 8
KB3185331
KB3188743
KB3174644
 
2008 R2 and Windows 7
KB3185278
KB3185330
KB3192391
KB3175024

KB3172605

If the public key length is shorter than 2048 bytes (e.g. RSA 512), Windows may cancel the HTTPS connection. 

Solution

To fully resolve the problem, regenerate vCenter or NetApp certificates so that the public key length is at least 2048 bit.
Please note that KB3172605 update causes issues with SHA-1 certificates even with the longer public key. For that one, you may want to generate the SHA-2 certificate.

As a workaround you can uninstall the updates if you have them installed.

More Information

Microsoft Security Bulletin Summary for September 2016  https://technet.microsoft.com/en-us/library/security/ms16-sep.aspx

Uninstalling Veeam Backup & Replication plug-in for Labtech

$
0
0

Challenge

Uninstalling the plug-in using Plugin Manager only removes it from the list of plugins but leaves Veeam components (reports, monitors, etc.) and database data intact.

Cause

Uninstalling the Veeam Backup & Replication Plug-in by means of the LabTech Plugin Manager is not supported. If you accidentally remove the plug-in using the Plugin Manager, you will additionally need to delete the backup historical data from the LabTech database. To delete data, you must run a SQL script provided by Veeam Software against the LabTech database.

Solution

To correctly uninstall the plug-in follow this procedure:
  1. Download the following script if you are running plugin version 1.1 and the following script if you are running plugin version 1.1.2.
  2. Open the LabTech Control Center.
  3. From the Control Center menu, choose Help > Plugin Manager > Disable Veeam Backup & Replication Plugin.
  4. Click Advanced, select Manage Plugins > Remove plugin.
  5. If you are running LabTech 10 or 11 click Advanced, select Reload Plugins > Reload DBAgent Plugins. In the displayed dialog box, click Yes. Wait for LabTech to restart the database agent. If you are running LabTech 2013 skip this step.
  6. Run the script from step 1 against LabTech database.
  7. Restart the Control Center.
  8. After you restart the Control Center, clear the LabTech system cache. To do this, from the Control Center menu, choose Tools > Reload System Cache.

More Information

Veeam monitors may not disappear from the UI at once. To force this procedure, restart LabTech MYSQL service.

Hot-Add/Virtual Appliance Failures

$
0
0

Challenge

All hot-add/Virtual Appliance Jobs fail with the following:
 

BackupDiskTrad failed
Client error: Failed to open VDDK disk [[My_Datastore] VM_Name/Disk1.vmdk] ( is read-only mode - [true] )
Logon attempt with parameters [VC/ESX: [vcenter];Port: 443;Login: [domain\veeam_user];VMX Spec: [moref=vm-5555];Snapshot mor: [snapshot-55];Transports: [hotadd]] failed because of the following errors:

Server error: End of file


 

Also, the Veeam box may have been moved or cloned recently, causing the UUID of the Veeam box to change.



The issue may be that the Veeam box has been moved or cloned, and the UUID of the Veeam box has changed. After this occurs, the host that Veeam is on can no longer find the Veeam VM by the UUID properly. You will see the following in the logs:

[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'SOAP'] Sending soap request to [TCP:vcenter:443]: findByUuid\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Could not find virtual machine.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Transport mode hotadd failed pre-flight check. Ignoring.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Checking transport mode nbdssl.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Transport mode nbdssl passed pre-flight checks. Adding to list of available transports.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Checking transport mode nbd.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Transport mode nbd passed pre-flight checks. Adding to list of available transports.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Narrowing down applicable transport modes.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'transport'] Disregarding transport mode hotadd.\n
[timestamp] <34> Info           [AP] (Client) output: VixDiskLib: No transport modes availble to access disks.\n
[timestamp] <34> Info           [AP] (Client) output: \n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 info 'App'] Successfully released all resources.\n
[timestamp] <34> Info           [AP] (Client) output: [timestamp 00832 trivia 'SOAP'] Sending soap request to [TCP:vcenter:443]: logout\n
[timestamp] <18> Info           [AP] (Client) error: Failed to open VDDK disk [[My_Datastore] VM_Name/Disk1.vmdk] ( is read-only mode - [true] )
[timestamp] <18> Info           [AP] (Client) error: --tr:{
[timestamp] <18> Info           [AP] (Client) error: --tr:VDDK error: 3.One of the parameters was invalid

Cause

After moving/cloning or copying the Veeam box, the UUID on the VMware side and the UUID that Microsoft gives a machine will go out of sync. Also the host we are trying to do hot-add/virtual appliance mode with cannot find the UUID provided, as it has changed and is no longer valid.

Solution

To resolve, run the following PowerShell script on the Veeam box:
 

$uuid = (Get-WmiObject Win32_ComputerSystemProduct).UUID
$array_uuid = @()
$array_uuid += $uuid.split("-")
$tempArray = @()
for($j = 0; $j -lt 5;$j++)
{
    $tempArray = @()
    for($i=0;$i -lt $array_uuid[$j].length;$i+=2)   
    {
        $tempArray += $array_uuid[$j][$i] + $array_uuid[$j][$i+1]
    }
    if($j -lt 3){ [array]::Reverse($tempArray)}
   
    $array_uuid[$j] = $tempArray
}
$array_uuid[3][0] = $array_uuid[3][0].Insert(0, "-")
$myUuid = ($array_uuid | Out-String).Replace("`n"," ").Replace(" -","-")
if((Get-WmiObject -Class Win32_ComputerSystem).SystemType -match "(x86)")
    {Set-ItemProperty "HKLM:\software\VMware, Inc.\VMware Virtual Disk Development Kit\volatile\" -Name "UUIDHost" -value $myUuid}
else
    {Set-ItemProperty "HKLM:\software\Wow6432Node\VMware, Inc.\volatile\" -Name "UUIDHost" -value $myUuid}

 

Windows 10 VM Hyper-V Guest processing skipped on Host W2K12R2

$
0
0

Challenge

Guest Windows 10 VMs will fail to engage VSS when application-aware image processing is enabled generating an error:
Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
 

Cause

Integration services on Windows 10 VM are newer than the integration services on the host and not (or at least at this point) backwards compatible. You can verify this by opening Hyper-V host Event Viewer and go to:
 
Event Viewer -> Applications and Serivces Logs -> Microsoft -> Windows -> Hyper-V Integration -> Event ID 4000

User-added image

If you log into the VM and try to update the integration services, you will get a message:

User-added image
 

Solution

Disable application aware image processing for the Windows 10 VM and use "Hyper-V quiescence" instead or engage Microsoft for further information regarding integration services compatibility.
 

More Information

For more information on "Hypver-V quiescence" please go to our Helpcenter.

High Veeam ONE Database IOPS Index

$
0
0

Challenge

Starting with version 9, Veeam ONE database server shows high IOPS usage of approximately 700-800 operations per second with regular spikes exceeding 2500 operations every 5 minutes.

Cause

As a real-time monitoring solution, Veeam ONE actively performs input-output operations against its database. Version 9 uses modified monitoring engine. Data collection and database data writing procedures have been overhauled, increasing the list of collected performance counters, the number of operations per second in regards to SQL database and WMI. The data are collected and written to the database using sophisticated algorithms for processing performance data and alarms.

The aforementioned processes cause relatively stable IOPS index with regular 5-minute spiking that references to the frequency with which bulk insert into the database takes place.

Solution

This behavior is by design. Possible solutions to balance the load include:
  • Migrating Veeam One database to a dedicated datastore to minimize the impact on other applications
  • Changing the Scalability mode as per KB2017

More Information

If you have any questions, contact Veeam Support

Deduplication Ratio Does Not Reflect Deduplicating Storage

$
0
0

Challenge

In the backup properties (pictured below), the “Data Size” or “Backup Size” is larger than expected, or the “Deduplication” column in the backup statistics is different from the ratio reported by duplicating storage appliances.
User-added image

Cause

Veeam Backup & Replication does not request information from storage appliances about the size of files as stored on the appliance. All values in this user interface would be the same if the data was written to non-deduplicating storage.

This limitation applies to all storage, whether integrated (HPE StoreOnce, EMC DataDomain, ExaGrid) or not.

Effect of Job Settings on Deduplication Ratio

Virtual disks are stored in each backup file as a combination of data blocks and tables of pointers to those blocks.

When inline data deduplication is enabled in the backup job settings, the deduplication table will contain many pointers to a smaller number of blocks. For example, a full backup of a single virtual disk might contain ten thousand blocks; if many of these blocks are identical, the backup file would contain a table of ten thousand pointers to only a few thousand actual data blocks.

When inline data deduplication is disabled (such as when using the default settings for writing to a deduplication appliance), each entry in the table of blocks for a virtual disk either points to a data block or is ‘sparse’, representing a block containing no data. In the above example, a full backup of a 40 GB virtual disk containing 30 GB used space becomes a backup file containing 10 GB of sparse blocks and 30 GB of actual blocks. Incremental backups of such a VM would usually not contain a significant number of data blocks containing zero data, because incremental backups do not read unchanged data. However, exclusion of deleted file blocks and VM guest files will result in sparse blocks being stored in an incremental backup file. In the above example, over 8 GB of the free space in the VM consists of deleted file or “dirty” blocks, so the incremental data size is 8.27 GB when deduplication is disabled. This occurs even though the zero blocks are not actually read during the incremental backup: in the example image, the VM was powered off, so no data (0.0 KB) was read from the VM disk during any incremental backup.

When inline data deduplication is enabled, these zero blocks are instead handled by the deduplication table in such a way that they do not contribute to the deduplication ratio or to the “Data Size” statistic. In the example image above, deduplication was enabled for the most recent incremental backup, so the “Data size” is negligible.

The deduplication ratio listed in the backup properties is the ratio of blocks in tables in the backup file to actual blocks stored in the file. For that reason, when the backup file contains a large number of sparse blocks, the listed deduplication ratio will be very high, or will be listed as 0.0x.
 

Solution

This user interface correctly reflects the backup file contents, and is working as designed.
 

More Information

The description of the backup file format in this article is simplified for clarity.

“Backup Size” may be larger than the size of data stored within the VM with the default settings for storage appliances (4 MB blocks, no inline deduplication, decompress before storing), even after accounting for deleted or hidden files.  This is because of the large block size: a 4 MB block containing a very small of data will still contribute 4 MB to the Backup Size. This empty space will be deduplicated by the storage appliance.
 

Veeam One - Steps to Compile Logs

$
0
0

Challenge

In some cases, logs will need to be reviewed by Veeam Support to help diagnose issues.

Solution

In Veeam One Monitor console navigate to Options > Server Settings > Other Settings > Launch Support Utility

or

Browse to or run "C:\Program Files\Common Files\Veeam\Veeam ONE Settings\VeeamOneSettings.exe". In the lower left corner click Export Logs.

User-added image

Exporting logs may take a while. When it is finished, you will be asked to choose the location for the logs to be saved to.

User-added image

Please attach these to your case, or in the event that the total attachment size is under 3.5MB, you may attach via email reply.

User-added image

If none of these options for upload are working, please notify Veeam Support and an engineer will provide FTP credentials for the case. Once you receive the FTP link, follow the below instructions to upload the logs via FTP. You can start by copy and pasting the URL with credentials, as this has the credentials built into the link. Once you have copied the URL, you can use a Windows\File Explorer browser (windows key + e) and paste the link into that browser, or your preferred FTP client.

Note: Internet Explorer, Firefox, or any other browser will not work for the FTP upload.


1. Open Windows\File Explorer.

User-added image


 2. Paste URL with credentials.

User-added image

 3. Drag and Drop the Zipped folder or individual logs into the browser.

User-added image

 4. Once completed, e-mail your Veeam Support engineer to inform them the logs have been successfully uploaded to ensure the quickest review and response.

More Information

Below are the relative log path lists for Veeam One. If there are difficulties compiling the logging requested via the Veeam One support tool instructions above, please obtain the files from the appropriate directory below and provide Veeam Support a .zip file copy.

Veeam ONE Monitor - C:\Users\<USER>\AppData\Local\Veeam\Veeam ONE Monitor
Veeam Reporter\Business View Service logs - C:\Users\<USER>\AppData\Local\Veeam\Veeam ONE
Veeam Reporter Web logs - C:\Program Files\Veeam\Veeam ONE\Veeam ONE Reporter Web\Veeam\Veeam ONE Reporter
Veeam Business View Web logs - C:\Program Files\Veeam\Veeam ONE\Veeam ONE Business View Web\Generated
VeeamOneSettings - C:\Users\<USER>\AppData\Local\Veeam\VeeamOneSettings

Changing SQL Server or database connection settings for Veeam ONE

$
0
0

Challenge

You need to change SQL Server or database connection settings for Veeam ONE.

Solution

To change SQL Server or database connection settings for Veeam ONE Server, you can use the Veeam ONE Settings utility.

To modify settings, follow the instructions below.

Provision a domain account for Veeam ONE Monitor and Veeam ONE Reporter on the new SQL Server. This can be one account or two different accounts. Move the database to the new SQL Server. Grant to the account(s) db_owner permissions on the VeeamOne database.  

Next, modify the accounts used by monitoring and reporting components of Veeam ONE.

First, modify connection settings for Veeam ONE Monitor. To accomplish this, follow the next steps:
  1. Launch the Veeam ONE Settings utility available from Veeam ONE Monitor console > Server Settings > Other Settings > Launch Support Utility or under the following path:

    C:\Program Files\Common Files\Veeam\Veeam ONE Settings\VeeamOneSettings.exe

  2. In the menu on the left, select General, open the Database tab and make the necessary changes in the SQL Server section.

    User-added image

  3. Make sure this account has sufficient privileges on the Veeam ONE database on the SQL Server side, as it will be used for performance data collection.

  4. Click the Test Connection button to check whether new configuration settings are correct.

  5. Save changes.

  6. Restart Veeam ONE Monitor Server service.

Then modify connection settings for Veeam ONE Reporter and Business View. To accomplish this, follow the next steps:

  1. In the list of services, locate Veeam ONE Reporter Server service and choose to modify its properties. Open the Log On tab and specify the necessary account for the service in the Log on as section.

  2. Make sure this account has sufficient privileges on the Veeam ONE database on the SQL Server side, as it will be used for data collection.

  3. Launch the Veeam ONE Settings utility available under the following path:

    C:\Program Files\Common Files\Veeam\Veeam ONE Settings\VeeamOneSettings.exe

  4. In the menu on the left, select General, open the Web Identity tab and make the necessary changes in the Web identity section.

    User-added image

  5. Make sure this account has sufficient privileges for Veeam ONE's database on a SQL Server side, as it will be used for overall Web UI operation and reports creation.

  6. Save changes.

  7. Restart Veeam ONE Reporter Server service.

More Information

If you have any questions or problems, please contact Veeam Support at http://www.veeam.com/support.html

Veeam One Tuning for Larger Environments

$
0
0

Challenge

Veeam One dropping data due to slow database performance when monitoring 1K+ VMs.

Cause

For efficient use of Veeam One in environments with 1 000+ VMs it is recommended to make scalability adjustments.

Solution

For better scalability it is possible to change real-time data to 1 datapoint in 5 minutes and change the granularity of level 2 aggregated data to 1 point in 20 minutes. Here are the steps:

1. Take a snapshot of Veeam One Server VM
 
2. Stop Veeam One Monitor Server and Veeam One Reporter Server services
 
3. Take a backup of Veeam One database
 
4. In the registry of Veeam One Server open HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam ONE Monitor\Service and create value RtLevel2QueryInterval (DWORD) = 300 (decimal)
 
5. Execute the following script on Veeam One DB:
 
UPDATE monitor.PerfCounterInfo SET rt_interval = 300, level2_interval = 1200
 
6. Start Veeam One Monitor Server and Veeam One Reporter Server services

More Information

For Veeam ONE version 8.x and higher refer to this KB Article.

Contact us at the Customer Support Portal if you have any questions

How to create Veeam ONE database manually

$
0
0

Challenge

  • You need to pre-create the Veeam ONE database on a Microsoft SQL Server before installing Veeam ONE.
  • You need to re-create the Veeam ONE database on a Microsoft SQL Server.

Solution

  1. Log on to a SQL instance intended to host the Veeam ONE database using SQL Management Studio.
  2. Right-click on the Databases node > New Database > specify the name. This will create the container for the new Veeam ONE database.
  3. Locate the Veeam ONE database creation script in Veeam ONE .iso file's Addins\SQLScript directory.
  4. Right-click on the newly created Veeam ONE database > New query and load the contents of the script and execute it. This will create all necessary Veeam ONE  tables, stored procedures, etc.
  5. Configure the connection to the database following the steps from this KB article. This step is omitted if you are pre-creating the database before software installation.
  6. To install Extension Modules for Veeam Reporter, locate them in Veeam ONE .iso file's Addins\ExtensionModules directory, copy and place them into "%PROGRAMFILES%\Veeam\Veeam ONE\Veeam ONE Reporter Server" folder on Veeam ONE server > restart Veeam ONE Reporter Server service. Make sure these files disappear after the service restart.
NOTE: The database will be of a release version. If Veeam ONE installation is patched you must install the patch once again after configuring the SQL connection and installing Extension Modules. Failure to do so will result in database/software versions mismatch and product nonoperation.
 
 

More Information

If you have any questions, contact Veeam Support.

CID Mismatch in VMware on Restore with Backup and Replication V9 installed on Server 2008R2

$
0
0

Challenge

Upon restore using Veeam Backup and Replication V9X you receive a CID mismatch error when you are on vSphere 5.5 Update 3b

Cause

This is a known issue with restoring with the option to use VM tags is enabled in vSphere 5.5 Update 3b

Solution

Known workarounds:
  1. Upgrade Backup and Replication server to Windows 2012 or 2012R2
  2. Deselect 'Restore VM tags'
  3. Disable the following regkey https://technet.microsoft.com/en-us/library/dn786418(v=ws.11).aspx#BKMK_SchannelTR_ClientCacheTime

Enhanced encryption must be enabled for vCenter 6.5 with disabled TLS 1.0

$
0
0

Challenge

You get the following error trying to add VMware vCenter 6.5 or higher to Veeam Virtualization Extensions UI:
 
Cannot connect to "vCenter_Name on port 443". The underlying connection was closed: An unexpected error occurred on a send. Authentication failed because the remote party has closed the transport stream. 

Cause

This is expected behavior if TLS 1.0 and SSLv3 are disabled on the target VMware Server.

Solution

To force .Net assemblies to use more secure encryption follow steps described in the following VMware KB article.

More Information

If you have any questions, please contact Veeam Support.

Using the Diskshadow Utility to manually test VSS operations

$
0
0

Challenge

This article describes the steps needed to manually create a volume shadow copy using the Diskshadow command line utility in Windows.

Cause

Veeam products use the Microsoft Volume Shadow Copy Service (VSS) for a variety of tasks:
  • In VMware environments, VSS ensures transaction consistency when backing up Windows VMs.
  • In Hyper-V on Windows Server 2008R2, 2012, and 2012R2, VSS is the primary component of the backup API.
  • Veeam Agent for Windows (formerly Veeam Endpoint Backup) uses VSS to help make sure data is consistent.
  • VSS is used to trigger transaction log truncation for Microsoft Exchange.
When troubleshooting, it is sometimes necessary to test if the Volume Shadow Copy Service is operating correctly. A quick test for a single volume can be performed using the Shadow Copies utility (see KB1794). Windows Server Backup is generally sufficient for isolation tests (and may be preferred by Microsoft Support), but it does not support all possible configurations.

The Diskshadow command line utility provides a greater degree of support, and is a better match for how Veeam products use the VSS API. However, it is only available on Windows Server 2008 or newer; it is not available for client versions of Windows.

Solution

Test shadow copy creation, then review Interpreting the Results (below).
 

VMware Environments or Agent Backups

 

Copy-Only Mode

Use this method when transaction log truncation is not relevant. For best comparison, specify copy-only in the Veeam job settings.

From an administrator command prompt run the following commands:

diskshadow /l <output path for log file> (the /l parameter is optional)
set verbose on
set context volatile

 
Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.

Examples:
add volume \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx}\
add volume X:
add volume C:\ClusterStorage\Volume1


See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

create

Normally, shadow copies created this way will be automatically deleted when Diskshadow exits or is reset, but the delete shadows command is available if needed.

Note: Some issues may be isolated to a single volume. To identify which volume is responsible for an error, add only one volume to the backup set, create the shadow copy, then exit or reset Diskshadow before starting over with the next volume.
 

Full Mode

Full mode is the default setting for Veeam jobs. This method will notify VSS writers that a backup has occurred. Some applications, including Microsoft Exchange, will truncate transaction logs automatically in response to this notification. Other applications, such as Microsoft SQL Server, record that a backup has been taken but do not truncate transaction logs.

To test log truncation along with the creation of the shadow, run the following commands from an administrator command prompt:

diskshadow /l <output path for log file> (the /l parameter is optional)
set verbose on
set context volatile


Next, add each volume that will be part of the backup set. Typically, this will be all volumes, including the system reserved partition.

Examples:
add volume \\?\Volume{3c8d1bcb-9d95-11e5-80b0-806e6f6e6963}\
add volume X:
add volume C:\ClusterStorage\Volume1


See Finding Volume GUIDs at the end of this article for more information. Once all volumes have been added, start shadow copy creation:

begin backup
create

 
Once the shadow copy is created, inform the VSS writers that the backup has completed. For some writers, this will trigger transaction log truncation:
 
end backup
 
Normally, shadow copies created this way will be automatically deleted when the end backup command is completed, but the delete shadows command is available if needed.
 

Hyper-V Environments

Note: The following information is applicable to Server 2008R2, 2012, and 2012R2.

Veeam Backup & Replication creates shadow copies of volumes containing virtual machines. For transactionally-consistent backups (Application-Aware Image Processing or Hyper-V Quiescence), the Hyper-V VSS Writer triggers the Hyper-V Volume Shadow Copy Requestor service in each VM to create shadow copies of volumes within the VM. Error messages generated by shadow copy creation failure generally do not indicate whether the problem is isolated to a hypervisor volume or an in-guest volume.

Start by isolating whether the failure occurs only when using Application-Aware Image Processing or Hyper-V Quiescence. If the problem does not occur when both options are disabled, test shadow copy creation from within the virtual machine as described in the VMware Environments or Agent Backups section, above.

If snapshot creation is failing for crash-consistent backups, or if in-guest shadow copy creation does not result in any errors, it may be necessary to test shadow copy creation on the Hyper-V host. Because Hyper-V backup can be complex, a scripted approach is recommended. Veeam Support can run a scripted test that also collects a significant amount of useful logging. Contact support to schedule a remote session, and ask about hv_mkshadow.

Warning: Attempting to perform a shadow copy of a volume containing virtual machines will trigger in-guest shadow copy creation on all VMs on that volume unless specifically excluded. This may have negative effects on performance or responsiveness of these VMs and the Hyper-V host. Veeam Backup and the hv_mkshadow script exclude unnecessary VSS writer components from the shadow copy.
 

Interpreting the Results

Typically, if shadow copy creation fails via both Diskshadow and Veeam products, that indicates that the Veeam application is functioning correctly, and the problem is isolated to shadow copy creation. Troubleshoot the errors reported by Diskshadow and any events appearing in Event Viewer.

In some cases, shadow copy creation will succeed via Diskshadow, but error or warning events will be generated in the event log. If similar errors are generated when Veeam backup or replication jobs fail to create a shadow copy, that usually indicates that the Veeam application is aborting the shadow copy in response to the errors. In this situation, the Veeam application can usually be assumed to be functioning correctly, so it is best to focus troubleshooting on any events appearing in Event Viewer.

Troubleshooting shadow copy creation or transaction log truncation may require the assistance of Microsoft technical support.

The error “COM call "lvssObject4->GetRootAndLogicalPrefixPaths" failed” can be ignored. For more information, see Microsoft KB3025158.
 

Finding Volume GUIDs

To add volumes that lack a mount point (such as the System Reserved Partition), use the volume GUID obtained through either the mountvol command-line tool, or the Shadow Copies utility.

To access the Shadow Copies utility, right click any volume and choose Configure Shadow Copies. On older operating systems, this may instead be a tab in Properties. In the Shadow Copies utility:
  1. Select the volume displayed as a GUID;
  2. Click Settings…
  3. Copy the volume GUID from the settings window and paste it into DiskShadow.
User-added image
Alternatively, run “mountvol” from a command prompt; below the usage information will be a list of GUIDs and their associated mount points, if any.

 

Manually installing Dashboards and Report Packs as Extension Modules

$
0
0

Challenge

After installing Veeam ONE, you are missing specific dashboards and/or reports or do not have any dashboards and/or reports at all.  

Cause

This can be caused by any reason — from pre-installing the SQL database or installing Veeam ONE in the Advanced Mode, to missing installation files or the installer being corrupt and so on.

Solution

To manually install missing Dashboards and Report Packs as Extension Modules, follow the next steps:

  1. Mount the Veeam ONE image to the Veeam ONE Server (or extract files from the .ISO file to the local drive).

  2. Open the Veeam ONE Reporter Web UI, navigate to Configuration > Extension Modules and click the Install link:

    User-added image

  3. In the Choose File to Upload window, navigate to the Addins\ExtensionModules folder located in the Veeam ONE installation image directory (if you extracted .ISO files to a local drive, navigate to the Addins\ExtensionModules folder in the target directory):

    User-added image

  4. Install all necessary Report Packs one by one.

  5. Make sure that installed packs are available in the Extension Modules view of the Veeam ONE Reporter Web UI.


To install Extension Modules altogether: 

  1. Copy and place them into "%PROGRAMFILES%\Veeam\Veeam ONE\Veeam ONE Reporter Server" folder on Veeam ONE server.

  2. Restart Veeam ONE Reporter Server service. 

  3. Make sure these files disappear after the service restart.

More Information

If you have any questions or problems, please contact Veeam Support at http://www.veeam.com/support

Hyper-V backup job fails to create shadow copy with default shadow storage limit

$
0
0

Challenge

Backup or replication of a Hyper-V VM fails. In the Windows event logs on the Hyper-V host server, there is an error with ID 8193 from source VSS:

Volume Shadow Copy Service error: Unexpected error calling routine Cannot find anymore diff area candidates for volume \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx}\ [0]. 

Note: This KB specifically addresses the error “cannot find anymore diff area candidates for volume.” There are a variety of possible VSS errors with ID 8193.

Veeam Backup & Replication reports a general error about snapshot creation failure:

Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Unknown status of async operation
The shadow copy provider had an unexpected error while trying to process the specified operation.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Cause

The Volume Shadow Copy Service is unable to allocate disk space to create a shadow copy because the maximum size of the shadow storage area is too small.

Solution

Increase the shadow copy storage area for the volume listed in the error event, or set the maximum size to “No limit.”

To identify the volume listed in the event, run “mountvol” from a command prompt; below the usage information will be a list of volume GUIDs and their associated mount points, if any.

This size of the shadow storage area can be changed in the Shadow Copies utility, or from the command line.

Note: By default, each volume stores its own shadow copy. For example, a shadow copy of a CSV mapped to C:\ClusterStorage\Volume8 would also be stored on C:\ClusterStorage\Volume8. Do not change the location of the shadow copy storage area on a Hyper-V host server.


Shadow Copies Utility

To access the utility, right click any volume and choose Configure Shadow Copies.

In the Shadow Copies utility:
  1. Select the volume with insufficient space;
  2. Click Settings…
  3. In the “Maximum size” box, either increase the limit, or choose “No limit”.
User-added image
A limit at least 20% of the total volume size may be needed on busy servers.



Command Line

All commands below should be run from an administrator command prompt.

Use vssadmin list shadowstorage to view existing shadow storage associations. If no association exists for any volume, the command response will be “No items found that satisfy the query.”

If no association is listed for the volume, run vssadmin add shadowstorage to set the size of the shadow storage area. For example, on a 1 TB volume labeled D:

vssadmin add shadowstorage /for=D: /on=D: /maxsize=200GB

If a shadow storage association already exists, run vssadmin resize shadowstorage to set the size of the shadow storage area. For example, on a 1 TB volume labeled D:

vssadmin resize shadowstorage /for=D: /on=D: /maxsize=200GB

The /maxsize parameter is not optional, but can be set as /maxsize=UNBOUNDED

 

More Information

It is not necessary to enable shadow copies for shared folders. “Next Run Time” should show “Disabled.”
 

Cannot find anymore diff area candidates for volume

$
0
0

Challenge

Backup or replication of a Windows OS fails. In the Windows event logs of the machine being backed up, there is an error with ID 8193 from source VSS:

Volume Shadow Copy Service error: Unexpected error calling routine Cannot find anymore diff area candidates for volume \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx}\ [0]. 

Note: This KB specifically addresses the error “cannot find anymore diff area candidates for volume.” There are a variety of possible VSS errors with ID 8193.
 

Cause

The Volume Shadow Copy Service is unable to allocate disk space to create a shadow copy because the maximum size of the shadow storage area is too small.
 

Solution

Note: If this error occurs on a Hyper-V host when backing up VMs on that host, see KB1846.

Increase the shadow copy storage area for the volume listed in the error event, or set the maximum size to “No limit.”

To identify the volume listed in the event, run “mountvol” from a command prompt; below the usage information will be a list of volume GUIDs and their associated mount points, if any.

This size of the shadow storage area can be changed in the Shadow Copies utility, or from the command line.
 

Shadow Copies Utility (Server OS Only)

To access the utility, right click any volume and choose Configure Shadow Copies. On older server operating systems, this may instead be a tab in Properties.
In the Shadow Copies utility:
  1. Select the volume with insufficient space;
  2. Click Settings…
  3. In the “Maximum size” box, either increase the limit, or choose “No limit”.
User-added image

A limit of 15-20% of the total volume size may be needed on busy servers.
 

Command Line (Client or Server OS)

All commands below should be run from an administrator command prompt.

Use vssadmin list shadowstorage to view existing shadow storage associations. If no association exists for any volume, the command response will be “No items found that satisfy the query.”

Note: When no shadow storage association exists for a volume, the shadow copies utility will display 10% of the volume or MinDiffAreaFileSize, whichever is greater. However, Windows may set a different maximum size during shadow copy creation. While the default behavior works well for most volumes, setting a limit of 15-20% of the total volume size may be needed on busy servers.

If no association is listed for the volume, run vssadmin add shadowstorage to set the size of the shadow storage area. For example, on a 1 TB volume labeled D:

vssadmin add shadowstorage /for=D: /on=D: /maxsize=200GB

If a shadow storage association already exists, run vssadmin resize shadowstorage to set the size of the shadow storage area. For example, on a 1 TB volume labeled D:

vssadmin resize shadowstorage /for=D: /on=D: /maxsize=200GB

The /maxsize parameter is not optional, but can be set as /maxsize=UNBOUNDED


 

More Information

By default, each volume stores its own shadow copy. For example, a shadow copy of a CSV mapped to C:\ClusterStorage\Volume8 would also be stored on C:\ClusterStorage\Volume8. The location of the shadow storage area may be changed as described in KB1794.

It is not necessary to enable shadow copies for shared folders. It is normal for “Next Run Time” to show “Disabled.”
 

Insufficient Storage for VSS Error "0x8004231F or Error: VSSControl: -2147467259"

$
0
0

Challenge

The following error occurs during backup or replication:
 
VSSControl: -2147467259 Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x8004231f].

Cause

This error is usually caused by insufficient disk space on one volume in the Windows OS being backed up. This can include the System Reserved Partition.
 
It can also be caused by defining shadow storage associations with a maximum size smaller than necessary to create the shadow copy.

Solution

Verify Sufficient Free Space

Within the VM guest OS that cannot be backed up or replicated, use the Disk Management utility (diskmgmt.msc) to view free space on each NTFS volume:
User-added image

If a volume is low on disk space, the solution is to delete files, expand the volume, or redirect shadow copies to another volume. For virtual machines, the simplest solution is usually to increase the size of the VM’s hard disk, then expand the volume in Disk Management.

Although it is sometimes possible to create a shadow copy using less than 1% of the free space on a volume, 15%-20% free space is often required on busy volumes. The smallest amount of space that can be allocated is configurable, but Microsoft Support has recommended maintaining at least 42 MB free on the system reserved partition.

If it is not clear which volume has insufficient space, see “How to Check Which Volume is Causing the Error”, below.

 

(Optionally) Redirect Shadow Copies

It is possible to change the shadow storage association to use a volume with sufficient free space; The location can be changed from the Shadow Copies utility, or from the command line. To access the utility, right click any volume and choose Configure Shadow Copies.
Warning: This is not recommended in a Hyper-V environment.

Shadow Copies Utility (Server OS Only)

To access the utility, right click any volume and choose Configure Shadow Copies. On older server operating systems, this may instead be a tab in Properties.
  1. Select the volume with insufficient space;
  2. Click Settings…
  3. In the dropdown labeled Located on this volume, select a volume with sufficient space to store the shadow copies.
User-added image

Command Line (Client or Server OS)

All commands below should be run from an administrator command prompt.

Use vssadmin list shadowstorage to view existing shadow storage associations. If no shadow storage exists for any volume, the command response will be “No items found that satisfy the query.”

If no association is listed for the volume, run vssadmin add shadowstorage to change the location of the shadow storage area. For example, to redirect shadows from D: to E:


vssadmin add shadowstorage /for=D: /on=E: /maxsize=200GB

If a shadow storage association already exists, run vssadmin resize shadowstorage to change the size of the shadow storage area. For example, to redirect shadows from D: to E:

vssadmin resize shadowstorage /for=D: /on=E: /maxsize=200GB

The /maxsize parameter is not optional, but can be set as /maxsize=UNBOUNDED. Otherwise, a value of 15-20% of the source volume’s size is generally recommended for busy servers.
 

How to Check Which Volume is Causing the Error

In some cases, it may not be obvious which volume needs to be modified. To isolate the problem, perform the following steps within the VM guest OS that cannot be backed up or replicated.
Create and then delete a shadow copy of each volume, one at a time. To access the GUI, right click any volume and choose Configure Shadow Copies. On older operating systems, this may instead be a tab in Properties. In the Shadow Copies utility:
  1. Select a volume;
  2. Click Create Now;
  3. If this is successful, select the shadow copy and click Delete Now;
  4. Repeat steps 1-3 for each volume.
User-added image
This can also be performed using the Diskshadow utility, which may provide a better match for the behavior of the VSS API used by Veeam Backup & Replication.

If the affected volume has sufficient free disk space, the shadow storage limit may be too small. Redirect the shadow copies as shown above, or increase the shadow storage limit.

 

More Information

It is not necessary to enable shadow copies for shared folders. It is normal for “Next Run Time” to show "Disabled".

ExaGrid Storage Best Practices

$
0
0

Challenge

This article documents general best practices and configuration advice when using an ExaGrid backup storage appliance with deduplication.

Solution

Restores
Restores from recent backups or backup copies will utilize the ExaGrid landing zone as a high-speed disk cache, eliminating the need to rehydrate the backup data.  All Veeam vPower operations such as Instant VM Recovery and SureBackup will experience VM boot times of seconds to minutes.
Use the "Per-VM backup files" option on the Storage Compatibility Settings in Repository's "Repository" page (Advanced button).  After switching to this option, do a manual active full before the next schedule backup runs. Set the repository’s “Limit maximum concurrent tasks to:" 10 and work with your ExaGrid support engineer to further tune this setting.
 
Utilize the ExaGrid Veeam Accelerated Data Mover option whenever possible for best performance and fastest synthetic full and backup copy jobs – since with the Data Mover option, I/O is done directly on the ExaGrid backup storage appliances rather than having to pull Veeam backup data across a network into the proxy server and back out again to the ExaGrid.
You can convert a CIFS repository to use the ExaGrid Veeam Accelerated Data Mover – contact your ExaGrid Customer Support engineer for assistance.

 

Backup Jobs

  • Use “Incremental” backup mode
  • Check “Create synthetic full backups periodically” – recommend one day per week
  • Check “Enable inline data deduplication” – the ExaGrid will further deduplicate the Veeam backups.  See below for more about combined deduplication.
  • Change the Compression Level to “Dedupe-friendly”
  • Change Storage optimization to “Local”
  • If you need to retain more than 14 recovery points, see the section below on how to use Backup Copy Jobs for longer retention
 
 
Backup Copy Jobs
When it comes to deciding the best value for number of backup recovery points to retain, keep in mind that the daily incremental backups are, by definition, largely just data that has changed in the VM blocks, and so consume a larger amount of backup storage space since deduplication is not able to significantly reduce the size on disk.  ExaGrid standard sizing expects the retention of backup recovery points to be 14 or less.  To retain backups for longer than 14 recovery points, utilize Veeam Backup Copy Jobs as follows:
  • For each repository configured to use an ExaGrid backup storage appliance (either CIFS or Data Mover), create a 2nd repository with the same settings but a different name – e.g. add “_copy” to the name of existing repositories. 
  • Keep the number of restore points at the minimum of 2, since you will use the backup job’s restore points for your primary source of restore points. Set the copy job interval as long as possible given the GFS settings. So, if GFS settings include weekly retention, set the copy interval to 7 days.
  • Create Veeam Backup Copy job(s) to make a copy of the backups you need to retain longer, using the 2nd repository. Enable the "Read the entire restore point from source backup instead of synthesizing it from increments" option on the "Target" tab of the Backup Copy Job.
  • Set the retention parameters on the Backup Copy Job to match your needs – i.e. retain 16 weeks, 6 monthlies, 1 yearly, etc.
Using the ExaGrid Veeam Accelerated Data Mover type repository will significantly speed up Backup Copy Jobs, since the reads and write I/O to make the copy all run directly on the ExaGrid backup storage appliance, using the high-speed RAID 6 disks in the ExaGrid appliance rather than having to move all backup data back and forth across between the ExaGrid and the backup proxy across a network.
 
Because the repository that has the backup jobs is the same ExaGrid share that receives the backup copies (via the 2nd repository you create), the copied backups will be deduplicated against the backups, allowing much longer retention of Veeam backups compared to using straight disk.
 
Contact your ExaGrid Customer Support engineer for assistance in setting up the Veeam Backup Copy jobs for longer retention.
 
Deduplication
The ExaGrid backup storage appliance will perform deduplication of the Veeam backups, which themselves have been deduplicated by Veeam (see backup job settings above).  The ExaGrid web-based administrative interface will show you the ExaGrid-achieved deduplication, but not the Veeam deduplication.  For example, if Veeam is deduplicating a backup job 2:1 (typical), and ExaGrid’s GUI show 5:1, the combined dedup is 10:1.
 
ExaGrid Customer Support has a PowerShell script that collects Veeam deduplication information and produces a combined deduplication result for each Veeam Backup Job; contact your support engineer to obtain the script and assistance using it. Please note that for Veeam Backup and Replication v9, you will need an updated script from ExaGrid Support.

More Information

The best practice guides can be download on the Exagrid’s appliance under [Help]->[Online Library] then expand the “Name VM Backup Applications”.

User-added image

 

Release Notes for Veeam Backup & Replication 9.0 Update 2

$
0
0

Challenge

Release Notes for Veeam Backup & Replication 9.0 Update 2

Cause

Please confirm you are running version 9.0.0.902 or 9.0.0.1491 prior to installing this update. You can check this under Help | About in Veeam Backup & Replication console. If you are using partner preview build 9.0.0.773, you must upgrade to GA build 9.0.0.902 first by installing Day 0 Update (Note: Please do NOT install Update0 from KB2084 over Update1. Please see the bottom of this article for the link Update2)

After upgrading, your build will be version 9.0.0.1715

Prior to installing this update please reboot the Veeam server to clear any locks on the Veeam services and when the reboot is done, please stop all the Veeam jobs and services before applying the update.

Ensure that Veeam service account has the db_owner role on the Veeam configuration database, so that the required database updates can be made during the first service start up after update. Additionally, either give Veeam service account the sysadmin role on the SQL Server hosting the configuration database, or just the right to view system table content (GRANT VIEW SERVER STATE TO "DOMAIN\ACCOUNT"). All of the above-mentioned roles and rights can be revoked after update is completed.

After installing the update, during the first start of the Veeam Backup Service, required modifications will be made to the configuration database automatically to optimize its performance. These modifications may take up to 10 minutes to complete. Please do not reboot the Veeam server, or attempt to stop the service during this operation. If there is concern regarding the time that the Veeam Backup Service takes to start after upgrade, please contact Veeam Customer Support.

Once Veeam Backup Service starts, please open the console and allow Veeam Backup & Replication to update its remote components.

If you have Enterprise Manager installed on a separate server, you need to install the update on it as well.

To obtain this update, please click here (you need to be logged in to download the update).
To obtain Day 0 update, please refer to 
KB2084
Checksum: aa94a88c164aa83b27cb2b1c51788aff

Solution

As a result of on-going R&D effort and in response to customer feedback, Update 2 includes over 300 enhancements and bug fixes, the most significant of which are listed below:

New platform support
  • Cisco HyperFlex HX-Series support for Direct NFS backup mode.
  • EMC Unity support for Backup from Storage Snapshots and Veeam Explorer for Storage Snapshots functionality.
  • EMC Data Domain DD OS 5.7 support for DD Boost integration.
  • ExaGrid 4.8.0.351.P28 is now the minimal ExaGrid firmware version supported.
  • NetApp Data ONTAP 8.3.2 support.
  • VMware vCloud Director 8.10 support.
  • VMware VSAN 6.2 support.

Engine
  • Backported a number of isolated Enterprise Scalability enhancements from 9.5 code branch to improve transaction log backup, tape backup and user interface performance.
  • Updated OpenSSH client to version 7.2 to enable out of the box support for modern Linux distributions.
  • Improved iSCSI target performance (iSCSI target is used to mount backup remotely in certain file-level and item-level recovery scenarios).
  • iSCSI mount operations are now retried automatically to workaround occasion “The device is not ready” errors which happen when mount operation takes too long. By default, the mount is retried 6 times every 10 seconds. To change the number of retries, create IscsiMountFsCheckRetriesCount (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server.

Backup Copy
  • To reduce backup server load, Backup Copy jobs targeting shared folder or deduplicating appliance backed repositories with the gateway server setting set to Automatic selection will now start the data mover on the mount server associated with the backup repository (as opposed to the backup server). In cases when the mount server is unavailable, the data mover will be started on the backup server as before.
  • Backup Copy performance should now be more consistent due to preserving backup files cache when the job is switching to idle mode.
  • Minor reliability improvements in GFS full backup creation algorithm.

Microsoft Hyper-V
  • Find-VBRHvEntity cmdlet performance has been improved significantly when used against Hyper-V cluster.
  • Backup infrastructure resource scheduler should schedule Guest Interaction Proxy resource dramatically faster in large infrastructures (for example, 5 seconds instead of 15 minutes).

Microsoft SQL Server
  • Improved performance and reduced resource consumption of Microsoft SQL Server transaction log backups.

Microsoft Exchange
  • Added ability to force CAS server for Veeam Explorer for Microsoft Exchange (instead of automatically detecting one) via DefaultCASServer (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server.
  • Added ability to change the order of Exchange autodiscovery policies for Veeam Explorer for Exchange (support only setting).

Oracle
  • Added ability to restore Oracle databases while preserving certain parameters which are critical to RMAN in scenarios such as database name change via new <Parameter Name="diagnostic_dest" StartupPath="true"/> parameter in PFileParameters.xml file of Veeam Explorer for Oracle.

Cloud Connect Replication
  • Support for Planned Failover functionality with cloud replicas. You can now perform planned failover to achieve zero data loss, for example when a natural disaster can be predicted in advance.

Veeam Cloud & Service Provider Partners
  • Update 2 introduces important changes and fixes around rental licensing, including a pilot functionality of usage reporting directly from the user interface. For additional information, as well as the list of other service provider specific enhancements and bug fixes included in this update, please refer to the issue tracking topic in the private VCSP forum. If you are VCSP but don't have access, please apply to Cloud & Service Providers group using Veeam forum's User Control Panel.

More Information

Ensure that Veeam service account has the db_owner role on the Veeam configuration database, so that the required database updates can be made during the first service start up after update. Additionally, either give Veeam service account the sysadmin role on the SQL Server hosting the configuration database, or just the right to view system table content (GRANT VIEW SERVER STATE TO "DOMAIN\ACCOUNT"). All of the above-mentioned roles and rights can be revoked after update is completed.

If you have Enterprise Manager installed on a separate server, you need to install the update on it as well.

To obtain this update, please click here (you need to be logged in to download the update).
Checksum: aa94a88c164aa83b27cb2b1c51788aff

To obtain the base version of Veeam Backup and Replication 9, please click here
 
Viewing all 4469 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>