Increase in API Calls when Performing Backups Directly to Immutable Object Storage

KB ID:	4470
Product:	Veeam Backup & Replication \| 12 \| 12.1
Published:	2023-07-11
Last Modified:	2024-06-04

Veeam Backup & Replication 12.1.2 Changes

This article has been updated to reflect the new Block Generation intervals in Veeam Backup & Replication (VBR) 12.1.2.

30 days — for Amazon S3 object storage and IBM Cloud object storage.
10 days — for all other types of object storage repositories.

Prior to VBR 12.1.2, all object storage repositories used the same 10-day generation interval.

Challenge

When backing up directly to Object Storage with immutability enabled, an increase in to the Object Storage is noticed. This behavior is observed when using either a simple Object Storage Repository or when targeting a Scale-Out Backup Repository that uses an Object Storage Repository as its Performance Tier. The spike in API calls occurs at regular intervals, typically around every 10 days or 30 days for AWS S3 and IBM cloud storage.

Cause

This situation is caused by the way in which backup file immutability is maintained when using Immutable Object Storage as a primary backup destination. As explained in the How Block Generation Works section of the User Guide, Veeam Backup & Replication utilizes Block Generations to extend the immutability of groups of backup files in intervals. This approach is implemented to reduce I/O operations with the storage over time.

Solution

The immutability extension API calls occurring on an interval is behavior by design.

More information can be found here: How Block Generation Works.

More Information

How to Reduce the Number of Immutability Extension API Calls

For most customers, using the software's default settings will be practical and cost-efficient. However, for customers looking to optimize their API vs Storage costs, there are options to shift the cost load from API calls to Storage. It may be possible to reduce overall costs in some cases, but each environment is different, and data size, block size, and immutability requirements must be considered carefully.

There are two ways to reduce the overall quantity of API calls needed to maintain immutability.

However, while either of these will result in fewer API calls, storage usage will increase.

Option 1: Reduce the Quantity of Blocks by Increasing Block Size

To decrease the number of API calls, consider increasing the block size (Storage Optimization) of the backup files generated by jobs that perform direct backups to immutable object storage. Increasing the block size used when creating the backup files will reduce the number of blocks involved in the immutability extension process but will increase the size of the incremental restore points (on average, twice as large).

The tradeoff with this method is fewer API calls but more storage usage.

Remember that changing the block size (Storage Optimization) used by a backup job will only take effect during the next Active Full backup session. Additionally, if backup copy jobs target the immutable object storage, the block size change must be implemented in the source backup jobs first, followed by forcing an Active Full backup in both the source backup and backup copy jobs.

Option 2: Adjust the Immutability Extension Frequency

The immutability extension frequency, which affects the number of API calls, can be changed by adjusting the Block Generation interval. By increasing this interval, the occurrence of immutability extensions within a given time frame can be reduced. However, this reduction in the frequency between bursts of API calls to extend immutability duration comes at the cost of setting the maximum immutability for blocks longer than the minimum immutability setting defined in the repository settings. Resulting in fewer API calls but more storage usage.

Example

Consider the following before adjusting the Immutability Generation Frequency.

An Immutable Object Storage Repository has been configured to have an Immutability of 50 days. When the first data block (a full backup) is uploaded, its immutability period by default is set to 50 + 10 = 60 days. The first full backup starts its generation, which will be appended with the incremental backups. All the incremental backups within the generation (that is, within the 10-day period) will have the same immutability expiration date as the full backup. For instance, a data block offloaded on day 9 will have the same immutability expiration date as a data block offloaded on day 1. Thus, we ensure that the immutability period for all the related and dependent data blocks within a generation is never less than the immutability setting (50 days).

To maintain backup consistency, Veeam Backup & Replication can extend immutability expiration for all data blocks in all backup chains (both active and inactive) and assign those blocks to a new generation. For example, within a forward incremental backup chain, the full backup file can not be removed before its dependent incremental backup files. Therefore, the immutability period must be extended for all data blocks in a given backup chain.

If one were to alter the Immutability Generation Frequency (Days) from 10 to 25, it would cause the extensions of immutability to happen less frequently. Still, it would cause some backups to have an immutability of 75 days (50 from the repository setting and 25 from the new generation value).

Changes starting in Veeam Backup & Replication 12.1.2

VBR 12.1.2 also brought new registry values to provide more granular control over block generation based on object storage repository type (vendor). In previous versions, one registry value (ObjectStorageImmutabilityGenerationDays) controlled the interval for all object storage repository types. That registry value is still valid in VBR 12.1.2 but will take a lower priority than the vendor-specific registry values documented in this article.

I understand and still wish to change the immutability generation frequency.
(click to expand and view registry value details)

Default Values are Hardcoded

The default values shown below are hardcoded into the software. If a registry value does not exist, the default value shown is what the software will use.

Global Generation Interval Control

Note: This registry value was added in Veeam Backup & Replication build 12.0.0.1420, and remains functional but is subordinate to any vendor-specific generation interval registry setting.