Challenge

Cause

Solution

Challenge

Storage space is a concern for backup copy jobs in Veeam. If the repository is insufficiently provisioned, retention can fail, leading to the repository filling up and the job failing. If this happens, the job must often be restarted from scratch.

Cause

First and most important to note is that backup copy job retention happens differently from local backups. As a copy job is optimized for off-site storage, we assume there may be connection issues over a WAN that aren’t present on a LAN. Merges, as a result, are done via temp file, instead of writing the VIB incremental data directly into the VBK full.
 
This temp file is going to be as big as (if not slightly larger than, due to changing data) the VBK. Once the merge is complete, the VIB and the old VBK are deleted and the temp file takes their place.
 
For GFS retention, every Weekly, Monthly, Quarterly, and Yearly backup is itself a VBK, and of similar size.

Solution

To estimate the space requirements for a backup copy job:
 

• Add the sizes of a full backup of all local jobs that will be included in the copy job. You may end up with a slightly smaller copy job due to compression and deduplication of similar blocks, but it’s preferable to plan for more.

• Determine how many full backups you’ll have:

  • One full for the regular retention

  • One full for every Weekly, Monthly, Quarterly, and Yearly GFS archival point

• Add one more point to the total for the temp file used in the merge process

• Multiply that total by the size estimate for the full backup. Ensure your repository will have space for at least this much data, and a bit more for incrementals/variance in data.

 
Example:
 
Consider two local backup jobs being written to a single backup copy job. A full backup of the first job is around 750GB and a full backup of the second is 500GB. Add the two together. 1.25TB should be accounted for the combined full of the copy job, and a minimum of another 1.25TB should be allowed for the merge.
 
The same two local backup jobs have a combined daily rate of change of 170GB (Local Backup 1, on a given week, has increments sized 100, 120, 100, 150, 125, and 175GB for an average of 128.334~ and Local Backup 2 has increments of 50, 40, 55, 30, 45, and 30GB for an average of 41.667). If the backup copy job will retain 14 points, allow at minimum 14 * 170GB or 2.38TB for increments. Given that one cannot consistently predict rates of change, it’s best to oversize here rather than to be left wanting.
 
Assuming no GFS retention on this example job, the backup copy repository should have:
 
[1.25TB (full)] + [1.25TB (merge overhead)] + [2.38TB (incremental points)] = 4.88TB
 
If we add GFS retention of 4 monthly points, those four full restore points add 4 * 1.25TB = 5.0TB. In this case we would need:
 
[1.25TB (full)] + [5.0TB (GFS points)] + [1.25TB (merge overhead)] + [2.38TB (incremental points)] = 9.88TB
 
As a general rule, a bit of additional space is recommended for unforeseen issues. Insufficient sizing for a merge can lead to running out of space, and if a copy job repository ends up with insufficient space to perform a merge—and subsequently is filled with more increments than specified by retention—the only recourse in almost every case is to clear the repository and let the copy job start with a new full, or provision more space for the repository if it is scalable.

Challenge

When accessing backup files on a CIFS (SMB) repository, Veeam Backup and Replication reports either of the following errors:
 
The specified network name is no longer available. (code 59, 0x8007003B)
 
An unexpected network error occurred. (code 64, 0x80070040)
 
Veeam Backup & Replication uses SMB connections for backup repositories, guest processing, and when first adding a Windows server to the Backup Infrastructure. This KB only addresses error messages that occur in the context of a backup file path, such as:
 
The specified network name is no longer available. Failed to read data from the file [\\10.0.0.81\backups\SQL VM New\SQL VM New2015-09-23T200441.vbk].
 
Note: If the failure is occurring in a tape job and no UNC path is specified in the error message, please contact Veeam Support.

Cause

These error messages are passed directly from Microsoft Windows, and indicate that some unknown problem led to a sudden failure of the connection between the repository gateway (acting as Server Message Block client) and the SMB server (typically a NAS appliance).



There are many possible reasons the SMB connection might fail: 

• Network issues are a common reason, constituting a potentially unlimited variety of hardware failure or network hardware configuration problems; such issues may or may not involve packet loss.
• Simple firewalls may block a connection consistently, while advanced firewalls might destroy an existing connection seemingly at random or under very specific conditions.
• Either client or server might close the connection due to one of a number of timeouts or retry limits either at the SMB protocol layer or the underlying TCP layer; such limits are often encountered due to network congestion or excessive load on the storage.
• The NAS may stop responding entirely due to hardware or software issues; it may in some cases be misconfigured. Appliances using post-process deduplication may stop responding if the process starts prematurely.

Solution

• If the repository share is located on a Windows server, try creating a Windows repository on that server instead of a CIFS (SMB) repository.
• Reboot both the SMB client machine (repository gateway) and the SMB server (NAS).
• If other storage is available, try writing the backup to the other storage as both verification of the problem and as a temporary workaround.
• Consider reconfiguring the NAS as iSCSI, a Linux repository, or an NFS mount on a separate Linux repository server, depending on what is supported by the NAS. This change may involve data loss. iSCSI usually offers the best performance.
• Writing backup files to SMB shares over a slow or unreliable network, such as a WAN connection, is not recommended. Deploy a gateway server with a fast connection to the share.
• To rule out interference from a firewall, temporarily disable any software firewalls (including certain antivirus products) on the SMB client or server, as well as any hardware firewalls between them.

• Open the repository settings and enable Limit combined data rate. If you can measure the existing data rate, start with a limit between 50% and 70% of that value to verify that reducing throughput prevents the error. If the network is still congested after limiting the repository data rate, and the source of the congestion is due to other Veeam processes, enable throttling for the relevant connections. 

• Change the gateway server in the repository settings to test whether particular routes to the share perform better than others. For example, if connections from any Windows server on an ESXi host tend to fail, but connections from a physical Windows machine do not fail, the root cause probably involves that ESXi host or its network connection. 

• If the SMB connection consistently fails after 10 hours, it may be due to expiration of a Kerberos service ticket. The most reliable workaround is to create non-domain credentials on the NAS, then specify those credentials in the repository settings. For more information and other possible workarounds, see this SAMBA mailing list discussion and the Authentication Expiration Timer heading in this MSDN blog post.

Challenge

Storage space is a concern for backup copy jobs in Veeam. If the repository is insufficiently provisioned, retention can fail, leading to the repository filling up and the job failing. If this happens, the job must often be restarted from scratch.

Cause

First and most important to note is that backup copy job retention happens differently from local backups. As a copy job is optimized for off-site storage, we assume there may be connection issues over a WAN that aren’t present on a LAN. Merges, as a result, are done via temp file, instead of writing the VIB incremental data directly into the VBK full.
 
This temp file is going to be as big as (if not slightly larger than, due to changing data) the VBK. Once the merge is complete, the VIB and the old VBK are deleted and the temp file takes their place.
 
For GFS retention, every Weekly, Monthly, Quarterly, and Yearly backup is itself a VBK, and of similar size.

Solution

To estimate the space requirements for a backup copy job:
 

• Add the sizes of a full backup of all local jobs that will be included in the copy job. You may end up with a slightly smaller copy job due to compression and deduplication of similar blocks, but it’s preferable to plan for more.

• Determine how many full backups you’ll have:

  • One full for the regular retention

  • One full for every Weekly, Monthly, Quarterly, and Yearly GFS archival point

• Add one more point to the total for the temp file used in the merge process

• Multiply that total by the size estimate for the full backup. Ensure your repository will have space for at least this much data, and a bit more for incrementals/variance in data.

 
Example:
 
Consider two local backup jobs being written to a single backup copy job. A full backup of the first job is around 750GB and a full backup of the second is 500GB. Add the two together. 1.25TB should be accounted for the combined full of the copy job, and a minimum of another 1.25TB should be allowed for the merge.
 
The same two local backup jobs have a combined daily rate of change of 170GB (Local Backup 1, on a given week, has increments sized 100, 120, 100, 150, 125, and 175GB for an average of 128.334~ and Local Backup 2 has increments of 50, 40, 55, 30, 45, and 30GB for an average of 41.667). If the backup copy job will retain 14 points, allow at minimum 14 * 170GB or 2.38TB for increments. Given that one cannot consistently predict rates of change, it’s best to oversize here rather than to be left wanting.
 
Assuming no GFS retention on this example job, the backup copy repository should have:
 
[1.25TB (full)] + [1.25TB (merge overhead)] + [2.38TB (incremental points)] = 4.88TB
 
If we add GFS retention of 4 monthly points, those four full restore points add 4 * 1.25TB = 5.0TB. In this case we would need:
 
[1.25TB (full)] + [5.0TB (GFS points)] + [1.25TB (merge overhead)] + [2.38TB (incremental points)] = 9.88TB
 
As a general rule, a bit of additional space is recommended for unforeseen issues. Insufficient sizing for a merge can lead to running out of space, and if a copy job repository ends up with insufficient space to perform a merge—and subsequently is filled with more increments than specified by retention—the only recourse in almost every case is to clear the repository and let the copy job start with a new full, or provision more space for the repository if it is scalable.

Challenge

When accessing backup files on a CIFS (SMB) repository, Veeam Backup and Replication reports either of the following errors:
 
The specified network name is no longer available. (code 59, 0x8007003B)
 
An unexpected network error occurred. (code 64, 0x80070040)
 
Veeam Backup & Replication uses SMB connections for backup repositories, guest processing, and when first adding a Windows server to the Backup Infrastructure. This KB only addresses error messages that occur in the context of a backup file path, such as:
 
The specified network name is no longer available. Failed to read data from the file [\\10.0.0.81\backups\SQL VM New\SQL VM New2015-09-23T200441.vbk].
 
Note: If the failure is occurring in a tape job and no UNC path is specified in the error message, please contact Veeam Support.

Cause

These error messages are passed directly from Microsoft Windows, and indicate that some unknown problem led to a sudden failure of the connection between the repository gateway (acting as Server Message Block client) and the SMB server (typically a NAS appliance).



There are many possible reasons the SMB connection might fail: 

• Network issues are a common reason, constituting a potentially unlimited variety of hardware failure or network hardware configuration problems; such issues may or may not involve packet loss.
• Simple firewalls may block a connection consistently, while advanced firewalls might destroy an existing connection seemingly at random or under very specific conditions.
• Either client or server might close the connection due to one of a number of timeouts or retry limits either at the SMB protocol layer or the underlying TCP layer; such limits are often encountered due to network congestion or excessive load on the storage.
• The NAS may stop responding entirely due to hardware or software issues; it may in some cases be misconfigured. Appliances using post-process deduplication may stop responding if the process starts prematurely.

Solution

• If the repository share is located on a Windows server, try creating a Windows repository on that server instead of a CIFS (SMB) repository.
• Reboot both the SMB client machine (repository gateway) and the SMB server (NAS).
• If other storage is available, try writing the backup to the other storage as both verification of the problem and as a temporary workaround.
• Consider reconfiguring the NAS as iSCSI, a Linux repository, or an NFS mount on a separate Linux repository server, depending on what is supported by the NAS. This change may involve data loss. iSCSI usually offers the best performance.
• Writing backup files to SMB shares over a slow or unreliable network, such as a WAN connection, is not recommended. Deploy a gateway server with a fast connection to the share.
• To rule out interference from a firewall, temporarily disable any software firewalls (including certain antivirus products) on the SMB client or server, as well as any hardware firewalls between them.

• Open the repository settings and enable Limit combined data rate. If you can measure the existing data rate, start with a limit between 50% and 70% of that value to verify that reducing throughput prevents the error. If the network is still congested after limiting the repository data rate, and the source of the congestion is due to other Veeam processes, enable throttling for the relevant connections. 

• Change the gateway server in the repository settings to test whether particular routes to the share perform better than others. For example, if connections from any Windows server on an ESXi host tend to fail, but connections from a physical Windows machine do not fail, the root cause probably involves that ESXi host or its network connection. 

• If the SMB connection consistently fails after 10 hours, it may be due to expiration of a Kerberos service ticket. The most reliable workaround is to create non-domain credentials on the NAS, then specify those credentials in the repository settings. For more information and other possible workarounds, see this SAMBA mailing list discussion and the Authentication Expiration Timer heading in this MSDN blog post.

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

Challenge

Cause

Solution

More Information

Challenge

Solution

1. Create a new folder IE: (If the current repository is E: Backups, created E:\Backups-temp)
2. Move the backup files related to the job from E:\Backups to E:\Backups-temp (a move rather than copy should be very quick)
3. Create a new simple repository in your infrastructure, pointing it to E:\Backups-temp (and allow the rescan to see the files you've moved there)
4. Edit the job pointing to the new (temp) repository and follow standard procedures to "Map" it to its new location
5. Once you know the job is configured with the backup data in its new (temp) location, then you can add that (temp) repository to your SOBR.
6. The backup data should now be migrated into your SOBR and the job will be automatically updated
7. Now place the temp repository into "Maintenance" mode
8. Now "Evacuate" the temp repository which will automatically re-distribute your backup data
9. Once the repository has been evacuated, you can delete the extent from the SOBR, then you can delete the simple repository

Cause

Calculate the needed Cache sizes for both Source and Target WAN Accelerators.

Solution

WAN Accelerator Cache/Digest Provisioning

More Information

Challenge

Product Information: 

Product Family: ETERNUS CS 
Status: Veeam Ready - Archive 
Classification Description: Verified disk archive storage that can be used as a Backup Copy target.  Synthetic full backups, granular restores, and vPower features may not provide sufficient performance or be supported. 

Solution

Product Details: 

Model number: CS800 S6 
Number of Drives: 12 
Drive type: SAS(6), SATA(6) 
Firmware version: 3.2.3_S6 
General product family overview: FUJITSU Storage ETERNUS CS800 is a turnkey data protection appliance and provides a simple and affordable solution for customers which follow a disk backup strategy with deduplication. The advanced deduplication technology reduces typical disk capacity requirements for disk to disk backup by up to 95%. 

http://www.fujitsu.com/fts/products/computing/storage/data-protection/cs800/  

Veeam Details: 

Veeam Build Number: 9.5 
Veeam Settings:  

• Repository Type: Shared Folder (CIFS) 
• Deduplication: No 
• Compression: None 
• Storage Optimization: LAN Target 
• Per-VM Backup Files: Yes 
• Decompress before storing: No 
• Align backup file blocks: No 

More Information

Company Information: 

Company name: Fujitsu 
Company overview: Fujitsu is the leading Japanese information and communication technology (ICT) company, offering a full range of technology products, solutions and services. Approximately 156,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers. Fujitsu Limited (TSE: 6702) reported consolidated revenues of 4.7 trillion yen (US$40 billion) for the fiscal year ended March 31, 2016. For more information, please see http://www.fujitsu.com.  

Challenge

Product Information: 

Product Family: ETERNUS CS8000 VTL 
Status: Veeam Ready - Tape 
Classification Description: Tape device where available hardware features have been tested to work with Veeam.

Solution

Product Details: 

Model number: CS8000 
Library firmware version: V6.1 SP03 Virtual juke box (VJUK) emulation 
Drive firmware version: V6.1 SP03 Virtual juke box (VJUK) emulation 
Driver for tape drive: Microsoft IBM Ultrium-TD4 SCSI Sequential Device 6.3.9600.16384 
Driver for media changer: Microsoft Unknown Media Changer 6.3.9600.16384 
Media Type: LTO Ultrium 4 (emulation) 
General product family overview: ETERNUS CS8000 is a datacenter solution for backup storage for mainframe and open systems. Using intelligent process automation and the pooling of storage capacities, backup data is automatically managed between different storage tiers, including disk, deduplication and tape technology as well as different performance and availability levels. From backup application point CS8000 is working as a virtual tape library = VTL. 

Veeam Details: 

Veeam Build Number: 9.5.0.580 

More Information

Company Information: 

Company name: Fujitsu 
Company overview:  Fujitsu is the leading Japanese information and communication technology (ICT) company, offering a full range of technology products, solutions and services. Approximately 156,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers. Fujitsu Limited (TSE: 6702) reported consolidated revenues of 4.7 trillion yen (US$40 billion) for the fiscal year ended March 31, 2016. For more information, please see http://www.fujitsu.com  

Challenge

Product Information: 

Product Family: Cohesity C-series 
Status: Veeam Ready - Repository 
Classification Description: Verified backup storage that supports all Veeam backup and restore features. 

Solution

Product Details: 

Model number: C2000 
Number of Drives: 12 x 8TB SAS HDD, 4 x 1.6TB NVME 
Drive type: SAS & NVME 
Firmware version: 3.5.1 
Additional support: Any hardware configuration with an equal or greater hardware configuration 
General product family overview: Cohesity’s mission is to redefine secondary storage. Cohesity provides the only hyperconverged platform designed to eliminate secondary storage silos by consolidating all secondary data and associated management functions on one unified solution – including backups, files, objects, test/dev copies, and analytics. 

Veeam Details: 

Veeam Build Number: 9.0 
Veeam Settings:  

• Repository Type: Shared folder 
• Deduplication: No 
• Compression: None 
• Storage Optimization: Local Target 
• Per-VM Backup Files: Yes 
• Decompress before storing: No 
• Align backup file blocks: No 

More Information

Company Information: 

Company name: Cohesity 
Company overview: Cohesity makes large organizations productive by consolidating, protecting and sharing your non-mission-critical data assets. Your essential data is instantly available when you need it, where you need it.  Our ground-breaking distributed systems technology hyperconverges all secondary storage workloads into an efficient, agile and infinitely scalable resource pool. This greatly simplifies both your infrastructure and the resources to administer it. 

Challenge

Product Information: 

Product Family: Promise 
Status: Veeam Ready - Repository 
Classification Description: Verified backup storage that supports all Veeam backup and restore features. 

Solution

Product Details: 

Model number: VTrak E5320f 
Number of Drives: 24 
Drive type: 9 Disks Seagate ST3000NM00 SAS, 3 Disks ST6000NM00 SAS, 4 Disks ST4000NM01 SAS, and 8 Disks HUS7260 SAS 
Firmware version: 11.2.0000.00 
Veeam Build Number: 9.0 
Additional support: Any hardware configuration with an equal or greater hardware configuration  
General product family overview: Storage subsystem with two storage controllers with 4 FC ports on each storage controller 

Veeam Details: 

Veeam Build Number: 9.0.0.902 
Veeam Settings:  

• Repository Type: Windows 
• Deduplication: No 
• Compression: No 
• Storage Optimization: Local Target 
• Per-VM Backup Files: No 
• Decompress before storing: No 
• Align backup file blocks: No 

More Information

Company Information: 

Company name: Promise Technology Inc. 
Company overview: Promise Technology Inc. is a recognized global leader in the storage industry and the leading developer of high-performance storage solutions, designed for the data center, surveillance, cloud and rich media markets. Always striving to meet the rigorous demands of our customers, Promise has earned a reputation for developing innovative storage solutions for vertical markets which deliver practical answers to the business challenges facing large enterprise corporations, small to medium businesses, security integrators and creative professionals. 

Promise is focusing on opening up new data storage markets, redefining storage possibilities and seeking opportunities for integrated development. This passion for innovation has kept us at the forefront of the storage industry and led us to form strategic alliances with leading storage-related companies worldwide. Promise has quietly become the preferred storage provider for the world’s top resellers and integrators, who are proud to sell our technology and products through their vertical markets and channels. 

Challenge

Product Information: 

Product Family: 3PAR 
Status: Veeam Ready - Integrated 
Classification Description: Integrated storage where joint development activities between the manufacturer and Veeam have occurred to create advanced backup or restore functionalities.

Solution

Product Details: 

Model number: All 
Number of Drives: Any 
Drive type: Any 
Firmware version: 3.1.2 - 3.2.2 
Additional support: Any 3PAR configuration with supported firmware 
General product family overview: HPE 3PAR StoreServ was built to meet the extreme requirements of massively consolidated cloud service providers. It’s remarkable speed—3M+ IOPS—and proven system architecture has been extended to transform mainstream midrange and enterprise deployments, with solutions from a few TBs up to more than 20PB scale.  

Veeam Details: 

Veeam Build Number: 9.5.0.823 

More Information

Company Information: 

Company name: Hewlett Packard Enterprise 
Company overview: HPE delivers high-quality, high-value products, consulting, and support services in a single package. That’s one of their principal differentiators. They have industry-leading positions in servers, storage, wired and wireless networking, converged systems, software, services and cloud. 

Challenge

Product Information: 

Product Family: StoreVirtual 
Status: Veeam Ready - Integrated 
Classification Description: Integrated storage where joint development activities between the manufacturer and Veeam have occurred to create advanced backup or restore functionalities.  

Solution

Product Details: 

Model number: All 
Number of Drives: Any 
Drive type: Any 
Firmware version: 9.5 – 12.6 (iSCSI only) 
Additional support: Any StoreVirtual configuration with supported firmware 
General product family overview: Versatile and reliable, the HPE StoreVirtual family provides affordable storage for a virtualized infrastructure. Advanced, shared storage technology provides the foundation for a composable data fabric, opening new possibilities for simplified management and scalability across your infrastructure. Our systems are extremely flexible, support continuous data growth and include high-availability features that keep you up and running. 

Veeam Details: 

Veeam Build Number: 9.5.0.823 

More Information

Company Information: 

Company name: Hewlett Packard Enterprise 
Company overview: HPE delivers high-quality, high-value products, consulting, and support services in a single package. That’s one of their principal differentiators. They have industry-leading positions in servers, storage, wired and wireless networking, converged systems, software, services and cloud. 

Challenge

Solution

Challenge

Cause

Solution

Challenge

Processing mailbox: mailbox@name.com...
Syncing folder items: name...
Sync time: 0.562066
Changed items: 26, deleted items: 0
Estimated changed items size: 1.77 MB
Error: Object reference not set to an instance of an object.
Stack:
   at Veeam.Archiver.Source.SyncPage.SaveChanges(IFolderRepository repository)
   at Veeam.Archiver.Source.ItemProcessor.ProcessFolder(Byte[] id, ExFolder folder, Byte[] currentState)
   at Veeam.Archiver.Source.ItemProcessor.<>c__DisplayClass6_0.<Process>b__0()
   at Veeam.Archiver.Source.FolderRetry.<>c__DisplayClass14_0`1.<Process>b__0()
   at Veeam.Archiver.Source.FolderRetry.Process(Action action)
   at Veeam.Archiver.Source.FolderRetry.Process[T](Func`1 func)
   at Veeam.Archiver.Source.ItemProcessor.Process(ExFolder folder)
   at Veeam.Archiver.Source.FolderProcessor.DoMailboxWork(CancellationToken cancel)
   at Veeam.Archiver.Source.MailboxAction.DoWork(CancellationToken cancel)
Mailbox processing failed: mailbox@name.com

Solution

If you running version 1.0.0.860 or older, the issue has been resolved in Veeam Backup for Microsoft Office 365 1.0 Update 1.
If you running version 1.0.0.912 or newer, contact Veeam support directly to obtain the fix.

Challenge

Cause

Solution

More Information