Challenge

Product Information:

Company name: Fujitsu (Fujitsu Technology Solutions)
Product Family: ETERNUS CS
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: CS8000 V7.1
Storage Category: RAID Storage Array
Drive quantity, size, type: 24 – 950GB SSD 2.5” drives
Storage configuration: RAID5
Firmware version: V7.1A2140-SP00
Connection protocol and speed: CIFS 10GbE
Additional support: All models and configurations of ETERNUS CS8000 with specifications equivalent or greater than the above

Veeam testing configuration:

Veeam Build Number: 9.5.4.2615

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• No array deduplication used in testing
• No array compression used in testing
• Jumbo frames used in testing (MTU=9000)

Challenge

There is a need to create a stable backup of your VCSA or restore it properly.

Solution

VCSA with Embedded Database:

Backup:

To backup the VCSA with Veeam, it's important to understand that Veeam does not directly support Postgres SQL (VMware's embedded DB). Instead, it's optimal to perform backup in 2 parts:

1. Image Level Backup with Veeam
2. Database backup via VMware Best Practice

Part 2 is not required, however, but please beware that restoring the image-level backup ensures the same state the Database was in at the point in time of the backup. VMware Best Practice shows to backup the database as regular maintenance. Then you would be able to restore the server from step 1, then restore the database to whichever point in time with the backup from step 2.

To backup the VCSA with Veeam (step 1):

1. Create a typical backup job;
2. VMware Tools Quiescence is disabled on the job level;
3. Ensure Application-Aware Processing and Guest File System Indexing are either disabled or excluded for the VCSA VM;
4. Customize other settings to desired configuration.

To backup the VCSA Database (step 2):

1. See this VMware KB on how to backup the database.

Restore:

When restoring the VCSA, there are some Considerations and Limitations from VMware. Once you understand the considerations and limitations, it's important to understand how to restore your vCenter when there is no vCenter in which to point Veeam. Then you are ready to perform a Full VM Restore or Instant VM Restore.

Once the VCSA is fully restored, if database backups were created, you are now ready to restore the Embedded Database to the desired point in time using the steps mentioned in the above KB.

VCSA with External DB

Backup:

Backup is the same as above with the difference that you may also create an image-level backup of the database server if you so choose. The benefit here is that you may restore the entire VM to a specified state. This state will be of the time in which the image backup was created and without capability of point-in-time database restores. 

For Microsoft SQL and Oracle databases, Veeam can perform log backups to provide point-in-time restores. 

1. For requirements, details, and how to enable log processing for Microsoft SQL, click here.
2. For requirements, details, and how to enable log processing for Oracle, click here.

RESTORE

The restore process is yet the same as the previous steps with these exceptions:

1. VCSA requires the database to function. If needed, the database/server should be restored first.
   1. Server restore:
      1. Instant VM Restore
      2. Entire VM Restore
   2. Database restore, if Veeam was used to backup the database transaction logs you may restore with their corresponding tool:
      • Veeam Explorer for Microsoft SQL Server
      • Veeam Explorer for Oracle
2. Once the database is restored you can restore the VCSA via Full VM Restore or Instant VM Restore.

Challenge

Cause

Solution

More Information

Challenge

Cause

[28.03.2019 08:50:09] <01> Info     [CTapeSourceBackupJob][<Job name>] Last backed up full tape storage creation time: 15/03/2019 0:00:00, backupId: 028285e1-8086-4f52-a44d-0094521b6a09, objectId: 00000000-0000-0000-0000-000000000000 mediaSet: Simple
[28.03.2019 08:50:09] <01> Info     [CTapeDailySynthesizedBackupCalendarSpecifics] Only latest virtual full backup is required. Skipping older virtual full backup creation
[28.03.2019 08:50:09] <01> Info     [CTapeDailySynthesizedBackupCalendarSpecifics] Virtual full backup dates between 16/03/2019 0:00:00 and 28/03/2019 8:50:09: 28/03/2019 0:00:00
[28.03.2019 08:50:09] <01> Info     [CTapeSynStorageCandidatesFinder] Analyzing syn candidates for 28/03/2019 0:00:00 to Simple mediaset
[28.03.2019 08:50:09] <01> Info     [CTapeSourceBackupJob] Vbk in tape creation time 05/03/2019 0:00:00, but in repository creation time is 09/03/2019 0:00:00. Supposed vbk '<File name>.vbk' was transformed
[28.03.2019 08:50:09] <01> Info     [CTapeSynDateAnalyzeResult] Full point was found. Details: Synthesized full backup was not created for 28/03/2019 because there is no VBK map for this date
[28.03.2019 08:50:09] <01> Info     [CTapeSynStorageCandidatesFinder] completed
[28.03.2019 08:50:09] <01> Info         [CTapeSynStorageCandidatesFinder] Found candidates:<Empty>
[28.03.2019 08:50:09] <01> Info     [CTapeSynStorageCandidatesFinder] Build virtual full storages for object: 00000000-0000-0000-0000-000000000000 is completed: 0,1064889 sec
[28.03.2019 08:50:09] <01> Info             [CTapeStorageFilterParams] Skip increment according settings: <File name>.vib:d6131d07-ef49-4bff-801f-0768fd894326:11/03/2019 22:45:10:11/03/2019 0:00:00:KbBlockSize1024:512
[28.03.2019 08:50:09] <01> Info             [CTapeStorageFilterParams] Skip increment according settings: <File name>.vib:36fc42d7-4c6b-4a41-bf83-1e5ff627e0f2:12/03/2019 22:39:30:12/03/2019 0:00:00:KbBlockSize1024:512
[28.03.2019 08:50:09] <01> Info             [CTapeStorageFilterParams] Skip full according settings: <File name>.vbk:e50cea1b-e22b-4433-85f9-31ffe8dc2611:28/03/2019 2:36:23:09/03/2019 0:00:00:KbBlockSize1024:512
[28.03.2019 08:50:09] <01> Info             [CTapeStorageFilterParams] Skip increment according settings: <File name>.vib:ae314a8c-ac10-42e3-8aeb-3e4e3dec942b:27/03/2019 2:21:54:27/03/2019 0:00:00:KbBlockSize1024:512

Solution

More Information

Challenge

Cause

Solution

• Microsoft Windows 10 May 2019 Update and Microsoft Windows Server version 1903 support as guest VMs, and for the installation of Veeam Backup & Replication and its components.
• Microsoft Windows 10 May 2019 Update and Microsoft Windows Server version 1903 support by the Veeam Agent for Windows 3.0.2 (included in the update).
• Linux Kernel version 5.0 support by the updated Veeam Agent for Linux 3.0.2 (included in the update).
• Ubuntu 19.04, SLES 15 SP1, RHEL 8, CentOS 8 and openSUSE Leap 15.1 distributions support in the Veeam Backup & Replication agent management functionality.
• VMware vCloud Director 9.7 compatibility at the existing Update 4 feature level.
• VMware vSphere 6.5 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. This record will be updated with further information and the official support statement based on results.
• VMware vSphere 6.7 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. This record will be updated with further information and the official support statement based on results.

• Oracle 19c on Linux support for RMAN-based and OCI-based application-aware processing, redo log backup and Veeam Explorer for Oracle in most configurations. Due to the recent availability of Oracle 19c, testing of DataGuard and RAC configurations is still being performed. This record will be updated with further information and the official support statement based on results.
• Oracle 19c on Windows support for RMAN-based and OCI-based application-aware processing, redo log backup and Veeam Explorer for Oracle in basic configurations only. Due to the most recent availability of Oracle 19c for Windows, testing of advanced configurations (DataGuard, RAC, ASM, Pluggable DB, flashback, virtual accounts) is still to be performed. This record will be updated with further information and the official support statement based on results.

• Veeam Backup & Replication will now periodically update the list of supported Amazon AWS regions from the Veeam update server. The list will be updated as Veeam QC validates compatibility with the newly added Amazon region. This functionality requires that you opt in for automatic product update check via the existing Check for product and hypervisor updates periodically check box in the Notification settings.

• Dell EMC Unity OE version 4.5 support.
• The proprietary NFS client has been updated for compatibility with the Western Digital IntelliFlash (formerly Tegile) Universal Storage API plug-in. The plug-in will be made generally available along with the Update 4b release.

• Tape virtual full backup reliability and performance, tape drive detection logic, WORM media support and NDMP backup reliability.
• Agent-based backup jobs may create unnecessary full backups under certain circumstances, or fail in the environments with very long backup repository paths.
• Scale-out backup repository race condition causes synthetic backup operations to dead lock when a very powerful backup proxy server is used.
• PowerShell Direct calls may hang under certain circumstances in the application-aware processing logic of Hyper-V backup.
• Veeam Agent for Windows: CBT driver block map reset issue; inability to failover to backup cache when it is hosted on a USB storage; inability to open a support case from the product UI under a restricted user account; improved command line tool for the Recovery Media creation.
• Veeam Agent for Linux: Dell servers with NUMA enabled may crash during the snapshot creation; intermittent backup jobs failures when processing XFS volumes; entire computer backup failure on Ubuntu 18.04 with Intel Software RAID; enhanced encrypted devices compatibility through redesigned device enumerator; CLI improvements for backup scope specification.
• NetApp ONTAP Snapvault retention processing in 7-mode clusters.
• Performance and resource usage optimizations in a number of backup infrastructure components.

More Information

If you have 9.5 Update 4 installed (RTM build 9.5.4.2399, GA build 9.5.4.2615, or Update 4a build 9.5.4.2753), you can upgrade to Update 4b using the following update package:

[[DOWNLOAD|DOWNLOAD UPDATE|https://www.veeam.com/download_add_packs/vmware-esx-backup/update4b_exe/]]
MD5: 5b75cb6da06b656c18b7ba782109273d
SHA1: a80968efe6d265dd4494ff1854a0132092c5a0fa

For new installations and upgrades from the previous versions, use the 9.5 Update 4b ISO image:

[[DOWNLOAD|DOWNLOAD ISO|https://www.veeam.com/download_add_packs/vmware-esx-backup/update4b_iso/]]
MD5: 9bd7f50c393bf368f42d6bd58dd9d3c1
SHA1: 24e230f19244d64d23db2b3660ebbc991835076f

Should you have any questions at all, please contact Veeam Support and we'll be happy to assist you with the upgrade.

Challenge

Cause

• the SQL instance have databases with names ending in a space character. 
• the account under which SQL VSS Writer service is running doesn't have sysadmin role on a SQL server
• SQL VSS Writer service stuck in an invalid state

Solution

• Please rename the database to a new name (without a space in it)
• Grant the SQL VSS Writer service user a sysadmin role (Instructions in KB here: https://www.veeam.com/kb1978)
• Restart SQL VSS Writer service (Instructions in KB here: https://www.veeam.com/kb2041)
• In the case of SBS machines that are also Domain Controllers, ensure that the SQL Writer is running as a domain administrator and not local system
• Allow the SQL Writer service account access to the Volume Shadow Copy service via the registry:
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl
  If the DWORD value “NT SERVICE\SQLWriter” is present in this key, it must be set to 1.
  If the Volume Shadow Copy service is running, stop it after changing this registry value. Do not disable it.
• To check if your database has space in the name you can run the following query:

select name from sys.databases where name like '% '

More Information

• In SQL Server 2008R2 and earlier, this means that the writer service account appears to SQL Server as “NT AUTHORITY\System”.
• In SQL Server 2012 and later, the writer service account appears to SQL Server as “NT Service\SQLWriter”.

Challenge

Cause

Please confirm that you are running version 9.0 Update 2 (build 9.0.0.1715) or later prior to installing this update. You can check this under Help | About in Veeam Backup & Replication console. After upgrade, your build number will be 9.5.4.2753.

You will also see the vendor name changed from Veeam Software AG to Veeam Software Group GmbH, reflecting the legal company name change. Please note that the name change has also resulted in a change of the code signing certificate.

Solution

Veeam Backup & Replication 9.5 Update 4a is a cumulative hotfix rollup that addresses issues reported by customers on the original build of Update 4, and adds the following platform support enhancements:

• VMware vSphere 6.7 U2 readiness including support for VMs with virtual hardware version 15. Please note that the testing has been performed on pre-release 6.7 U2 builds, while the official support will still require a comprehensive testing of the GA build. UPDATE: vSphere 6.7 U2 is officially supported with Update 4a.
• Microsoft System Center Virtual Machine Manager 2019 is now supported as a source for Microsoft Hyper-V VMs backup jobs.
• Dell EMC Data Domain using DD OS 6.2 is now supported as the backup repository via both DDBoost and CIFS/NFS protocols.
• NSS trustee rights on files and folders are now restored during file-level recovery from NSS volumes hosted on Micro Focus Open Enterprise Server (OES).
• Experimental support for keyboard-interactive authentication can be enabled by creating the LinuxKeyboardInteractiveAuth (DWORD) = 1 registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. By default, we will look for the "password" substring in the SSH console prompts. You can override this using the LinuxKeyboardInteractiveAuthPrompt (REG_MULTI_SZ) value, where multiple substrings can be specified one per line.
• New Japanese era compatibility. 新元号の変更検証済み

• Certificate verification issues cause the backup console taking too long to open, or failing to open with "Object reference not set to an instance to an object" from backup console or "Could not load file or assembly" error in the log.
• Upgrade to Update 4 fails with the "Database version downgrade detected ... Reboot and restart the setup" error on machines where system time regional format uses a.m. and p.m. (lowercase with dots).
• SureBackup jobs start failing after upgrading to Update 4 due to virtual lab settings incorrectly updated by the configuration upgrade, in case of lack of free disk space in the vPower NFS cache folder.
• On Hyper-V versions prior to 2016, jobs with application-aware processing disabled fail with "The system cannot find the file specified" error.
• Backup of Windows XP SP1 VMs fails with the "Failed to call RPC function Vss.GetFileFromGADir" error.
• Various primary storage integration issues with HPE Nimble, NetApp, IBM and Dell EMC VNX storage.
• Tape support reliability issues, including a number of UI performance issues with tape management functionality.
• HPE StoreOnce Catalyst library memory leak issue (solved by updating the library to version 24.90.0.8).
• VeeamZIP™ operation incorrectly consumes a license for VMs that were previously licensed by regular backup jobs.

More Information

Challenge

Cause

Solution

More Information

Challenge

Cause

Solution

Challenge

• Instant VM Recovery
• Windows and Linux File Level Recovery
• Cloud Tier
• Cloud Mobility
• DataLabs
• SureBackup
• Staged Restore
• Universal Restore (U-AIR)
• On-Demand Sandbox
• Secure Restore
• Backup and Backup Copy Jobs when configured with one of the following:
• Health Check
• Compact processing
• Synthetic Fulls
• Forever Forward Incremental (Merges)
• Reverse Incremental
• Backup to Tape Jobs, specifically when performed with synthesized fulls
• Replication when performed from backup as a source
• Backup Copy Job processing on source side

Cause

Solution

• Best Practices for HPE
• Best Practices for Exagrid
• Best Practices for Dell EMC DataDomain
• Best Practices for Quantum

Challenge

Cause

Solution

Challenge

[11.07.2019 20:07:56] <  5652> mt |   Drive encryption status:
[11.07.2019 20:07:56] <  5652> mt |   Encryption Mode: Disabled
[11.07.2019 20:07:56] <  5652> mt |   Decryption Mode: Disabled
[11.07.2019 20:07:56] <  5652> mt |   Raw Decryption Mode Disabled (RDMD): Disabled
[11.07.2019 20:07:56] <  5652> mt |   Volume Contains Encrypted Logical Blocks (VCELB): Disabled
[11.07.2019 20:07:56] <  5652> mt |   Logical Block encryption parameters: Application Managed
[11.07.2019 20:07:56] <  5652> mt |   Hardware encryption is available
[11.07.2019 20:07:56] <  5652> mt | ERR | [MT]SCSI PASS THROUGH DIRECT ended with Scsi Error Fixed format, current;  Sense key: Illegal Request. Additional sense: Invalid field in command information unit
[11.07.2019 20:07:56] <  5652> mt | >>  |tr:Failed to enable hardware encryption, tapeHandle: 0x00000000000006a4
[11.07.2019 20:07:56] <  5652> mt | >>  |tr:Failed to create TapeReader.
[11.07.2019 20:07:56] <  5652> mt | >>  |tr:Failed to create TapeFileReader, setPba: 614675, localFla: 136589546, restFileSize: 4621392384
[11.07.2019 20:07:56] <  5652> mt | >>  |An exception was thrown from thread [5652].

1. The tape library is hardware.
2. Encryption is enabled in the target media pool settings (https://helpcenter.veeam.com/docs/backup/vsphere/add_media_pool_encryption.html?ver=95u4).
3. In the tape library settings, the encryption is set to Application managed (this setting is configured on the vendor side).

Solution

1. Download the hotfix. It contains the agents both for x64 and x86 agents.
2. Check the current version of Veeam Backup & Replication. It must be 9.5.4.2866.
3. Make sure that no jobs are running, close the console, and stop all Veeam services.
4. Stop Veeam services on the tape server.
5. Go installation folder C:\Program Files (x86)\Veeam\Backup Transport\x86 on the tape server.
6. Rename VeeamAgent.exe to VeeamAgent.exe_original.
7. Copy VeeamAgent.exe from the Win32 hotfix folder to the installation folder.
8. Go installation folder C:\Program Files (x86)\Veeam\Backup Transport\x64 on the tape server.
9. Rename VeeamAgent.exe to VeeamAgent.exe_original.
10. Copy VeeamAgent.exe from the x64 hotfix folder to the installation folder.
11. Start Veeam services.

More Information

Challenge

Cause

Solution

In Veeam 9.5 we removed the setting for disabling Hosts right from the menu. Now we do it through the Managed VM’s tab. Excluding a host this way will also exclude the sockets that were used for that host. You should see the licensed socket count change after this KB has been finished. 

The first thing to do is go into the Server Settings for Veeam One.



After the server settings window is open we are going to go into the Monitored VM’s tab and hit the ‘Create New…’



This will open a new window where we will put in the information for this new exclusion rule. In this window name and describe the rule in any way you want. Click next when you are finished. 



In the next window we will click ‘Add…” and then ‘Infrastructure Tree’.



When the tree shows up in a new window, go ahead and find the host you no longer wish to be in Veeam One.



Once the host is selected click ‘OK’ and hit ‘Next’. This will send you to the last window which will give you an overview of the change. Just hit finish and you all done. 

More Information

Challenge

Cause

• Cloud Copies counter on the Overview dashboard does not return any data.
• Backup agents managed by Veeam Backup & Replication report 0 restore points available in the backup policy drill down view. 

• Portal administrators can log in only when using a full domain name of the Veeam Availability Console server. 
• Users belonging to a domain user group which is added to a Veeam Availability Console security group cannot log in to the portal.
• Users belonging to a nested domain group cannot log in to the portal.

• Under certain conditions, Veeam.MBP.Service might be consuming excessive memory.
• Under certain conditions, data collection job fails to retrieve information from a Cloud Connect server that has lots of task sessions on cloud gateway servers. This results in displaying these tenants’ companies as deleted in the Veeam Availability Console UI.
• Tenant password changed in Veeam Cloud Connect is not synchronized with Veeam Availability Console.

• Office 365 and Gmail SMTP servers are not supported for notifications.  

• Inaccessible computers cannot be located when using CSV and IP discovery rule options.

• Full Access UI mode cannot be applied to a backup agent.  
• When using Select All checkbox on the Managed Backup Agents tab agent settings are applied to all computers of all managed companies regardless of the selected scope in the Company\Reseller filter.  
• Licensing attribute is not updated for managed backup agents which leads to license expiration on the backup agent.    
• Backup agent files are not validated when the installation is performed from a local folder without using a master management agent. This may result in a corrupted installation/upgrade of the backup agent. 

• Under certain conditions, management agents fail to upgrade with the following error message “The system cannot find the file specified.”

• Workstation and server backup counters in the Veeam Cloud Connect usage report are mixed up.
• Under certain conditions License usage report for Veeam Backup & Replication servers is not generated. 

• Under certain conditions, all VMs with a cloud copy which belong to different locations are reported as VMs from the default location in the invoice. 

• Querying tenant information with SetLinks parameter may take considerable time.
• The vMsBackudUpToCloud counter does not display data about VM and Agent backup copy jobs targeted to a cloud repository.  

• Cloud repository usage is not updated in ConnectWise Manage agreements when there are unknown files (not related to Veeam backups) in the repository. 
• Tickets state synchronization may stop working when triggering a mass-resolve operation in the ConnectWise Manage application. 

Solution

msiexec /update c:\VAC.ApplicationServer.x64_3.0.0.2725.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

msiexec /update c:\VAC.WebUI.x64_3.0.0.2725.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACWebUI.txt

More Information

Challenge

This KB article provides a list of all versions of Veeam Backup & Replication and the respective build numbers.

Solution

The following table lists Veeam Backup & Replication versions and their build numbers. Also provided is a link to the specific version release notes.

Challenge

Cause

Solution

Cause

Solution

More Information

Challenge

Cause

Solution

More Information

Challenge

Product Information:

Company name: Fujitsu
Product Family: ETERNUS CS
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: CS8000
Drive quantity, size, type: 10 - SAS HDD
Firmware version: V6.1A7120-SP03
General product family overview:

ETERNUS CS8000 is a datacenter solution for backup storage for mainframe and open systems. Its intelligent process automation and pooling of storage capacity reduces the TCO for complex backup environments by 40%. Unique scalability in capacity and performance makes ETERNUS CS8000 a viable solution, delivering ongoing cost savings.

Veeam testing configuration:

Veeam Build Number: 9.5.x

Job Settings:

Repository Settings:

Challenge

Cause

Solution

More Information