Restoring domain controller from an application-aware backup

August 21, 2018, 1:09 pm

≫ Next: How to restore vCenter Server without a vCenter Server

≪ Previous: Microsoft SQL Server processing with Update 3a

Challenge

Application Aware Image Processing is required to be enabled and working as per Microsoft in order to functionally restore from a DC.

Since Active Directory implements multi-master replication, where multiple domain controllers sync changes with each other, one of the key challenges is the DC recovery process. This article outlines different DC restore scenarios and goes into some specifics of when and why this or that type of restore is required as well as gives instructions on the manual steps to perform proper DC recovery from backup created with Veeam B&R.

Before going into details, it is worth stressing that by default Veeam B&R performs automated non-authoritative restore of domain controller and in most cases when you need to recover failed DC, authoritative restore is not required.

The following situations are possible:

Restoring single lost DC in a multi-DC environment
Restoring entire AD infrastructure (AKA “all DC’s are lost”)
Restoring from Active Directory corruption

Depending on the scenario, different steps (or no steps at all) are required to perform DC restore. All of the scenarios assume application-aware image processing was enabled in the backup job that backed up the DC being restored.

Solution

Restoring single lost DC in a multi-DC environment or in environment with only a single DC

This scenario, actually the most common one, incurs restoring just one of the multiple DC’s when there are still other functional DC’s in the environment that the restored DC can replicate changes from.
DC recovery with Veeam B&R in this case is fully automated and does not require any user interaction. If your backup was done with application-aware image processing enabled in the backup job settings, Veeam B&R performs a non-authoritative restore of the DC, where the restored VM should first boot in Directory Services Restore Mode (DSRM) mode and then reboot automatically immediately to boot up next time normally.
The domain controller itself will understand that it has been recovered from backup and will allow normal replication to update everything that has been changed since the backup took place.
The automatic recovery should also work for environments with only a single DC.

Restoring entire AD infrastructure (AKA “all DC’s are lost”)

As mentioned above, the automatic recovery process performs a non-authoritative restore, where the DC reboots and starts looking for other DC’s to sync up. However, in a scenario where all DC’s are gone, there are no other partners available and replication may take quite long (15-30 minutes) to start. To avoid wasting the time attempting to contact replication partners, it is recommended to restore two of the domain controllers at once, power them on, wait for their reboot and force one of them to become authoritative for SYSVOL, so that they can start replicating. Then restoring other DC’s will be similar to the first scenario, i.e. will be 100% automatic.

Note: During the restore procedure, make sure the restored DC’s DNS records point to available DNS servers (e.g. to itself).

The procedure for designating DC as authoritative for SYSVOL varies based on whether FRS or DFS-R is used for SYSVOL replication. To determine if you are using FRS or DFSR for SYSVOL in the production environment check the value of the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DFSR\Parameters\SysVols\Migrating Sysvols\LocalState registry subkey. If this registry subkey exists and its value is set to 3 (ELIMINATED), DFS-R is being used. If the subkey does not exist, or if it has a different value, FRS is being used.

If domain level is Windows 2008 or above you could also use the command dfsrmig.exe /getglobalstate to monitor if we are in the ‘ELIMINATED’ state and we are using DFSR.
If you are still using the FRS to replicate SYSVOL you need to perform an authoritative restore of the SYSVOL on first DC restored using burflags
To perform an authoritative restore of the SYSVOL when using FRS, use the following steps:

Start the Registry Editor
Navigate to "HKLM\System\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup"
Double-click on "BurFlags"
Assign it a value of D4 (hex) or 212 (dec)
Stop the NTFRS Service
Start the NTFRS Service

Or you can use the following commands

REG ADD "HKLM\System\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup" /v BurFlags /t REG_DWORD /d 212 /f
NET STOP NTFRS
NET START NTFRS

You could monitor the status of the replication by searching for the Event ID 13516
“The File Replication Service is no longer preventing the computer <Computer_name> from becoming a domain controller. The system volume has been successfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL.”

If you are using DFS-R as the more widely used these days the steps to implement basically consist of setting the following registry values:

Key: HKLM\System\CurrentControlSet\Services\DFSR\Restore
Value: SYSVOL (REG_SZ) = “authoritative”

Key: HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore\SystemStateRestore
Value: LastRestoreId (REG_SZ) = any GUID value (e.g. 10000000-0000-0000-0000-000000000000)

If the first restored DC already hosts operations master roles, set the following registry value in order to bypass initial synchronization requirements and not to wait for another partners to replicate the directory partitions:
Key: HKLM\System\CurrentControlSet\Services\NTDS\Parameters
Value: Repl Perform Initial Synchronizations (REG_DWORD) = 0

Note: Don’t forget to reset this value back to 1 after domain recovery is completed, so that domain controller has successful replication with its partners before starting to service client requests.

After setting the values above, restart the domain controller.

Notes:
• If you’re restoring DC without FSMO roles, you might want to transfer them to it manually after the restore, using the ntdsutil seize command.
• This type of restore is similar to what Veeam B&R performs automatically when restoring DC within SureBackup isolated virtual lab.

Restoring from Active Directory corruption

Scenario where no DC’s are actually lost, however, AD itself is damaged in some way (corrupt objects or schema) and you need to restore from the backup created before corruption occurred. In this case you need to restore one of the multiple DC’s when other DC’s are still operating a damaged copy of AD and force all of them to accept replication changes from the restored DC. This is where authoritative restore of the DC is required.

Note: It is recommended to perform restore with network disabled to prevent DC from accepting changes from other controllers after the default non-authoritative restore.

To perform an authoritative restore:
1. Restore the DC and let it complete the default non-authoritative restore (wait until it reboots second time).
2. During this second boot, press F8 to get to DSRM mode.
3. Log in with DSRM account and password.
4. Open a command prompt and run ntdsutil command.
5. At the "ntdsutil:" prompt, type "authoritative restore" and press Enter.
6. At the "ntdsutil authoritative restore:" prompt, type "restore database" and press Enter.
7. At the Authoritative Restore Confirmation dialog box, click Yes.
8. Upon restore completion, type "quit" and press Enter to exit the ntdsutil utility.
9. Reboot server.
10. Perform an authoritative restore of the SYSVOL, as was already discussed above.

Note: For an easier item-level recovery of Active Directory objects (without the need to restore the domain controller itself), consider using Veeam Explorer for Active Directory.

More Information

Veeam recovery of a domain controller
Active Directory backup and recovery with Veeam
Recovering Your Active Directory Forest
Windows Server - How to Perform an Authoritative Restore of Active Directory Object
Restoring The SYSVOL (Non-)Authoritatively When Either Using NTFRS Or DFS-R (Part 1)
Restoring The SYSVOL (Non-)Authoritatively When Either Using NTFRS Or DFS-R (Part 2)
Restoring The SYSVOL (Non-)Authoritatively When Either Using NTFRS Or DFS-R (Part 3)

↧

How to restore vCenter Server without a vCenter Server

August 21, 2018, 1:52 pm

≫ Next: Considerations for Office 365 organizations with no SharePoint Online or OneDrive for Business plans.

≪ Previous: Restoring domain controller from an application-aware backup

Challenge

The vCenter Server is not available and the vCenter Server VM needs to be restored.

Solution

In order to restore the vCenter Server you will need to add one of individual ESX(i) hosts to the Veeam console. This will allow Veeam to restore the VM directly to the host.

1. Add the standalone host:

https://helpcenter.veeam.com/docs/backup/vsphere/add_vmware_server.html?ver=95

2. Restore the vCenter VM to the standalone host:

https://helpcenter.veeam.com/docs/backup/vsphere/full_recovery.html?ver=95
(Be sure to choose the option to restore to a different location with different settings.)

More Information

If you receive the error “VMware ESXi with name ‘<hostnme/fqdn/ip>’ already exists.

User-added image

Please try using an alternative way to contact the server. For example, if the name you tried to use was the IP address, try using the hostname. The hostname for a host can be found in a vSphere Client under Configuration>DNS & Routing.

User-added image

↧

Considerations for Office 365 organizations with no SharePoint Online or OneDrive for Business plans.

August 9, 2018, 9:29 am

≫ Next: How to restore vCenter Server without a vCenter Server

≪ Previous: How to restore vCenter Server without a vCenter Server

Challenge

Some Office 365 plans (e.g. Exchange P1) include neither SharePoint Online nor OneDrive for Business features. When working with such organizations, please keep the following considerations in mind:

To add such organization into Veeam Backup for Microsoft Office 365, the SharePoint Administrator role must still be assigned to a user you add an organization with. Otherwise you would see the following error: “Role is not assigned”
Backup jobs will fail with the error “The remote name could not be resolved: 'example-admin.sharepoint.com'”.

Solution

To solve the “Role is not assigned” issue:
The role can be assigned within Microsoft 365 admin center at https://portal.office.com. Unchecking “SharePoint Server” checkbox will not skip this check. However, when you upgrade from a previous version, the organization is not re-checked, which will eventually cause an issue from the point 2; hence the role still must be added.
To solve the “The remote name could not be resolved: 'example-admin.sharepoint.com'” issue:
Site and OneDrive types of items must be excluded from all backup jobs through the processing options, else the job will fail with the error Please refer to our User Guide for steps.

↧

How to restore vCenter Server without a vCenter Server

August 21, 2018, 1:52 pm

≫ Next: Backing up Hyper-V guest cluster based on VHD set

≪ Previous: Considerations for Office 365 organizations with no SharePoint Online or OneDrive for Business plans.

Challenge

The vCenter Server is not available and the vCenter Server VM needs to be restored.

Solution

https://helpcenter.veeam.com/docs/backup/vsphere/add_vmware_server.html?ver=95

2. Restore the vCenter VM to the standalone host:

https://helpcenter.veeam.com/docs/backup/vsphere/full_recovery.html?ver=95
(Be sure to choose the option to restore to a different location with different settings.)

More Information

If you receive the error “VMware ESXi with name ‘<hostnme/fqdn/ip>’ already exists.

User-added image

↧

Backing up Hyper-V guest cluster based on VHD set

August 22, 2018, 6:11 am

≫ Next: Configure Veeam Backup for Microsoft Office 365 to use an HTTP proxy

≪ Previous: How to restore vCenter Server without a vCenter Server

Challenge

VHD Set is a new shared Virtual Disk model for guest clusters in Windows Server 2016. VHD Set files can be included in application-consistent checkpoints and backed up, but there are some limitations.

Cause

Please make sure the following requirements are met:

each guest VM should have the cluster feature installed
all members of guest cluster should be on-line
VHDSet shouldn't be used as Cluster Shared Volume (CSV)
folder on CSV containing VHDSet files should have proper permissions settings
guest operating system should be Windows 2016
All VHDSet files must reside on CSV or SMB shares
Microsoft supports legacy shared VHDX on Windows 2016, but this feature cannot be used like a workaround, because it cannot be included in application-consistent checkpoints, therefore Veeam doesn't support backup of such disks on Windows Server 2016

Otherwise backup job might fail on checkpoint creation with one of the errors listed below.

Solution

Error code: '32768'. Failed to create checkpoint on collection 'Hyper-V Collection'
This issue occurs because Windows can't query the cluster service inside the guest VM. To fix this issue, make sure that the cluster feature is installed on all guest VMs and cluster service is running.

Error code: '32770'. Active-active access is not supported for the shared VHDX in VM group
This issue occurs because the VHDS disk is used as a Cluster Shared Volume (CSV), which cannot be used for creating checkpoints. To fix this issue, you need to use each disk as a shared disk instead of a Cluster Shared Volume. This can be done by using the "Remove from Cluster Shared Volume" option in cluster manager GUI.

Error code: '32775'. More than one VM claimed to be the owner of shared VHDX in VM group 'Hyper-V Collection'
This issue occurs because the shared drive was offline in the guest cluster. To fix this issue, make sure that all shared drives in the cluster that are part of the backup are online.

Error Event 19100 Hyper-V-VMMS 19100 'BackupVM' background disk merge failed to complete: General access denied error (0x80070005) This issue occurs because of a permission issue. To fix this issue, the folder that holds the VHDS files and their snapshot files must be modified to give the VMMS process additional permissions. To do this, follow these steps:

Determine the GUIDS of all VMs that use the folder. To do this, start PowerShell as administrator, and then run the following command:
```
get-vm | fl name, id
```
sample output:
```
Name : BackupVM
Id : d3599536-222a-4d6e-bb10-a6019c3f2b9b
Name : BackupVM2
Id : a0af7903-94b4-4a2c-b3b3-16050d5f80f
```

For each VM GUID, assign the VMMS process full control by running the following command:

icacls <Folder with VHDS> /grant "NT VIRTUAL MACHINE\<VM GUID>":(OI)F

Example:

icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant "NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2":(OI)F
icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant "NT VIRTUAL MACHINE\d3599536-222a-4d6e-bb10-a6019c3f2b9b":(OI)F

↧

Configure Veeam Backup for Microsoft Office 365 to use an HTTP proxy

August 10, 2018, 7:55 am

≫ Next: Backing up Hyper-V guest cluster based on VHD set

≪ Previous: Backing up Hyper-V guest cluster based on VHD set

Challenge

An HTTP proxy is required by a corporate security policy, and Veeam Backup for Microsoft Office 365 fails to run any backups or add Office 365 organizations to its scope because of the proxy.

Solution

Currently only non-transparent proxies without user authentication are supported
To make Veeam Backup for Microsoft Office 365 use an HTTP proxy, the following must be done:

Using the netsh command, a system proxy must be configured in order for Exchange Online PowerShell to work properly. Please start an elevated cmd.exe and input:
```
Netsh
Winhttp
Set proxy <proxyFQDNorIP>
```
Ensure that no Backup or Restore jobs are running, then stop both the Veeam Backup for Microsoft Office 365 Service and the Veeam Backup Proxy for Microsoft Office 365 Service
Navigate to %ProgramData%\Veeam\Backup365
Edit Config.xml, adding <Ews UseSystemProxy="True" /> under <Veeam>, but above <Archiver>
Do the same thing with Proxy.xml
Navigate to the Veeam Backup for Microsoft Office 365 installation path (by default %ProgramFiles%\Veeam\Backup365), edit the file Veeam.Archiver.Service.exe.config,and add the following lines under <configuration>:
```
<system.net>
<defaultProxy>
<proxy proxyaddress="http://yourproxyFQDNorIP:port" usesystemdefault="true" />
</defaultProxy>
</system.net>
```
And the same thing for Veeam.Archiver.Proxy.exe.config
Pay attention to proxy address format – ‘http://yourproxyFQDNorIP’ and ‘port’ are mandatory, otherwise you will see errors like this one

If you are using a remote backup proxy, then steps 1, 5, and 7 should also be repeated there, yet Veeam.Archiver.Proxy.exe.config will be in %WinDir%\Veeam\Backup365Proxy.

↧

Backing up Hyper-V guest cluster based on VHD set

August 22, 2018, 6:11 am

≫ Next: Entire VM restore fails with 'An error occurred while taking a snapshot: Error.'

≪ Previous: Configure Veeam Backup for Microsoft Office 365 to use an HTTP proxy

Challenge

Cause

Please make sure the following requirements are met:

each guest VM should have the cluster feature installed
all members of guest cluster should be on-line
VHDSet shouldn't be used as Cluster Shared Volume (CSV)
folder on CSV containing VHDSet files should have proper permissions settings
guest operating system should be Windows 2016
All VHDSet files must reside on CSV or SMB shares
Microsoft supports legacy shared VHDX on Windows 2016, but this feature cannot be used like a workaround, because it cannot be included in application-consistent checkpoints, therefore Veeam doesn't support backup of such disks on Windows Server 2016

Otherwise backup job might fail on checkpoint creation with one of the errors listed below.

Solution

Determine the GUIDS of all VMs that use the folder. To do this, start PowerShell as administrator, and then run the following command:
```
get-vm | fl name, id
```
sample output:
```
Name : BackupVM
Id : d3599536-222a-4d6e-bb10-a6019c3f2b9b
Name : BackupVM2
Id : a0af7903-94b4-4a2c-b3b3-16050d5f80f
```

For each VM GUID, assign the VMMS process full control by running the following command:

icacls <Folder with VHDS> /grant "NT VIRTUAL MACHINE\<VM GUID>":(OI)F

Example:

icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant "NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2":(OI)F
icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant "NT VIRTUAL MACHINE\d3599536-222a-4d6e-bb10-a6019c3f2b9b":(OI)F

↧

Entire VM restore fails with 'An error occurred while taking a snapshot: Error.'

August 23, 2018, 5:13 am

≫ Next: Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

≪ Previous: Backing up Hyper-V guest cluster based on VHD set

Challenge

A VM restore fails with the error:

Entire VM restore fails with 'An error occurred while taking a snapshot: Error.'

Cause

This can occur when there exists a workingDir entry in the VMX that is causing the VM to direct snapshots to a different datastore that is not present in the site where the restore is taking place.

Solution

In order to restore the VM a VM Files restore must be performed. This type of restore will restore the VM Files directly to the datastore, without attempting to register the restore VM.

1. The procedure for perform a VM Files restore can be found here: http://helpcenter.veeam.com/backup/70/vsphere/performing_vmfile_restore.html
Please keep in mind you must manually specify the host and folder on the datastore where the restored information will be placed.

2. After the VM Files restore completes, edit the VMX and remove any workingDir entries. Tips for editing VMX: kb.vmware.com/kb/1714

3. After correcting the VMX, add the VM to inventory. kb.vmware.com/kb/1006160

If you are unsure on how to proceed, please contact VMware support for VMX modifications.

More Information

Information regarding the worklingDir parameter can be found here: kb.vmware.com/kb/1002929

↧

Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

August 23, 2018, 9:02 am

≫ Next: Backup configuration is not saved after configuring backup job

≪ Previous: Entire VM restore fails with 'An error occurred while taking a snapshot: Error.'

Challenge

After installing Veeam Backup & Replication 9.5 Update 3a, backup duration could be significantly increased for jobs using Appliance transport mode (HotAdd). Nbd and Direct SAN/NFS transport modes are not affected.

Cause

This is a known issue affecting Virtual Disk Development Kit 6.5.2, Virtual Disk Development Kit 6.5.3 and Virtual Disk Development Kit 6.7 https://kb.vmware.com/s/article/55418

Solution

Hotfix for Veeam version 9.5.0.1922 is available to switch to unaffected VDDK versions. To install the hotfix:

Stop Veeam jobs and Veeam services

Make a backup copy of the following files and folders, from the backup server and all backup proxies:

C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_5
C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_7
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe

Replace the files listed in the previous step with those found in the hotfix package
Start Veeam services on server

Note: in case after applying the fix HotAdd stops working entirely, please install Visual C++ Redistributable Packages for Visual Studio 2013 package.

More Information

Download hotfix

↧

Backup configuration is not saved after configuring backup job

August 13, 2018, 5:00 am

≫ Next: Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

≪ Previous: Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

Challenge

After installing Veeam Agent for Microsoft Windows and configuring a backup job, the settings are not shown in the Control Panel, and Veeam Agent for Microsoft Windows keeps prompting to configure backup settings.

Cause

The Windows OS was installed with non-English region and culture settings. As a result, these settings were applied to the Windows system account.
Veeam Agent for Microsoft Windows is based on Microsoft .NET Framework which has some limitations in term of particular OS cultures

Solution

You will need to set Format to English (United States) for the system account.

Go to the Region settings (press Win + R > type intl.cpl > press Enter)

On the Formats tab
1. set Format to English (United States)
On the Administrative tab
1. click Copy Settings
  1. check that Format is set to English for Current user
  2. Enable the Welcome screen and system accounts checkbox
  3. Click OK to save settings and return to the Administrative tab.
Click OK to close the Region settings and save settings.
Restart your machine

Once the necessary changes are applied to the system account, a non-English Format can be set for the Current user account. Note, that the Format has to remain English (United States) for the system account.

More Information

Here is an example of a working configuration:

↧

Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

August 23, 2018, 9:02 am

≫ Next: Could not invoke guest operation error in a job when using vSphere API for Guest OS connection

≪ Previous: Backup configuration is not saved after configuring backup job

Challenge

Cause

This is a known issue affecting Virtual Disk Development Kit 6.5.2, Virtual Disk Development Kit 6.5.3 and Virtual Disk Development Kit 6.7 https://kb.vmware.com/s/article/55418

Solution

Hotfix for Veeam version 9.5.0.1922 is available to switch to unaffected VDDK versions. To install the hotfix:

Stop Veeam jobs and Veeam services

Make a backup copy of the following files and folders, from the backup server and all backup proxies:

C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_5
C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_7
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe

Replace the files listed in the previous step with those found in the hotfix package
Start Veeam services on server

Note: in case after applying the fix HotAdd stops working entirely, please install Visual C++ Redistributable Packages for Visual Studio 2013 package.

More Information

Download hotfix

↧

Could not invoke guest operation error in a job when using vSphere API for Guest OS connection

August 24, 2018, 2:54 am

≫ Next: Release Notes for Pure Storage Plug-In for Veeam Backup & Replication

≪ Previous: Backup jobs using Appliance transport mode (HotAdd) take longer after installing 9.5 Update 3a

Challenge

When Microsoft User Profile Disks (for example, in a Windows RDS farm) are used, Veeam Application Aware Image Processing through vSphere API may fail with the following error: Could not invoke guest operation.

Cause

When processing Guest OS data through vSphere API, Veeam needs to deploy a runtime component in a temporary location. C:\Windows\Temp\ folder had been used for these purposes previously. However, starting from 10.2.0 VMware tools’ version, the Temp folder location has been changed to C:\Users\username\Temp\.

When User Profile Disks feature is used, Windows has no User’s profile temp folder unless a User is logged in (once a User is logged out, Temp folder is deleted). Hence, Veeam has no temp folder to place its components in.

Solution

VMware is planning to address this issue in one of the next VMware tools releases. As a workaround, you may want to downgrade VMware tools version to an earlier version or set up a direct network connection to the Guest OS from a Guest Interaction Proxy

More Information

https://helpcenter.veeam.com/docs/backup/vsphere/runtime_process.html?ver=95

↧

Release Notes for Pure Storage Plug-In for Veeam Backup & Replication

August 24, 2018, 8:48 am

≫ Next: Computer connection status is "Rejected" in Veeam Availability Console

≪ Previous: Could not invoke guest operation error in a job when using vSphere API for Guest OS connection

Challenge

Release notes Release Notes for Pure Storage Plug-In for Veeam Backup & Replication 1.1.40

Cause

Please confirm that you are running version 9.5.0.1536 or later prior to installing this update. You can check this under Help | About in Veeam Backup & Replication console.

Solution

This update features the following enhancements:
•    Added support for volume snapshots created as part of a Pure Storage Protection Group.

This update also resolves most common support issues including the following:
•    The default HTTP timeout value was Increased from 5 to 60 seconds to make the plug-in more resilient to network issues.
•    vVol volumes are no longer displayed in the list of volumes.

More Information

[[DOWNLOAD|DOWNLOAD UPDATE|https://www.veeam.com/download_add_packs/vmware-esx-backup/purestorage/]]

MD5 checksum for PureStoragePlugin_1.1.40.zip is 8b88b326613cd1658d46b17050dfc6fb

↧

Computer connection status is "Rejected" in Veeam Availability Console

August 24, 2018, 8:53 am

≫ Next: Protecting the Veeam Server and Database: Preparation and Considerations

≪ Previous: Release Notes for Pure Storage Plug-In for Veeam Backup & Replication

Challenge

You may see following status for the computer in Veeam Availability Console -> "Discovery" -> "Discovered Computers":

User-added image

If you login to the computer in question, launch C:\Program Files\Veeam\Availability Console\CommunicationAgent\Veeam.MBP.AgentConfigurator.exe, double click on VAC icon in the system tray and you will see following:

User-added image

ClientAgent.log contains:

[INFO ] 1352 [__42] Veeam.SPP.Agent.ClientAgent: Registering agent <ID> with User <login>.
[ERROR ] 1352 [__42] Veeam.SPP.Agent.ClientAgent: Agent can't pass registration. Reconnecting...

Server.log contains:

[INFO ] 344 [__60] Veeam.SPP.AgentManagement.AgentManager: Registering agent VEEAM, with id <ID> for user <userID>
[ERROR ] 344 [__60] Veeam.SPP.AgentManagement.AgentManager: Failed to register agent VEEAM, with id <ID> : Veeam.SPP.Utilities.Exceptions.AgentHasBeenRejectedException: The VAC Agent has been rejected
at Veeam.SPP.AgentManagement.AgentManager.SaveAgent(UserNode user, Guid agentId, AgentInfo agentInfo)
at Veeam.SPP.AgentManagement.AgentManager.RegisterAgent(UserNode user, Guid agentId, AgentInfo agentInfo)

Solution

Follow VAC UI > "Discovery" > "Discovered Computers", select computer in question, press "Change settings…" and turn on "Accept new connections"

User-added image

↧

Protecting the Veeam Server and Database: Preparation and Considerations

August 24, 2018, 8:56 am

≫ Next: Usage of a predefined VMware ESXi extension (VIB) for Veeams Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

≪ Previous: Computer connection status is "Rejected" in Veeam Availability Console

Challenge

When performing a Failover to a Replica containing the Veeam Database or restore that VM from a backup file, several steps must be taken before resuming normal operation of Veeam.

Cause

Veeam Database may not be consistent with changes made to the production database since the last run of the replication/backup job. Should a failover or restore be required, the VM may experience difficulties in taking over normal operations due to these inconsistencies.

Solution

The server hosting the Veeam Database should be replicated/backed up in isolation from other jobs, ideally when no other data is processing. The server hosting the Veeam Database should have sufficient resources to ensure that VSS processing is fast and does not cause interruptions.

It is recommended that you perform and rely primarily on the built in Configuration Backup as a means for backing up and recovering the Veeam Configuration. This will preserve the current state of the Veeam Database, and also preserve all job and infrastructure related configurations within Veeam.
If the Configuration Backup was taken more recently than the last replica, be sure to Restore from the Configuration Backup to ensure that the most recent state of the Veeam Database is used.

You may need to contact Veeam Support for assistance should you experience any difficulties when failing over to a Replica of the Veeam Server or perform restore.

↧

Usage of a predefined VMware ESXi extension (VIB) for Veeams Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

August 25, 2018, 3:42 am

≫ Next: Manual Firewall changes for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

≪ Previous: Protecting the Veeam Server and Database: Preparation and Considerations

Challenge

This article describes the usage of a predefined VMware ESXi extension (VIB) for Veeams Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing.

IMPORTANT: With Cisco HyperFlex 3.0 the needed Firewall changes have been implemented in the OS image and no further, manual steps are required. Please follow the KB below only if you are running a HyperFlex version below 3.0. For new customers, we recommend to install the HyperFlex cluster with HX 3.0 and for existing customers we recommend to upgrade to HX 3.0 to benefit from the new Firewall changes.

Cause

To achieve optimal balancing within the Cisco HyperFlex data network at Backup from Storage Snapshot processing, it is needed to change the ESXi host firewall.
See more background information here.
One of the Methods to change the ESXi host firewall is by a pre-defined VIB that can be found at the Veeam Community GitHUB site.
To implement this, follow the below instructions.

Solution

Install the Firewall VIB on ESXi:

Repeat the following steps on all Cisco HyperFlex nodes in your cluster.

1. Enable ssh and log in to your ESXi host by using a tool like PuTTY
User-added image

2. Copy the VIB file to the ESXi host's tmp folder using HTTP or a SCP client
User-added image

3. Install the VIB
Command:

esxcli software vib install -v /tmp/VeeamCiscoHXFirewall.vib -f

4. Verify the VIB was installed
Command:

esxcli software vib list | grep 'Veeam'

5. Verify the new firewall rule is active
Command:

esxcli network firewall ruleset list

Note: If the VIB installation fails, you might need to set the acceptance level to CommunitySupport and retry the installation.
Command:

esxcli software acceptance set --level=CommunitySupported

Set the Veeam Proxy Servers

1. Enable allowed IP list for the new firewall rule
Command:

esxcli network firewall ruleset set -r "VeeamCiscoHXFirewall" -a false

2. Set the Veeam proxy server data network IP that is on the Hyperflex ("Storage Controller Data Network")
Repeat the following command for all Veeam proxy server or set a subnet:

esxcli network firewall ruleset allowedip add -r "VeeamCiscoHXFirewall" -i "172.17.53.53."

3. Verify the IPs are set
Command:

esxcli network firewall ruleset allowedip list | grep -v "All"

Note: Veeam recommends to set the all IPs of Veeam proxy servers in the firewall rule. Otherwise the firewall rule is enabled for all incoming connections. You can specify either the IP address or a subnet. Use one command per proxy.

Check if everything is configured correctly

1. Check the Security Profile on the ESXi hosts

For HX systems < HX2.5 using the relevant VIB.
User-added image

For HX system >= HX2.5 using the relevant VIB.
User-added image

2. Check the VIB
Command:

esxcli software vib list | grep 'Veeam'

3. Check the ruleset
Command:

esxcli network firewall ruleset list

4. Check which Veeam Proxy IPs are assigned
Command:

esxcli network firewall ruleset allowedip list | grep -v "All"

↧

Manual Firewall changes for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

August 25, 2018, 3:44 am

≫ Next: Creating a VMware ESXi extension (VIB) for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

≪ Previous: Usage of a predefined VMware ESXi extension (VIB) for Veeams Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

Challenge

This article describes making manual firewall changes for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing.

IMPORTANT: With Cisco HyperFlex 3.0 the needed Firewall changes have been implemented in the OS image and no further, manual steps are required. Please follow the KB below only if you are running a HyperFlex version below 3.0. For new customers, we recommend to install the HyperFlex cluster with HX 3.0 and for existing customers we recommend to upgrade to HX 3.0 to benefit from the new Firewall changes.

Solution

To achieve the optimal balancing within the Cisco HyperFlex data network at Backup from Storage Snapshot processing, it is needed to change the ESXi host firewall. See more background information here.

One of the methods to change the ESXi host firewall is by manual creation of an ESXi host firewall rule. This configuration is reset by an ESXi host reboot and can be used for test environments.

To open ports on ESX(i) hosts for Cisco HX < 2.5, add the following firewall rule to the services.xml file on an ESX(i) host.

<ConfigRoot>
   <service id='9230'>
     <id>VeeamCiscoFirewall</id>
     <rule id='0000'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>
         <begin>0</begin>
         <end>65535</end>
       </port>
     </rule>
</service>
</ConfigRoot>

To open ports on ESX(i) hosts for Cisco HX >= 2.5, add the following firewall rule to the services.xml file on an ESX(i) host.

<ConfigRoot>
   <service id='9230'>
       <id>VeeamCiscoHXFirewall</id>
       <rule id='0000'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>111</port>
       </rule>
       <rule id='0001'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>2049</port>
       </rule>
       <rule id='0002'>
       <direction>inbound</direction>
       <protocol>tcp</protocol>
       <porttype>dst</porttype>
       <port>2449</port>
       </rule>
       <enabled>true</enabled>
       <required>false</required>
       </service>
</ConfigRoot>

The following example shows all steps required to open the firewall at an ESXi host SSH connection:
1. Back up the current services.xml file by running the command:

cp /etc/vmware/firewall/service.xml /etc/vmware/firewall/service.xml.bak

2. Modify the access permissions of the service.xml file to allow writes by running the chmod command:

chmod 644 /etc/vmware/firewall/service.xml

chmod +t /etc/vmware/firewall/service.xml

3. Open the service.xml file in a text editor:

vi /etc/vmware/firewall/service.xml

4. Add the rule to the service.xml file (see example above)
5. Revert the access permissions of the service.xml file to the read-only default by running the command:

chmod 444 /etc/vmware/firewall/service.xml

6. Refresh the firewall rules for the changes to take effect by running the command:

esxcli network firewall refresh

7. Enable the new firewall rule:

esxcli network firewall ruleset set -r "VeeamCiscoFirewall" -e true -a false

8. Bind the firewall rule to all Veeam proxy server data network IPs. This is the IP on the HyperFlex “Storage Controller Data Network”. Repeat the command for each proxy server:

esxcli network firewall ruleset allowedip add -r "VeeamCiscoFirewall" -i "<yourVeeamProxyIP>"

9. Check the IP binding

esxcli network firewall ruleset allowedip list | grep -v "All"

10. Check if the firewall rule is enabled

esxcli network firewall ruleset list

More Information

For more information about custom firewall rule creation, click here.

↧

Creating a VMware ESXi extension (VIB) for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

August 25, 2018, 3:46 am

≫ Next: “Error: Specified argument was out of the range of valid values.” in SharePoint Online and OneDrive for business backup jobs.

≪ Previous: Manual Firewall changes for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

Challenge

This article contains instructions on how to create a VMware ESXi extension (VIB) for Veeams Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing.

To achieve the optimal balancing within the Cisco HyperFlex data network at Backup from Storage Snapshot processing over NFS, it is needed to change the ESXi host firewalls. See more background information here.

One of the Methods to change the ESXi host firewall is by a newly created VIB file that can be created with help of the VMware VIB Author Software.
Please follow the next steps to create the VIB.

IMPORTANT: With Cisco HyperFlex 3.0 the needed Firewall changes have been implemented in the OS image and no further, manual steps are required. Please follow the KB below only if you are running a HyperFlex version below 3.0. For new customers, we recommend to install the HyperFlex cluster with HX 3.0 and for existing customers we recommend to upgrade to HX 3.0 to benefit from the new Firewall changes.

Solution

Create a VIB in SLES11

SLES11 can be downloaded here.
VMware VIB Author can be downloaded here.
All steps are performed as the root user from the root (/) directory.

1. Prepare SLES

zypper install python-lxml
zypper install python-urlgrabber

2. Install VIB Author

cd /tmp
rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
cd /

3. Create File Directory

mkdir stage
mkdir stage/payloads
mkdir stage/payloads/payload1
mkdir stage/payloads/payload1/etc
mkdir stage/payloads/payload1/etc/vmware
mkdir stage/payloads/payload1/etc/vmware/firewall

4. Copy the required files to the folder tree
The "descriptor.xml" (link here) must be copied to /stage

descriptor.xml sample:
<vib version="5.0">

<type>bootbank</type>
<name>VeeamCiscoHXFirewall</name>
<version>1.0.0-0.0.1</version>

<vendor>Veeam</vendor>
<summary>Veeam Firewall rule for Cisco HyperFlex</summary>
<description>Adds inbound ports required by Veeam</description>

<relationships>
<depends></depends>
<conflicts/>
<replaces/>
<provides/>
<compatibleWith/>
</relationships>
<software-tags>
</software-tags>
<system-requires>
<maintenance-mode>false</maintenance-mode>
</system-requires>
<file-list>
<file></file>
</file-list>
<acceptance-level>community</acceptance-level>
<live-install-allowed>true</live-install-allowed>
<live-remove-allowed>true</live-remove-allowed>
<cimom-restart>false</cimom-restart>
<stateless-ready>true</stateless-ready>
<overlay>false</overlay>
<payloads>
<payload name="payload1" type="vgz"></payload>
</payloads>

</vib>

The “VeeamCiscoHXFirewall.xml” <download link> must be copied to /stage/payloads/payload1/etc/vmware/firewall

The VeeamCiscoHXFirewall.xml for Cisco HX version < 2.5:
<ConfigRoot>
<service id='9230'>
    <id>VeeamCiscoHXFirewall</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>
        <begin>0</begin>
        <end>65535</end>
      </port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
</service>
</ConfigRoot>

The VeeamCiscoHXFirewall.xml for Cisco HX version >= 2.5:
<ConfigRoot>
       <service id='9230'>
              <id>VeeamCiscoHXFirewall</id>
              <rule id='0000'>
              <direction>inbound</direction>
              <protocol>tcp</protocol>
              <porttype>dst</porttype>
              <port>111</port>
              </rule>
              <rule id='0001'>
              <direction>inbound</direction>
              <protocol>tcp</protocol>
              <porttype>dst</porttype>
              <port>2049</port>
              </rule>
              <rule id='0002'>
              <direction>inbound</direction>
              <protocol>tcp</protocol>
              <porttype>dst</porttype>
              <port>2449</port>
              </rule>
       <enabled>true</enabled>
       <required>false</required>
       </service>
</ConfigRoot>

5. Create the VIB using vibauthor:

vibauthor -C -t stage -v VeeamCiscoHXFirewall -f

6. Creation finished, ready for download
The VIB is now created and available in the root (/) directory. You can use the SCP client to download the VIB to your local operating system.

7. Install on ESXi

Install the Firewall VIB on ESXi:

Repeat the following steps on all Cisco HyperFlex nodes in your cluster.

a. Enable ssh and log in to your ESXi host using a ssh tool like PuTTY
User-added image

b. Copy the VIB file to the ESXi host's tmp folder using HTTP or a SCP client
User-added image

c. Install the VIB
Command:

esxcli software vib install -v /tmp/VeeamCiscoHXFirewall.vib -f

User-added image

d. Verify that the VIB was installed
Command:

esxcli software vib list | grep 'Veeam'

e. Verify that the new firewall rule is active
Command:

esxcli network firewall ruleset list

Note: If the VIB installation fails, you may need to set the acceptance level to CommunitySupport and retry the installation.
Command:

esxcli software acceptance set --level=CommunitySupported

Set the Veeam Proxy Servers

1. Enable allowed IP list for the new firewall rule
Command:

esxcli network firewall ruleset set -r "VeeamCiscoHXFirewall" -a false

2. Set the Veeam proxy server data network IP that is on the Hyperflex "Storage Controller Data Network"
Repeat the following command for each Veeam proxy server:

esxcli network firewall ruleset allowedip add -r "VeeamCiscoHXFirewall" -i "172.16.3.10"

3. Verify that the IPs are set
Command:

esxcli network firewall ruleset allowedip list | grep -v "All"

Note: Veeam recommends to set the IPs of each Veeam proxy server that is on the HyperFlex “Storage Controller Data Network” in the firewall rule. Otherwise the firewall rule is enabled for all incoming connections. Issue this command once per IP Address. It is important to use the IP Address on the “Storage Controller Data Network”, and not the public, or management IP address.

Check if everything is configured correctly

1. Check the Security Profile on the ESXi hosts
User-added image

2. Check the VIB

esxcli software vib list | grep 'Veeam'

3. Check the ruleset

esxcli network firewall ruleset list

4. Check which Veeam Proxy IPs are assigned

esxcli network firewall ruleset allowedip list | grep -v "All"

↧

“Error: Specified argument was out of the range of valid values.” in SharePoint Online and OneDrive for business backup jobs.

August 26, 2018, 3:12 am

≫ Next: “Unauthorized” error is thrown by SharePoint Online and/or OneDrive for business backup jobs

≪ Previous: Creating a VMware ESXi extension (VIB) for Veeam Backup from Storage Snapshot with Cisco HyperFlex IOvisor processing

Challenge

SharePoint Online and/or OneDrive backup jobs fail with the following error:

25/07/2018 10:05:17 PM   61 (6188) Error: Specified argument was out of the range of valid values.
25/07/2018 10:05:17 PM   61 (6188) Type: System.ArgumentOutOfRangeException
25/07/2018 10:05:17 PM   61 (6188) Stack:
25/07/2018 10:05:17 PM   61 (6188)    at Microsoft.SharePoint.Client.ChunkStreamBuilder.ReadonlyChunkStream.set_Position(Int64 value)

Cause

Due to a bug in SharePoint Online API, incorrect length of a file is returned to Veeam Backup for Microsoft Office 365 during a backup job run. This results in treating an EOF marker as an ungraceful termination of a download stream and initiating a retry from the last good position. Since the last good position is, in fact, an EOF marker, a read request is issued with an argument which is out of the range of valid values.

Solution

While a bug in SharePoint Online API is yet to be fixed by Microsoft, a hotfix with improved handling of such situations is already available for Veeam Backup for Microsoft Office 365. To get it, please contact Veeam technical support.

↧

“Unauthorized” error is thrown by SharePoint Online and/or OneDrive for business backup jobs

August 26, 2018, 3:17 am

≫ Next: How to Backup a Windows Failover Cluster with Veeam Agent for Microsoft Windows

≪ Previous: “Error: Specified argument was out of the range of valid values.” in SharePoint Online and OneDrive for business backup jobs.

Challenge

Backup jobs for SharePoint Online and/or OneDrive items fail with any of the following errors:

07-08-2018 11:11:39 3 (6640) Error: The request failed with HTTP status 401: Unauthorized. 
07-08-2018 11:11:39 3 (6640) Type: System.Net.WebException 
07-08-2018 11:11:39 3 (6640) Stack: 
07-08-2018 11:11:39 3 (6640) at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

Error: Cannot contact web site 'https://example.sharepoint.com/' or the web site does not support SharePoint Online credentials. The response status code is 'Unauthorized'. The response headers are 'X-SharePointHealthScore=1, X-MSDAVEXT_Error=917656; Access+denied.+Before+opening+files+in+this+location%2c+you+must+first+browse+to+the+web+site+and+select+the+option+to+login+automatically."

Cause

Legacy authentication protocols are disabled within Organization’s settings. As a result, applications using the SharePointOnlineCredentials class to authenticate the SharePoint Online resources are denied to access. Veeam Backup for Microsoft Office 365 uses the above-mentioned class to connect to SharePoint Online.

Solution

To resolve the issue, legacy protocols must be allowed. There are 2 ways to enable legacy protocols:

In SharePoint Online admin center under “Access control” -> “Apps that don’t use modern authentication”, choose “Allow”
Use Set-SPOTenant cmdlet as in this Microsoft article.

↧