Challenge

Cause

Solution

More Information

Challenge

Product Information:

Company name: QSAN Technology, Inc.
Product Family: XCubeNAS XN8000
Status: Veeam Ready - Repository
Classification Description: Verified backup storage that supports all Veeam backup and restore features.

Solution

Product Details:

Model number: XN8012R
Storage Category: Hybrid RAID Storage
Drive quantity, size, type: 4 – 2TB SSD
Storage configuration: RAID5
Firmware version: QSM 3.1.1
Connection protocol and speed: iSCSI, 10 GbE
Additional support: All models and configurations of XCubeNAS XN8000 with specifications equivalent or greater than the above

QSAN Storage Manager 3 (QSM 3) is a NAS operation system specially designed for QSAN XCubeNAS series. The core of QSM is Linux kernel and in-house fine-tuning 128-bit ZFS. The ZFS has high scalability and can be managed easily, also with the ability to maintain the data integrity. QSM 3 not only inherits the amazing native features of ZFS, its powerful storage features ensure persistent, reliable storage management, protection against data corruption, seamless capacity expansion, several data integrity mechanisms, pool and disk encryption protection, unlimited snapshots, and unlimited clones.

Veeam testing configuration:

Veeam Build Number: 9.0.0.1922

Job Settings:

Repository Settings:

Vendor recommended configuration:

Hardware Settings:

• Array deduplication used in testing – No
• Array compression used in testing – Yes

Challenge

Solution

 Implementation step 1 - Backup & Replication  

1. Use a new Windows Server and install Veeam Backup & Replication v9.5 Update 4 or newer if you do not have a Veeam Backup Server. The Server can run within any VMware cloud on AWS SDDC, AWS S3 or on-premises environments, if the network connection to the VMC vCenter/Veeam Servers potentially the VMs (for Guest processing) is possible.
2. Add DNS network settings so that this Server can resolve Internet DNS names to be able to resolve the fully qualified domain name of the VMC vCenter server
3. Check the below information carefully for any known limitations and configuration steps before you proceed.

Implementation step 2 - VMware Cloud on AWS

Firewall Configuration for vCenter connection

1. Usage of the vCenter public IP for customers with NSX-v (default) 
   • Open Port TCP 443 from Backup Server and Proxy Server to the predefined vCenter object on the Compute Network.
   • Allow the Compute Gateway Public IP to communicate over TCP 443 with the predefined vCenter object on the Management Network.
     
2. Usage of a VPN tunnel for customers with VMware NSX-v 
   To be able to directly access the vСenter within VMC, please follow the VMC internal guidelines to create a VPN tunnel from the compute network to the management network: https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.getting-started/GUID-30BED7B3-D312-4DF3-BD7A-66F8D1C619DC.html
   Please update your DNS Servers to resolve the FQDN of the vСenter to its private IP address. If you want to use hosts entries on the Veeam Server for it, add them on all Veeam Backup and Proxy Servers.
   If your Backup & Replication (Management) Server is outside of the VMC cluster, please implement the same VPN connection for it.
3. Usage of the local connection for customers with VMware NSX-t 
   NSX-t allows VMC customers to directly access the management network over the built-in firewall. TCP Port 443 needs to be opened from all Veeam Backup and Veeam Proxy Servers as a Source with the vCenter internal IP as a target.
   a) Configure DNS entry of the vCenter for local IP address usage.

   Go to your SDDC Management – Settings – vCenter FQDN and select the Private vCenter IP address.
   Hint: If you configure the vCenter DNS record for the internal IP address, you will lose VMC connection from Backup and Replication Server outside of VMC. You can use the local hosts file or any other DNS method to resolve the vCenter FQDN with the public IP address on the Veeam Server outside of VMC. Optionally, use the Public IP address for the VMC internal and external Veeam Server.

   b) Open firewall ports for vCenter Server access

   On the Management Network
   
   On the Compute Gateway
   

1. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
2. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

1. On the Compute Network:
   1. Open TCP 22 (SSH) port from Veeam Backup server and Veeam proxy server to the Amazon VPC where the EC2 Server was installed. You can as well define the exact IP addresses of the repository server as Destination.
   2. Open TCP 2500-5000 ports for Veeam Data Transport in both directions for same servers. It is recommended to use the VMware Cloud on AWS integrated high throughput/low latency ENI network connection to avoid any traffic costs.
2. Open the same ports on the Inbound Firewall of the Amazon EC2 server used as a repository server. As the Firewall Rule Source you should add all Veeam Backup Servers (including Proxy/Repository/MountServer/Console/…) instead of 0.0.0.0/0                                                                                                                                                                 

1. Veeam Backup Copy Job to a second EC2 Server used as an additional Repository. The second EC2 Server can be placed on another AWS Availability Zone or AWS Geo Location.
2. AWS Storage Gateway Software in VTL mode can be used to emulate a Tape Library to write data to S3. Veeam Backup to Tape Jobs can be used with it. For details see: https://www.veeam.com/wp-using-aws-vtl-gateway-deployment-guide.html
3. Veeam Backup Copy Job to on premises or Veeam Cloud Connect (Enterprise). There is no special configuration needed for this use case beside network and firewall connections. For standard Repository usage on premises it is recommended to create a VPN tunnel from VMware Cloud on AWS to the on premises datacenter. This can be done by the VMC integrated VPN functionality, by Veeam PN or Third Party.

1. VMware Cloud on AWS used as Restore target. 
   1. Implementation steps 1-4 are needed.
2. Veeam VM Replication.
   1.  Implementation steps 1-5 are needed. The Repository Server (when NOT used for Backups, can run within the VMware Cloud on AWS SDDC to store the Veeam Replication data. On premises to VMC, VMC to VMC and VMC to on premises is possible. Usage of Veeam Availability Orchestrator is possible in specific scenarios, see VAO deployment guide: https://helpcenter.veeam.com/docs/vao/deployment/welcome.html?ver=10

More Information

VMware Cloud on AWS specific problems and solutions:
 

Problem

Solution

1. After restore, go to the VM configuration within the vCenter
2. Delete the network card and save the configuration
3. Create a new network card for the VM
4. Apply the needed Network card settings within the Operating System as usual

Problem

Solution

5. Create a vCenter User with required rights (Active Directory linked mode) described here, or use the cloudadmin@vmc.local user.
6. When adding a vCenter server, specify the fully qualified domain name (FQDN) that ends with vmwarevmc.com or vmc.vmware.com (depending on the URL shown in the VMC interface for the vCenter).

Problem

Solution

• vsanDatastore datastore
• Management VMs folder
• Mgmt-ResourcePool resource pool

Problem

Solution

Problem

Solution

Challenge

Solution

1. Close the Veeam Backup & Replication console.
2. Open Regedit.exe
3. Navigate to the key HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
4. Add a new DWORD ‘EnableDBExclusions’
5. Set the Value to 1

More Information

Challenge

Solution

Requesting rights to manage cases

To access this new feature you need to be logged into the ProPartner portal.

Find the desired contract and submit a request to manage cases by clicking the Request permission button. Once the request is submitted the button is replaced with Pending approval status, and an e-mail about the new request is sent to the Primary License Administrator of the selected contract.

Approval of requests

The Primary License Administrator of the contract will receive an email notification about the new request to manage cases. Additionally, all requests will be visible to any License Administrators and can be Accepted or Declined in their My Account:



Any users with the following roles on the contract can approve a request:

• Primary license administrator

• Secondary license administrator

• License administrator

After the request is approved or declined, the user from the partner's account who submitted the request will receive a communication from Veeam about the request status.

Opening a case via the ProPartner Portal

Once the request is approved, ProPartner users will see an Open a Case button on the ProPartner page:

Clicking the button will take you to a predefined Open a Case form.
Fill in the remaining required fields, click Next and submit the case as you normally would with any other case.

More Information

Challenge

Solution

Challenge

Cause

Solution

Importing Signatures

If Veeam ONE Monitor server has no Internet connection, you can manually import the file with the latest version of signatures:

1. Download the signatures.
2. Open Veeam ONE Monitor. For details, see Accessing Veeam ONE Monitor.
3. At the bottom of the inventory pane, click Data Protection View.
4. In the inventory panel, select the main node.
5. Open the Veeam ONE Agent tab.
6. At the top of the page, click Import Signatures.
7. Specify a path to the .package file with signatures obtained from Veeam Support and click Open.

More Information

Challenge

Cause

Solution

msiexec /update c:\VAC.ApplicationServer.x64_2.0.2.1807.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACApplicationServerSetup.txt

msiexec /update c:\VAC.ConnectorService.x64_1.0.0.254.msp /l*v C:\ProgramData\Veeam\Setup\Temp\VACConnectorServiceSetup.txt

More Information

Challenge

Cause

Solution

Importing Signatures

If Veeam ONE Monitor server has no Internet connection, you can manually import the file with the latest version of signatures:

1. Download the signatures.
2. Open Veeam ONE Monitor. For details, see Accessing Veeam ONE Monitor.
3. At the bottom of the inventory pane, click Data Protection View.
4. In the inventory panel, select the main node.
5. Open the Veeam ONE Agent tab.
6. At the top of the page, click Import Signatures.
7. Specify a path to the .package file with signatures obtained from Veeam Support and click Open.

More Information

Challenge

Cause

1. Too many jobs are running at a time for the number of concurrent tasks set across all proxies available to the job that failed.
2. Proxies may be unavailable (Shutdown, not connected to the network, etc.).
3. The existing set of backup proxies does not have enough resources to process the backup jobs in a timely manner.
4. The virtual machine might be being processed by another job
5. New LUNs created and/or VMs migrated to new LUNs

Solution

1. Stagger your backup jobs and replication jobs at different time slots so that a job isn’t waiting on available proxies when it starts. (Ex. Instead of scheduling your jobs to all start at 8:00PM, start one job at 8:00, another at 8:30, and another at 9:00.)
2. Increase the number of concurrent tasks allowed on your proxies. However, keep in mind the system requirements.  See How to set max concurrent tasks. If your backup proxy is a virtual machine, you should increase the amount of CPU and RAM available to the proxy.

• VERSION 7.X through to VERSION 9.X with default job settings requires 1 CPU core and 200 MB RAM per task that you wish the proxy to run at a time, plus 2 GB RAM for the operating system and Veeam services. Although only 200 MB per task is required, a 2GB per CPU core ratio is recommended for best performance and reliability. (2 Tasks at once = 2 CPU cores and 2.4 GB RAM required, 4GB recommended)
  • If using high compression, 2 cores per concurrent task are required.
  • Memory usage may be greater than the minimum system requirements. If memory usage on the proxy frequently exceeds 80%, consider increasing the available memory.
  • VERSION 7.X through to 9.X runs tasks in parallel, but can run sequentially with the parallel processing option disabled. Processing multiple disks on one VM will reserve multiple concurrent tasks. See Enabling Parallel Processing.
• Remember to not under-allocate your Veeam Backup server, especially if it is your backup proxy.  If the configuration database is installed on a local SQL instance (the default installation), then SQL Express should be allocated 1 CPU core and 1-1.5 GB RAM when planning your resource usage. A full version of SQL may use significantly more resources in a large environment. 500 MB RAM per concurrent task is recommended for the Veeam Backup services and the SQL server combined; 4 GB minimum.

3. Deploy additional backup proxies from within “Backup Infrastructure->Backup Proxies”.
   • This can be a physical server, an existing VM, or a newly-deployed VM. If adding existing virtual machines, bear in mind that Changed Block tracking will be disabled when backing up these VMs, which can substantially increase processing time.
4. Make sure that your backup job or replication job is selecting the correct proxies. Choose “Automatic selection” to have Veeam try to choose the best proxy from your available pool of proxy machines, or choose a specific set of proxies. The latter can be useful if you have set up a specific proxy with a large amount of resources to be used on a large backup or replication job.

• However, be careful as a backup job set to use a single proxy can fail with the above error if that proxy is unavailable. It might be beneficial to choose more than one if manually specifying proxies.
• Also note that automatic proxy selection will detect available transport modes for each proxy, and then wait for a proxy with the best available transport mode. The modes are ranked as follows: Direct SAN (SAN) > Virtual Appliance (HOTADD) > Network (NBD). For example, if you have a virtual machine proxy on each of three hosts, and all of your VMs are on local storage, a job set to automatic will wait for the proxy that has access to the local storage (HOTADD) rather than use an available proxy that can only read the disk over the network (NBD). For more information, see Transport Modes.

5. Investigate backup job performance. If specific jobs are taking longer to process than normal, check for warnings, compare the bottleneck statistics to previous jobs sessions, and try to isolate the problem to a specific proxy, repository, host, or datastore.
6. Present newly added LUNs to the Veeam Server

More Information

Challenge

You would like to know the ports used for Veeam Backup & Replication and what they apply to.

Solution

VMware
Veeam Backup & Replication 9.5 - https://helpcenter.veeam.com/docs/backup/vsphere/used_ports.html?ver=95
Veeam Backup & Replication 9.0 - http://helpcenter.veeam.com/backup/vsphere/used_ports.html
Veeam Backup & Replication 8.0 - http://helpcenter.veeam.com/backup/80/vsphere/used_ports.html
Veeam Backup & Replication 7.0 - http://helpcenter.veeam.com/backup/70/vsphere/used_ports.html
Veeam Backup & Replication 6.5 - http://helpcenter.veeam.com/backup/65/vsphere/used_ports.html
 
Hyper-V
Veeam Backup & Replication 9.5 - https://helpcenter.veeam.com/docs/backup/hyperv/used_ports.html?ver=95
Veeam Backup & Replication 9.0 - http://helpcenter.veeam.com/backup/hyperv/used_ports.html
Veeam Backup & Replication 8.0 - http://helpcenter.veeam.com/backup/80/hyperv/used_ports.html
Veeam Backup & Replication 7.0 - http://helpcenter.veeam.com/backup/70/hyperv/used_ports.html
Veeam Backup & Replication 6.5 - http://helpcenter.veeam.com/backup/65/hyperv/used_ports.html

Challenge

Cause

Solution

To resolve this issue, we will need to Install the Windows Assessment and Deployment Kit *ADK* (Windows 8 and later), also known as Windows Automated Installation Kit *AIK* for Windows 7. The Server OS versions will use the same Kits as their matching Desktop OS. In this article, we will cover the installation for Windows 10 and Windows 7, as AIK and ADK have different installation steps.

For the first step, you will want to download the appropriate Kit from this link.

For Windows 10 1809 and later: Download the WinPE add-on for the ADK which is available here - https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install  

Windows 10 - ADK

Windows 7 - AIK

Setting Registry Values

ForceUseAdkForRecoveryMedia

• Name: ForceUseAdkForRecoveryMedia
• Type: REG_DWORD
• Value: 1

DismPath

• Name: DismPath
• Type: String
• Value: Full path to your EXE, including the dism.exe

More Information

Challenge

Cause

Solution

Challenge

Cause

Solution

Challenge

Cause

Solution

The compatibility issue will be remediated in Update 4 GA.
Hotfix is available for Veeam 9.5 Update 4 RTM, please contact Veeam Support for assistance.

Alternatively, you can switch to version v1_4 with the considerations below.

The following Cloud Connect functionality requires changing request's version from v1_3 to v1_4:

• Create, Edit tenants

The following may require changing request's content:

• Tenant edit. GET request now returns a new tenant type block. Adding this block to PUT requests should allow functionality to work as intended:
  <TenantType>
         <StandaloneTenant>
            <TenantCredentials>
               <Username>TENANT_NAME</Username>
            </TenantCredentials>
         </StandaloneTenant>
  </TenantType>   
  
  The element is described in the new XSD as follows: 

  <xs:element name="TenantType" type="CloudTenantType" minOccurs="0" maxOccurs="1"/>

• Task status. Complex tasks such as create/edit/delete tenant have got a new status OperationFinished which should be properly enumerated.

Challenge

Solution

zypper install python-lxml
zypper install python-urlgrabber

cd /tmp
rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
cd /

mkdir stage
mkdir stage/payloads
mkdir stage/payloads/payload1
mkdir stage/payloads/payload1/etc
mkdir stage/payloads/payload1/etc/vmware
mkdir stage/payloads/payload1/etc/vmware/firewall

vibauthor -C -t stage -v VeeamCiscoHXFirewall -f

esxcli software vib install -v /tmp/VeeamCiscoHXFirewall.vib -f

d. Verify that the VIB was installed
Command:

esxcli software vib list | grep 'Veeam'

esxcli network firewall ruleset list

esxcli software acceptance set --level=CommunitySupported

esxcli network firewall ruleset set -r "VeeamCiscoHXFirewall" -a false 

esxcli network firewall ruleset allowedip add -r "VeeamCiscoHXFirewall" -i "172.16.3.10"

esxcli network firewall ruleset allowedip list | grep -v "All"

esxcli software vib list | grep 'Veeam'

esxcli network firewall ruleset list

esxcli network firewall ruleset allowedip list | grep -v "All"

More Information

Challenge

Cause

1. Tenant’s Veeam installation had been updated to the latest version (9.5 Update 4) before the SP's installation.
2. SP's Cloud Connect server was unavailable during the upgrade.

Solution

Challenge

To allow communications between Veeam Agents and VBR, TLS certificates are used. By default, Veeam Backup & Replication uses a self-signed certificate.

Solution

Please refer to the following User Guide page for further instructions: Using Certificate Signed by Internal CA

Challenge

Product Information:

Company name: Dell/EMC
Product Family: Unity
Status: Veeam Ready - Integrated
Classification Description: Integrated storage where joint development activities between the manufacturer and Veeam have occurred to create advanced backup or restore functionalities.

Solution

Product Details:

EMC Unity sets the new standards for midrange storage with a powerful combination of simplicity, modern design, affordable price point, and deployment flexibility –perfect for resource-constrained IT professionals in large or small companies. Unity is perfect for midsized deployments, Remote Office/Branch Office locations, and cost-sensitive mixed workload environments. It is designed for all-flash, delivers the best value, and is available in purpose-built (all flash or hybrid), converged deployment (through VCE), and as a software -defined virtual edition. With all-inclusive software, new differentiated features, internet-enabled management, and a modern design, Unity is where powerful meets simplicity.

Veeam testing configuration:

Veeam Build Number: 9.5.0.823

Challenge

Product Information:

Company name: DELL EMC
Product Family: VNX/e
Status: Veeam Ready - Integrated
Classification Description: Integrated storage where joint development activities between the manufacturer and Veeam have occurred to create advanced backup or restore functionalities.

Solution

Product Details:

• VNX Operating Environment for block 05.33.008.5.119
• VNX Operating Environment for File 8.1.8.121
• EMC Unisphere 1.3.8.1.0119

The VNX family delivers industry-leading innovation and enterprise capabilities for file and block storage in a scalable, easy-to-use unified storage solution. VNX storage combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the affordability, efficiency and performance needs of today’s enterprises. All of this is available in a choice of systems ranging from affordable entry-level solutions to high-performance, petabyte-capacity configurations servicing the most demanding mixed workload requirements.

Veeam testing configuration:

Veeam Build Number: 9.5.0.823