NFC connectivity troubleshooting steps

Challenge

A job fails with an error related to NFC (Network File Copy) connectivity. For example:

Error: NFC storage connection is unavailable.

Failed to create NFC download stream.

Failed to create NFC upload stream.

Cause

The cause of the majority of NFC errors fall in to 3 primary categories:

• Port (902)
• Permissions*
• DNS

*If the account that Veeam Backup & Replication is using to communicate with the VMware Environment has granular permissions set please confirm all permissions are set according to the Granular Permissions Guide.
 
Due to the natue of NFC as a client server type connection, there is also potential that all things may be correct on the Veeam client side and there exists an issue within the VMware Infrastructure itself.

Solution

Note: When investigating this issue it is advisable to reconfigure the job to use a specific proxy. This is done to isolate an issue specific to a single proxy, as well as make it easier to identify which server the logs will found on.

Port, Permission, or DNS: Investigation

To investigate the potential of a Ports, Permissions, or DNS issue you will need to review the Agent logs.

1. Navigate to the following location, on the Veeam Proxy:

   %programdata%\Veeam\Backup

2. Open the folder that matches the name of the job that is having an issue.
3. Within this folder you will need to find the agent log for the specific VM.
   • For a Backup Job:

     Agent.<JobName>.Source.<VMName>.log

   • For a Replication Job:

     Agent.<JobName>.Source.<VMName>.log

     Agent.<JobName>.Target.<VMName>.log

4. Search for the following:

   Creating NFC download stream

5. Here you will want to look for the following items that could point to the cause of the error.
   1. Port
      If there is an issue with ports, there will be an error after the following:

      nfc| Establishing connection with the host [esx1]. Port: [902]. Failed.

   2. Permissions
      If there is an issue with permissions, there will be an error after the following:

      nfc| Sending authd message: [SESSION ID NUMBER].
      nfc| Sending authd message: [Name of Host].
      nfc| Waiting for the authd reply message...

   3. DNS
      If there is an issue with DNS, the following error will be found.

      nfc| Resolving host name (esx1) to IP address...
      nfc| Resolving host name (esx1) to IP address... Failed.

Port Issues

Further testing can be done using the following PowerShell command, the destination should be the hostname/FQDN/IP shown in the logs for the host.

Test-NetConnection -ComputerName "<destination_host>" -Port 902

If the Proxy is unable to reach port 902, check all applicable Firewalls between the server and the ESXi host.
 

Permissions Issues

When a vCenter server or standalone ESXi host is added to the [Backup Infrastructure] section of Veeam Backup & Replication, the user assigns an account to be used. That account should have sufficient privileges to all objects related to the VM being backed up. 
 

DNS Issues (Most Common)

The proxy must be able to resolve the hostname or FQDN of the hosts in the Virtual Infrastructure.
 

Host-side Issues

Restarting the Management Agents for the ESXi host may be required to correct NFC connection issues.



Other Isolation Ideas:
These are not solutions, they are only meant to help pinpoint a cause.

• Try a different proxy. (Isolates issue to specific Veeam Proxy)
• Try vMotioning the VM to a different host. (Isolates issue to specific ESXi host)
• Try Storage vMotioning the VM to a different datastore. (Isolates possible missing permission at datastore level)
• Try adding the ESXi host the VM is on as a standalone host to Veeam, then create a backup job to pull the VM from the standalone host.
  (This will help isolate if the Management Agents need to be restarted on the ESXi host)
  Remember to remove the job and standalone host from Veeam's configuration after the issue is resolved.

More Information

The proxy being used to process a VM can be identified by opening the “Task” log specific to the VM from inside the folder named after the job in %programdata%\Veeam\Backup. Searching the Task log for the words “starting agent” will allow for the identification of which server performed the task.

Further information can be found in this forum post: http://forums.veeam.com/veeam-backup-replication-f2/failed-to-create-nfc-download-stream-t2991-15.html#p69806

The NFC connection requires the following:

• DNS Resolution of target host
• Port 902 is open to/from Backup Server/Proxy to ESX(i) host
• Permissions to download files via ESX(i) host and/or vCenter

 
An issue with Port 902 may represent an issue with a firewall on the ESXi host, Veeam Proxy, or the connection between the two.