Challenge
After installing Windows April update KB4015553 Veeam jobs and services fail at different operations with:
"An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full"
Cause
This particular update (and all consequent rollups that include it) causes the system to run out of free RPC ports when there are failed iSCSI connections.Solution
Repairing or removing unsuccessful iSCSI connections, followed by the server reboot, was found to fix the issue. Additionally, remove any unused targets from the ‘Favorite Targets’ tab that are associated with these connections as they may be reconnected after reboot.
This issue has been addressed by Microsoft. Apply the appropriate update below to resolve the issue
For 2016: https://support.microsoft.com/en-us/help/4025334
For 2012 R2: https://support.microsoft.com/en-us/help/4025335
-------------------------------------------------------------------------------------------------------------------------------------------------
The update that resolved the above Buffer Space iSCSI issue (KB4025335) has caused another iSCSI issue that only affects Backup from Storage Snapshots using an iSCSI connection.
After installing Windows Update KB4025335, KB4034681, KB4025333, or KB4025336, Veeam jobs utilizing Backup from Storage Snapshots using an iSCSI connection fail with the following errors:
“Error: Cannot retrieve physical extents for connecting iSCSI target IQN iqn.2007-11.com.nimblestorage:lun-nim-oracle-veeam-lun-nim-oracle-v069a713ff86811d1.000000d1.fef260ee.s069a713ff86811d1.000000de.0000b1cb on proxy IQN iqn.1991-05.com.microsoft:mtl-s-fs0005.solotech.local”
In the logs, it can be seen with this error:
Task Log:
“[25.07.2017 09:44:54] <61> Error Failed to find physical drive for LUN with lun Id 'WwnOrIqn: 'iqn.1992-04.com.emc:cx.apm00140946425.a1', LunId: '10',
[25.07.2017 09:44:54] <61> Error Unable to retrieve devices for iSCSI session. number of retry attempts: '20'. Interval: '100'. (Veeam.Backup.Common.CCppComponentException)
[25.07.2017 09:44:54] <61> Error in c++: void __thiscall disklib::vmfs_san::iscsi_api::iscsi_discovery::GetDevicesForSessionWithRetry(struct _ISCSI_UNIQUE_SESSION_ID &,class std::vector<struct ISCSI_DEVICE_ON_SESSIONW,class std::allocator<struct ISCSI_DEVICE_ON_SESSIONW> > &,int,int) failed.
[25.07.2017 09:44:54] <61> Error in c++: void __thiscall disklib::vmfs_san::iscsi_api::iscsi_discovery::LoginTarget(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,class std::vector<struct disklib::vmfs_san::iscsi_api::IscsiDriveInfo,class std::allocator<struct disklib::vmfs_san::iscsi_api::IscsiDriveInfo> > &) failed.”
Proxy Resources Log:
“[04.08.2017 13:50:55] < 4016> cli| VMFSSAN.ConnectISCSITarget
[04.08.2017 13:50:55] < 4016> cli| (EString) TargetName = iqn.1992-04.com.emc:cx.apm00140946425.a1
[04.08.2017 13:50:55] < 4016> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[04.08.2017 13:50:55] < 4016> dsk| Retrieving devices for target 'iqn.1992-04.com.emc:cx.apm00140946425.a1'.
[04.08.2017 13:50:55] < 4016> dsk| WARN|The data area passed to a system call is too small.”
The above issue is resolved by installing Windows Update KB4034663, which can be found here:
https://support.microsoft.com/en-us/help/4034663/windows-8-1-windows-server-2012-r2-update-kb4034663
More Information
https://social.technet.microsoft.com/Forums/SharePoint/en-US/492f62ce-aec8-4777-98f0-0c9dbf302954/problem-with-kb4015553?forum=winserverwsushttps://support.microsoft.com/en-us/help/4015550
This security update introduced an issue in which, if an iSCSI target becomes unavailable, attempts to reconnect will cause a leak. Initiating a new connection to an available target will work as expected.
If after performing the steps outlined above the issue still occurs, a support case with Microsoft may be needed.