Challenge
Backup job fails, indicating that a file is locked by another process/task. The console may indicate specifically what file is locked. Identification of the process that is locking the file(s) must take place.Solution
Windows:There are a few utilities that can show File Locks. Pre-installed is Resource Monitor.
In this example, there is a PID for the VeeamAgent process, meaning Veeam has a lock on this file. The agent responsible for this lock can be confirmed in logging, or with the assistance of support. The PID of a given agent exists at the beginning of any source or target agent log. In the case of file locks, the target agent log should be examined.
< 23740> Windows agent.
< 23740> Path to the executable module: C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe
< 23740> Agent version: 9.5.0.1536
< 23740> Installed memory, MB: 8191
< 23740> PID: 18232
Linux:
There are a few utilities that can be used to determine File Locks on a Linux repository. This will cover lslocks, however there may be other specific utilities and methods based on your distribution. It is important to differentiate between Locks and Open Files, though. A file can be locked, but not open, so a command like lsof may not produce any results.
LSLOCKS – Requires util-linux package
http://man7.org/linux/man-pages/man8/lslocks.8.html
LSOF
Simply typing LSOF will show a list of all open files belonging to active processes. You can also specify a specific file if the file name is known, using:
lsof “\path\to\file”
Manually investigating /proc/locks can also be done, like so:
sudo find -L /proc/*/fd -maxdepth 1 -print -exec readlink {} \;
In either scenario, one must first verify that the file is not being actively modified. File Locks can come from a variety of sources. If a job is unexpectedly terminated due to a network drop, then it is plausible that the Veeam Agent finished, but never received a terminate command. If the repository is using deduplication, the storage may have too aggressive of a profile active and is locking file(s) as soon as Veeam releases a lock on them.
Once it has been verified that the file being locked is no longer being modified, then it is safe to manually kill any process still maintaining a lock on the file.
Failure to verify that the file is no longer being modified may result in a corrupted file.