How to forcibly stop jobs that are stuck in ‘stopping’ status

Challenge

This KB documents the process for forcibly terminating all jobs for a given Veeam Backup & Replication server. The steps outlined in this KB will terminate all jobs.

Note: Please understand that some jobs may take some time to stop, please allow up to 60 minutes for a jobs to stop on their own before forcibly terminating them. If you are not sure if a job is stuck, please create a Veeam Support case and provide logs for the job.

Solution

The problem may occur because a process or task that the job is waiting for has not been completed. Such a task can be snapshot removal. Therefore, as the first step it is recommended to check the vSphere Client to see if there is a snapshot removal process pending/working on the vSphere side.

The following steps will forcibly terminate ALL jobs/restores.

1. Close the Veeam Backup & Replication console.
2. Stop all services that begin with the word Veeam.
3. Open the Task Manager on the Veeam Server and kill all VeeamAgent.exe processes.
   Note: Some VeeamAgent.exe process will be located on Source Proxies and Windows Repositories that are not the Veeam Server.
4. Wait 5-10 minutes for the tasks to timeout and fail. 
5. Remove snapshots from VM(s) that are part of the stuck jobs. (If they have not been already removed).
6. Start the services that were stopped in step 2.

***Repeat Steps 2 –4 on each Proxy and Repository used by the Job.

Note. If you use the Virtual Appliance (HOTADD) mode, before removing the snapshots make sure there are no stuck disks on the Veeam Backup server or one of the backup proxies. Otherwise, the snapshots can be orphaned. https://vee.am/kb1775 

If the jobs are still listed as “Stopping” after performing the steps outlined above please proceed through the following steps.
For information on how to apply SQL scripts please review https://vee.am/kb1443.

1. Backup the Veeam database. (http://vee.am/kb1471)
2. Run the following query against the VeeamBackup database:

   SET QUOTED_IDENTIFIER ON
   UPDATE [Backup.Model.JobSessions]
   SET [state] = '-1'
   WHERE [state] != '-1'

Note: If running this under SQL Server Management Studio, typically the statement SET QUOTED_IDENTIFIER ON is not needed. However, if using SQLCMD, it is almost always required, else the query will fail.

Once run, you should see a specified number of rows reported affected by the query. Allow up to 60 second for the UI to update to reflect this.