VMware vCenter server experiences performance problems while waiting for tasks to complete

Challenge

Hung tasks, slow response time or disconnects from VMware vCenter server in VMware vSphere client or in any monitoring product which uses VMware vCenter server for gathering data.

Regular disconnects from VMware vCenter server.

Cause

If you consistently have slow response times in the VMware vSphere client while being connected to VMware vCenter server, please check your ESX hosts for resource problems.

A possible scenario is that an ESX host has several third party agents installed that are consuming Service Console memory. Backup and enterprise monitoring agents are common examples. It is also possible that the resource needs of the virtual machines have exceeded the host’s physical capabilities. Although cpu and memory over-allocation is possible with ESX, there is always a limit to the number of guests that can coexist together peacefully. That number depends on guest workload types and individual VM virtual hardware configuration. Multiple VMs with applications that heavily use memory or that require multiple virtual cpus usually do not perform well if contained on the same ESX host.

Performance troubleshooting will ultimately pinpoint the exact cause of the problem, but there are some general changes that can be made first.

Solution

Increasing the amount of RAM assigned to the ESX Server service console is the VMware Knowledge Base article that explains how to increase the Service Console allocated memory (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003501&sliceId=2&docTypeID=DT_KB_1_1&dialogID=64395968&stateId=0%200%2064393864).

From the KB article:
“By default the service console on an ESX Server host is usually not configured to utilize the maximum amount of RAM.” [ ... ] On ESX Server 3.x the amount of RAM is set to 272MB.

The default values are usually sufficient, however there are several instances where VMware recommends to increase the RAM assigned:

When third-party system management agents are installed or crashing on the service console.

When a backup agent is installed on the service console.

When heavy swap file utilization is noticed on the service console

Hopefully you already have a swap partition that will support bumping your Service Console ram up to the 800 mb maximum. If not, try increasing the memory to at least 512 mb and see if you notice any improvement.

Also from the same KB article, Here are the step by step instructions for increasing the Service Console memory:

“To change the amount of RAM assigned to the service console:

1. Log in to VirtualCenter from the Virtual Infrastructure Client with a user that has administrative rights.(Note: If you do not have VirtualCenter, log in directly to the ESX Server host as root.)
2. From the Inventory select the ESX Server host.
3. Click the Configuration tab.
4. Click Memory.
5. Click Properties.
6. On the Memory window enter a value between 256MB and 800MB for the service console parameter.(Note: For troubleshooting purposes, VMware recommends that you increase the service console RAM to 800MB.)
7. Click OK.

Changes do not take effect until the ESX Server is rebooted. For more information on rebooting an ESX Server host, see Rebooting an ESX Server (1003530). http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003530

Checking for resource starvation of the ESX service console is a VMware Knowledge Base article that explains how to use top utility to troubleshoot for (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003496&sliceId=2&docTypeID=DT_KB_1_1&dialogID=64395968&stateId=0%200%2064393864):

High CPU utilization on an ESX Server

High memory utilization on an ESX Server

Slow response when administering an ESX Server