Hal Rosenberg has released a document to the VMware Communities titled “Performance Troubleshooting for VMware vSphere 4”. Hal’s document utilizes a flow chart approach in order to identify specific performance issues. For each problem covered, it includes a discussion of the possible root-causes and solutions. It is a must read for anyone administering a vSphere environment. Thanks Hal. You can find his post and download link here

     My team was deploying a new SharePoint portal for a customer and were seeing some strange behavior from some of our VMs. Our environment consisted of 3 BL685 servers running ESX and 9 VMs ( I know this is overkill, but our customer was very wary of the portal performance in a virtual environment, and our hope was to introduce a platform that could lead them into further virtualization and server consolidation). We were receiving some rather “interesting” errors while adding domain users to the local administrator group of some of the machines. While doing a “check name”, AD would return the account name along with some non-printable characters ( blocks, smiley face, etc…). We were also having issues copying files to the VM through UNC. Some would copy, others would not stating “source no longer available”. We were not seeing this behavior across all the VMs, which was disturbing as they were all deployed from the same template. We did manage to narrow it down to guests running on just one of the hosts.

     My initial thought was the possibility of bad memory installed on this host. After running some memory tests and the eventual replacement, our errors continued. We then replaced the host with another blade and swapped the hard drives so we wouldn’t need to reinstall ESX. After powering up the host and migrating a couple of VMs back, we were astonished that the issue still remained. It was then that we reinstalled ESX as we thought that was the only thing left.

   After reinstalling ESX, it was then that we found some additional problems. We added the host to the cluster and were attempting to configure HA. This would fail at random intervals with often different errors including, path not found, SSL errors, and others. It was at this point my attention turned to the network. This server was configured with 2 NICs with all network trunked and vlan tagged. I disabled one of the NICs and all traffic continued to work. I switched over to the remaining NIC and everything came to a halt. Finally, it looked like we were narrowing down the cause. When the blade was swapped out, the NICs were moved from the original. We swapped out the NIC and everything worked as advertised.

Although the initial errors did not point to the network, after all other troubleshooting did not result in success the NICs were the only thing left.

      Some how in my late night troubleshooting, I managed to delete the virtual disk (vmdk) file associated with one of my VMs. I was greeted with a very unpleasant error message attempting to power on my VM. Of course I had to try and power it on a couple of times (maybe it would fix itself…)

Looking at the storage, I still had the *-flat.vmdk file, which is the actual disk, but I did not have the configuration file. I decided to manually recreate the file and see if I could save my VM from eminent destruction. You can see the contents of the file below:

# Disk DescriptorFile
version=1
CID=24a5eeb6
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 41943040 VMFS "Wordpress-flat.vmdk"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.toolsVersion = "7299"
ddb.uuid = "60 00 C2 92 45 db 09 47-c5 9d a9 22 a6 cd 24 e9"
ddb.geometry.cylinders = "2610"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

I created a new file called wordpress.vmdk, ensuring it was pointing to the correct -flat.vmdk file. I was then able to edit my virtual machine and assign the new disk and power up the VM.

     It’s been a while since I’ve done any sort of O&M, but I have always taken pride in my ability to troubleshoot difficult problems. I was having an issue with one of my Ubuntu VMs. Everything “appeared” to be fine, but in hindsight, I now realize they weren’t. After checking on everything I thought was the problem (CPU usage on the VM, network issues, and anything else I could think of), I was at a loss as to why I had continual PHP timeouts on an apache web-server (yes the one that hosts this blog). I decided, as a last ditch effort, that I would bring my entire infrastructure down and bring it back up cold. It was after I did this that I realized I had been looking in the wrong place for the cause.

     As you can see in the charts, my CPU, Memory and Disk utilization on my ESX server was not only high, but pretty erratic. If I had noticed this before the reboot I could of investigated the cause or at least used top to figure out what was chewing up the CPUs. Now I will never know what was causing the problem, unless of course it returns. If it does I will be better prepared to solve the issue, as I will be armed with all the information. The moral of this story is to keep the troubleshooting to it’s basic level before assuming the problem exists higher up. Approach the problem in a logical manner working from the bottom up. I hope you learn from my mistakes, as I know I sure have