Hal Rosenberg has released a document to the VMware Communities titled “Performance Troubleshooting for VMware vSphere 4”. Hal’s document utilizes a flow chart approach in order to identify specific performance issues. For each problem covered, it includes a discussion of the possible root-causes and solutions. It is a must read for anyone administering a vSphere environment. Thanks Hal. You can find his post and download link here

     We were deploying a new Moss portal for one of our customers and everything was working quite well. The only problem we found was after doing a search, any link we clicked on from the results page would take over a minute to return. We were perplexed that the portal could be performing as well as it was, but this one page would grind to a screeching halt. After numerous Google searches, we finally surrendered and called MS professional services. While on the phone with PS, we performed numerous netmon and fiddler captures trying to diagnose what was causing this behavior to no avail. One of our team members continued to search Google and found a link by Peter Stilgoe that matched the issue that we were seeing. It turns out that federated search was set to return results from an external provider vice the local index. Our Web Front ends are on a network that does not allow internet access and this caused the results page to wait until the timeout was reached (default is 60 seconds) before allowing us to continue. We took a different approach than outlined in Peter’s article. We edited the results page and removed the federated search webparts from the page. Now we are up and running with the same performance across all SharePoint pages. Thanks Bill for finding Peter’s page.

     It’s been a while since I’ve done any sort of O&M, but I have always taken pride in my ability to troubleshoot difficult problems. I was having an issue with one of my Ubuntu VMs. Everything “appeared” to be fine, but in hindsight, I now realize they weren’t. After checking on everything I thought was the problem (CPU usage on the VM, network issues, and anything else I could think of), I was at a loss as to why I had continual PHP timeouts on an apache web-server (yes the one that hosts this blog). I decided, as a last ditch effort, that I would bring my entire infrastructure down and bring it back up cold. It was after I did this that I realized I had been looking in the wrong place for the cause.

     As you can see in the charts, my CPU, Memory and Disk utilization on my ESX server was not only high, but pretty erratic. If I had noticed this before the reboot I could of investigated the cause or at least used top to figure out what was chewing up the CPUs. Now I will never know what was causing the problem, unless of course it returns. If it does I will be better prepared to solve the issue, as I will be armed with all the information. The moral of this story is to keep the troubleshooting to it’s basic level before assuming the problem exists higher up. Approach the problem in a logical manner working from the bottom up. I hope you learn from my mistakes, as I know I sure have