Get a Handle on Windows Performance Analysis

As an administrator, you probably know firsthand that manually analyzing Performance Monitor logs is a time-consuming process, requiring both a deep-level knowledge of performance counters and familiarity with Windows architecture. The Microsoft Global Escalation Services team gets thousands of calls from customers requesting help in analyzing Performance Monitor logs either to help solve an existing problem or determine whether their servers are optimized for the best possible performance. I’ll tell you about some tools that Microsoft support uses for system performance analysis, which can help you more easily and effectively analyze your systems’ performance and troubleshoot performance issues.

Two-Part Analysis
Performance analysis comprises two main tasks:

• Data collection: Collecting Performance Monitor logs locally or remotely can be challenging as the number of systems to be monitored increases. Another challenge is determining which set of performance counters to collect and at what intervals.
• Data analysis: Once the data is collected, analyzing it correctly is yet another challenge, and it’s in the analysis portion of the process that Microsoft support receives the most calls for assistance.

Let’s look at tools you can use to handle each performance-analysis task.

Data-Collection Tools
Performance Monitor Wizard (perfwiz.exe) walks you through creating local and remote Performance Monitor logs. PerfWiz simplifies the process of gathering Performance Monitor logs on Windows Server 2003, Windows XP (x86 only), and Windows 2000 systems by configuring the correct counters to collect and suggesting the most appropriate sample intervals and log-file sizes. You can download PerfWiz here.

Logman.exe is a built-in Windows command-line tool that manages and schedules performance counter collections on local and remote systems and runs on Windows Server 2008, Windows Vista (x86 and x64), Windows 2003, and XP. You can find information about Logman syntax and examples of usage here.

Here’s an example of how you’d use logman.exe from the command line to generate a Performance Monitor log called High-CPU-Perf-Log capturing data at 5-second intervals:

Logman.exe create counter High-CPU-Perf-Log
  -f bincirc -v mmddhhmm -max 250
  -c "\LogicalDisk(*)\*" "\Memory\*"
    "\Network Interface(*)\*" "\Paging File(*)\*"
    "\PhysicalDisk(*)\*" "\Process(*)\*"
    "\Redirector\*" "\Server\*" "\System\*"
    "\Thread(*)\*" -si 00:00:05

(Be sure to type the entire command without the line breaks shown in the example here.)

Executing the logman.exe command generates a .blg file. To view this file, open Performance Monitor and navigate to the Data Collector Sets folder, as Figure 1 shows.

When you’re running Performance Monitor to try to pinpoint the source of a problem on a system, I recommend you reboot the server before starting a Performance Monitor log and let the log run from reboot until the condition you’re tracking occurs (e.g., system hang, sluggishness). By allowing the Performance Monitor log to collect data until the problem returns, you’re ensuring that the log captures all relevant data about your problem. Gathering as much data as possible in the log will improve the reliability of the analysis. Rebooting first, then starting the Performance Monitor log helps in identifying trouble spots because you can see the resource consumption (memory, CPU, and disk) from the very beginning all the way through the problem period.

You should regularly collect baseline performance data on your most essential servers. Doing so could save you a lot of pain and effort in proving or disproving an issue exists. Often at Microsoft support, customers ask us whether a certain performance statistic is “bad” or “good.” Although Microsoft provides performance guidelines, it’s difficult to know what’s typical for your environment unless you have baseline performance data. Creating baselines also helps you to focus on the problem at hand when there are other problems on the system. Certain performance statistics can be falsely blamed as the culprit for a new issue, but having a baseline from before the new problem occurred will help keep the focus off irrelevant statistics. (You can find Microsoft guidelines for gathering baseline data here.)

Data Analysis
A tool that Microsoft support relies on to analyze Performance Monitor logs is the Performance Analysis of Logs (PAL) Tool. Clint Huffman, a Microsoft senior premier field engineer, wrote the 6,000-line VBScript tool, which is free and open source. PAL lets administrators easily analyze Performance Monitor logs without requiring them to be experts in performance counters or Windows architecture.

PAL contains a wizard-based UI that asks specific information about the system, which PAL passes as arguments to the VBScript program. PAL picks up where other log analyzers leave off, such as taking into account whether the system is 64-bit or 32-bit, whether the /3GB switch is used, and how much physical memory is installed—all variables that affect system performance. PAL uses these variables along with known thresholds, which were determined by engineers with years of experience, to determine the analysis that’s displayed. PAL provides a chronological order of alerts, so that you can correlate your system’s performance to any problems that you noticed at specific times.