About LoadSim Data
As I mentioned earlier, LoadSim takes about an hour to reach steady state. If you throw away the first hour (time 00:00 through 00:59, also referred to as hour 0) and use the second through fourth hours (time 01:00 through 03:59), the scores are more realistic. LSLOG, the LoadSim log analysis program, makes this process easy. It provides options for preparing the data in the .log files. Otherwise, you'd have to analyze the .log file by hand.
Even if you follow this advice, the score will have some margin of error. Typically, the longer you let LoadSim run, the more accurate the score is. But if you are considering very long runs (more than 12 hours) to solve all your problems, you need to know that as LoadSim runs for long periods, conditions can degrade. For example, if you let a test run for days, you will see fluctuations in the response-time trends. Longer is not necessarily better when it comes to LoadSim runs; take the first 4 to 5 hours of data to get a good idea of how the system will run.
Identical runs don't always produce identical scores, but they are usually within 100ms. Often, they are within 50ms. The theoretical minimum score (depending on the Exchange Server hardware) is about 150ms. Any score less than 150 is meaningless.
Comparing Results from LoadSim 4.0 and LoadSim 5.0
I don't recommend that you compare the results obtained from LoadSim 4.0 with LoadSim 5.0. If you perform identical runs on identical LoadSim clients on identical Exchange servers, the scores produced by LoadSim 4.0 and LoadSim 5.0 will probably be different. Usually, the LoadSim 5.0 scores will be lower.
This difference in score does not invalidate any of your tests with LoadSim 4.0, because you're looking at response time trends, not absolute response time. Rather, changes in the LoadSim 5.0 engine make it more efficient and more accurate in the load it imparts on Exchange. Compare LoadSim 4.0 scores to other LoadSim 4.0 scores and LoadSim 5.0 scores to other LoadSim 5.0 scores.
Gathering Useful Data
You can gather useful data with LoadSim in different ways. I'll talk about two methods: using scores from multiple clients and using a dedicated monitor client.
You can run multiple, identically configured LoadSim clients to ensure that each LoadSim client is producing accurate scores. You can then average the scores for accuracy.
Using the example of testing 300 users and based on the guidelines in part 2 of this series (February 1998), you'll need three LoadSim clients. Each machine is a 100MHz Pentium with at least 48MB of RAM and a fast disk. Each LoadSim client can easily handle 100 Medium users, and the score reflected is reliable because the LoadSim client is not bottlenecked. Run LSLOG, and take the average score of the LoadSim clients to produce the final score for the run. If the tests have been run correctly, each score will be within 50 to 100 points of the average.
This technique is probably the best, but it requires a lot of hardware resources, especially if you intend to simulate many clients. Keeping track of all the .log files as the number of LoadSim clients increases is difficult. However, the scores this technique produces are probably as accurate as possible.
Another technique for gathering accurate LoadSim scores is to dedicate a LoadSim client as a score machine. The key is to ensure that the dedicated LoadSim client is underloaded. This machine is always assured of having a fair score regardless of whether the other LoadSim clients are overloaded.
Although a heavily loaded LoadSim client might not produce an accurate score, it still produces an accurate load on the server. You probably can't simulate 500 Medium users on one LoadSim client, but you can squeeze more users out of a LoadSim client if you don't care about the score the client produces.
For example, in the scenario with 300 Medium users, simulate 250 of the 300 users on two slightly less powerful machines--for example, 66MHz Pentium machines with at least 32MB of RAM and a fast disk. You don't have to configure the machines the same way. Simulate 125 users for each LoadSim client rather than 100. Then, with a separate LoadSim client--for example, a 100MHz Pentium with 64MB of RAM and a fast disk--simulate the remaining 50 users. This machine is the dedicated monitor client. Use the score from this monitor client, and disregard the scores from the other two machines.
Using a dedicated monitor client is helpful if you don't have enough hardware to configure multiple identical high-performance LoadSim clients. The LoadSim clients that produce the bulk of the load don't have to be identical, and you need only one high-performance machine to get the score from. You won't get the benefit of averaging scores, but scores compared from identical runs should be within 100 points of each other.
Analyzing Data
After you've gathered your LoadSim data, you will need to analyze it to see what was going on inside Exchange Server. Let me offer some suggestions for analyzing the data.
Plot a users-per-server graph. The most useful piece of information that you can glean from this simulation exercise is the number of users a server supports. You can use the techniques I outlined above to plot a graph with this information.
If you take the sample scenario one step further, you can see how much headroom the server has. You already know the example Exchange server yields a score of 270 for 300 Medium users. But how does it handle up to 800 Medium users? Reconfigure the test for 100 users, and repeat the process to get another data point. Repeat the test for 200 users, 400 users, and so on, until you test 800 users. Then plot the data points. The resulting graph might look like Graph 1.
You can get useful information at a glance from a users-per-server graph. In Graph 1, the current hardware on the Exchange server takes you to 600 users before it starts encroaching on the subsecond response time requirement. This information will help you anticipate the need for a server upgrade. And you have concrete data to characterize the performance capabilities of your server for this particular load.
Correlate scores with Perfmon data. The data points for a users-per-server graph take time to generate, but the resulting graph is an excellent picture of how the server performs. To augment this picture, I recommend gathering Perfmon data during each LoadSim run. But, do not run the Perfmon application on the Exchange server. Run Perfmon from another machine and collect the metrics from the Exchange server remotely. I recommend collecting data at an interval of 10 seconds or longer to keep the overhead on the server to a minimum. Remember, part of the goal is to disturb the system under test as little as possible.
Observe the Perfmon data from the LoadSim run to see what's going on in your server. I recommend you capture all Perfmon objects except Thread because of the overhead that collecting this object will impose on the server. You'll have all relevant counters at your disposal. Look at objects such as CPU utilization, paging, available memory, Exchange message queues, and disk utilization.
You can easily view a graph of the data with Perfmon, or you can export the data into Excel for detailed analysis. You can correlate the scores LoadSim produces with the Exchange server's Perfmon data from each run. Very quickly, you'll begin to spot trends for which subsystems in the server are beginning to get used up. At that point, you have excellent insight into what's going on with Exchange. Then you're well on your way to mastering the performance of your server.
Stay consistent. When you're conducting performance tests, stay consistent. If you select a method of gathering data, producing scores, or running tests, stay with it. Consistency is the most important aspect of getting meaningful performance data. Any variables you introduce into the process can create havoc and produce puzzling results. Remember, performance is not an exact science; some subjectivity is involved. If test results do not make sense, run the test again. Investing time to be sure of your results is better than proceeding on a false premise. If the results look good and they're repeatable, you are on the right track.
LoadSim Complete
This series covered a lot of ground. You learned about the philosophy behind LoadSim, how to use LoadSim, and how to collect and analyze data in a useful fashion. LoadSim veterans learned what's new in LoadSim 5.0. And I offered tips and advice about how to proceed with your own performance testing.
LoadSim is a powerful tool, but with that power comes a degree of complexity. If you are willing to invest the time and effort, the results can be useful.
Now you are ready to set up LoadSim and start running simulations. For information about where to find LoadSim, current releases, and protocol support, see the sidebar, "Updated Information on LoadSim." If you're looking for additional resources, refer to Microsoft's white paper that covers LoadSim and the concept of users per server, "MS Exchange Performance: Concurrent Users Per Server." This white paper was available with the May 1996 edition of Microsoft TechNet, and you can find it at http://www.microsoft.com/syspro/technet/boes/bo/mailexch/exch/technote/upswpfnl.htm. Another Microsoft white paper from Tech Ed '96, "How Many Users Can I Put On a Single Exchange Server?" is available from MSDN. It covers general Exchange Server performance issues. You can also refer to the Microsoft Knowledge Base article Q155417, "XADM: LoadSim, Microsoft Exchange Server Load Simulation Tool," which was revised April 29, 1997 (http://support.microsoft.com/support/kb/articles/q155/4/17.asp). Finally, LoadSim 5.5 comes with a LoadSim User Guide that explains concepts and methodology for using LoadSim 5.5.
End of Article
Prev. page
1
[2]
next page -->
