SQL Server Magazine

Quest Software's Benchmark Factory 2.5 is a highly scalable workload generator for database, Web, mail, and file server applications. You can use the product to load-test an application for performance tuning, expose and diagnose application reliability problems, and determine the maximum throughput that a hardware configuration will support.

You license Benchmark Factory by application driver. Twelve drivers are available—File Server, Web Server, DB2, Informix, Microsoft SQL Server, Oracle 7, Oracle 8, Sybase, ODBC, Messaging API (MAPI), IMAP4, and POP3. The Professional edition includes the Benchmark Foundation Classes, which are benchmark components that you can use with Microsoft Visual C++ to write custom benchmarks. The license key that Benchmark Factory includes unlocks the features you have licensed. I used a key that unlocked all drivers and benchmarks but not the Benchmark Foundation Classes.

Installing Benchmark Factory
The minimum hardware requirement for Benchmark Factory Agent machines and the Visual Control Center (VCC) machine is a Pentium 90 processor, 32MB of RAM, and Windows 2000, Windows NT, or Windows 9x. Quest Software recommends a Pentium II processor running Win2K or NT 4.0, with 32MB of RAM for Agent systems and 64MB of RAM for the VCC machine. Benchmark Factory supports TCP/IP, the IPX and SPX protocols, and any network that supports Winsock.

I installed Benchmark Factory from a CD-ROM to a 450MHz Pentium II Compaq running NT 4.0 Service Pack 5 (SP5). After sharing the Netinstall directory that the installation process created, I ran Setup from the directory to install the Agent to a Win2K Professional-equipped client on the network.

Benchmark Factory has three primary components: the VCC, which runs benchmark tests and manages results; a runtime version of Sybase's Adaptive Server Anywhere, which stores benchmark results; and the Benchmark Factory Agent, which runs on one or more computers and generates a workload that simulates one or more users per Agent. You need to install Microsoft Internet Explorer (IE) 5.0 to operate the VCC. If you want Benchmark Factory to write test results to a Microsoft Excel spreadsheet, you need to install Excel on the VCC computer. The installation routine doesn't check for the presence of IE 5.0 or Excel, so you need to install those programs first to avoid problems with Benchmark Factory.

Benchmark Factory's installation routine is easy to follow and contains several standard installation scenarios and a Custom option. The installation process lets you choose locations in which to install Benchmark Factory and the results repository that Adaptive Server Anywhere accesses. If you need to collect detailed transaction statistics during testing, the repository can quickly grow to several hundred megabytes or more. Benchmark Factory uses ODBC to write to the repository, so you can use another database, such as SQL Server, in place of Adaptive Server Anywhere.

Getting Started
Benchmark Factory supports multiuser testing, which uses networked Agents that the VCC controls, and single-user testing, which runs directly from the VCC. To use Benchmark Factory, you create a project in which you choose a benchmark type (e.g., TPC-C or File Server). Each project contains at least one predefined test and is the framework for customizing how you want each test to run. Each test has one or more transactions, which are discrete test units (e.g., reading an email message, viewing a Web page). Transactions are static or dynamic. Static transactions reside in the benchmark's .dll file and have fixed functionality. You can define dynamic transactions (e.g., executing a SQL query), and you can use one or more static or dynamic transactions to create a multiuser or single-user test. A Benchmark Factory Profile points to the server you are testing, which Benchmark Factory refers to as the System Under Test (SUT).

When you run a multiuser test, Benchmark Factory passes the test parameters to one or more Agents, which run test transactions repeatedly during a time period that you specify. In contrast, the VCC runs single-user tests without using an Agent. The VCC runs a single-user workload a fixed number of times.

To run a test, place it in the Run Queue and schedule the test to run immediately or at a later time. Jobs in the Run Queue run one at a time in the order you place them in the queue. While a job is running, you can use the VCC to view the job's progress. When a job finishes, you can use the VCC to view the results or have Benchmark Factory export the results to an Excel spreadsheet.

I ran a TPC-C workload, a transaction-processing test that is one of Benchmark Factory's standard database benchmarks. The test has no connection to the Transaction Processing Performance Council's (TPC's) TPC-C test.

I used Benchmark Factory's New Project Wizard to base a project on the TPC-C workload. Then, I used the New Profile Wizard to give Benchmark Factory the information that the program needs to run the test against a specific server. Information that the program required for the TPC-C test included the server name, the database driver for the test to use (in this case, SQL Server), the database for the test to use, and the User ID (UID) and password for the Agents to use to authenticate their access to the database.

After the New Profile Wizard was completed, the New Project Wizard regained control and displayed the Scale Selection screen. Scale sets the test size. For database testing, Scale specifies the database size in rows and controls the number of rows that Benchmark Factory generates to fill the test database. Scale also controls the range of key values that the test queries.

The New Project Wizard checked the server for the necessary database objects and created jobs to delete, recreate, load, and index the tables. Figure 1 shows the VCC as these jobs were running. The Run Queue is in the screen's upper-right pane, a list of transactions and benchmarks appears in the screen's upper-left corner, and a Results summary appears in the pane at the bottom of the screen. After the database creation jobs finished, I had completed the basic preparation necessary to run a test.

Letting the New Project Wizard create database objects that you need for testing is a very handy feature of Benchmark Factory. However, when you create large databases, the wizard's database creation jobs can take a long time to run. To speed this process, Benchmark Factory has a Data Generator tool that creates each table's data in a flat-file format. Programs such as SQL Server's bulk copy program (bcp) can use the file to bulk-load the tables at high speed.

Creating a Benchmark Test
For each driver, Benchmark Factory includes at least one predefined test. The TPC-C workload contains the TPC-C Transaction Mix test. To customize how the test would run, I right-clicked the test in the VCC and selected Properties. The Properties screen of every benchmark test has six tabs—General, Transactions, User Load, Latency, Timing, and Options. The General tab lets you type a test description and shows when you last modified the test.

The Transactions tab lets you specify what percentage of the test will consist of each type of transaction. For example, if you simulate the workload of a company that processes most payments between 4 p.m. and midnight each day, you might set Benchmark Factory's simulated workload to include 30 percent payment-processing transactions during that shift. During simulation of another shift, you might set payment-processing transactions at 2 percent of the workload.

The User Load tab specifies how test iterations will run and how many virtual users will take part in each test iteration. By running multiple test iterations with different numbers of load-generating users, you can create a throughput curve for the application. This curve shows the maximum throughput (transactions per second—tps) the server will process, the transaction response time at each load level, and the degree to which performance decreases under increasing loads.

Latency dictates how long each virtual user waits between transactions, in effect determining how hard each virtual user will work the SUT. The Latency tab contains several options for setting fixed- or random-length delays before and after each transaction is executed.

On the Timing tab, you set the duration of each of the four benchmark iteration intervals. The test begins with Quiet Time, which is a waiting time before each iteration. Quiet Time lets server activity from the previous iteration stop before the next iteration begins. During Ramp Up, each virtual user begins running transactions that the test prescribes, but Benchmark Factory doesn't report those transactions as a part of the test results. During the Execution Time interval, Benchmark Factory measures and reports results from all transactions. During Ramp Down, virtual users continue to begin new transactions, but Benchmark Factory doesn't record the transactions. Ramp Down ensures that the SUT's workload continues unabated while the transactions that began during Execution Time are completed.

   Prev. page   [1] 2     next page