SQL Server Magazine

Editor's Note: Windows NT Magazine is solely responsible for the benchmark results in this article. Send comments or questions about the benchmark results to John Green at jgreen@winntmag.com.

When you sit down at your desk in the morning and prepare for the day's challenges, what's the first thing you do? If you're like most of us, you'll answer, "Check my email." Email has become indispensable for business, and productivity and work routines are disrupted when email systems go down. Every enterprise wants a reliable mail system that supports the mail management and distribution features employees want and that can grow with the company. This month, I review six SMTP mail servers that are suitable for small, midsized, and, in a couple of cases, large companies. I don't review messaging suites such as Microsoft Exchange Server and Lotus Notes but focus instead on products that serve companies with modest requirements. (Messaging suites add workgroup features that many companies don't need, and such products usually support fewer users on a given hardware platform.)

To select six servers for review, I looked at mail servers that have a few basic features and target small to midsized businesses. In addition to the requirement that the mail server must run under Windows NT, each server must support the three basic mail protocols: SMTP, POP3, and Internet Message Access Protocol 4 (IMAP4). For an overview of email protocols, see the sidebar "Electronic Mail Standards," page 128. Each server also must include a list server to support a variety of business communication needs. Although I didn't require it for inclusion in the review, I looked for the ability to authenticate access to user mailboxes with a NT user ID. I also looked for telephone and email technical support.

In this review, I first describe the key feature set for each product and my experience installing and configuring the product. Table 1, page 126, compares the six products' important features, and Table 2, page 127, summarizes product pricing. I used Microsoft Outlook Express 5.0 and QUALCOMM's Eudora Pro 4.1, mail clients that support POP3 and IMAP4 access to the mail server, to test each product. I tested built-in Web servers that support Web access to mail in those products that include this feature. Finally, I stress-tested each server with a workload consisting of SMTP and POP3 transactions to derive a metric that lets you compare the servers' maximum throughput on a given hardware platform.

Mail and Authentication Options
Although all the systems I tested had good feature sets, some features stood out. For example, Ipswitch's IMail Server 5.0 and Gordano's NTMail can both automatically enable email support for NT user accounts, which saves administrative overhead. All six products support alternatives to sending a plain-text password across the network for POP3 and IMAP client access, although Stalker Software's CommuniGate Pro and ISOCOR's N-PLEX Global 4.0 have the most complete support for secure authentication standards.

Documentation
In an ideal world, products work reliably with an intuitive user interface (UI) and need no documentation. But in the real world, documentation often makes the difference between struggle and ease of use. All the products I tested include detailed and usable documentation. Five of the products make their documentation available in a printable electronic format—Portable Document Format (PDF), Microsoft Word, or DynaText. With the exception of CommuniGate Pro, whose documentation is available only in HTML format, the products ship with printed documentation.

Installation
None of the products I tested were difficult to install and configure for basic operation. All use an installation wizard to install software components. NTMail and Rockliffe's MailSite 3.3.1 were the easiest of the group to install because they both require minimal additional configuration after the installation wizard completes. N-PLEX Global's installation wizard installs only the product's Management Center; you must install mail support as an additional installation procedure.

Configuration and Management
The six products that I tested differed significantly from one another in configuration and management. MailSite was the easiest to administer, partly because of the MailSite Console's Explorer-like interface, and partly because of the product's relative lack of tuning options. CommuniGate Pro's Web-based administration tools were also easy to use and offered a useful variety of performance-tuning options. QUALCOMM's Eudora WorldMail Server 2.0 and N-PLEX Global both have an Explorer-like interface that is similar to MailSite's, and WorldMail Server and N-PLEX Global offer a variety of useful configuration and performance-tuning options. IMail's primary administration tool also offers an Explorer-like interface, but you can't use the tool remotely.

Remote Web-based administration is possible with IMail, although some loss of functionality exists in comparison with IMail's GUI tool. NTMail's Web-based administration was a little more cumbersome to use than CommuniGate Pro's but was more complete than IMail's. Except for IMail, each product lets you use the same tool for local and remote administration.

Differences between the products were also apparent in the type of mail accounts each supports. In addition to server-based mailbox accounts, which each product supports, all products except WorldMail Server support mailboxes that use NT user account authentication. CommuniGate Pro, IMail, and NTMail also support mailboxes that rely on an external database; MailSite includes this support at additional cost.

Mail Access
I found no significant differences between the products for POP3 and IMAP access with Outlook Express and Eudora Pro. CommuniGate Pro, IMail, and NTMail also offer Web-based mail access as a standard feature, although I was less than impressed with NTMail in this respect.

List-Server Features
CommuniGate Pro, MailSite, and IMail include full-featured list servers. NTMail includes a limited-use list server, and you can purchase NTMail's companion product, NTList, separately as a standalone list-management server. N-PLEX Global's and WorldMail Server's list servers can't reflect digested lists to members but are otherwise functional products.

Performance Testing
I performed mail-server benchmark testing in partnership with Client Server Solutions. CSS produces Benchmark Factory, an NT-based performance measurement and benchmarking tool that supports a broad spectrum of benchmark tests. (For more information about Benchmark Factory, go to http://www.benchmarkfactory.com.) I tested each of the mail servers with a mix of SMTP and POP3 transactions. Figure 1 combines the test results for all six products in one graph.

I tested each SMTP server package on a Compaq ProLiant 6500 quad-capable server configured with two Pentium Pro 200MHz processors and 256MB of RAM. I installed the programs to a RAID 5 array of seven Seagate Cheetah 9.1GB hard disks connected to a Compaq Smart-2DH Array Controller. Two HP Vectra XU 6/200 Pentium Pro 200MHz computers with 128MB of RAM generated the client workload. A Compaq Netelligent 5506 100Base-T switch was the network's heart.

I used Benchmark Factory's POP3 benchmark to devise a test for this review. For each server, I ran the test at varying user loads—from 100 users to 1000 users in 100-user increments. I generated the test's think time (i.e., how long each simulated user waits before executing the next transaction) randomly by using a negative exponential distribution with a 10-second mean. Because this think time is short, each simulated user generated a much higher workload than even the most active of email users would generate. Each time I repeated the test, a simulated user sent mail randomly to the other simulated users that participated in the test run. I intended this transaction mix to simulate an accelerated heavy-usage pattern for a POP3 user. In such a pattern, the POP3 client periodically connects to the server to check for new mail, download the mail, and delete the messages from the server. The benchmark test consisted of the following mix of transactions:

• 2 percent send mail to 10 recipients with a 10KB message
• 9 percent send mail to 3 recipients with a 10KB message
• 9 percent send mail to 3 recipients with a 10KB message and a 10KB attachment
• 40 percent read all messages, including message header, body, and attachment
• 40 percent delete all messages from the server

Results
Benchmark Factory measures the average number of transactions per second (tps) for each user load tested. As Figure 1 shows, with the exception of IMail, all the servers had comparable throughput for between 100 and 300 simulated users. At the 400-users level, NTMail's performance begins to drop behind the pack. At 400 users and above, a spread in performance is evident, with MailSite topping out at more than 76tps. N-PLEX Global placed second at almost 64tps, and CommuniGate Pro follows closely at 60.3tps.

Five of the six servers let NT user IDs and passwords authenticate access to mailboxes. The testing shows that using NT authentication can have a significant negative impact on overall POP3 mail-server performance. Because WorldMail Server doesn't support using NT accounts for mailbox access, I ran the primary series of tests with each mail server's user ID and password scheme. I ran additional tests in which IMail and MailSite used NT local machine accounts to authenticate mailbox access. IMail's throughput dropped to 58 percent of its throughput rate with native IMail authentication. For MailSite, the throughput cost of using NT account authentication was even greater: 48 percent of the throughput rate I measured with native MailSite authentication.

To help you better understand the test results, and to propose an explanation for the anomalous data point in the MailSite test results, let me explain SMTP mail delivery. SMTP mail delivery to a user mailbox is a two-step process. When a mail message first arrives at the server, the server places the message in a delivery queue. Messages usually travel from this incoming mail queue to their final destination (e.g., a local mailbox or another SMTP server) in a first in/first out (FIFO) sequence. The relative priority that any mail server gives to the SMTP receive operation in relation to the SMTP deliver operation is a function of the server's resource allocation bias and any tuning parameters available in the server. Of the products I tested, only CommuniGate Pro specifically lets administrators specify the maximum number of local delivery threads that mail servers can use. MailSite is the only product I tested that didn't provide a mechanism for performance tuning. All the other products let an administrator specify the maximum number of threads that the server can use. During benchmark testing, I observed that local mail delivery from the SMTP queue into user mailboxes would continue for a time after the end of a particular test run. To adjust for this phenomenon in CommuniGate Pro, I specified 25 local delivery threads. The other tunable products let me specify a number of threads for SMTP protocol use, but I couldn't specifically allocate the threads to local mail delivery.

The tuning bias can affect benchmark results in many ways. In MailSite, I observed that local delivery continued for a significant time following the end of a benchmark test run. MailSite apparently gives an absolute priority to SMTP receive activity and performs local delivery with the leftover system resources. As a result, under very high SMTP receive loads, no local mail delivery would occur. The absence of local mail delivery contributed to efficient SMTP transaction processing during the benchmark test and to efficient POP3 client transaction processing because the mailboxes had a high probability of being empty. In a conversation I had with Rockliffe's technical support department, the technician confirmed that problems with SMTP receipt and delivery processing in MailSite 3.3.1 cause delivery-queue backing under heavy loads. Rockliffe reports that it has resolved this problem in version 3.4.3.

For CommuniGate Pro, tuning to add local delivery threads gave the server a bias away from absolute SMTP receive priority. That is, allocating more resources to local delivery processing takes resources away from receipt processing. A reasonable assumption is that CommuniGate Pro would have scored higher had I configured fewer local delivery threads.

Keep in mind that with a 10-second mean time between transactions, the stress-testing load was at least an order of magnitude heavier than any real user would generate. Although the test generated an unrealistically heavy load, the results illustrate a need for vendors to tune their products to handle those exceptional loads that can sometimes catch a mail-system administrator by surprise.

Recommendations
The testing makes clear that I can't recommend one product for all situations. Despite performance variations, each SMTP server has plenty of throughput capacity for small to midsized organizations. For the small price-conscious company, WorldMail Server is a great value. With support for Lightweight Directory Access Protocol (LDAP) clients, a list server, and plenty of performance, WorldMail Server more than meets the needs of small organizations. Large companies would do well with either CommuniGate Pro or MailSite. MailSite posted the highest maximum throughput of the group, was the easiest to install and configure, and has the lowest price for more than 100 users. CommuniGate Pro offers solid performance and adds a good Web-based mail client, a more feature-rich list server than the other products, and the ability to add multiple-server support for growing organizations.

   Prev. page   [1] 2 3 4 5 6 7     next page