In late January, a worm called SQL Slammer shut down a Bank of America ATM network, Continental Airlines' online ticketing system, and an emergency call center in Seattle as well as cutting off Internet access for millions of PC users worldwide. Slammer also revealed a hidden problem in the SQL Server community: Many customers aren't promptly applying service packs and hotfixes.

Slammer was a devastating worm. But it wasn’t the first and won’t be the last virus to hit the Internet. Attackers will continue to find holes in software, and vendors will have to patch those holes as they’re discovered. Still, patches are useless unless customers install them. The same goes for service packs. SQL Server Magazine heard from hundreds of readers who consciously decided not to apply the patch for the buffer-overflow/escalation-of-privilege vulnerability that Slammer took advantage of.

In this interview with Microsoft Vice President of SQL Server Gordon Mangione, SQL Server MVP Brian Moran explores why customers aren’t applying patches, Microsoft’s plans to address these problems, and the future of security for SQL Server.

One of the most common reasons users gave for not applying the original hotfix, the cumulative patch, or SQL Server 2000 Service Pack 3 (SP3) was that their ISVs didn’t support the latest patch or service pack. So in a Catch-22 situation, customers know that a patch exists and that they should apply it, but they also know that doing so might invalidate their ISV service contract or break an application. How does Microsoft currently work with the ISV community to roll out patches and service packs, and how do you plan to improve this process?

We learned through this latest service-pack process that we have to make it much easier for ISVs and customers to upgrade to the latest service packs. Our product team is committed to frictionless installs. ISVs generally test, support, and certify at the service-pack level, not at the individual-fix or cumulative-fix level. So, most customers could install the cumulative hotfix and maintain ISV support while waiting for formal certification of a new service pack.

For our service packs, we’ve also started beta programs and joint-development programs to give ISVs early drops of the code before releasing it to the public. We encourage ISVs to test their applications with the service packs not only to give us feedback about the quality of the service packs but also to pre-certify their applications running against the service packs. We’re also working with ISVs to get playbacks of their applications so that we can test the database with their applications even before we release the code to them.

Another common reason that users gave for not installing SP3 was that they can’t uninstall service packs. Readers said they have time to roll out the service pack and deal with a few minutes of downtime to apply the patch and take it off if something breaks. But they don’t have time to completely rebuild their servers if there’s an issue with the service pack. Does Microsoft plan to enable users to roll back service packs?

Customers tell us they want to be able to uninstall service packs, security patches, and Quick Fix Engineering (QFE) updates. We’re focused in the short term on providing the capability to roll back security fixes. We also absolutely have the goal of allowing rollbacks of service packs, although that’s a much more complicated process and will take longer to implement in the code.

Many customers said they probably would have installed the patch sooner if it had come with an installation program. Manually copying files might seem like a simple task, but it opens the door to manual errors, which would be difficult to trace if an incorrect file was accidentally overwritten. Will Microsoft issue all new security hotfixes with an installer?

The weekend Slammer hit, we re-released patch MS02-061 along with a hotfix installer to make installation easier. The installer will be available with any future security hotfixes.

You and many other Microsoft SQL Server officials are visiting customers who were hit by Slammer. What do your visits and Microsoft’s research say about other reasons why SQL Server customers aren’t applying service packs and patches in a timely manner?

The key issues we’ve heard include difficulty installing patches, lack of time and resources to update systems, uncertainty about which patches are critical, and lack of awareness of unmanaged SQL Server or MSDE systems. And in the specific case of SQL Server 2000 SP3, users noted the lack of time to adequately test SP3; we released it on a Monday, and Slammer hit the following Friday.

What are the differences between a hotfix patch and a service pack? Should users always apply all hotfixes? If not, how do users know for sure which hotfixes to apply? Should they apply all service packs? If a user applied all hotfixes consecutively as Microsoft released them, is that the same as if the user applied a cumulative patch or a service pack—in other words, could the user just skip the cumulative patch or service pack?

The Microsoft Security Response Center issues a bulletin for any product vulnerability that could, in our judgment, result in multiple customers' systems being impacted, no matter how unlikely or limited the impact. However, this approach to identifying vulnerabilities has made it difficult for some customers to identify vulnerabilities that represent especially significant risks. So Microsoft recently adopted a new rating system for security patches to help customers understand the severity.

   Prev. page   [1] 2 3 4 5     next page



You must log on before posting a comment.

If you don't have a username & password, please register now.

Reader Comments

Here's another reason why SQL Server support staffs don't immediately apply service packs. Just because a service pack is available doesn't mean that it is completely tested and error free for your systems. We have a server that we just applied SP3 on last week. Immediately after the upgrade, we found applications that used to run in 2 minutes running over 55 minutes. After running and re-running sp_updatestats, we had no improvement. fter running Update Statistics TABLE with fullscan (a supposed solution if the SP_updatestats doesn't help), we again had no improvement. After studying all other possible causes, including memory settings, we could find no solution. I finally uninstalled SQL Server 2000 with SP3 (sure would be nice to have an uninstall) and then re-installed with SP2--the problem immediately went away, and we had our 2 minute response time. Now, you might say we could tune our applications and add indexes to solve this. Probably true. However, when I upgrade the server to the new and improved service pack, I shouldn't have to retune the performance of software that we have installed at 100+ customer sites. Therefore, you can see why a DBA might be reluctant to start running new service packs when they are announced as available. Beta testing service packs is not a cost-efficient endeavor.

Thanks..Ted Henderson

Ted Henderson- April 28, 2003

The ongoing emphasis by Microsoft on not using mixed mode authentication is particularly frustrating. Microsoft needs to do a better job of motivating 3rd party software vendors to support Windows authentication. NOt only don't they support it now, but when you ask when they will your question is met with silence. They have no plans to move to it. It is these vendors that make the decision for us. We have no discretion in choosing our authentication mode! Kay

Kay Conheady- May 01, 2003

To the question of why DBAs didn’t apply the patch Mr. Mangione responds, “The key will be to make these patches easier for customers to understand and deploy..." But problem is that Microsoft is becoming a victim of its our own “ease of use” success. So I was very disappointed to hear this response.

Lets be honest, “ease of use” is a double edged sword. On one side we have a product that should take care of mundane tasks BUT on the other it can dumb down a shop, lower everyone’s salaries, and ends up causing a downward spiral (dumb people doing dumb things requiring even more “ease of use”) eventually leading customers to dump SQL Server for more robust products (products that require skilled keepers). I have experience seeing this happen. Its to the point where many IT managers consider SQL Server DBAs to be mere babysitters preferring Oracle DBAs for the “real” work. And I was even been told by a MS SQL Server evangelist that for enterprise class shops they would recommend hiring Oracle DBAs.

IT shops are not hiring qualified people for MS products because MS pitches the idea that its products are self-maintaining and requires only semi-skilled labor (read as lower wages/cost). I would really like to see an honest discussion about if this minimally qualified work force is where we want to take things or not. I think if this trend continues the creative people in this field will leave perhaps for other careers or other platforms. And if we don’t want to be replaced by monkeys or robots, having all our salaries lowered to minimum wage, and work in dreadfully boring spaces with dreadfully boring people then how can we improve SQL DBA training/certification? Currently, the training and cert programs are not well respected nor do they adequately prepare one for being a DBA/database developer. BUT I don’t see anyone talking about this issue! At least not in this magazine.

Mike- May 01, 2003