In late January, a worm called SQL Slammer shut down a Bank of America ATM network, Continental Airlines' online ticketing system, and an emergency call center in Seattle as well as cutting off Internet access for millions of PC users worldwide. Slammer also revealed a hidden problem in the SQL Server community: Many customers aren't promptly applying service packs and hotfixes.
Slammer was a devastating worm. But it wasn’t the first and won’t be the last virus to hit the Internet. Attackers will continue to find holes in software, and vendors will have to patch those holes as they’re discovered. Still, patches are useless unless customers install them. The same goes for service packs. SQL Server Magazine heard from hundreds of readers who consciously decided not to apply the patch for the buffer-overflow/escalation-of-privilege vulnerability that Slammer took advantage of.
In this interview with Microsoft Vice President of SQL Server Gordon Mangione, SQL Server MVP Brian Moran explores why customers aren’t applying patches, Microsoft’s plans to address these problems, and the future of security for SQL Server.
One of the most common reasons users gave for not applying the original hotfix, the cumulative patch, or SQL Server 2000 Service Pack 3 (SP3) was that their ISVs didn’t support the latest patch or service pack. So in a Catch-22 situation, customers know that a patch exists and that they should apply it, but they also know that doing so might invalidate their ISV service contract or break an application. How does Microsoft currently work with the ISV community to roll out patches and service packs, and how do you plan to improve this process?
We learned through this latest service-pack process that we have to make it much easier for ISVs and customers to upgrade to the latest service packs. Our product team is committed to frictionless installs. ISVs generally test, support, and certify at the service-pack level, not at the individual-fix or cumulative-fix level. So, most customers could install the cumulative hotfix and maintain ISV support while waiting for formal certification of a new service pack.
For our service packs, we’ve also started beta programs and joint-development programs to give ISVs early drops of the code before releasing it to the public. We encourage ISVs to test their applications with the service packs not only to give us feedback about the quality of the service packs but also to pre-certify their applications running against the service packs. We’re also working with ISVs to get playbacks of their applications so that we can test the database with their applications even before we release the code to them.
Another common reason that users gave for not installing SP3 was that they can’t uninstall service packs. Readers said they have time to roll out the service pack and deal with a few minutes of downtime to apply the patch and take it off if something breaks. But they don’t have time to completely rebuild their servers if there’s an issue with the service pack. Does Microsoft plan to enable users to roll back service packs?
Customers tell us they want to be able to uninstall service packs, security patches, and Quick Fix Engineering (QFE) updates. We’re focused in the short term on providing the capability to roll back security fixes. We also absolutely have the goal of allowing rollbacks of service packs, although that’s a much more complicated process and will take longer to implement in the code.
Many customers said they probably would have installed the patch sooner if it had come with an installation program. Manually copying files might seem like a simple task, but it opens the door to manual errors, which would be difficult to trace if an incorrect file was accidentally overwritten. Will Microsoft issue all new security hotfixes with an installer?
The weekend Slammer hit, we re-released patch MS02-061 along with a hotfix installer to make installation easier. The installer will be available with any future security hotfixes.
You and many other Microsoft SQL Server officials are visiting customers who were hit by Slammer. What do your visits and Microsoft’s research say about other reasons why SQL Server customers aren’t applying service packs and patches in a timely manner?
The key issues we’ve heard include difficulty installing patches, lack of time and resources to update systems, uncertainty about which patches are critical, and lack of awareness of unmanaged SQL Server or MSDE systems. And in the specific case of SQL Server 2000 SP3, users noted the lack of time to adequately test SP3; we released it on a Monday, and Slammer hit the following Friday.
What are the differences between a hotfix patch and a service pack? Should users always apply all hotfixes? If not, how do users know for sure which hotfixes to apply? Should they apply all service packs? If a user applied all hotfixes consecutively as Microsoft released them, is that the same as if the user applied a cumulative patch or a service pack—in other words, could the user just skip the cumulative patch or service pack?
The Microsoft Security Response Center issues a bulletin for any product vulnerability that could, in our judgment, result in multiple customers' systems being impacted, no matter how unlikely or limited the impact. However, this approach to identifying vulnerabilities has made it difficult for some customers to identify vulnerabilities that represent especially significant risks. So Microsoft recently adopted a new rating system for security patches to help customers understand the severity.