August 21, 2008 03:21 PM

Can the Transaction Log Tell Us What Happened?

Rating: (11)
SQL Server Magazine
InstantDoc ID #100076

In all my years of teaching SQL Server classes and answering questions on public forums, one of the most commonly asked questions has always been “How can I see the information in a database’s transaction log?” Most people asking this question seem to want this information to be able to recover data that was lost due to the infamous "user error." Some people seem to get upset, or even irate, when told that Microsoft doesn't provide a facility for viewing the contents of the log, as if it's their unalienable right to be able to see this information. In my opinion, the log contents are akin to the source code, and buying a license to use the product should in no way guarantee that you'll be able to see exactly how SQL Server does what it does. The log format and its contents are proprietary information; part of the reason for not making this information freely available is that it would expose too much information about the way SQL Server actually works.

If you, or your users, perform an action, you should know what you did. You can prevent unwarranted actions, such as running DELETE operations that remove too many rows, by using triggers or by allowing DELETEs only through carefully coded and tested stored procedures. If you let your users do things they shouldn’t, you’ll need to clean up your own mess. You can also set up custom auditing mechanisms to get the information you want get from the log by using SQL Trace. SQL Server 2008 provides an even richer array of auditing capabilities.

The transaction log is intended for SQL Server’s internal use, such as when performing a rollback operation or restoring data changes that you've captured in a log backup. In addition, the log lets SQL Server protect your database from being corrupted, which could happen if there were a system failure in the middle of a data modification operation. Think of what might happen if the data was updated, but SQL Server hadn't yet updated all the index pointers before the failure occurred. If there were no transaction log recording what changes were being made, SQL Server wouldn't realize upon system restart that the index structures were inconsistent and corrupt. The log lets SQL Server run recovery on system restarts and undo any incomplete operations.

Although Microsoft doesn’t provide you with the ability to read the transaction log, there are third-party products that provide this functionality. The vendors developing these tools based their work on an undocumented command, DBCC LOG, which was initially added to SQL Server to help support providers track down strange system behavior. Back when I worked for product support at Sybase, we used DBCC LOG regularly, and in fact the online Sybase documentation (at http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.39996_1250/html/svrtsg/svrtsg102.htm) still includes this command.

The first commercial log reader tool, from Lumigent, was actually acquired from a small consulting company that was run by a Sybase consultant. The consultant had written a log reader tool based on his knowledge of DBCC LOG and used this tool in his consulting business. Lumigent acquired the tool and made it commercially available. Other vendors began offering log reader tools, and SQL Server Magazine published a comparative review of the Lumigent and Red Gate Software log reader products in the article "When SQL Server Tool Vendors Compete, DBAs Win," at http://www.sqlmag.com/Articles/ArticleID/93174/93174.html.

Looking at the article now, it's not obvious which SQL Server version is being discussed. The structure of the log records and the exact details that are returned with the DBCC LOG command changed substantially between SQL Server 2000 and SQL Server 2005, so any vendor that wrote a tool and examined the results of DBCC LOG would have to completely rewrite their product for SQL Server 2005. Throughout the review, no SQL Server version is mentioned. Finally, at the end of the review, a third log reader tool from ApexSQL is mentioned with the comment that it supports SQL Server 2005, which leads us to assume that Lumigent and Red Gate Software's tools don't support SQL Server 2005. And none of the vendors have announced support for SQL Server 2008. 

So what can you do? You can search the Internet for references to using DBCC LOG and the function that wraps around it: fn_dblog. You can try using the output from one of those commands to get an idea of what has been done to your data. However, your best course of action is to plan ahead, either by preventing unwanted changes before they happen or by defining a trace to your own specifications that you can analyze when needed.

Add a Comment

FYI, the Red Gate 'SQL Log Rescue' product specifically tells you that it only works with SQL Server 2000 when you try to run it against a newer version.

Dan8/25/2008 4:17:02 PM


Most of us manage just fine without a log reader tool. I don't think it should be top of the agenda, but it would be a very welcome addition to SQL Server.

I had a play about with the Lumigent tool a little while ago. I think I remember that it was able to roll back individual transactions in the log file - this could potentially be a lifesaver.

DBAs benefit when 3rd party vendors write tools for SQL Server. We still want to see as many features as possible squeezed into the RTM though. Despite popular belief, I don't think MS is trying to take over the world - I'm sure they balance the demands of it's users without setting out to alienate 3rd party vendors. Adding a feature like this would be a major blow to a company like Lumigent and could possibly make other vendors wary about developing apps to enhance SQL Server.

On a similar note - I wonder how the new intellisense in SQL 2008 will affect RedGate? They are probably pleased about the lack of intellisense support for SQL2000/2005 in SSMS 2008. Intellisense is a popular feature I think MS would need to include it in SQL Server at some point. I guess it probably didn't come as that much of a surprise. Also, they seem to have a number of other tools that I can't see been included in SQL Server any time soon...

David8/23/2008 5:06:06 PM


Wow... I have never gotten so many comments so quickly about one of my articles. I must really have touched a nerve here!

There is a difference between the actual data rows referenced by the log, and the log format. It's the log format, and giving people full details about what is in the log, that is propriatary information. There is nothing specifically bad about giving people that information. However, calling me names because I don't stand up on a soapbox and DEMAND that MS add this functionality seems a little extreme. There are plenty of other things MS could do with the product and providing a log reader tool is way down on the list.

Yes I realize it is important to some people, but there are many other ways to get this information through tracing etc. If the developer resources are limited at MS, I would much prefer they spend their time on more important stuff. MS knows it's important that people have this information, that's why they added a great deal of additional tracing capabilities in SQL Server 2008.

Also, keep in mind that a log reader tool wouldn't help you debug problems with logic, or with bad reports due to faulty SELECTs. If your WHERE clause was written badly, a log reader tool could tell you which rows were affected, but not WHY. You'd need a tracing tool for that. Vogelm's comment that a log reader tool would help troubleshoot bad queries from 3rd party apps is not true; you need to see the statements for that, not just the affected data.

I do appreciate kbreneman's comment that the real problem is one of perception. MS should make clear that the transaction log is not an audit tool; if you want auditting, you need to set it up on your own, because you're the only one who knows what's important for you to capture.

Thanks for all the feedback!
Kalen

KALEN8/23/2008 3:31:06 PM


It is disappointing to see a highly regarded professional like you being mediocre. You need to demand more from a tool like SQL Server and expect it to rise up and provide a log reader. Oracle has done this long ago and is a great diagnostic/audit tool. It is not rocket science. Microsoft has already opened up full .NET source code. So I don't see much substance in the claim that somebody will misuse the proprietary log format; well if they do, kudos to them - I hope their product will have a log reader ;)

Ravindra8/21/2008 9:21:27 PM


Dear Kalen,
We expect more from you.
We do not want to open the Log file using a binary editor and read everything. We want a tool to read the basic data and show us what has happened. Atleast they should disclose some information on LOG, may be from DBCC LOG or fn_dblog()

If Lumigent and RedGate can provide the data, why can't Micorsoft. Is it MS way of supporting the clients. I see this as a move for earning revenue through Product Support.

Preethiviraj8/21/2008 2:25:30 PM


Berating people for demanding Microsoft’s “proprietary information” was the wrong way to start this article. A user being suspected of doing something they shouldn’t have is a fact of life and a manager wanting to know exactly who did what and when is legitimate. To me the problem is the perception that the Transaction Log is also an Audit Log. For someone with this perception it is understandably incomprehensible that there would be a “log” which they are not “allowed” to read. I suppose Microsoft could do better making it clear SQL Server DOES NOT COME WITH an audit log feature and that if you want one it must be third-party. But is it right to expect companies to talk about the features their products *don’t* have?

KEN8/21/2008 11:37:02 AM


Unfortunately, there are many times that we have no control over the application and/or database code - like with 3rd party applications whose databases we support. A number of these packages are poorly written and use in-line T-SQL, etc. Being able to read the log would help greatly in troubleshooting issues with these applications.

Maria8/21/2008 9:46:32 AM


"The log format and its contents are proprietary information; part of the reason for not making this information freely available is that it would expose too much information about the way SQL Server actually works."

Please... that comment makes no sense to me. It's like saying we won't provide a way for you to access the contents of your database because that would expose the proproetary information of the DBMS.

A tool that would allow you to view the contents of the log file would make perfect sense and would help a great deal in determining the source of the database failure.

STEVE8/21/2008 8:53:02 AM


I'd have to respectfully disagree. Oracle has had LogMiner since 8i, and SQL DBA's have had...third party, partially supported products. Oracle apparently doesn't feel they're opening up their proprietary transaction logs, so I don't understand why Microsoft would. They'd only be providing a tool to see the transactions, not to take apart the log itself. During those times that you're investigating missing/corrupted data that isn't audited, and maybe should have been, a product like LogMiner can provide that bit of information so your company can rest easier. If folks have the right to modify the data, but aren't volunteering (or don't realize) that they missed a where clause in their update, or are using a tool they don't realize is corrupting data (and it happens sporadically) it's a heck of lot easier to find with a log related tool.

Rick8/21/2008 8:24:54 AM


You must log on before posting a comment.

Are you a new visitor? Register Here

Rename Virtual Server (Cluster)

I am going to rename a clustered instance (default).1. Change SQL Network Name2. Take offline3. Bring online4. Flush DNS, Cache5. Test failover6. Rebu...222-96209

GOOGLE LINKS
SPONSORED LINKS
FEATURED LINKS