SQL Server Magazine

You've implemented IPsec to protect traffic on your organization's LAN, and although you've followed all the technical documents carefully, you aren't convinced that the traffic on your network is actually protected from eavesdroppers. How can you reassure yourself that IPsec is truly encrypting your computers' network traffic?

Let's take a look at how to ensure that everything's running smoothly, then—should you discover problems in the process—look at the larger picture of troubleshooting IPsec-related authentication problems. In the process, I'll show you some built-in Windows Server 2003 tools that you can use to diagnose IPsec problems if you do find that your traffic isn't encrypted. (Note, however, that the techniques I discuss in this article don't necessarily apply to special IPsec applications, such as IPsec cards and VPN-based IPsec.)

1. Checking Whether IPsec Is Working
Even if you've successfully deployed IPsec, a lot can go wrong with it. When you suffer a loss of general network connectivity on a Windows Server 2003 or Windows XP computer, the OS makes sure that you know you have a problem. Unfortunately, there's no similar method to keep you aware of IPsec status. Depending on the IPsec deployment, when you have a problem, either you'll lose all network connectivity or—more likely and more insidious— network communications will continue working but with no encryption. Imagine your shock when you find that your network traffic is completely unsecure when you believed yourself to be protected.

The quickest way to check whether IPsec is working is to use Network Monitor to capture packets to and from your computer and check whether they're encrypted. Network Monitor ships with Windows 2003, and you can install it by opening the Control Panel Add or Remove Programs applet and accessing Add/Remove Windows Components. After you install Network Monitor, you can start a packet capture by selecting Start from the Capture menu. Then, to create some network activity and generate some data, perform a common activity such as browsing a shared folder on another computer. Finally, select Stop and View from the Capture menu.

Figure 1 shows the results of two packet captures. In the left capture, performed when a computer isn't configured to use IPsec, you see a variety of protocols listed in the Protocol column. In the right capture, performed when a computer is configured to use IPsec, you see only one protocol listed: Encapsulating Security Payload (ESP). When a Windows 2003 computer is configured to use default IPsec policies, the only protocols that will be present in a packet capture will be ESP and Internet Control Message Protocol (ICMP). ICMP will be present because the default IPsec policies permit ICMP traffic. So, if you're seeing a variety of protocols when you capture network traffic, you can reasonably assume that IPsec isn't functioning properly.

Almost all IPsec problems occur because of authentication difficulties during the Internet Key Exchange (IKE) phase of authentication. When two computers attempt to form a security association (SA), they go through a process in which they authenticate each other's identity. IKE is the algorithm that negotiates the formation of the SA. Identity authentication occurs through either a preshared key, a digital certificate, or Kerberos. The default Windows 2003 IPsec policies use Kerberos. In the large majority of cases, trouble-shooting IPsec means trouble-shooting the authentication process.

2. Restarting IPsec Services
Once you're sure that IPsec isn't working, you should first try restarting IPsec services. Restarting IPsec services will clear the IKE negotiation state completely. This strategy often restores functionality when IPsec becomes nonfunctional after significant changes in policy. Two advantages of this solution are that it works without requiring a reboot of the server and it takes only a few moments to implement. You can restart IPsec services from the command prompt on a Windows 2003 computer by issuing the following Net commands:

net stop policyagent 
net start policyagent 

Now, perform the network packet capture test again, or run one of the Netsh command tests that I talk about later.

3. Diagnosing IKE Problems in the Event Log
If restarting IPsec services doesn't solve the problem, you can move on to examining the security event log. The acts of creating and deleting SAs are audited as network logon events. If you enable success and failure auditing for the Audit Logon Events policy, these events will be written to the log. When everything is working properly, the success codes that appear are 541, 542, and 543. However, this article discusses what you should do when IPsec isn't working, so in our case, be on the lookout for the failure codes that Table 1 shows. Figure 2 shows failure event 547.

One problem with the logging of SA events is that the events can quickly congest your log. If you typically audit logon events but don't want your security event log filled with IKE entries, you can disable IKE auditing by creating the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Audit\DisableIKE Audits subkey and setting its value to 1.

4. Using Shared-Key Authentication
Because it relies on a shared string rather than more complex methods such as Kerberos or certificates, shared-key authentication can provide a fallback method for getting IPsec to work. Of course, if you're already using shared-key authentication and are having problems, you should verify that the pre-shared keys on each computer actually match. IPsec won't work if one of your pre-shared keys contains a typo.

If you can get IPsec working with shared-key authentication, you'll know that the Kerberos or certificate-authentication mechanisms require more investigation. To shift to sharedkey authentication from Kerberos on the default IPsec policies, perform the following steps:

1. Disable any Active Directory (AD) IPsec policies that might apply.
2. Go to Start, Run, and enter gpedit.msc to begin editing the local policy.
3. Navigate to the Computer Configuration\Windows Settings\Security Settings\IP Security Policies node of the Group Policy Object Editor.
4. Right-click the policy that's causing problems, and select Properties.
5. On the Rules tab, select All IP Traffic in the Filter List and click Edit.
6. In the Edit Rule Properties dialog box, select the Authentication Methods tab and click Add to add a new authentication method.
7. Select the Use this string radio button and enter a pre-shared key. Click OK.
8. Use the Move Up button on the Authentication Methods tab to move the Preshared Key method to the top of the list.

These instructions assume that you haven't heavily customized your IPsec policies. In general, you should always create a new policy rather than edit an existing one. However, in this case, you're merely adding an authentication method, which you can demote later after you resolve the problem. Changing to shared-key authentication significantly simplifies the authentication process, so it's likely to get IPsec working on your network.

   Prev. page   [1] 2     next page