Windows Server 2003 : Planning an IPSec Implementation (part 1) - Evaluating Threats & Introducing IPSec

Evaluating Threats

When you log on to an FTP server on your network, you have to supply a user name and a password to be granted access. The FTP client program you use probably does not display the password on the screen as you type it, but of course the password must be included in the data packets the client sends over the network to the FTP server. Figure 1 shows a screen capture from Microsoft Network Monitor, which is displaying the contents of an FTP packet that the program captured from the network.

In this packet, you can clearly see the password (which is “password”) associated with the user account that the client is supplying to the server. If you are a network administrator, and you use the Administrator account to access the FTP server, someone capturing the packets in this way could learn the Administrator password and possibly wreak havoc on the network.

Not all applications transmit passwords in clear text this way, however. When you log on to Active Directory, for example, the computer transmits your password in encrypted form. This is just an example of how easy it is for unauthorized people to capture and access your data as it is being transmitted. A user running a protocol analyzer such as Network Monitor can capture the packets containing your data files, your e-mail messages, or other confidential communications, and reconstruct the data for their own use.

There are many ways that unauthorized personnel can use this captured data against you, including the following:

• Compromising keys In the same way that captured packets can contain passwords, they can also contain encryption keys. An intruder capturing a key can then decrypt any data using that key. The Public Key Infrastructure (PKI) used on networks running Microsoft Windows is not threatened by this practice, because it uses separate public and private keys for encryption and decryption, and the private keys are never transmitted over the network. However, other encryption systems use a single key to encrypt and decrypt data, and if an intruder captures that key, the entire security system is compromised.

• Spoofing Spoofing is digitally masquerading as another person by using captured IP addresses and other information. By capturing network packets, an intruder can discover an actual user’s IP address, packet sequence numbers, and the other personal information needed to create new packets that to have originated from the actual user’s computer. Using this method, the intruder can send messages in the victim’s name, receive data that was meant for the victim, and even engage in financial or other transactions using the victim’s accounts. Sometimes an attacker will simultaneously initiate a denial-of-service attack on the victim’s computer to prevent the victim from sending any further messages while the attacker assumes the victim’s identity.

  Security Alert

  Even when you use applications that encrypt your passwords for transmission, it is still sometimes possible for intruders to use those passwords by simply pasting the encrypted string into a spoofed message. Even though the intruder doesn’t actually know what the password is, the authenticating system could decrypt it and accept it as genuine.

  
  

• Modifying data When intruders capture data packets from the network, they can not only read the information inside, they can also modify it, then send the packets to the recipient. The packets arriving at the destination therefore might contain information that the true sender did not create, even though the packets appear to be genuine.

• Attacking applications In addition to modifying the data in captured packets, intruders might add their own software to the packets and use the packets to introduce the software into the destination computer. Viruses, worms, and Trojan horses are just some of the dangerous types of code that can infiltrate your network in this way.

Introducing IPSec

IPSec is designed to protect data by digitally signing and encrypting it before transmission. IPSec encrypts the information in IP datagrams by encapsulating it, so that even if the packets are captured, none of the data inside can be read. Using IPSec protects your network against all the threats listed in the previous section.

Because IPSec operates at the network layer, as an extension to the IP protocol, it provides end-to-end encryption, meaning that the source computer encrypts the data, and it is not decrypted until it reaches its final destination. Intermediate systems, such as routers, treat the encrypted part of the packets purely as payload, so they do not have to perform any decryption; they just forward the encrypted payload as is. The routers do not have to possess the keys needed to decrypt the packets, nor do they have to support the IPSec extensions in any way.

There are other protocols besides IPSec that provide network traffic encryption, such as Secure Sockets Layer (SSL), but these are application layer protocols that can encrypt only specific types of traffic. For example, SSL only encrypts communications between Web clients and servers. IPSec can encrypt any traffic that takes the form of IP datagrams, no matter what kind of information is inside them.

IPSec Functions

In addition to encrypting IP datagrams, the IPSec implementation in Windows Server 2003 provides a variety of security functions, including the following:

• Key generation For two computers to communicate over the network using encrypted IP datagrams, both must have access to a shared encryption key. This key enables each computer to encrypt its data and the other computer to decrypt it. However, the key cannot be transmitted over the network without compromising the security of the system. Therefore, computers preparing to communicate with each other using IPSec both use a technique called the Diffie–Hellman algorithm to compute identical encryption keys. The computers publicly exchange information about the calculations that enable them to arrive at the same result, but they do not exchange the keys themselves or information that would enable a third party to calculate the key.

• Cryptographic checksums In addition to encrypting the data transmitted over the network, IPSec uses its cryptographic keys to calculate a checksum for the data in each packet, called a hash message authentication code (HMAC), then transmits it with the data. If anyone modifies the packet while it is in transit, the HMAC calculated by the receiving computer will be different from the one in the packet. This prevents attackers from modifying the information in a packet or adding information to it (such as a virus). IPSec supports two hash functions: HMAC in combination with Message Digest 5 (MD5) and HMAC in combination with Secure Hash Algorithm-1 (SHA1.) HMAC-SHA1 is the more secure function, partly due to SHA1’s longer key length (SHA1 uses a 160-bit key as opposed to the 128-bit key used by MD5). HMAC-MD5 is strong enough for a normal security environment, but HMAC-SHA1 is the better choice for a high-level security environment and it meets the United States government’s security requirements for high-level security.

• Mutual authentication Before two computers can communicate using IPSec, they must authenticate each other to establish a trust relationship. Windows Server 2003 IPSec can use Kerberos, digital certificates, or a preshared key for authentication. Once the computers have authenticated each other, the cryptographic checksum in each packet functions as a digital signature, preventing anyone from spoofing or impersonating one of the computers.

• Replay prevention In some cases, it is possible for attackers to use data from captured packets against you, even when the data in the packets is encrypted. Using traffic analysis, it is possible to determine the function of some encrypted packets. For example, the first few packets that two computers exchange during a secured transaction are likely to be authentication messages. Sometimes, by retransmitting these same packets, still in their encrypted form, attackers can use them to gain access to secured resources. IPSec prevents packet replays from being effective by assigning a sequence number to each packet. An IPSec system will not accept a packet that has an incorrect sequence number.

• IP packet filtering IPSec includes its own independent packet filtering mechanism that enables you to prevent denial-of-service attacks by blocking specific types of traffic using IP addresses, protocols, ports, or any combination of the three.