Securing Exchange Server 2010 with ISA Server : Managing and Controlling Simple Mail Transfer Protocol (SMTP) Traffic

3/29/2011 11:22:26 AM

The Simple Mail Transfer Protocol (SMTP) is the second most commonly used protocol on the Internet, after the web HTTP protocol. It is ubiquitously used as an email transport mechanism on the Internet, and has become a critical tool for online collaboration.

Unfortunately, SMTP is also one of the most abused protocols on the Internet. Unsolicited email (spam), phishing attacks, and email-borne viruses all take advantage of the open, unauthenticated nature of SMTP, and it has become a necessity for organizations to control and monitor SMTP traffic entering and leaving the network.

ISA Server 2006’s application-layer inspection capabilities allow for a high degree of SMTP filtering and attack detection. By default, ISA supports the protocol as part of standard rules and policies.

ISA Server 2006 is an ideal candidate for deployment in an environment with an Exchange Server 2010 Edge Transport server, as it can scan the traffic bound for that server for irregularities and potential exploits before delivering messages to the Exchange Server environment.

Note

The next version of ISA Server, Threat Management Gateway (TMG) will support installing an Exchange Edge Transport Server directly on the TMG server, improving the overall security and removing any unnecessary hop.

Publishing the SMTP Server for Inbound Mail Access

The first step toward securing SMTP traffic is to create a server publishing rule that protects the internal SMTP server from direct external access. This can be done through an SMTP server publishing rule. In this scenario, an Exchange Server 2010 Edge Transport server in the DMZ of the ISA firewall is published:

1.	From the ISA console, select the Firewall Policy node from the console tree.
2.	In the tasks pane, click the Publish Mail Servers link.
3.	Enter a descriptive name for the rule (for example, SMTP Inbound to Edge Transport Server), and click Next.
4.	Select Server-to-Server Communication: SMTP, NTTP, and click Next.
5.	In the dialog box shown in Figure 1, choose SMTP, and click Next. Figure 1. Setting up an inbound SMTP publishing rule.
6.	Enter the internal IP address of the Edge Transport server, and click Next.
7.	Select to which networks the ISA server will listen by checking the boxes next to them, and click Next. This is usually the external network. ISA then listens to this network for SMTP requests sent to it and relays them to the Edge Transport server.
8.	Click Finish, click Apply, and then click OK.

Creating an SMTP Access Rule in ISA Server 2006

ISA can be easily configured to allow SMTP traffic and to scan that traffic as it passes through the ISA firewall. This can be configured with a simple access rule, set up as follows:

1.	On the ISA server, open the ISA console, and choose the Firewall Policy node from the console tree.
2.	In the tasks pane, click the Create Access Rule link.
3.	Enter a descriptive name, such as Allow SMTP Outbound from Edge Transport Server, and click Next.
4.	Select Allow from the rule action list, and click Next.
5.	Under This Rule Applies To, select Specified Protocols.
6.	Click Add under Protocols, then drill down and choose Common Protocols, SMTP. Click Add.
7.	Click Close and then click Next.
8.	Under Access Rule Sources, click Add.
9.	Drill down to Networks, and select Perimeter (or the network where the Edge Transport server is), click Add, and then click Close. Note that it is often better practice to restrict this to the specific IP address of the Edge Transport server, rather than to a whole network.
10.	Under Access Rule Destinations, click Add.
11.	Under Networks, select External, click Add, and then click Close.
12.	Click Next, click Next, and then click Finish.
13.	Click Apply at the top of the details pane, and then click OK to confirm.

Customizing the SMTP Filter

After SMTP rules have been set up to allow the traffic to flow through the SMTP Screener, the ISA SMTP filter can be customized to block specific types of SMTP commands and content. To access the SMTP filter settings on the ISA server, do the following:

1.	From the ISA console, click the Add-ins node in the console tree.
2.	Under Application Filters in the details pane, double-click on SMTP Filter.
3.	Examine and configure the settings in the SMTP Filter Properties dialog box, some of which are shown in Figure 2. Figure 2. Configuring SMTP filter settings.