Installing and Configuring the Windows Server 2003-based ISA Server 2000 SMTP Filter and Message Screener on the ISA Server Firewall

 

 

The ISA Server 2000 SMTP Application Filter and Message Screener components can be used to protect your Exchange Server from spam and malicious content attached to SMTP mail. These two components work together to protect your organization:

 

 

The SMTP Application Filter is installed on ISA Server firewalls by default. However, ISA Server does not automatically enable the SMTP Application Filter. After you enable this filter, it will examine all incoming SMTP messages that pass through a Server Publishing Rule. The filter protects the published SMTP server from buffer overflow attacks.

 

 

The SMTP Message Screener extends the protection provided by the SMTP Application Filter by delving deep into the SMTP application layer content. The SMTP Message Screener examines the SMTP messages for source domain and address, keywords in the message subject and body, and attachments. If an SMTP message contains an unapproved source address, attachment type or text string, then it can be deleted, forwarded or quarantined.

 

*       Note:
The SMTP Message Screener can evaluate the entire content of a plain text message. The SMTP Message Screener provides limited screening support for HTML messages.

 

The SMTP Application Filter always runs on the ISA Server firewall itself; you can’t install it on another machine that acts as an SMTP relay. However, you do have the option of installing the SMTP Message Screener on the ISA Server firewall itself, or on an SMTP server on the internal network located behind the ISA Server firewall. In both circumstance, the SMTP Message Screener communicates with the SMTP Application Filter to carry out its tasks.

 

In this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document we will cover the procedures required to create a co-located SMTP relay, SMTP Message Screener and SMTP Message Filter. These procedures include:

 

 

*       Note:
This ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document is designed to be used in conjunction with ISA Server 2000 Exchange Server 2000/2003 Deployment Kit documents Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay and Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay. Please read these article first for details on how to create an anonymous and authenticating SMTP relay on the ISA Server firewall computer.

 

 

Install Windows Server 2003 on the ISA Server Firewall Computer

 

The computer that will become the ISA Server 2000 firewall/SMTP relay must meet the following minimum requirements:

 

 

The ISA Server firewall and Web caching components work very well on very modest hardware. This is true even when the SMTP filter is enabled and protecting the published co-located SMTP server. However, the SMTP Message Screener can be very processor intensive. This is why I recommend that you use a processor with a minimum of rating of 1.5 MHz.

 

 

Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer

 

The SMTP Message Screener requires the IIS SMTP service. You will need to install the SMTP service because Windows Server 2003 does not install IIS by default. Perform the following steps to install the IIS 6.0 SMTP service:

 

  1. Click Start, point to Control Panel and click the Add or Remove Programs command (figure 1).

 

Figure 1

 


  1. Click the Add/Remove Windows Components button on the left side of the Add or Remove Programs window (figure 2).

 

Figure 2

 

 


  1. In the Windows Components dialog box, click on the Application Server entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 3).

 

Figure 3

 

 


  1. In the Application Server dialog box, click on the Internet Information Services entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 4).

 

Figure 4

 

 


  1. On the Internet Information Services (IIS) page, put a checkmark in the SMTP Service checkbox. The Internet Information Services Manager checkbox will be automatically selected for you. Click OK.

 

Figure 5

 

 


  1. Click OK in the Application Server dialog box (figure 6).

 

Figure 6

 

 


  1. Click Next on the Windows Components page (figure 7).

 

Figure 7

 

 


  1. The Windows Components Wizard installs the IIS SMTP service (figure 8).

 

Figure 8

 

 

  1. Click Finish when the Wizard completes.

 

 

Disable SMTP Service Socket Pooling

 

By default, the SMTP service listens on all IP addresses on all adapters installed on the ISA Server firewall. You must disable socket pooling to prevent the SMTP service from listening on all IP addresses on all adapters. Socket pooling prevents Server Publishing Rules from working correctly. However, you do not need to disable socket pooling if you plan to use packet filters to make the SMTP relay co-located on the ISA Server firewall available to external users.

 

Its good practice to disable socket pooling for any IIS service installed on the ISA Server firewall. Perform the following steps to disable socket pooling for the IIS 6.0 SMTP service:

 

1.       Click Start and then click the Command Prompt link. In the Command Prompt window, switch to the Inetpub\AdminScripts folder. Then type in the following command and press ENTER (figure 9):

 

Adsutil.vbs set /smtpsvc/1/DisableSocketPooling 1

 

Figure 9

 

2.       If the SMTP service is installed and you entered the command correctly, you should see what appears in figure 10.

 

Figure 10

 

 

3.       Close the command prompt window.

 

At this point the SMTP service continues to listen on all IP addresses on all interfaces. You must configure the service to listen on specific IP addresses to limit the server to listening on a subset of addresses.

 

 

Configure the IIS 6.0 SMTP Service Relay Properties and Remote Domains

 

The next step is to configure the ISA Server firewall as an SMTP relay. There are two ways you can do this:

 

 

The advantage of using an SMTP Server Publishing Rule is that the Server Publishing Rule exposes the incoming SMTP messages to the buffer overflow protection of the SMTP Application Filter. The disadvantage of using an SMTP Server Publishing Rule is that you will not be able to protect communications with the SMTP relay using TLS encryption. The reason is that the SMTP Application Filter does not support TLS encryption.

 

The advantage of using an SMTP Server packet filter is that you can use TLS encryption to encrypt communications between the SMTP client and server. The disadvantage of using the SMTP packet filter to allow inbound connections to the SMTP relay is that the packet filter does not expose the SMTP messages to the SMTP Application filter’s buffer overflow protect.

 

*       Note:
The SMTP Message Screener is able to inspect SMTP messages moving through the SMTP relay co-located on the ISA Server firewall in both the SMTP Server Publishing and SMTP packet filter scenarios.

 

Please review ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay for detailed information on how to configure the SMTP relay parameters for the SMTP server co-located on the ISA Server firewall.

 

Please review ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay for detailed information on how to configure the SMTP relay properties for the SMTP server co-located on this ISA Server firewall. This article also discusses how to configure an authenticating SMTP relay for your remote uses who require a secure SMTP relay.

 

 

Install ISA Server 2000 onto the Windows Server 2003 Firewall Computer

 

The next step after installing and configuring the SMTP service on the ISA Server firewall is to install ISA Server 2000 with the SMTP Filter and Message Screener on to the Windows Server 2003 computer.

 

Please review the ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing ISA Server 2000 on Windows Server 2003 for instructions on how to install ISA Server 2000 on Windows Server 2003. This document provides instructions on how to install all ISA Server 2000 components onto the Server. If you need to remove components, you can remove them later.

 

 

Configuring Packet Filters or Server Publishing Rules on the ISA Server Firewall

 

You need to configure the ISA Server firewall to allow incoming connections to the co-located SMTP relay. As mentioned in earlier in this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document, you can use either Server Publishing Rules or packet filters to allow these incoming SMTP connections. There are advantages and disadvantages to each of these approaches.

 

Please review ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay for detailed information on how to configure the packet filters and Server Publishing Rules required to allow inbound access to an non-authenticating SMTP relay.

 

Please review ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay for detailed information on how to configure the packet filters and Server Publishing Rules required to allow inbound access to an authenticating SMTP relay.

 

 

 

 

 

Enable and Configure the ISA Server 2000 SMTP Filter and Message Screener

 

You must perform the following actions to realize the full benefit of the SMTP relay co-located on the ISA Server firewall:

 

·         Enable the SMTP Application Filter

·         Confirm that the SMTP Message Screener is installed

·         Install the SMTP Message Screener if it is not installed

·         Configure the SMTP Filter and SMTP Message Screener Properties

 

Enable the SMTP Application Filter

 

You must enable the SMTP Application Filter before the ISA Server firewall begins to examine SMTP packets. Perform the following steps to enable the SMTP Application Filter:

 

  1. Open the ISA Management console, expand the Servers and Arrays node and then expand your server name. Expand the Extensions node and click on the Application Filters node (figure 11). Right click on the SMTP Filter entry in the right pane of the console and click Enable.

 

Figure 11

 

  1. Select the Save the changes and restart the service(s) option in the ISA Server Warning dialog box. (figure 12). This will cause the Firewall service to automatically restart. Click OK.

 

Figure 12

 

 


  1. The icon for the SMTP Filter Application Filter shows that the filter is no longer disabled (figure 13).

 

Figure 13

 

 

The SMTP Application Filter is now enabled. At this point any incoming SMTP messages coming inbound through an SMTP Server Publishing Rule will be exposed to the buffer overflow protection provided by the SMTP Application Filter.

 

 

Confirm that the SMTP Message Screener is Installed

 

You may have installed the ISA Server 2000 software before you considered the possibility of using the firewall as a filtering SMTP relay. In that case, you may have not installed the SMTP Message Screener component.

 

Perform the following steps to determine if you have configured the SMTP Message Screener on the ISA Server firewall:

 

1.       Click Start and click on the Run command. Type Regedit in the Open text box and click OK. The Windows Server 2003 Registry Editor opens (figure 14).

 

Figure 14

 

 

2.       Use the Registry Editor’s Find feature to find the following Registry entry:

 

HKEY_CLASSES_ROOT\CLSID\{4F2AC0A5-300F-4DE9-821F-4D5706DC5B32}

 

Click the Edit menu and click the Find command. Enter the above string between the brackets into the Find what text box (figure 15). Click the Find Next button.

 

Figure 15

 

 

3.       If the SMTP Message Screener is installed on the ISA Server firewall/SMTP relay computer, the Registry entry will be found, as seen in figure 16. If the Registry entry is not found, then the SMTP Message Screener is not installed and you will need to install it before you can filter SMTP messages based on address, attachments and keywords.

 

Figure 16

 

4.       Close the Registry Editor.

 

 

Install the SMTP Message Screener if it is Not Installed

 

You can use the Add or Remove Programs Control Panel applet to install the SMTP Message Screener if it was not installed with the ISA Server 2000 software. Perform the following steps to install the SMTP Message Screener after the initial ISA Server 2000 software installation:

 

1.       Click Start and point to Control Panel. Click on Add or Remove Programs. In the Add or Remove Programs window (figure 17), click on the Change button for the Microsoft Internet Security and Acceleration Server entry.

 

Figure 17

 

 

2.       The setup routine will search for currently installed components. On the Microsoft ISA Server Setup dialog box, click the Add/Remove button (figure 18).

 

Figure 18

 

 

3.       On the Microsoft ISA Server – Customer Installation page, click on the Add-in services entry (do not put a checkmark in its checkbox!) and click the Change Option button (figure 19).

 

Figure 19

 

 

4.       Put a checkmark in the Message Screener checkbox on the Microsoft ISA Server – Add-in services dialog box (figure 20). Click OK.

 

Figure 20

 

 

5.       Click Continue on the Microsoft ISA Server – Customer Installation page (figure 21).

 

Figure 21

 

 

6.       The Message Screener is installed. Click OK on the Microsoft ISA Server Setup dialog box informing you that the setup was completed successfully (figure 22).

 

Figure 22

 

 

7.       Close the Add or Remove Programs window.

 

 

Configure the SMTP Filter and SMTP Message Screener Properties

 

The SMTP filter and SMTP Message Screener are configured using the same interface in the SMTP Filter Properties dialog box. However, keep in mind that the SMTP filter and SMTP Message Screener are two distinct entities. It is possible to use the SMTP filter and not use the SMTP Message Screener and it is possible to use the SMTP Message Screener and not use the SMTP filter.

 

For example, you can use the SMTP Filter without using the SMTP Message Screener by not installing the SMTP Message Screener. The SMTP filter will then protect an published SMTP server against buffer overflow attacks, including the SMTP server co-located on the ISA Server firewall.

 

You can use the SMTP Message Screener and not the SMTP Filter by using a packet filter to allow inbound access to the SMTP relay co-located on the ISA Server firewall. The SMTP Message Screener examines the incoming SMTP messages when they are accepted by the IIS 6.0 SMTP service. The SMTP Filter does not protect against buffer overflow attack in this scenario because incoming SMTP messages accepted via a packet filter are not exposed to the SMTP filter.

 

*       Note:
You must install ISA Server 2000 Feature Pack 1 if you want to support authenticating with a Server Published SMTP server. Pre-Feature Pack 1 versions of the SMTP Filter did not support the AUTH command and would not allow users to authenticate against a Server Published SMTP server. You can authenticate with a Server Published SMTP server after installing Feature Pack 1. Under no circumstances can you use TLS encryption with a Server Published SMTP server when the Message Screener is enabled.

 

Perform the following steps to configure the SMTP filter and SMTP Message Screener components:

 

1.       Open the ISA Management console, expand the Servers and Arrays node and expand your server name. Expand the Extensions node and click on the Application Filters node. Right click on the SMTP Filter entry in the right pane of the console and click on the Properties command (figure 23).

 

Figure 23

 

 

2.       The General tab is the first thing you see when the SMTP Filter Properties dialog box opens (figure 24). You can enable or disable the filter by adding or removing the checkmark in the Enable this filter checkbox. Click on the Keywords tab.

 

Figure 24

 

 

3.       You can enter a prioritized list of keyword to filter on the Keywords tab. The SMTP Message Screener mediates the keyword filtering function. The SMTP filter does not examine SMTP messages for keyword. Click the Add button to add a keyword (figure 25).

 

Figure 25

 

 

4.       Confirm the there is a checkmark in the Enable keyword rule checkbox (figure 26). Type in a keyword that you want the SMTP Message Screener to look for in the Keyword text box. Note that the SMTP Message Screener does not search for whole words; the filter only looks at text strings.

 

Select one of the following options in the Apply action if keyword is found in frame:

 

Message header or body

If the keyword is found in either the message header or message body, then the Action you configure for the rule will be applied.

Message header

If the keyword is found in the header (subject line), then the Action you configure for the rule will be applied.

Message body

If the keyword is found in the body of the message, then the Action you configure for the rule will be applied

 

Click the down arrow for the Action drop down list box. You have the following options:

 

Delete message

The SMTP message is deleted without being saved or informing anyone that it has been deleted.

Hold Message

The SMTP message is held in the BADMAIL directory in the SMTP service’s folder hierarchy. You can view components of the held message, but the message is not saved in a format that you can easily forward to the recipient.

Forward message to

The SMTP message is forwarded to an email address you configure in this rule. Each rule can have a different email address that the message is forwarded to.

 

Click OK on the Mail Keyword Rule dialog box.

 

Figure 26

 

 

5.       The keyword rule appears in the keywords list on the Keywords tab (figure 27). Click on the Users / Domains tab.

 

Figure 27

 

 

6.       You can configure the SMTP Message Screener to block messages based on the sender’s user account or email domain on the Users / Domains tab. Enter a user email account in the Sender’s name text box and click Add. The senders email address appears in the Rejected Sender’s list. Type in a email domain in the Domain name text box and click Add. The email domain appears in the Rejected Domains list.

 

Email messages processed by the SMTP Message Screener matching email addresses or email domains found in these lists are deleted. These messages are not stored anywhere on the server, nor are they forwarded to any user or administrator. If a message from a rejected sender or rejected domain also contains a keyword that matches a keyword rule, and that keyword rule is configured to hold the message, the message will not be held because it is rejected before the keyword search begins.

 

Click Apply and then click OK. Click on the Attachments tab.

 

Figure 28

 

 

7.       You can block messages with certain types of attachments on the Attachments tab (figure 29). Click Add to add an attachment rule.

 

Figure 29

 

 

8.       Confirm that there is a checkmark in the Enable attachment rule checkbox on the Mail Attachment Rule dialog box (figure 30). You have three options in the Apply action to messages containing attachments with one of these properties frame:

 

Attachment name

Select this option and type in a name for the attachment, including file name and file extension, in the text box next to this option. Use this option if you don’t want to block all attachments with a particular file extension, but you do want to block a specific file name. For example, you do not want to block all .zip files, but you do want to block a file named exploit.zip.

 

Attachment extension

It is more common to block all files with a specific file extension. For example, if you want to block all files with the exe file extension, select this option and then type in either exe or .exe in the text box to the right of this option.

 

Attachment size limit (in bytes)

You can also block attachments based on their size. Select this option and type in the size of the file extension you want to block.

 

Click the down arrow for the Action drop down list box. You have the following options:

 

Delete message

The SMTP message is deleted without being saved or informing anyone that it has been deleted.

Hold Message

The SMTP message is held in the BADMAIL directory in the SMTP service’s folder hierarchy. You can view components of the held message, but the message is not saved in a format that you can easily forward to the recipient.

Forward message to

The SMTP message is forwarded to an email address you configure in this rule. Each rule can have a different email address that the message is forwarded to.

 

In this example we’ll select the Forward message to option so that you can see how to enter the forwarding address.

 

Figure 30

 

 

9.       When you select the Forward message to option, a text box appears that allows you to enter an email address to forward the message to. However, the server must be able to resolve the address of the mail domain of this user.

 

For example, in figure 31we have entered the email address smtpsecurityadmin@internal.net. The ISA Server firewall must be able to access an MX record for the internal.net domain. The ISA Server firewall forwards the message to the SMTP server responsible for the internal.net mail based on the information in the MX record.

 

In this example the firewall is configured with a DNS server address of a DNS server on the internal network that can resolve both internal and external network names. The message is forwarded to the internal address of the Exchange server. You must configure a split DNS infrastructure if the internal.net domain is available to both internal and external users.

 

*       Note:
Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring DNS to Support Exchange Server Publishing for information on how to create a split DNS to support SMTP server publishing.

 

Click OK in the Mail Attachment Rule dialog box. Click on the SMTP Commands tab.

 

Figure 31

 

 

10.   The settings on the SMTP Commands tab are mediated by the SMTP filter component. The SMTP Message Screener does not evaluate SMTP commands and it does not protect against buffer overflow conditions. The commands in the list are limited to a pre-defined length. If an incoming SMTP connection sends a command that exceeds the length allowed, then the connection is dropped. In addition, if a command is sent over the SMTP channel is not on this list, it is dropped.

 

Click the Add button to add an SMTP command to the list (figure 32).

 

Figure 32