Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay

 

In the ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay you learned how to configure an SMTP relay on the ISA Server computer. That SMTP relay can accept incoming SMTP messages destined for your email domains and relay these messages to the Exchange Server on your internal network.

 

While that SMTP relay configuration works well when you want to allow Internet SMTP servers to forward mail to your Exchange Server, this configuration does not allow external users to send SMTP messages to any mail domain. The problem with allowing the SMTP relay to relay SMTP mail to all domains is that spammers can use the SMTP relay on the ISA Server firewall to forward spam to any domain on the Internet.

 

The solution to this problem is to create an authenticating SMTP relay that requires that users authenticate before the SMTP relay will relay mail. The authentication requirement prevents spammers from hijacking your SMTP server to forward spam email. Your users can use authenticate to the SMTP relay and send mail to your own email domains, or any other domain on the Internet.

 

Advantages of creating an authenticating SMTP relay include:

 

·         External users that do not log into a local ISP can use the authenticating SMTP relay to send SMTP messages

 

Many external users connect to the Internet via a wired or wireless link that does not require logging onto a local ISP. These links can be found in hotels, restaurants and airports. The service provider does not provide your users an SMTP server address. This can be a problem for users who use POP3/SMTP or IMAP4/SMTP clients. These users will be able to read their email but won’t be able to respond to it if they cannot access an SMTP server. Your authenticating SMTP server allows them to send and receive email.

 

·         The authenticating SMTP relay can be configured to force TLS security on SMTP connections

 

Most SMTP servers do not require any type of authentication. Almost all ISPs allow “on network” hosts send SMTP messages and do not allow SMTP relay for off network users. Your users will be on networks outside of your administrative control and you have no idea what level of security is applied to the network. Malicious types may be running network analyzers in an attempt to capture user passwords.

 

You can configure your authenticating SMTP server to require TLS encryption. This protects the user credentials and the data moving between the SMTP client and SMTP server. The secure connection prevents people listening to activity on the wire from stealing user passwords and content contained within email messages.

 

·         Even for users who log onto a local ISP that provides an SMTP server, you can force these users to use the authenticating SMTP relay so that sensitive corporate information is not passed through the Internet “in the clear

 

You may have users to connect to the Internet via a local ISP that provides them with an SMTP server address. You may wish to force your email clients to use your authenticating SMTP server to protect user credentials, and more importantly, the data contained in the SMTP messages. You can not be sure of the level of security of any network between the client and the authenticating SMTP server. The best course of action may be to require users to connect to your authenticating SMTP server and use TLS to protect the data.

 

You need to carry out the following procedures to creating your authenticating SMTP relay:

 

·         Install Windows Server 2003 on the machine that will be the ISA Server firewall/SMTP relay

·         Install a Certificate Server on your network or obtain a Web site certificate from a third party

·         Install the Internet Information Services (IIS) 6.0 SMTP services on the ISA Server firewall/SMTP relay computer

·         Disable Socket Pooling on the ISA Server firewall/SMTP relay

·         Create a second virtual SMTP server on the ISA Server firewall/SMTP relay

·         Request and install a Web site certificate that can be used to create and force the TLS connection between SMTP client and server and force TLS encryption

·         Configure the SMTP server to control relay and user authentication

·         Create Remote Domains for your own domains so that authenticated users can send mail to your internal domains and configure the Remote Domain to authenticate with the Exchange Server’s SMTP service

·         Install ISA Server 2000 on the Firewall/SMTP relay computer

·         Configure inbound and outbound packet filters to support the authenticating SMTP relay

·         Configure the SMTP client to use the authenticating SMTP relay and install the CA certificate into the client’s Trusted Root Certification Authorities certificate store

 

In this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document we will assume that you want to provide unauthenticated and authenticated relay services. The unauthenticated SMTP relay server is used by Internet SMTP servers to send mail to mail domains under your administrative control, and the authenticated SMTP relay is used by your users who need to relay mail to domains that are under your administrative control and those that are not under your control.

 

However, we will review steps that both authenticating and non-authenticating SMTP relay have in common so that you can use this document to create an authenticating SMTP relay without first configuring a non-authenticating SMTP relay.

 

       Note:
Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay for detailed information on how to configure the non-authenticating SMTP relay.

 

The remainder of this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit will cover the required procedures outlined in the list above.

 

 

 

Installing Windows Server 2003 on the Firewall Computer

 

The computer that will become the ISA Server 2000 firewall/SMTP relay must meet the following minimum requirements:

 

• A personal computer with a 1.5 MHz or higher Intel/AMD-compatible CPU
• For the operating system, Windows 2000 Service Pack 4 or Windows Server 2003
• 256 MB of memory (RAM)
• 20 MB of available hard disk space for program files
• Two network adapters that is compatible with Windows 2000 or Windows Server 2003 , for communication with the internal and external networks
• One local hard disk partition that is formatted with the NTFS file system for log files and Web caching (if you wish to run the ISA Server firewall’s Web caching facilities)

 

The ISA Server firewall and Web caching components work very well on very modest hardware. This is true even when the SMTP filter is enabled and protecting the published co-located SMTP server. However, the SMTP Message Screener can be very processor intensive. This is why I recommend that you use a processor with a minimum of rating of 1.5 MHz. This is especially true if you plan on running an authenticating and non-authenticating SMTP relay on the same computer.

 

Installing a Certificate Server on the Internal Network

 

The first thing you need to do before you install the certificate server is to install the IIS 6.0 Web server. The Web server component is required to host the Web enrollment site for the CA.

 

Installing the IIS 6.0 SMTP Services on the Windows Server 2003 Firewall Computer

 

The SMTP Message Screener requires the IIS SMTP service. You will need to install the SMTP service because Windows Server 2003 does not install IIS by default. Perform the following steps to install the IIS 6.0 SMTP service:

 

1. Click Start, point to Control Panel and click the Add or Remove Programs command (figure 1).

 

Figure 1

 

2. Click the Add/Remove Windows Components button on the left side of the Add or Remove Programs window (figure 2).

 

Figure 2

 

 

3. In the Windows Components dialog box, click on the Application Server entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 3).

 

Figure 3

 

 

4. In the Application Server dialog box, click on the Internet Information Services entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 4).

 

Figure 4

 

 

5. On the Internet Information Services (IIS) page, put a checkmark in the SMTP Service checkbox. The Internet Information Services Manager checkbox will be automatically selected for you. Click OK.

 

Figure 5

 

 

6. Click OK in the Application Server dialog box (figure 6).

 

Figure 6

 

 

7. Click Next on the Windows Components page (figure 7).

 

Figure 7

 

 

8. The Windows Components Wizard installs the IIS SMTP service (figure 8).

 

Figure 8

 

 

9. Click Finish when the Wizard completes.

 

 

 

Disabling Socket Pooling on the ISA Server Firewall SMTP Relay Computer

 

 

You will need to disable socket pooling if you intend to use the Server Publishing method. Perform the following steps to disable socket pooling for the Windows Server 2003 IIS 6 SMTP service:

 

       Note:
Socket pooling allows a service to listen on all IP addresses and all interfaces. This prevents Server Publishing Rules from binding to the socket required listen for incoming SMTP messages.

 

 

1.       Click Start and then click the Command Prompt link. In the Command Prompt window, switch to the Inetpub\AdminScripts folder. Then type in the following command and press ENTER (figure 9):

 

Adsutil.vbs set /smtpsvc/1/DisableSocketPooling 1

 

Figure 9

 

2.       If the SMTP service is installed and you entered the command correctly, you should see what appears in figure 10.

 

Figure 10

 

 

3.       Close the command prompt window.

 

The SMTP service will continue to listen on all IP addresses on all interfaces. You must configure the service to listen on specific IP addresses to limit the server to listening on a subset of addresses.

 

Creating a Second Virtual SMTP Server on the ISA Server 2000 Firewall SMTP Relay Computer

 

You must create a second SMTP virtual server for your authenticating SMTP relay. You cannot use a single virtual server for your authenticating and non-authenticating SMTP relay. There are two reasons for this:

 

• Internet SMTP servers cannot negotiate TLS sessions with the SMTP relay

• Internet SMTP servers cannot authenticate with the SMTP relay

 

You want to force TLS encryption for the SMTP traffic moving between the SMTP client and server. Internet SMTP servers sending mail to your domains will not negotiate TLS encryption with your SMTP relay. Because you want to force TLS encryption for all SMTP connections on the authenticating SMTP relay, you must create a second SMTP virtual server.

 

You also want to force authentication on the authenticating SMTP relay computer because this machine is capable of relaying mail to any email domain. Spammers will use your mail server to relay spam if you do not force authentication.

 

       Note:
The second virtual SMTP server cannot listen on the same IP address as the first virtual SMTP server. If you choose to run both an authenticating and non-authenticating SMTP relay on the same computer, you must bind at least two IP addresses to the external interface of the ISA Server firewall/SMTP relay. If you choose to create only an authenticating SMTP relay, you do not need to create the second virtual SMTP server.

 

Perform the following steps to create the second virtual SMTP server on the ISA Server firewall/SMTP relay computer:

 

1. Open the Internet Information Services (IIS) Manager console. Right click on your server name, point to New and click on SMTP Virtual Server (figure 11).

 

Figure 11

 

2. Type in a name for your virtual server in the Name text box on the Welcome to the New SMTP Virtual Server Wizard page (figure 12).

 

Figure 12

 

 

3. On the Select IP Address page, click the down arrow in the Select the IP address for this SMTP virtual server drop down list box (figure 13). Select an IP address on the external interface of the ISA Server firewall that is not being used by any other SMTP virtual server or any SMTP Server Publishing Rule.

 

       Note:
No two virtual SMTP servers can listen on the same IP address. In addition, on virtual SMTP server can use an IP address that is already in use by an SMTP Server Publishing Rules. The SMTP Server Publishing Rule needs to bind the socket. If you configure the SMTP virtual server to use the same IP address as an SMTP Server Publishing Rule, the Publishing Rule will no longer function.

 

Figure 13

 

4. On the Select Home Directory page, type in the path to a folder on the local computer that will store the SMTP server’s folder hierarchy. The Wizard will create the folder “on the fly”, so you do not need to create it in advance (figure 14). Click Next.

 

Figure 14

 

 

5. On the Default Domain page, type in a bogus name in the Domain text box. We do not want this SMTP server to be the endpoint for any SMTP messages. This SMTP server’s only job is to relay messages. In this example, I’ve used the name SMTPAUTH to help me identify this virtual SMTP server. This name appears in SMTP headers sourcing from this virtual SMTP server and you can use this information when troubleshooting SMTP connections with Network Monitor (figure 15). Click Next.

 

Figure 15

 

 

6. The new virtual SMTP server appears in the left pane of the Internet Information Services (IIS) console (figure 16).

 

Figure 16

 

 

Request and Install a Web Site Certificate on the Authenticated SMTP Relay Server

 

The authenticating SMTP relay server requires a certificate to create the TLS connection between itself and the SMTP client. There are several ways you can obtain a Web site certificate for the virtual SMTP server. The most convenient method is to obtain a certificate from an online certificate authority. Two conditions must be met in order to obtain a certificate from an online certificate authority:

 

·         You have installed an enterprise CA

·         The ISA Server firewall/SMTP relay belongs to the same domain as the enterprise CA

 

If the ISA Server firewall/SMTP relay does not belong to the same domain as the enterprise CA, then you must submit an offline request and manually request and install the Web site certificate.

 

       Note:
For information on how to submit an offline request for a Web site certificate, please see ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document How to Obtain a Web Site Certificate

 

Perform the following steps to create the online request and install the certificate:

 

1.       In the Internet Information Service (IIS) Manager console, Right click on the authenticating virtual server’s name and click the Properties command (figure 17).

 

Figure 17

 

 

2.       In the authenticating virtual server’s Properties dialog box, click on the Access tab. On the Access tab, click on the Certificate button in the Secure communication frame (figure 18).

 

Figure 18

 

 

3.       Read the information on the Welcome to the Web Server Certificate Wizard page and click Next (figure 19).

 

Figure 19

 

 

4.       On the Server Certificate page, select the option that fits your requirements (figure 20). You have the following options:

 

Create a new certificate

This allows you to request a new certificate for the SMTP virtual server. If you do not already have a certificate, then this is the option you should select.

Assign an existing certificate

If you already have a certificate for this virtual server, then you can bind the certificate to the SMTP virtual server using this option. The certificate must already be installing into the machine’s certificate store

Import a certificate from a Key Manager backup file

If you have a certificate from an IIS 4.0 site, you can import the certificate from a Key Manager backup file using this option

Import a certificate from a .pfx file

If you have a certificate that has been exported with its private key into a .pfx file from another site, you can import that certificate into the machine’s certificate store and assign it to the virtual SMTP server

Copy or Move a certificate from a remote server to this site

If you have another server with the same certificate, and you want to use that same certificate on this virtual SMTP server, then select this option. The server should be located somewhere on the internal network.

 

We do not have a certificate for this virtual SMTP server, so we must request a new certificate. Select the Create a new certificate option and click Next.

 

Figure 20

 

 

5.       Select the Send the request immediately to an online certificate authority option on the Delayed or Immediate Request page (figure 21). This allows the Wizard to automatically forward the request to the enterprise CA on the internal network. The Prepare the request now, but send it later option creates a text file that you can submit to any CA and obtain a certificate. You must then manually install the certificate after you receive it. Click Next.

 

Figure 21

 

 

6.       Type in a “friendly name” in the Name text box on the Name and Security Settings page (figure 22). This is a descriptive name only and does not effect the functionality of the certificate. Chose a bit length for the encryption key. The longer the bit length, the more processor intensive the encryption process will be. The default value of 1024 is reasonably secure. Click Next.

 

Figure 22

 

 

7.       Type an Organization and Organizational unit name in the text boxes provided on the Organizational Information page (figure 23). Click Next.

 

Figure 23

 

 

8.       The Your Site’s Common Name page is very important and the correct Common name must be entered into the text box (figure 24). The common name is the name the client application users to connect to the site. For example, if the common name on the certificate is smtpauth.internal.net, then the client must connect to the virtual SMTP server using this name.

 

In addition, this name must resolved to the IP address that is listening for the virtual SMTP server that uses this certificate. In our current example the authenticating virtual SMTP server is listening on 131.107.0.3. The fully qualified domain name smtpauth.internal.net must resolve to 131.107.0.3 so that the client can send the request to the correct IP address the virtual SMTP server is listening on.

 

Note that the SMTP email client software must be configured to use the FQDN of the SMTP relay and not the IP address. The client need to match the name on the certificate the SMTP relay presents to it with the name that its connecting to. You will see an error message on the SMTP email client if these names do not match.

 

Enter the correct FQDN in the Common name text box and click Next.

 

 

Figure 24

 

9.       Type in a State/province and City/locality on the Geographical Information page (figure 25). Use the drop down list box to select a Country/Region. Click Next.

 

Figure 25

 

 

10.   Your enterprise CA will appear in the Certificate authorities drop down list box on the Choose a Certificate Authority page (figure 26). If you have more than a single enterprise CA on the network, you can choose one of them from the list. In this example we have a single enterprise CA, so we will go with the default. Click Next.

 

Figure 26

 

 

11.   Review the information on the Certificate Request Submission page (figure 27). Confirm that the Common Name (listed as the Issued To entry on this page) matches the name users will use to access this virtual SMTP server. Click Next.

 

Figure 27

 

 

12.   Click Finish on the Completing the Web Server Certificate Wizard page (figure 28).

 

Figure 28

 

 

13.   The SMTP virtual server now has a certificate installed that it can use to create the TLS sessions between itself and the SMTP email client. The next step is to force a TLS session so that SMTP email clients can’t create a non-secured connection. Click the Communication button in the Secure communication frame (figure 30).

 

Figure 30