Internet attacks against network services are increasingly based on application layer weaknesses. Traditional packet filtering firewalls are not application layer aware and allow inbound packets to network servers based on source and destination port and IP address. The buffer overflow attack is one of the most popular carried out against network services. ISA Server 2000 firewalls can block buffer overflow attacks against DNS and POP3 servers via its DNS and POP3 application filters. This article includes discussions on the nature of buffer overflow attacks, why traditional packet filtering firewalls are unable to stop them, and how ISA Server 2000 application layer firewalls use the DNS and POP3 filters to block buffer overflow attacks against POP3 mail servers and DNS servers.

Step by step procedures on how to configure DNS and POP3 server publishing rules to use ISA Server 2000 application filters are provided and a discussion of how to secure an Internet accessible DNS completes this document.

The Problem: Attackers Use Buffer Overflow Attacks to Take Down DNS and POP3 Servers

A buffer overflow happens when an application or service tries to store more data in a memory (RAM) buffer than it was intended to hold. These buffers are temporary storage areas where data is stored until it is acted upon. Buffers are created to contain a pre-defined amount of data which is consistent with the typical amount of data handled by the service or application. When a buffer overflow condition takes place, extra information placed in the buffer exceeds the amount of space dedicated to the specific buffer used by the application or service. The excess amount of information cannot fit into the pre-defined buffer space and can overflow into buffers designed to be used by other applications and processes.

When the excess information leaks into buffers not dedicated to the service being attacked, information in these other buffers can become corrupt, and this can even lead to overwriting of the valid data held in the other buffers. Sometimes a buffer overflow condition can happen accidentally because of a programming error somewhere within the application or service. However, the buffer overflow attack is an increasingly common application layer attack against network servers and services.

Buffer overflow attacks differ from accidental buffer overflows. In contrast to the accidental buffer overflow, the extra data contains commands designed to carry out exploits against the server. This has the effect of sending new commands to the attacked computer. These commands can damage the user files, change server or database data, or disclose confidential information.

DNS and POP3 servers on the internal network are potentially susceptible to buffer overflow attacks. An attacker on the Internet can send abnormal commands to the DNS or POP3 servers behind the firewall and corrupt the DNS and POP3 services. This can result in disabling the servers, cause the servers to run erratically or even allow the attacker to take control of the servers.

Conventional packet filtering firewalls examine the source and destination IP address and port numbers of the inbound packets and allow connection requests to be forwarded to the DNS or POP3 servers on the internal network. An attacker can take advantage of this situation by creating special packets that appear to be legitimate POP3 or DNS requests, but in fact contain buffer overflow exploits.

Figure A shows how an attacker is able to leverage this weakness in the conventional packet filtering firewall.

The Solution: The ISA Server 2000 DNS and POP3 Application Layer Filters

The solution to the problem of POP3 and DNS buffer overflow attacks is an application layer aware firewall that has an understanding of correct protocol behavior. Unlike the traditional packet filtering firewall, the application layer aware firewall can protect internal network DNS and POP3 servers from buffer overflow attacks. The application layer filtering (stateful filtering) firewall knows the maximum legitimate length of POP3 and DNS commands and blocks incoming connection attempts that include too-large command strings.

Figure B shows the ISA Server 2000 application layer filtering firewall blocking the incoming buffer overflow attacks.

Placing ISA Server 2000 on Your Network

The ISA Server 2000 firewall can be the only firewall on your network, or you can integrate ISA Server 2000’s powerful application layer filtering protection with your existing firewall infrastructure. Some of the common ISA Server 2000 network topologies are:

ISA Server 2000 acting as front-end firewall
ISA Server 2000 acting as back-end firewall
ISA Server 2000 acting as front-end and back-end firewalls
ISA Server 2000 acting as application layer filtering gateway in a perimeter network

ISA Server 2000 Front-end Firewall Topology

Smaller organizations or organizations that do not already have a large investment in a current firewall infrastructure may prefer to make the ISA Server 2000 firewall a front-end firewall. The front-end firewall has a network interface on the corporate network and a network interface directly connected to the Internet. All communications into and out of the corporate network are exposed to ISA Server 2000’s deep application layer inspection.

The advantages of this configuration include:

All communications into and out of the corporate network are exposed to firewall policy
You only need to learn how to configure the ISA Server 2000 firewall software; this avoids the potential for firewall misconfiguration when multiple vendor firewalls are used
All inbound and outbound access can be controlled on a granular, user or group basis. Users only access the content and servers you want them to access, based on the rules you configure
This configuration is easy to set up and maintain

Figure D shows the network topology for the ISA Server 2000 front-end firewall placement.

ISA Server 2000 Back-end Firewall Topology

Organizations that already have an existing firewall infrastructure may prefer to leave the current firewalls in place and put the ISA Server 2000 firewall behind the current firewalls. This topology allows third party firewalls to provide high speed packet filtering (stateful filtering) before forwarding the remaining packets to the application aware ISA Server 2000 firewall. The network between the third party front-end firewalls and the back-end ISA Server firewall is a perimeter network where publicly accessible services can be placed.

The third-party packet filtering firewalls have an interface directly connected to the Internet and an interface connected to a perimeter network between the third-party packet filtering firewalls and the ISA Server 2000 application layer aware firewall. The ISA Server 2000 firewall has an interface on the perimeter network and an interface on the protected, corporate LAN.

Advantages of this configuration include:

Organizations do not need to perform a major redesign of their current firewall infrastructures
Third party hardware-based firewalls can perform high-speed packet filtering. This offloads packet filtering overhead from the ISA Server 2000 firewall and increases the resources available on the ISA Server 2000 firewall to perform deep application layer inspection
Resources located on the corporate network are protected by the ISA Server 2000 firewall’s enhanced application layer inspection mechanisms
Granular inbound and outbound access control can be done on a user/group basis

Figure E shows the topology of the ISA Server 2000 back-end firewall topology.

ISA Server 2000 Front-end and Back-end Firewalls

The ISA Server 2000 front-end and back-end firewall configuration uses two ISA Server 2000 computers, one as the Internet edge firewall and one as the corporate LAN edge firewall. The front-end ISA Server 2000 firewall has an interface directly connected to the Internet and an interface on the perimeter network between the firewalls. The back-end ISA Server 2000 firewall has an interface on the perimeter network and an interface on the protected corporate LAN.

The advantages of this configuration include:

A single firewall system; this reduces the training overhead and the probability of a configuration error
Sophisticated application layer filtering protects hosts on the perimeter network and the corporate network
You can leverage Web Proxy chaining and firewall chaining to significantly increase access control from perimeter network servers and users on the internal network. This prevents attackers from using compromised servers on the perimeter network as a launch point for outbound attacks from the perimeter network
Granular outbound user/group based access control for hosts on both the corporate network and the perimeter network
Excellent support for highly secure VPN passthrough, allowing access to protected resources on the corporate network

Figure F shows the topology for the ISA Server 2000 front-end back-end firewall configuration.

ISA Server 2000 Application Layer Filtering Web Proxy in the Perimeter Network

Some organizations already have an existing firewall infrastructure that includes front-end and back-end firewalls. These organizations have a large investment in their current firewall infrastructures and prefer to leave them intact. You can still leverage ISA Server 2000’s application layer filtering features by making the ISA Server an application layer filtering proxy. This ISA Server 2000 proxy can be placed on the perimeter network between the front-end and back-end third party packet filtering firewalls or you can place the ISA Server 2000 application layer proxy on the corporate network.

Advantages of the application layer filtering proxy configuration include:

The ability to leave the current firewall infrastructure intact; you can “drop in” the ISA Server 2000 application layer filtering proxy virtually anywhere
The third party front-end and back-end packet filtering firewalls can pass packets at high speed while allowing the ISA Server 2000 to provide a very high level of security for communications passed through its application layer inspection mechanisms
A hardened ISA Server 2000 proxy can be placed on the perimeter network segment to reduce the attack surface
In reverse Web Proxy scenarios, the ISA Server 2000 application layer filtering proxy can forward user credentials across the back-end firewall to pre-authenticate remote users

Figure G shows the topology of the application layer filtering proxy configuration.

Configuring the DNS and POP3 Filters

ISA Server 2000 firewalls make it very easy to configure strong application layer protection for DNS and POP3 servers you publish on the corporate network. The DNS and POP3 filters integrate with DNS and POP3 Server Publishing Rules. In addition, these filters tie into built-in Alerts that can automatically take actions and inform you of the attack.

In this section we will discuss the following step by step configurations:

Configuring the DNS Filter, the DNS Server Publishing Rule and DNS Filter Alert
Configuring the POP3 Filter, POP3 Server Publishing Rule and POP3 Alert

Configuring the DNS Filter, DNS Server Publishing Rule and DNS Filter Alert

The DNS filter is one of the many application layer filters included with ISA Server 2000 right out of the box. You do not need to add components or perform a separate installation procedure. The first step in configuring DNS filtering is to confirm that the DNS filter is enabled and that the DNS attack types you wish to block are selected.

Perform the following steps at the ISA Server 2000 firewall computer:

Open the ISA Management console. Expand the Servers and Arrays node and then expand your server name. Expand the Extensions node and click on the Application Filters node. Right click on the DNS intrusion detection filter and click Properties.

In the DNS intrusion detection filter Properties dialog box, place checkmarks in the checkboxes for the DNS attacks you want to block. You should always have checkmarks in the DNS host name overflow, DNS length overflow, DNS zone transfer from high ports checkboxes. You may want to remove the checkmark from the DNS zone transfer from privileged ports (1-1024) option if you want to allow external DNS servers to perform zone transfers from your published DNS servers on the internal network.

Click OK after making your selections.

The DNS filter is enabled by default. If for some reason the filter is disabled, you can enable the filter by right clicking on the DNS intrusion detection filter entry in the right pane of the console and then clicking the Enable command.

The next step is to create a DNS Server Publishing Rule. The Server Publishing Rule allows external users to query a DNS server on the internal network. Publishing your own DNS servers allows you complete control over your public DNS zones. There are several procedures required before you can publish your own public DNS servers:

You must register your public domain names with a domain registrar
After registering a public domain name, you must inform the domain registrar of the IP addresses of two publicly accessible DNS servers. This requires that you have at least two IP addresses bound to the external interface of the ISA Server 2000 firewall so that you can publish two DNS servers on the internal network. You inform the domain registrar of the public addresses on the external interface of the ISA Server 2000 firewall that you are using to publish the two DNS servers
If you wish internal and external clients to connect to internal network servers using the same domain names, then you will need to create a split DNS. The split DNS requires that you create two DNS zones on two different DNS servers. One zone is an external zone that external network clients use to access your published resources, and the second zone is the private zone that internal network clients use to access resources on the internal network
Any DNS server exposed to the Internet should be hardened. Please refer to the section on creating a secure DNS server in this document for detailed instructions on how to configure a secure Internet accessible DNS server
You must install the ISA Server 2000 rollup hotfix. Please refer to Microsoft KB article 810493 at http://support.microsoft.com/?id=810493 for details of this hotfix and how to download the updated files

Perform the following steps to publish a DNS server on the internal network:

1. Open the ISA Management console. Expand the Servers and Arrays node and then expand your server name. Expand the Publishing node and click on the Server Publishing Rules node. Right click on the Server Publishing Rules node, point to New and click Rule.

2. Enter a name for the Server Publishing rule in the Server publishing rule name text box on the Welcome to the New Publishing Rule Wizard page and click Next.

3. On the Address Mapping page, enter the IP address of the internal network DNS server in the IP address of internal server text box. Enter the IP address on the external interface of the ISA Server 2000 firewall that you want to listen for incoming DNS queries in the External P address on ISA Server text box. You can use the Browse button and select the external IP address. Click Next.

4. On the Protocol Settings page, click the down arrow for the Apply the rule to this protocol list and select the DNS Query Server protocol. This allows external users to query the DNS server, but it does not allow zone transfers from external users and DNS servers. Click Next.

5. On the Client Type page, select the Any request option. You need to select this option because you do not know in advance who will query your DNS server. Click Next.

6. Review your settings on the Complete the New Server Publishing Rule Wizard page and click Finish.

7. In the right pane of the ISA Management console you will see the details of the DNS Server Publishing rule you created.

8. In the left pane of the ISA Management console, expand the Monitoring node and click on the Sessions node. You will see an active Firewall Session showing the IP address of the published DNS server in the Client Computer column.

The ISA Server 2000 firewall enables you to create an Alert that triggers when a DNS attack occurs. You can configure the Alert to perform an action, restart a service, and send an email message when a DNS attack occurs. Perform the following steps to enable and configure the DNS attack Alert:

1. In the left pane of the ISA Management console, expand the Monitoring Configuration node and click on the Alerts node. Right click on the DNS intrusion entry in the right pane and click Properties.

2. The General tab is the first one to appear in the DNS intrusion Properties dialog box. Make sure that there is a checkmark in the Enable checkbox.

3. Click on the Events tab. The settings on the Events tab determine when the actions you configure on the Actions tab are triggered. Settings in this dialog box include:

Event -- Displays a list of events defined by ISA Server. An event is an action that occurs as the result of an exception in ISA Server. An alert notifies you when specified events occur and can be configured to trigger a series of actions as a result of these events.

Addition Condition -- When applicable, displays a list of additional keys for the selected event. An additional key means an additional condition that must be fulfilled in order for the alert to be triggered.
Name of occurrences before the alert is issued -- Specifies a threshold level of how many events should pass before the alert is reissued. If a value is typed in here, and a value is also typed in Number of events per second before the alert is reissued, then both limits must be reached before an alert is reissued.

Number of events per second before the alert is issued -- Specifies whether the alert should be reissued if the specified number of events occur per second. If a value is typed in here, and a value is also typed in Number recurrences before the alert is reissued, then both limits must be reached before an alert is reissued.

Immediately -- Specifies that the action should be executed immediately after the specified conditions occur.

After manual reset of alert -- Specifies that the action should be executed only after the alert is manually reset.

If time since last execution is more than X minutes -- Specifies that the action should be executed if it has been at least the specified number of minutes since it was previously executed. In other words, if the administrator doesn’t respond within the specified time, the action (for example, sending an e-mail) will be executed again.

In this example, we will configure the Alert to trigger for Any DNS intrusion. The Number of occurrences is set to 1 and the number of events per second is also set to 1. The If time since last execution is more than setting is configured for 5 minutes. This will give the security administrator time to respond to the Alert and take corrective action.

4. Click the Actions tab. You have the following options on this tab:

Send e-mail -- Specifies whether an e-mail should be sent when the alert conditions are met. Select this option only if an internal Simple Mail Transfer Protocol (SMTP) server can be used to send the e-mail. In order to check whether an SMTP Server is available, click Servers and Arrays, click Name , click Extensions, click Application Filters and then click SMTP Filter and check Server properties.

SMTP server -- Provides a space for you to type the Internet protocol (IP) address or name of the Simple Mail Transfer Protocol (SMTP) server. An event is an action that occurs as the result of an exception in ISA Server.

To: -- Provides a space for you to type the e-mail address of the recipient.

An event is an action that occurs as the result of an exception in ISA Server.

Cc: -- Provides a space for you to type the e-mail address of the person who receives a copy of the mail. An event is an action that occurs as the result of an exception in ISA Server.

From: -- Provides a space for you to type the e-mail address of the sender.

Test -- Click to test e-mail facility.

Program -- Specifies whether a program should be executed when the alert conditions are met.

Run this program -- Provides a space for you to type the location and name of the program you want to run. If you are not sure of the program's location or file name, click Browse.

Use this account -- Displays the name of the selected user account or local system account that will be used to run the specified program. An event is an action that occurs as the result of an exception in ISA Server.

Report to Windows 2000 event log -- Specifies whether the event should be written to the Windows 2000 event log when the alert conditions are met.

Stop selected services -- Specifies whether the selected services should be stopped when the alert conditions are met. If you set the alert actions to start and stop a service, then the service is first stopped and then restarted.

Start selected services -- Specifies whether the selected services, Firewall Proxy and Web Proxy, should be started when the alert conditions are met. If you set the alert actions to start and stop a service, the service is first stopped then restarted.

In this example, we select the Send e-mail option and enter the IP address of an internal network SMTP server. The Alert message is sent to the security administrator and DNS administrator. In the From text box, we enter an email address for the Alert. This allows the administrators receiving the Alert to configure their Outlook email clients to alert them when the Alert message is sent to them by the ISA Server 2000 firewall.

The Report to Windows 2000 event log option is selected. This allows the alert to appear in the Event logs for later viewing. Both the Stop selected services and Start selected services options are enabled. For both of these selections, click the Select button and select the Firewall option.

5. Click Apply and then click OK.

The Alert will now be triggered after a DNS attack occurs and an email message will be sent to the addresses configured on the Actions tab.

Note:
No further configuration is required if you use the IP address of the mail server on the internal network. If you use the email server’s name, then the ISA Server 2000 firewall needs to be able to resolve the name to an IP address. If you choose to use an external SMTP server, then you will need to create a packet filter that allows outbound access to TCP port 25 from any local port. In addition, the ISA Server 2000 firewall will need to be able to resolve the name of the public SMTP server if you use a name instead of an IP address in the SMTP server text box.

Configuring the POP3 Filter, POP3 Server Publishing Rule and POP3 Alert

The ISA Server 2000 POP3 filter protects POP3 servers on the internal network that you have published using ISA Server 2000 Server Publishing Rules. The POP3 filter requires almost no configuration and begins working automatically after you configure the POP3 Server Publishing rule. ISA Server 2000 includes a built-in POP3 Alert that you can configure to warn you of an attack against your published POP3 servers.

Perform the following steps to confirm that the POP3 filter is enabled:

In the ISA Management console, expand the Servers and Arrays node and then expand your server name. Expand the Extensions node and click on Application Filters. In the right pane of the console, right click on the POP intrusion detection filter and click Properties.

On the General tab of the POP intrusion detection filter Properties dialog box, make sure that there is a checkmark in the Enable this filter checkbox. Click OK.

When the POP3 filter is enabled, it will automatically protect from buffer overflow attacks against your published POP3 server. The next step is to create the Server Publishing rule that publishes your POP3 server to the Internet.

Perform the following steps to publish your POP3 server:

1. In the ISA Management console, expand the Publishing node and right click on the Server Publishing Rules node. Point to New and click Rule.

2. Enter a name for the rule in the Server publishing rule name text box on the Welcome to the New Server Publishing Rule Wizard page and click Next.

3. On the Address Mapping page, enter the IP address of the POP3 server on the internal network in the IP address of internal server text box. Enter the IP address on the external interface of the ISA Server 2000 firewall that you want to listen for incoming POP3 connections in the External IP address on ISA Server text box. You can use the Browse button and select an external IP address as well. Click Next.

4. On the Protocol Settings page, click the down arrow for the Apply the rule to this protocol list and select the POP3 Server protocol. Click Next.

5. Select the Any request option on the Client Type page.

6. Review your settings on the Complete the New Server Publishing Rule Wizard page and click Finish.

7. In the ISA Management console, you will see the details of the new Server Publishing rule in the right pane.

8. Expand the Monitoring node in the left pane of the console and click on the Sessions node. You will see an active Firewall Session with the published POP3 server on the internal network.

9. The POP3 Server Publishing rule will begin to work automatically. You do not need to restart the server or any of the ISA Server 2000 services.

ISA Server 2000 includes a pre-built Alert you can use, which is triggered by an attack against your POP3 server. Perform the following steps to configure the Alert.

1. In the ISA Management console, expand the Monitoring Configuration node and click on the Alerts node. Right click on the POP Intrusion entry in the right pane and click Properties.

2. On the General tab of the POP Intrusion Properties dialog box, ensure that there is a checkmark in the Enable checkbox.

3. Click on the Events tab. The settings on the Events tab determine when the actions you configure on the Actions tab are triggered. Settings in this dialog box include:

Name of occurrences before the alert is issued -- Specifies a threshold level of how many events should pass before the alert is reissued. If a value is typed in here, and a value is also typed in Number of events per second before the alert is reissued, then both limits must be reached before an alert is reissued.

Number of events per second before the alert is issued -- Specifies whether the alert should be reissued if the specified number of events occur per second. If a value is typed in here, and a value is also typed in Number recurrences before the alert is reissued, then both limits must be reached before an alert is reissued.

Immediately -- Specifies that the action should be executed immediately after the specified conditions occur.

After manual reset of alert -- Specifies that the action should be executed only after the alert is manually reset.

If time since last execution is more than X minutes -- Specifies that the action should be executed again if at least the specified number of minutes have passed since it was previously executed.

In this example, we will configure the Number of occurrences to 1 and the number of events per second to 1. The If time since last execution is more than setting is configured for 5 minutes. This will give the security administrator time to respond to the Alert and take corrective action.

4. Click the Actions tab. You have the following options on this tab:

SMTP server -- Provides a space for you to type the Internet protocol (IP) address or name of the Simple Mail Transfer Protocol (SMTP) server.

To: -- Provides a space for you to type the e-mail address of the recipient.

Cc: -- Provides a space for you to type the e-mail address of the person who receives a copy of the mail.

From: -- Provides a space for you to type the e-mail address of the sender.

Test -- Click to test e-mail facility.

Program -- Specifies whether a program should be executed when the alert conditions are met.

Run this program -- Provides a space for you to type the location and name of the program you want to run. If you are not sure of the program's location or file name, click Browse.

Use this account -- Displays the name of the selected user account or local system account that will be used to run the specified program.

Report to Windows 2000 event log -- Specifies whether the event should be written to the Windows 2000 event log when the alert conditions are met.

In this example, we select the Send e-mail option and enter the IP address of an internal network SMTP server. The Alert message is sent to the security administrator and mail administrator. In the From text box we enter an email address for the Alert. This allows the administrators receiving the Alert to configure their Outlook email clients to alert them when the Alert message is sent to them by the ISA Server 2000 firewall.

Configuring a Secure Microsoft DNS Server

Any service you make accessible to the Internet must be secured as much as possible. DNS servers present special security concerns because you must allow anonymous access to the DNS server. In contrast, a POP3 server published to the Internet can be configured to require user credentials and those credentials can be protected using SSL/TLS. Although you must allow anonymous access, there are some things you can to do harden the public DNS server you publish to the Internet.

The published DNS server resolves names for external clients. These clients only query your DNS server when they need to resolve names for hosts in domains you own. The DNS server should not be able to resolve names for other domains. DNS servers configured in this way are often referred to as DNS advertisers. The DNS advertiser “advertises” names for hosts in your domains only. The DNS advertiser does not resolve names for hosts in other domains.

Note:
DNS servers configured to resolve names for hosts in domains for which they are not authoritative are often referred to as DNS resolvers. DNS resolvers are able to answer directly any queries for domains for which they are authoritative, and perform recursion to answer queries for domains for which they are not authoritative. For more information on DNS advertisers and resolvers, please refer to http://www.eu.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/idc/rag/ragc02.asp

Perform the following steps on the DNS server you plan to publish to the Internet:

Open the DNS console from the Administrative Tools menu. In the left pane of the console, right click on your server name and click Properties.

On the Interfaces tab in the server’s Properties dialog box, confirm that the DNS server is listening on a specific IP address.

Click on the Forwarders tab. Remove the checkmark from the Enable Fowarders checkbox. A DNS server uses a forwarder to resolve names in domains for which the DNS server is not authoritative. Since your publicly accessible DNS server does not resolve names for domains for which it is not authoritative, there is no need for a forwarder.

Click on the Advanced tab. Place checkmarks in the Secure cache against pollution and Disable recursion checkboxes. The Secure cache against pollution option prevents DNS-specific exploits that can cause false information to be cached by the DNS server. The Disable recursion option prevents the DNS server from querying other DNS servers to resolve names in domains for which it is not authoritative.

Click on the Root Hints tab. You should remove all entries from the root hints list by selecting each entry and then clicking the Remove button. The list should be empty, as seen in the figure below. The DNS server uses the root hints to resolve names for domains for which it is not authoritative. When this list is empty, the DNS server will only be able to answer DNS queries for domains configured on this DNS server.

Click Apply and then click OK.

Summary

Internet attacks against network services are increasingly based on application layer weaknesses. Traditional packet filtering firewalls are not application layer aware and allow inbound packets to network servers based on source and destination port and IP address. The buffer overflow attack is one of the most popular attacks carried out against network services. ISA Server 2000 firewalls can block buffer overflow attacks against DNS and POP3 servers via its DNS and POP3 application filters. This article includes discussions on the nature of buffer overflow attacks, how traditional packet filtering firewalls are unable to stop them, and how ISA Server 2000 application layer firewalls use the DNS and POP3 filters to block buffer overflow attacks against POP3 mail servers and DNS servers. Step by step procedures on how to configure DNS and POP3 server publishing rules to use the application filters are provided and a discussion of how to secure an Internet accessible DNS completes this document.