ISA Server 2000 is an advanced application aware firewall. As a sophisticated application aware firewall, ISA Server 2000 examines application layer content of communications moving through it. You can use ISA Server 2000’s advanced application layer filtering to help prevent unwanted email, worms and viruses from endangering your network. In this ISA Server 2000 Application Layer Filtering Kit document you will learn about the how unwanted email, viruses and worms enter the network and how they damage computer networks and the businesses owning them. You will also learn how ISA Server 2000 application layer aware firewalls protects your network from these dangers. ISA Server 2000 application layer filtering firewalls can be placed virtually anywhere on a corporate network and this allows a high level of flexibility for organizations that already have an existing firewall infrastructure that they do not wish to replace.

You will learn where you can place the ISA Server 2000 firewall on your network, and see step by step details how to configure the ISA Server 2000 firewall as a secure unwanted email and virus/attachment filtering gateway.

The Problems: unwanted email, Virus and Buffer Overflow Attacks

Networks are under constant attack. There are almost as many different types of attacks as there are attackers. However, the three most common attacks made against networks and network services revolve around three main areas:

unwanted email
Viruses and Worms
Buffer Overflows

The a large proportion of problems encountered on networks today can be traced to one of these issues. unwanted email, viruses, worms and buffer overflows clog mail servers, disable network services, destroy data, consume available network bandwidth, and cost companies thousands and potentially millions of dollars.

unwanted email

Unsolicited commercial email (commonly known as “unwanted email”) is the greatest problem facing corporate networks and the Internet today. unwanted email leads to the following problems:

Wasted bandwidth on Internet connections
Increased Internet bandwidth cost
Increased non-productive traffic on the corporate network
Decreased employee productivity due to reading and deleting unwanted email
Increased administrative costs as network administrators attempt to reduce the negative effects of unwanted email
Increased disk usage on mail servers
Increased processor and memory utilization on mail servers
Increased exposure to legal liability secondary to users who may view offensive unwanted email messages
Increased risk of corporate servers and desktops being used by unwanted email relay stations

It has been estimated that unwanted email e-mail messages consume up to 50% of total bandwidth usage on the Internet today. Recent trends suggest there will be an acceleration in the unwanted email volume curve and an increase in the resources required to review, store, report and delete unwanted email messages from mail servers and user workstations. Corporate networks have already reached a breaking point regarding unwanted email on their mail servers. unwanted email control and elimination is no longer an optional network activity; it’s a requirement.

The challenge is to determine which e-mail messages are unwanted email. You want to block unwanted email and allow valid email messages into your mail systems. An overly aggressive approach to unwanted email control could have an adverse effect on overall productivity. There are a number of methods available used to control unwanted email and each of these methods inspect application layer information. Devices designed to control the influx of unwanted email must be application layer aware. Application layer aware devices can inspect the SMTP messages transporting the unwanted email email and evaluate characteristics of the messages. These characteristics include the following:

Source email address
Source email domain
Keywords in the subject line
Keywords in the message body
Attachment name
Attachment extension
Attachment size

unwanted email can be blocked by restricting access to your mail servers from domains known for hosting spammers. In some circumstances you may want to restrict a specific e-mail address rather than an entire domain. Blocking unwanted email based on source mail domain or e-mail address is only a first step in controlling unwanted email.

Another powerful method used to block unwanted email is keyword matching. unwanted email is blocked based on content in the subject line and message body. unwanted email messages typically contain words never or seldom used in legitimate e-mail. You can leverage this fact by blocking email messages containing targeted keywords in the subject or message body.

E-mail attachments can be included with both unwanted email and non-unwanted email email. Attachments present a special problem for corporate networks and are discussed in the next section on viruses and worms.

Viruses and Worms

Viruses and worms cause a tremendous amount of damage to corporate networks today. Viruses and worm attacks are responsible for:

Destruction of data on servers and workstations
Denial of service attacks on servers and workstations
Lost employee productivity because a workstation or network server is unavailable
Distribution of corporate secrets via mass mailing worms
Increased administrative costs due to repairing damaged workstations and servers
Increased bandwidth use on the corporate network and Internet connection secondary to mass mailing worms and denial of service attacks
Destruction of corporate Web sites
Lost sales because of service unavailability

Historically, worms and viruses were introduced into the corporate network by employees. Floppy disks and CDs containing infected files allowed exploits into the corporate network. Floppy disks and CDs now represent a minor source of infection. Internet downloads using a variety of protocols are now responsible for the vast majority of corporate virus and worm infestations. The most common Internet protocol used to download viruses and worms into the network is SMTP, which is used to send mail to the network.

Virus writers realize that email is a vital function to all businesses and they take advantage of this fact by crafting viruses and worms that spread via email. Both sophisticated and unsophisticated users open email attachments that contain dangerous code. The code is released to the user’s computer and then spreads to the rest of the network. A single infected host can damage virtually every networked device in a short period of time.

An effective way to prevent e-mail borne attacks is to block all attachments at the perimeter. An application aware device can then examine the attachments and perform one of the following actions:

Scan the attachment for viruses, worms and other dangerous code
Hold the attachment for later examination; this is sometimes referred to as “quarantine”
Forward the message, along with its attachment, to a security administrator’s email account for inspection and analysis
Delete the message immediately
Forward the message to the mail server if the application layer filter determines that the attachment does not represent a risk to network security

Note:
Application layer inspection of outbound mail is also possible. An organization may wish to block outgoing viruses and worms in an effort to protect other Internet connected networks. In addition, outbound mail inspection prevents users from sending attachment documents and other files that contain proprietary corporate data.

Buffer Overflow Attacks

Attackers use buffer overflow attacks to disable network servers. Unlike the typical virus or worm attack, a buffer overflow cannot be protected against by blocking attachments. Hackers use buffer overflow exploits to disable specific server services with the intent of creating a denial of service – either by disabling a specific service on the target computer or by taking the entire machine offline. More elaborate buffer overflow exploits can be used to disable key security features and allow the attacker to run commands of his choice on the targeted machine.

SearchSecurity.com defines a buffer overflow in this way:

“A buffer overflow occurs when a program or process tries to store more data in a buffer (temporary data storage area) than it was intended to hold. Since buffers are created to contain a finite amount of data, the extra information - which has to go somewhere - can overflow into adjacent buffers, corrupting or overwriting the valid data held in them. Although it may occur accidentally through programming error, buffer overflow is an increasingly common type of security attack on data integrity. In buffer overflow attacks, the extra data may contain codes designed to trigger specific actions, in effect sending new instructions to the attacked computer that could, for example, damage the user's files, change data, or disclose confidential information. Buffer overflow attacks are said to have arisen because the C programming language supplied the framework, and poor programming practices supplied the vulnerability.”

Buffer overflow attacks can be mounted against an organization’s SMTP server and can stop the inflow of mail. The best way to prevent a buffer overflow attack against the SMTP server is to stop the attacker at the network perimeter, before the exploit ever finds its way into the corporate network. An application aware device can evaluate the SMTP commands sent through the firewall and stop the attack.

The Solutions: The SMTP Filter and SMTP Message Screener

ISA Server 2000 is a sophisticated application layer aware firewall that can solve these problems. The ISA Server 2000 firewall can be configured to stop buffer overflow attacks, dangerous viruses and worms, and unwanted email at the network perimeter. ISA Server 2000 performs deep inspection of SMTP messages moving through the firewall and blocks dangerous code and unwanted email from entering the corporate network.

ISA Server 2000 firewalls use two technologies to protect the corporate network:

The SMTP filter
The SMTP Message Screener

When used in combination, the SMTP filter and SMTP Message Screener become powerful allies in the war against SMTP attacks and unwanted email.

The SMTP Filter

The ISA Server 2000 SMTP filter examines SMTP commands sent by Internet SMTP servers and clients. This application layer filter intercepts the SMTP commands and checks to see if they are larger than they should be. SMTP commands that are larger than RFC limits are assumed to be attacks against the SMTP server and are stopped at the perimeter by the ISA Server 2000 firewall’s SMTP application layer filter.

Figure A shows the flow of information and where the SMTP filter blocks the buffer overflow attack.

Figure B shows a list of the default commands included with the SMTP filer.

Each SMTP command has a Maximum Length associated with it. This length represents the number of bytes allowed for each command. If an attacker sends a command that exceeds the number of bytes allowed for the command, then the ISA Server 2000 firewall drops the connection and prevents the attacker from communicating with the corporate mail server.

The SMTP Message Screener

The SMTP Message Screener works together with the ISA Server 2000 SMTP filter, which is an application filter to intercept all SMTP traffic arriving on TCP port 25 of the ISA Server 2000 firewall. The filter accepts the traffic, performs deep application layer inspection, and passes it on to the corporate SMTP server only if it passes inspection based on rules you configure.

The SMTP Message Screener component can filter incoming mail based on:

Source e-mail address
Source e-mail domain
Keywords in the e-mail subject line
Keywords in the e-mail message body
Attachment name
Attachment file extension
Attachment size

An alert can be generated if mail is received from specific users. In addition, the SMTP Message Screener can be configured to hold the e-mail for later inspection or forward the message to a security administrator’s account for further examination and analysis.

Figure C shows an example of how the SMTP filter can be configured to block keywords. One of the most common unwanted email messages received on corporate networks contains the keyword Viagra. This example shows how to block e-mail containing the word Viagra in the message header (subject line) or body. There are three actions that can be taken when the e-mail matches this rule:

Delete message
Hold message
Forward message to

Placing the ISA Server 2000 Firewall on Your Network

The ISA Server 2000 firewall can be the only firewall on your network, or you can integrate ISA Server 2000’s powerful application layer filtering protection with your existing firewall infrastructure. Some of the common ISA Server 2000 network topologies are:

ISA Server 2000 acting as front-end firewall
ISA Server 2000 acting as back-end firewall
ISA Server 2000 acting as front-end and back-end firewalls
ISA Server 2000 acting as application layer filtering gateway in a perimeter network

ISA Server 2000 Front-end Firewall Topology

Smaller organization or organizations that do not already have a large investment in a current firewall infrastructure may prefer to make the ISA Server 2000 firewall a front-end firewall. The front-end firewall has a network interface on the corporate network and a network interface directly connected to the Internet. All communications into and out of the corporate network are exposed to ISA Server 2000’s deep application layer inspection.

The advantages of this configuration include:

All communications into and out of the corporate network are exposed to firewall policy
You only need to learn how to configure the ISA Server 2000 firewall software; this avoids the potential for firewall misconfiguration when multiple vendor firewalls are used
All inbound and outbound access can be controlled on a granular, user or group basis. Users only access the content and servers you want them to access, based on the rules you configure
This configuration is easy to set up and maintain

Figure D shows the network topology for the ISA Server 2000 front-end firewall placement.

ISA Server 2000 Back-end Firewall Topology

Organizations that already have an existing firewall infrastructure may prefer to leave the current firewalls in place and put the ISA Server 2000 firewall behind the current firewalls. This topology allows third party firewalls to provide high speed packet filtering (stateful filtering) before forwarding the remaining packets to the application aware ISA Server 2004 firewall. The network between the third party front-end firewalls and the ISA Server firewall is a perimeter network where publicly accessible services can be placed.

The third-party packet filtering firewalls have an interface directly connected to the Internet and an interface connected to a perimeter network between the third-party packet filtering firewalls and the ISA Server 2000 application layer aware firewall. The ISA Server 2000 firewall has an interface on the perimeter network and an interface on the protected, corporate LAN.

Advantages of this configuration include:

Organizations do not need to perform a major redesign of their current firewall infrastructures
Third party hardware-based firewalls can perform high-speed packet filtering. This offloads packet filtering overhead from the ISA Server 2000 firewall and increases the resources available on the ISA Server 2000 firewall to perform deep application layer inspection
Resources located on the corporate network are protected by the ISA Server 2000 firewall’s enhanced application layer inspection mechanisms
Granular inbound and outbound access control can be done on a user/group basis

Figure E shows the topology of the ISA Server 2000 back-end firewall topology.

ISA Server 2000 Front-end and Back-end Firewalls

The ISA Server 2000 front-end and back-end firewall configuration uses two ISA Server 2000 computers, one as the Internet edge firewall and the other as the corporate LAN edge firewall. The front-end ISA Server 2000 firewall has an interface directly connected to the Internet and an interface on the perimeter network between the firewalls. The back-end ISA Server 2000 firewall has an interface on the perimeter network and an interface on the protected, corporate LAN.

The advantages of this configuration include:

A single firewall system; this reduces the training overhead and the probability of a configuration error
Sophisticated application layer filtering protecting hosts on the perimeter network and the corporate network
You can leverage Web Proxy chaining and firewall chaining to significantly increase access control from perimeter network servers and users on the internal network. This prevents attackers from using compromised servers on the perimeter network as a launch point for outbound attacks from the perimeter network
Granular outbound user/group based access control for hosts on both the corporate network and the perimeter network
Excellent support for highly secure VPN passthrough, allowing access to protected resources on the corporate network

Figure F shows the topology for the ISA Server 2000 front-end back-end firewall configuration.

Figure F:

ISA Server 2000 Application Layer Filtering Web Proxy in the Perimeter Network

Some organizations already have an existing firewall infrastructure that includes front-end and back-end firewalls. These organizations have a large investment in their current firewall infrastructures and prefer to leave them intact. You can still leverage ISA Server 2000’s application layer filtering features by making the ISA Server an application layer filtering proxy. This ISA Server 2000 proxy can be placed on the perimeter network between the front-end and back-end third party packet filtering firewalls or you can place the ISA Server 2000 application layer proxy on the corporate network.

Advantages of the application layer filtering proxy configuration include:

The ability to leave the current firewall infrastructure intact; you can “drop in” the ISA Server 2000 application layer filtering proxy virtually anywhere
The third party front-end and back-end packet filtering firewalls can pass packets at high speed while allowing the ISA Server 2000 to provide a very high level of security for communications passed through its application layer inspection mechanisms
A hardened ISA Server 2000 proxy can be placed on the perimeter network segment to reduce the attack surface
In reverse Web Proxy scenarios, the ISA Server 2000 application layer filtering proxy can forward user credentials across the back-end firewall to pre-authenticate remote users

Figure G shows the topology of the application layer filtering proxy configuration.

SMTP Filter and Message Screener Scenarios: SMTP Relay and Message Screener on the Corporate Network

The SMTP filter always runs on the ISA Server 2000 firewall computer. However, you can place the SMTP Message Screener on another computer located on the protected network behind the ISA Server 2000 firewall. The SMTP Message Screener can be installed in the following locations:

On the ISA Server 2000 firewall itself
On an independent SMTP relay located behind the ISA Server 2000 firewall
On the Exchange Server

Message filtering requires a significant amount of processing power. For this reason, most organizations prefer to put the SMTP Message Screener on the ISA Server 2000 firewall computer or on an SMTP relay located somewhere on the corporate network, so as not to overload the Exchange server.

The SMTP Message Screener can be installed on an SMTP relay computer on the corporate network running the IIS 5.0 or IIS 6.0 SMTP service. The ISA Server 2000 firewall publishes the SMTP relay on the internal network and the Message Screener blocks dangerous e-mail at the SMTP relay computer. The SMTP Message Screener communicates with the SMTP filter to obtain information about which e-mail messages should be blocked.

Figure H shows the topology of the SMTP Message Screener on a dedicated SMTP relay configuration.

SMTP Filter and Message Screener Scenarios: SMTP Relay and Message Screener on the ISA Server 2000 Firewall

Many organizations prefer to use a “one box” solution in which the SMTP Message Screener is located on the ISA Server 2000 firewall itself. This simplifies setup and management of the SMTP Message Screener and reduces the hardware and software configuration overhead.

In this scenario, the ISA Server 2000 firewall acts as an SMTP relay. The IIS SMTP service is installed on the ISA Server 2000 firewall and processes the incoming SMTP Messages. The SMTP Message Screener filters unwanted email, viruses and attachments and relays the safe e-mail messages to the Exchange Server on the corporate network.

Note:
In both this scenario and the one where the SMTP Message Screener is installed on a dedicated SMTP relay, the ISA Server 2000 firewall can be integrated into an existing firewall infrastructure. The ISA Server 2000 firewall can act as a back-end firewall or an application layer filtering SMTP proxy located on the perimeter network. The only requirement is that the front-end firewall must forward inbound SMTP messages to the ISA Server 2000 firewall machine.

This is the most popular configuration because of the lower hardware and configuration overhead. The remainder of this document provides detailed step by step procedures for configuring the ISA Server 2000 firewall as a secure unwanted email and virus filtering SMTP relay.

Installing and Configuring the SMTP Message Screener and SMTP Filter on the ISA Server 2000 Firewall

The ISA Server 2000 firewall can be used as a unwanted email and virus filtering gateway. The following steps are required:

· Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer

· Disable SMTP Service Socket Pooling

· Configure the IIS 6.0 SMTP Service Relay Properties

· Create Remote Domains to Support Your E-mail Domains and Enable Relay for Those Domains

· Install ISA Server 2000 onto the Windows Server 2003 Firewall Computer

· Configure Server Publishing Rules on the ISA Server Firewall

· Configure the SMTP Filter and SMTP Message Screener Properties

The remainder of this document provides detailed step by step procedures on how to configure the ISA Server 2000 firewall as a unwanted email and virus/attachment filtering gateway.

Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer

The SMTP Message Screener requires the IIS SMTP service. You need to install the SMTP service because Windows Server 2003 does not install the SMTP service by default. Perform the following steps to install the IIS 6.0 SMTP service:

Click Start, point to Control Panel and click the Add or Remove Programs command (figure 1).

Figure 1

Click the Add/Remove Windows Components button on