ISA Server 2000 is a sophisticated, intelligent application layer filtering firewall that can help protect networks against the network attacks of today and tomorrow. ISA Server 2000 firewalls can be used instead of traditional stateful filtering firewalls or in conjunction with an existing packet filtering firewall infrastructure. ISA Server 2000’s application layer filtering and inspection mechanisms provide the ideal level of network security and protection for Internet facing Microsoft servers and services.

This document details the ways that ISA Server 2000 uses intelligent application layer filtering to protect networks using built in filters and how the ISA Server 2000 firewall features can be expanded to meet the network attacks of the future.

Protecting Modern Networks with Application Layer Filtering

The number of attacks against networks from Internet based attackers increases each year. Along with the increase in the number of attacks comes an increase in attack sophistication. Firewalls in use to protect corporate networks from Internet attackers must be able to meet the challenges of increased number and increased sophistication of Internet based attacks.

Traditional Firewalls Perform Basic Packet Filtering

Traditional firewalls act as stateful packet filters. The stateful packet filter can use information in the layer 3 (network layer) and layer 4 (transport) headers of the TCP/IP communication and insure that no unsolicited inbound connections are allowed to the internal network and that sessions communicated through the firewall meet the specifications of the TCP/IP protocol suite and that sessions are not hijacked by attackers.

Unsolicited inbound connections are those that are not in response to a request from a user on the internal network. The stateful filtering firewall assumes that unless there is an explicit filter that allows inbound unsolicited connections, then those connections are not legitimate and potentially represent an attack against the firewall or the corporate network. The unsolicited inbound connection is characterized by a packet that contains the TCP SYN flag. The SYN flag indicates that the host that initiating the connection wishes to establish a new session.

Stateful filtering takes advantage of features of the Transmission Control Protocol (TCP). TCP supports a number of states that mark the status of a TCP session between two communicating hosts. The stateful filtering firewall can use this state information and manage valid connections between internal and external network hosts. In contrast to TCP, the User Datagram Protocol (UDP) transport protocol is not a session based protocol and does not make provisions for session state; it is up to the applications using UDP to manage the sessions between clients and servers. A stateful filtering firewall can address this limitation by enforcing a pseudo-statefulness on UDP-based communications.

Traditional stateful filtering firewalls use connection state, source and destination IP address and source and destination port number to determine whether a packet should be allowed through the firewall. For example, suppose an attacker sends an attack directed at the external IP address on the firewall to TCP port 25. This is the TCP port used to accept SMTP mail messages. If the firewall is configured to allow inbound connections through TCP port 25 from the IP address of the attacker, then the firewall allows the attacker through the firewall to the SMTP mail server on the corporate network. If inbound connections to TCP port 25 are not allowed by the firewall, then the connection attempt is dropped as an unsolicited inbound connection request.

The traditional stateful filtering firewall is effective at dropping connection requests to services you do not want allowed into the internal network. The stateful filtering firewall is not good at determining the validity of the connection request. In the example above, the inbound attack aimed at TCP port 25 was allowed because the stateful filtering firewall is unable to determine the legitimacy of the connection attempt. Security in the context of the stateful filtering firewall depends to a large extent to level of security employed on the internal network server.

While it is important to harden internal network servers as much as possible so that there is defense in depth, stateful filtering firewalls impose an unacceptably large security burden on the level of host-based security of servers on the internal network. Network attackers should be stopped at the perimeter. Stopping attackers at the network perimeter blocks attacks before they ever get to the corporate network and takes a significant load off of servers that need to be accessed by valid external users.

Intelligent Application Layer Filtering and Inspection Firewalls Prevent Attacks Against Network Services

The answer to comprehensive security at the perimeter is the application layer inspection or application layer filtering firewall (sometimes referred to as stateful inspection). The stateful inspection or application layer filtering firewall is able to block attacks by looking for anomalies in the application layer header and data sections of a communication. Application layer filtering firewalls build on the features of the traditional stateful filtering firewalls and enforce both valid connection states and valid application layer communications.

Application layer filtering firewalls are required to protect networks from modern attackers because attackers now focus their efforts on developing exploits against weaknesses in the services they attack. Attackers use a variety of application layer specific methods to exploit known and unknown weaknesses in server services to disable servers or take control of them. An application layer filtering firewall is able to examine the application layer commands and data and determine whether the content or commands being sent to a server on the corporate network fall outside the bounds of valid connection attempts.

Imagine that a company wants to allow their off-site employees access to the full range of Exchange Server features using the full Outlook MAPI client. The only secure way this can be accomplished with a stateful filtering-only firewall is to require remote users to establish a VPN connection to the corporate network and connect to the Exchange Server through the VPN link. VPN connections can potentially provide significant obstacles because of their inherent complexity and end-user confusion on how to install, maintain and manage a VPN client connection.

An application layer firewall that understands how the Outlook client communicates with the Exchange Server using the RPC protocol and Exchange UUIDs can manage the Outlook/Exchange communications and do it in a secure fashion. The application layer filtering firewall that understands the nature of valid RPC communications is able to drop exploits directed against the Exchange Server’s RPC interfaces and allow only valid Outlook connections.

Application layer filtering can also be used to prevent inappropriate communications from leaving the corporate network. Users may try to send proprietary corporate data out of the corporate network to individuals that should not have this information, or users may try to download files located on the Internet that contain viruses, worms or trojans that can be used to attack internal network servers. An intelligent application layer filtering firewall can prevent losses due to uploading sensitive corporate information or downloading dangerous code.

Application layer filtering firewalls can be used instead of, or in conjunction with traditional stateful packet filtering firewalls. When used instead of a traditional stateful packet filtering firewall, the application layer filtering firewall can stop application layer exploits at the Internet edge and prevent attacks at the front line, so that attack code never reaches perimeter networks or internal corporate networks. On the other hand, you may wish to leave high performance (but low security) network layer stateful packet filtering firewalls on the Internet edge and put the intelligent application layer filtering firewalls on the edge of corporate network segments that require the highest level of application layer filtering protection against Internet-based attackers.

How ISA Server 2000 Application Layer Filtering and Deep Content Inspection Protects Microsoft Networks

ISA Server 2000 represents the model of an application layer filtering firewall. Because ISA Server 2000 is a software based firewall, it is able to quickly accommodate the processing and inspection overhead that comes with deep application layer inspection and filtering. ISA Server 2000 application layer filtering firewalls have the ability to block a number of application layer attacks and unwanted email right out of the box. In addition, you can expand the already high level of application layer security provided by ISA Server 2000 firewalls by installing security add-ons.

An ISA Server 2000 firewall is also ideally suited to protect Internet facing Microsoft services. These include Internet Information Server services, Exchange Server services, SharePoint Portal Server services, VPN server services and many more. ISA Server 2000 firewalls leverages the unique level of understanding Microsoft has of its own network services and uses this knowledge to provide an impressive level of protection for Microsoft networks and network services.

This ISA Server 2000 Application Layer Filtering Kit focuses on ISA Server 2000’s sophisticated application layer filtering and inspection mechanisms and how they protect Microsoft servers and services. There are a number of ways ISA Server 2000 firewalls protect corporate networks against today’s application layer focused attacks. These include:

Blocking unwanted email and SMTP buffer overflows with the SMTP filter and SMTP Message Screener
Securing remote full Outlook MAPI client connections with the secure Exchange RPC filter
Preventing attacks from hiding inside SSL tunnels using ISA Server 2000 SSL to SSL bridging
Blocking suspicious Web connections at the perimeter using URLScan for ISA Server 2000
Protecting against DNS and POP3 buffer overflow attacks using the ISA Server 2000 DNS and POP3 filters
Improving incident response with ISA Server 2000 intrusion detection and filtering
Preventing attackers from learning about your internal computer network names and using that information against you by using the ISA Server 2000 Link Translator
Expanding ISA Server 2000 firewall application layer filtering and inspection with high security add-ons

Block Buffer Overflow Attacks and unwanted email with the SMTP Filter and SMTP Message Screener

Buffer overflow attacks against server services is one of the most common methods attackers use to disable a network service and potentially take control of the server running the network service. An attacker can craft a packet containing oversized SMTP commands and send these to an SMTP mail server. If the mail server implementation has a known or unknown buffer overflow weakness, the attack could disable or take over the server.

ISA Server 2000 comes with the SMTP filter that contains a pre-built list of SMTP commands and insures that no inbound SMTP connections are made that exceed the legitimate size of a valid SMTP command. The SMTP filter blocks the buffer overflow attempt at the firewall and prevents the attack from getting past the ISA Server 2000 firewall.

unwanted email represents one of the major threats to network security and stability today. unwanted email clogs email servers and impairs overall employee productivity. ISA Server 2000 includes the SMTP Message Screener that can with alone, or together with another unwanted email filtering solution to provide an anti-unwanted email defense in depth solution. The SMTP Message filter blocks unwanted email based on source email account or email domain, keywords in the subject line or body, and attachment type, name or size.

Read the ISA Server 2000 Application Layer Filtering Kit document Chapter 2: Block unwanted email and Viruses with the SMTP Filter and Message Screener for more information on how the SMTP filter and SMTP Message Screener application layer filters protect the corporate network.

Securing Full Outlook MAPI Client Connections with the Secure Exchange RPC Filter

Corporate network employees enjoy using the same email client application regardless of their location. The full Outlook MAPI client allows users on the internal network full access to the entire range of Exchange Server features when connected to the corporate network. Users often become dissatisfied and experience decreased productivity when they leave the office and must use another email client application to access information stored on the Exchange Server.

Traditional stateful packet filtering firewalls cannot be configured to allow remote users the high level of productivity afforded by the full Outlook MAPI client because of the large number of ports that must be allowed inbound and outbound. The traditional packet filtering firewall does not understand the Outlook/Exchange RPC connections and has no way to secure these connections. Either organizations must allow the Outlook MAPI clients VPN access to the network, or risk being infected by RPC worms and other exploits designed to take advantage of the large number of open ports on the stateful packet filtering firewall.

In contrast to the traditional stateful packet filtering firewall, ISA Server 2000 is an intelligent application layer filtering and inspection firewall that understands Outlook/Exchange RPC communications. The ISA Server 2000 secure Exchange RPC filter allows valid inbound connections from the full Outlook MAPI client to the Exchange Server and blocks illegitimate connection attempts. Because ISA Server 2000 is a sophisticated application layer filtering firewall, it can allow remote users full access to the array of Exchange Server services using the full Outlook MAPI client and protect against RPC worms and other RPC related exploits.

Read the ISA Server 2000 Application Layer Filtering Kit document Chapter 3: Prevent Attacks Against Microsoft Exchange Servers using ISA Server 2000 RPC Filters for more information on how the secure Exchange RPC filter perform intelligent application layer filtering and inspection to enable secure remote access for remote Outlook MAPI clients.

Prevent Attacks from Hiding inside SSL Tunnels using ISA Server 2000 SSL to SSL Bridging

Conventional stateful packet filtering firewalls can be configured to allow incoming connections to secure Web servers on the corporate network. Secure Web servers require that the connection between the Web client on the Internet and the Web server on the internal network use SSL to encrypt the username, password and data moving between the two. The encrypted information is protected because intruders cannot read the information moving inside the SSL tunnel.

The problem is that the conventional stateful packet filtering firewall cannot evaluate information inside the SSL tunnel. Even third party application layer firewalls are unable to inspect the contents of the communications between the Web client on the Internet and the Web server on the internal network because the application layer firewall is unable to determine the contents of the SSL tunneled communications.

ISA Server 2000 provides a unique level of protection and application layer filtering and inspection for secure Web servers. The ISA Server 2000 SSL to SSL bridging feature allows the application layer filtering and inspection features of ISA Server 2000 to decrypt the SSL tunnel and inspect the connection to insure that only valid communications are passed through the firewall. When the communications pass inspection, then the ISA Server 2000 firewall re-encrypts the communications and forwards them. Attackers are no longer able to hide attack code inside an encrypted SSL tunnel.

Read the ISA Server 2000 Application Layer Filtering Kit document Chapter 4: Prevent Virus and Hacker Attacks against Secure Web and OWA sites with SSL Bridging for more information on how ISA Server 2000 firewalls use SSL to SSL bridging to protect secure Web servers on the corporate network.

Block Suspicious Web Connections at the Perimeter with ISA Server 2000 URLScan

Application layer attacks are the most common type of attack directed against Web servers. Attacks can create special requests aimed at exploiting known and unknown weaknesses in Web server software. An application layer filtering firewall should be able to review the HTTP header information and data and be able to determine when a potential attack is taking place.

ISA Server 2000 firewalls use a special version of URLScan to review HTTP requests that are forwarded to Web servers on the internal network. This version of URLScan works very much like the URLScan that is installed on Internet Information Services Web servers. The advantage of using URLScan on the ISA Server 2000 firewall is that Web based attacks are stopped at the perimeter and are never forwarded to the Web server on the internal network. Only communications that pass the URLScan filtering mechanism are allowed through the firewall and forwarded to the corporate Web server on the internal network.

Read the ISA Server 2000 Application Layer Filtering Kit document Chapter 5: Block Hacker Attacks Against Web and OWA Sites with URLScan 2.5 for more information about how to use URLScan on the ISA Server 2000 firewall to block HTTP exploits at the network edge.

Protect Against DNS and POP3 Buffer Overflow Attacks with the DNS and POP3 Filters

Buffer overflow attacks are the most popular attacks launched against corporate servers exposed to the Internet. Buffer overflows can disable server services or even allow attackers to take control of the server. An application layer filtering firewall should be able to detect buffer overflow attacks at the perimeter and stop them before they ever reach the server on the corporate network.

ISA Server 2000 firewalls include the DNS and POP3 application layer filters. These filters protect DNS and POP3 servers from buffer overflow attacks launched against them from Internet intruders. The DNS and POP3 application layer filters can automatically protect your corporate DNS and POP3 servers when you publish them to the Internet.

Read ISA Server 2000 Application Layer Filtering Kit document Chapter 6: Block Buffer Overflow Attacks Against Published DNS and Mail Servers with DNS and POP3 Application Layer Filters to learn about how the DNS and POP3 application layer filters are used to protect your network.

Improve Incident Response with ISA Server 2000 Intrusion Detection and Alerting

In addition to network layer attacks, Internet-based attacks can use network layer attack methods to compromise your firewall and gain access to the corporate network. ISA Server 2000 includes a number of intrusion detection filters that allow you to detect these network layer attacks and prevent them from disabling the firewall or accessible internal network resources.

ISA Server 2000 can also be configured to Alert the security administrator of an ongoing attack. ISA Server 2000 Alerts can send information to the Event Logs and send an email to a security administrator who can take quick, corrective action. In addition, ISA Server 2000 Alerts can be configured to automatically run a script or program that can mitigate the effects of the attack.

Read ISA Server 2000 Application Layer Filtering Kit document Chapter 7: Warn and Protect Against Hacker Attacks Using ISA Server 2000 Intrusion Detection to learn about how the ISA Server 2000 firewall detects and warns of ongoing network layer attacks and how to configure Alert actions to mitigate the negative effects of the attack.

Hide Internal Network Names using the Link Translator

Network attackers use a variety of methods to gain access to information on your corporate network. Different attackers have different motivations; some want to destroy information or disable the network, either “just for fun” or for political or economic reasons (business competitors). Others do not wish to do damage, but instead want to steal information. This is often done for profit (corporate espionage). The latter care more about stealing private information than destroying or disabling network services.

Some Web sites return the names of private internal network servers. Attackers can use this information to help them steal private and proprietary information from your network. The ISA Server 2000 Link Translator can correct this problem by leveraging ISA Server 2000’s application layer filtering firewall features to return only public names to external users accessing the Web sites.

Read ISA Server 2000 Application Layer Filtering Kit document Chapter 8: Prevent Attackers from Learning About Network Infrastructure Names using the Link Translator to learn how the ISA Server 2000 Link Translator can be used to hide internal network names from Internet attackers.

Expanding ISA Server 2000’s Deep Inspection Capabilities with Security Add-ons

Network attackers continue to develop new and more sophisticated methods to attack corporate networks. Modern application layer firewalls must be able to adapt to evolving application layer attacks. ISA Server 2000 firewalls are ideally suited to meet this challenge. ISA Server 2000 allows you to expand the level of application layer protection provided by the firewall. This expandability enables the ISA Server 2000 firewall to meet the application layer firewall filtering needs of today and keep pace with changes taking place in the network attacks landscape.

Read ISA Server 2000 Application Layer Filtering Kit document Chapter 9: Increasing Security by Extending ISA Server 2000 Application Layer Filtering to learn more about how powerful security add-ons can be used to allow you to stay one step ahead of Internet attackers.