Protocol rules identify which protocols may be utilized for communication, between the internal network and external public domain or Internet. Protocol rules are processed at the application level of the OSI model.
Protocol rules dictate to ISA clients which protocols can be utilized to access resources on the Internet. Protocol rules can be configured to allow or deny the use of one or more protocol definitions.
Protocol definitions are defined inbound and outbound port settings that are assigned to a protocol name, and are a component under Policy elements within the ISA console.
Please note: Protocol rules only work with Secure NAT ISA clients and with Firewall ISA clients.
a) The diagram above shows the location of the protocol rules within the ISA MMC console.
Protocol rules at the enterprise and Array-level
Protocol rules can be used at both levels. When an array policy is allowed, protocol rules can then further restrict enterprise-level protocol rules, this enables array-level protocol rules to deny the use of specific protocols.
Protocol rule functionality.
Protocol rules can be configured to apply to a specific protocol and to protocol definitions and all IP Internet protocol traffic. Protocol rules are applicable in all installation modes of ISA server. The protocol rules function differently in each mode as described in the table below.
B) The table above describes what protocols protocol rules can be applied to when installed in its respective mode. (This information is according to Microsoft but I have noticed very little differences when testing the all IP in cache mode scenario)
ISA Server includes a catalog of preconfigured commonly used protocol definitions, ISA server allows you to also add or modify additional protocols to the protocol definition list.
c) The above diagram shows where protocol definitions can be found.
The steps below are used when creating a protocol definition.
Identify the port number. A specific number between 1 and 65535 that is used for the by the respective protocol for the initial connection.
Low-level protocol. Either the TCP (Transmission Control Protocol) or the UDP (User Datagram Protocol).
Direction. Outbound only, inbound only, or both outbound or inbound (for UDP protocol) or Inbound, Outbound (for TCP protocol).
Secondary connections. Constitutes of a series of port numbers, protocol name, and direction used for additional secondary connections or packets that follow the opening connection.
When an ISA client requests a web resource using a specific protocol, ISA Server first checks the protocol rules. If one of the protocol rules specifically denies the use of the protocol, then request is denied.
Requests are processed only if one of the protocol rules specifically allows the client to communicate using that specific protocol. A site and content rule must also specifically allow access to the requested object.
To access the internet both a site and content and a protocol rule must exist explicitly allowing access to the web resource.
A number of application filters create and install new protocol definitions. Please note: when disabling an application filter, all its protocol definitions are also disabled. Some protocol definitions have application filters that are common to two protocol definitions these are an exception to the protocol definition disable rule and the application filter can still be used by the protocol definition left enabled. (I have only found this to be true for some but not all of the scenarios tried please test your individual scenario before making this live.)
d) The above diagram shows where application filters are located.
Secure NAT clients and the interaction with Protocol rules
A protocol rule is applied to both firewall secure NAT clients if the protocol is defined by an application filter. If a protocol has secondary connections, and it is not defined by an application filter, then the protocol rule applies only to the primary connection. This means that this type of rule only applies to a firewall ISA client. When the rule applies to all IP traffic then it is relevant to Secure NAT ISA clients.
Protocol rule processing order
All rules that deny protocols are processed.
Then rules that allow access are processed.
Summary: Protocol rules are one of the key components of the ISA Access policy, and protocol rules function together with site and content rules to give clients using ISA access to the internet. Protocol rules determine what protocols can be used by the ISA clients to access the internet. This can be a powerful tool when used correctly as you can allow and deny access to particular protocols that best suits your organizations needs. I have outlined protocol rules in this tutorial, and this information should give you a better understanding of what protocol rules are and what they can do.