Configuring the ISA Server Firewall/VPN Server to Support L2TP/IPSec NAT Traversal Client Connections

 

You may want to support external L2TP/IPSec clients that are located behind NAT based firewalls to connect to your ISA Server firewall/VPN server. You need to do the following on the ISA Server firewall/VPN server to support the NAT-T RFC compliant L2TP/IPSec clients located behind a remote NAT device:

 

  • Create a packet filter for inbound UDP 500 (receive/send)
  • Create a packet filter for inbound UDP 4500 (receive/send)
  • Create a packet filter for inbound UDP 1701 (receive/send)

 

The UDP 500 receive/send packet filter allows for Internet Key Exchange (IKE) protocol packets to be received by the ISA Server firewall/VPN server. The ISA Server firewall/VPN server needs this packet filter in order to accept calls from both NAT-T L2TP/IPSec clients and non-NAT-T L2TP/IPSec clients.

 

The UDP 4500 receive/send packet filter is specific for NAT-T clients. The IPSec ESP header (IP Protocol 50) is encapsulated inside the UDP port 4500 header. When the Windows Server 2003 ISA Server firewall/VPN server receives the packet, it removes the UDP header and exposes the ESP header. The Windows Server 2003 uses this procedure to determine if the packet is from an L2TP/IPSec NAT-T client.

 

The UDP 1701 receive/send packet filter allows the L2TP control channel to be established and maintained. There are a number of control messages sent through the L2TP control channel. These control messages:

 

  • Establish the VPN tunnel
  • Maintain the VPN tunnel
  • Tear down (close) the tunnel

 

Figure A shows the structure of an L2TP/IPSec packet.

 

Figure A (fig299)

 

You create the three packet filters at the ISA Server firewall/VPN server that you want to accept NAT-T L2TP/IPSec connections from L2TP/IPSec clients located behind a NAT device. If you do not want to support NAT-T L2TP/IPSec clients, then you can use the ISA Server 2000 VPN Wizard to create the packet filters you require.

 

*       Note:
For more information on automatically creating VPN-related packet filters using the ISA Server 2000 VPN Wizard, please refer to ISA Server 2000 VPN Deployment Kit document Configuring the Windows Server 2003 ISA Server 2000/VPN Server.

 

Creating the Packet Filter for UDP Port 500

 

Perform the following steps to create the IKE packet filter for UDP Port 500:

 

  1. In the ISA Management console, expand the Server and Arrays node, then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter (figure 1).

 

Figure 1 (Fig300)

 

  1. Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page (figure 2). I recommend you name the packet filter UDP 500 (receive/send). Click Next.

 

Figure 2 (Fig301)

 

  1. Select the Allow packet transmission option on the Filter Mode page (figure 3). Click Next.

 

Figure 3 (Fig302)

 

  1. Select the Custom option on the Filter Type page (figure 4). Click Next.

 

Figure 4 (Fig303)

 

  1. Configure the details of the packet filter on the Filter Settings page (figure 5). Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 500. Select the All ports option in the Remote port drop down list box. Click Next.

 

Figure 5 (Fig304)

 

  1. Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page (figure 6). The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box in the network interface’s Properties dialog box. Click Next.

 

Figure 6 (Fig305)

 

  1. Select the All remote computers option on the Remote Computers page (figure 7). Click Next.

 

Figure 7 (Fig306)

 

  1. Review the settings on the Completing the New IP Packet Filter Wizard page (figure 8), then click Finish.

 

Figure 8 (Fig307)

 

 

Creating the Packet Filter for UDP Port 4500

 

Perform the following steps to create the packet filter for UDP port 4500:

 

1.       In the ISA Management console, expand the Server and Arrays node, then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter (figure 9).

 

Figure 9 (Fig308)

 

2.       Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page (figure 10). I recommend you name it UDP 4500 (receive/send). Click Next.

 

Figure 10 (Fig309)

 

3.       Select the Allow packet transmission option on the Filter Mode page (figure 11). Click Next.

 

Figure 11 (Fig310)

 

4.       Select Custom on the Filter Type page (figure 12). Click Next.

 

Figure 12 (Fig311)

 

5.       Configure the details of the packet filter on the Filter Settings page (figure 13). Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 4500. Select the All ports option in the Remote port drop down list box. Click Next.

 

Figure 13 (Fig312)

 

6.       Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page (figure 14). The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box, which is found in the external interface’s Properties dialog box. Click Next.

 

Figure 14 (Fig313)

 

7.       Select the All remote computers option on the Remote Computers page (figure 15). Click Next.

 

Figure 15 (Fig314)

 

8.       Review the settings on the Completing the New IP Packet Filter Wizard page (figure 16), then click Finish.

 

Figure 16 (Fig315)

 

Neither the Windows 2000/Windows Server 2003 server, nor the ISA Server services, need to be restarted. The packet filters will start working automatically. If you have a very busy machine and you need the packet filters to start working immediately, you should restart the Firewall service.

 

*       Note:
You can restart the firewall service by navigating to the Servers and Arrays/Server Name/Monitoring/Services node in the ISA Management console. Right click on the Firewall service entry in the right pane and click the Stop command. After stopping the service, right click the Firewall service entry again and click the Start command. You can also stop the Firewall service from the command prompt. Open a command prompt and type “net stop Microsoft firewall” (without the quotes). After the Firewall service stops, restart the Firewall service by typing “net start Microsoft firewall” (without the quotes).

 

Creating the Packet Filter for UDP 1701

 

Perform the following steps to create the L2TP control channel packet filter for UDP 1701:

 

1.       In the ISA Management console, expand the Server and Arrays node, and then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter (figure 17).

 

Figure 17 (Fig316)

 

2.       Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page (figure 18). I recommend you name it UDP 1701 (receive/send). Click Next.

 

Figure 18 (Fig317)

 

3.       Select the Allow packet transmission option on the Filter Mode page (figure 19). Click Next.

 

Figure 19 (Fig318)

 

4.       Select the Custom option on the Filter Type page (figure 20). Click Next.

 

Figure 20 (Fig319)

 

5.       Configure the details of the packet filter on the Filter Settings page (figure 21). Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 1701. Select the All ports option in the Remote port drop down list box. Click Next.

 

Figure 21 (Fig320)

 

6.       Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page (figure 22). The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box. Click Next.

 

Figure 22 (Fig321)

 

7.       On the Remote Computers page (figure 23), select the All remote computers option and click Next.

 

Figure 23 (Fig322)

 

8.       Review the settings on the Completing the New IP Packet Filter Wizard page and click Finish.

 

Figure 24 (Fig323)

 

The L2TP/IPSec NAT-T VPN clients are able to connect after you create all three packet filters. You do not need to restart the server or any of the ISA Server services. If the ISA Server firewall is very busy, it may take a while for the packet filters to take effect. You can manually restart the Firewall service if you need the packet filters to be applied immediately.