Understanding the Web Proxy and Firewall Client Automatic Configuration

by Stefaan Pouseele [Published on 11 June 2005 / Last Updated on 20 May 2013]

In this article we will explore how the ISA Server 2004 Web Proxy and Firewall Client Automatic Configuration really works from a client point of view. With that knowledge you should be able to decide which method is the most appropriate for your specific environment. Although this article is written with the ISA Server 2004 in mind, most of the principles apply also to an ISA Server 2000 environment because the Web Proxy and Firewall Client Automatic Configuration is mainly a client feature, not an ISA Server issue.

Understanding the Web Proxy and Firewall Client Automatic Configuration

By Stefaan Pouseele
June 2005
Last Update: 11/06/2005

1. Summary

In this article we will explore how the ISA Server 2004 Web Proxy and Firewall Client Automatic Configuration really works from a client point of view. With that knowledge you should be able to decide which method is the most appropriate for your specific environment. Although this article is written with the ISA Server 2004 in mind, most of the principles apply also to an ISA Server 2000 environment because the Web Proxy and Firewall Client Automatic Configuration is mainly a client feature, not an ISA Server issue.

Take note that it is not within the scope of this article to explain how you configure the Automatic Configuration in Internet Explorer and the Firewall client, and what infrastucture components should be in place to take advantage of this feature. This information is already covered in great detail in Tom's book Configuring ISA Server 2004 and many articles such as:

We will also use extensively the free network monitor tool Ethereal. This is a must have for anybody who wants to understand how things really work on the wire.

2. Overview

The Web Proxy client Automatic Configuration feature is available in Internet Explorer 5 and later. If you open in Internet Explorer the Tools menu, click Internet Options, then the Connections tab and select the LAN Settings you will see the following screen:

The upper part of the screen contains the Automatic Configuration settings. The lower part of the screen contains the manual Proxy settings. For the Automatic Configuration settings you have the choice between Automatically detect settings and Use automatic configuration script, or you can select both.

The Firewall client Automatic Configuration feature is available in the ISA Server 2000 Firewall client and later. If you right click the ISA 2004 Firewall client icon in the System Tray and select Configure you will see the following screen:

Again, the upper part of the screen contains the Automatic Configuration settings. The lower part of the screen contains the manual settings. Take note that for the Firewall client you have only one choice, namely Automatically detect ISA Server.

Automatically Detect

When you select Automatically detect settings in Internet Explorer or Automatically detect ISA Server in the Firewall client, you tell Internet Explorer or the Firewall client to first locate the server where the automatic discovery information is stored. This is done with the WPAD (Web Proxy Auto-Discovery) protocol to locate a WPAD entry in DHCP (Option 252) or DNS (Alias or CNAME). Once the location of the automatic discovery information is determined, the client will download the configuration script file through the HTTP protocol and configure itself according to the content of this file. For Internet Explorer the configuration script file has a predefined name wpad.dat and for the Firewall client the predefined name is wspad.dat.

Whether you use DHCP or DNS, in both cases the result is the same. The client will request the appropriate configuration script file with the URL http://<HOST>:<PORT><PATH>. The component HOST should match the FQDN of the server publishing the automatic discovery information. Normally this will be the FQDN of the Internal interface of your ISA server. The component PORT should match the TCP port number on which you have enabled the publishing of the automatic discovery information. If the discovery mechanism does not provide a port component, then the client must assume port 80. This is the case for the WPAD entry in DNS because you cannot specify a port number in a DNS Alias or CNAME record. Therefore it is adviced to always use TCP port 80 for the publishing of the automatic discovery information. The component PATH is /wpad.dat for Internet Explorer and /wspad.dat for the Firewall client. Be sure to use lowercase letters for the PATH component because ISA server is case-sensitive, otherwise the request will fail.

Note 1: if you use a WPAD entry in DNS, you can easely test the automatic discovery mechanism by entering the URL's http://wpad/wpad.dat and http://wpad/wspad.dat in Internet Explorer. Those requests should return the configuration script files which you can view with notepad. If you use a WPAD entry in DHCP, use the hostname or FQDN of the Internal interface of your ISA server as HOST component.

Note 2: if the Require all users to authenticate Web Proxy setting is configured for the Internal interface of your ISA 2004 server, the request for the configuration script file (wpad.dat or wspad.dat) must be authenticated also. This means that for Internet Explorer an authentication prompt will pop-up. However, the Firewall client does not handle the "401 Authentication Required" response. Therefore, that request will fail. To solve that problem, check out the Microsoft Knowledge Base Article 885683.

It should be obvious that the DHCP automatic discovery process can only work if the client host is configured to use DHCP to obtain his TCP/IP settings. Likewise, for the DNS automatic discovery process to work correctly, the Primary DNS Suffix assigned to the adapter should be correctly set. However, if no Primary DNS Suffix or an incorrect Primary DNS Suffix is assigned to the adapter then the DNS resolving of the WPAD entry might fallback to a Netbios Name Query of  WPAD<00>. This is standard name resolving behaviour for a Windows based OS.

2.2. Automatic Configuration Script

When you select Use automatic configuration script in Internet Explorer, you tell Internet Explorer immediately the exact location of the configuration script file. Therefore, the main difference with the option Automatically detect settings is that the discovery mechanism based on the WPAD protocol is completely bypassed. This means also that no specific DHCP (Option 252) or DNS (Alias or CNAME) configuration is needed. The default configuration URL is http://FQDN:8080/array.dll?Get.Routing.Script where FQDN is the name or IP address of the ISA Server computer. However, if you have to support third-party proxy servers in Internet Explorer, you should specify here the Proxy Auto-Configuration (PAC) file. The administrator of the third-party proxy server should be able to give you the exact URL (e.g. http://FQDN/proxy.pac).

Note: to learn more about the content of the wpad.dat or .pac file, check out Chapter 26 - Using Automatic Configuration, Automatic Proxy, and Automatic Detection in the Internet Explorer 6 Resource Kit.

3. Client OS Support

There has been some debate about the automatic discovery support for Firewall and Web Proxy clients for various operating systems. According to the ISA Server 2004 Help file and the Microsoft article Automatic Discovery for Firewall and Web Proxy Clients, the Firewall client 2004 supports both the DHCP and DNS automatic discovery process for all users. For Internet Explorer 5 and later, the DNS automatic discovery process is supported for all users on Windows 2003, Windows XP and Windows 2000. However, the DHCP automatic discovery process is supported for all users on Windows 2003 and Windows XP, and only for admin users on Windows 2000. On the other hand, the Microsoft Knowledge Base Article 312864 states explicitely that Automatic Proxy Discovery in Internet Explorer with DHCP Requires Specific Permissions in Windows 2000 and XP.

Because of the inconsistency in the Microsoft information I have done some extensive testing on Windows 2000, Windows XP and Windows 2003 to determine the real behaviour for an admin and normal user. The result is shown in the table below:

Client Operating system Internet Explorer 5 and later Firewall Client 2004
Windows 2000 All users (DNS)
Admin users only (DHCP)
All users (DHCP + DNS)
Windows 2000 SP4 All users (DNS)
Admin users only (DHCP)
All users (DHCP + DNS)
Windows XP All users (DNS)
Admin users only (DHCP)
All users (DHCP + DNS)
Windows XP SP1A All users (DNS)
Admin users only (DHCP)
All users (DHCP + DNS)
Windows XP SP2 All users (DHCP + DNS) All users (DHCP + DNS)
Windows 2003 All users (DNS)
Admin users only (DHCP)
All users (DHCP + DNS)
Windows 2003 SP1 All users (DHCP + DNS) All users (DHCP + DNS)

How do the above result match the Microsoft Knowledge Base article 312864, particular with the status "This behavior is by design"? If you look at the list of fixes included in Windows XP Service Pack 2 you will find a reference to that KB article 312864 in the category Networking. Apparently the behaviour described in the KB article 312864 is changed with Windows XP Service Pack 2. Therefore, that KB article 312864  is no longer valid and should be updated accordingly. For Windows 2003 Service Pack 1 I did not find any specific information.

4. Automatic Discovery Process

In the previous section you can find for various Windows operating systems which automatic discovery process will work depending on the user rights of the logged in user. However, how was I able to test that consistently and be sure that a full automatic discovery process was performed? First of all, you should have a good lab environment. How to build one is explained in my article How to build an ISA firewall lab with Virtual PC 2004. Secondly, you should familiarize yourself with a good network monitor tool. You can use the built-in network monitor in Windows 2000 and 2003, but be aware it can not capture data in promicious mode. Therefore, my favorite one is the free network monitor tool Ethereal. Last but not least, be willing to spend a lot of time and effort in testing and documenting the different scenario's.

One of the pitfalls in such kind of testing is to be aware that many dynamic gathered information is cached by the operating system or application. Therefore, if we want to see with a network monitor tool how things work on the wire, we should be able to clear the cached information before every test. The major ones involved in the automatic discovery process are the DNS resolver cache, the Internet Explorer cache and the fact that the DHCP parameters are also cached on an adapter base.

After some experimenting I found that a full automatic discovery process could be enforced on the client in the following way:

Internet Explorer: make sure you have only one instance of Internet Explorer open. In the Internet Options go to the General tab, configure there a blank home page and delete the temporary Internet files. On the Connection tab go to LAN Settings and uncheck all boxes for automatic and manual configuration. Then close Internet Explorer and clear the DNS resolver cache with the command 'ipconfig /flushdns'. Open Internet Explorer again, go to Internet Options > Connection > LAN Settings and configure the Automatic Configuration option you want to test and then browse to a simple external web site. If Automatically detect settings is checked, Internet Explorer will first go through a full automatic discovery process.

Firewall client 2004: first disable the Firewall Client Agent service on the client. Next, change the DHCP scope on the DHCP server so that you will be sure that a renew of the IP configuration on the client will result in a new IP address. Then on the client repair the interface or run the command 'ipconfig /renew' and check that a new IP address is allocated. Finally, enable the Firewall Client Agent service on the client. The Firewall client will now go through a full automatic discovery process assuming Automatically detect ISA Server is checked.

In the next sections we will walk through the automatic discovery process with DHCP and DNS for Internet Explorer on a Windows XP SP2 client. Keep in mind that the discovery process is exactly the same for Internet Explorer and the Firewall client. Next we will discuss the case where both the Automatically detect settings and Use automatic configuration script is checked in Internet Explorer although no specific DHCP or DNS configuration is done. For the three cases the DNS domain is intranet.splab.net and the hostname of the ISA Server 2004 is isa. The DNS/DHCP server has as IP address 192.168.22.2 and the ISA Server 192.168.22.1. The DHCP scope for the clients is 192.168.22.100 - 192.168.22.200. In the last section we will highlight some specific features of the Firewall client 2004.

4.1. DHCP Option 252

In this scenario the Automatically detect settings is checked and the Use automatic configuration script is unchecked in Internet Explorer. The DHCP Option 252 is populated with the URL http://isa.intranet.splab.net:80/wpad.dat and no DNS WPAD Alias is configured. Here is the corresponding Ethereal trace:

No Time      Source         Destination     Proto Info
 1  0.000000 192.168.22.100 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0xfe7e973d
 2  0.002204 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0xfe7e973d
 3  4.027730 192.168.22.100 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0xfe7e973d
 4  4.037363 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0xfe7e973d
-----------------------------------------------------------------------------------------
 5 13.216448 192.168.22.100 192.168.22.2    DNS   Standard query A isa.intranet.splab.net
 6 13.221231 192.168.22.2   192.168.22.100  DNS   Standard query response A 192.168.22.1
---------------------------------------------------------------------------------------------------------
 7 13.224327 192.168.22.100 192.168.22.1    TCP   1303 > http [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
 8 13.231428 192.168.22.1   192.168.22.100  TCP   http > 1303 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
 9 13.231452 192.168.22.100 192.168.22.1    TCP   1303 > http [ACK] Seq=1 Ack=1 Win=65535 Len=0
10 13.231654 192.168.22.100 192.168.22.1    HTTP  GET /wpad.dat HTTP/1.1
11 13.234891 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 200 OK (application/x-ns-proxy-autoconfig)
12 13.234912 192.168.22.1   192.168.22.100  HTTP  Continuation or non-HTTP traffic
13 13.234933 192.168.22.100 192.168.22.1    TCP   1303 > http [ACK] Seq=125 Ack=2921 Win=65535 Len=0
14 13.241378 192.168.22.1   192.168.22.100  HTTP  Continuation or non-HTTP traffic
15 13.361555 192.168.22.100 192.168.22.1    TCP   1303 > http [ACK] Seq=125 Ack=2936 Win=65520 Len=0
16 13.371409 192.168.22.1   192.168.22.100  TCP   http > 1303 [FIN, ACK] Seq=2936 Ack=125 Win=65411 Len=0
17 13.371512 192.168.22.100 192.168.22.1    TCP   1303 > http [ACK] Seq=125 Ack=2937 Win=65520 Len=0
18 13.371728 192.168.22.100 192.168.22.1    TCP   1303 > http [FIN, ACK] Seq=125 Ack=2937 Win=65520 Len=0
19 13.374219 192.168.22.1   192.168.22.100  TCP   http > 1303 [ACK] Seq=2937 Ack=126 Win=65411 Len=0
-----------------------------------------------------------------------------------------------------------
20 13.382720 192.168.22.100 192.168.22.1    TCP   1304 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
21 13.391508 192.168.22.1   192.168.22.100  TCP   8080 > 1304 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
22 13.391540 192.168.22.100 192.168.22.1    TCP   1304 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
23 13.391954 192.168.22.100 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
24 13.394582 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 407 Proxy Authentication Required 

To facilitate the analysis of the above Ethereal trace, I have splitted up the trace into four parts:

  • Frame 1 - 4: Check the DHCP Option 252.
  • Frame 5 - 6: Resolve the FQDN returned in the DHCP Option 252.
  • Frame 7 - 19: Get the configuration script file wpad.dat.
  • Frame 20 and later: Get the web page requested by the user.

Check the DHCP Option 252

In frame 1 the client request  in a DHCP INFORM packet a number of parameters from the DHCP server. Take note that this request is sent as a limited broadcast (destination IP address is 255.255.255.255). One of the parameters requested is Option 252 as shown in the figure below:

In frame 2 the DHCP server responds in a DHCP ACK packet with the requested parameters as long as they are configured on the DHCP server. In the Ethereal decode you can clearly see that the DHCP server responds in Option 252 with the location of the automatic discovery information, more precisely with the exact URL of the configuration script file as shown in the figure below:

Normally you would expect now a DNS request to resolve the FQDN returned in the DHCP Option 252 by the DHCP server. However, we see in frame 3 and 4 again a DHCP INFORM and DHCP ACK packet with exactly the same information. Moreover, frame 4 is sent by the client about a 4 seconds after the receiving of frame 2. This is happening for all Windows operating systems tested (Windows 2000 and above). So, the question is why does the client send after 4 seconds again a DHCP INFORM packet to request Option 252 although the answer was already given in frame 2?

Resolve the FQDN returned in the DHCP Option 252

Once the client has completed the DHCP part of the automatic discovery process, the client still needs to resolve the FQDN returned in the DHCP Option 252 by the DHCP server. Because we have flushed the local DNS resolver cache before starting the automatic discovery process, we can see the DNS request in frame 5 and the DNS response in frame 6 in the Ethereal trace. Although this DNS transaction completes perfectly well, you can find something very strange in the Ethereal trace:

No Time      Source         Destination     Proto Info
 4  4.037363 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0xfe7e973d
----------------------------------------------------------------------------------------
 5 13.216448 192.168.22.100 192.168.22.2    DNS   Standard query A isa.intranet.splab.net
 6 13.221231 192.168.22.2   192.168.22.100  DNS   Standard query response A 192.168.22.1

If you look very carefully to the timestamps in the Ethereal trace, then you should see that the DNS request in frame 5 is sent about a 9 seconds after the DHCP response has been received in frame 4. Again, the question is why does the client wait another 9 seconds before the client starts the DNS resolving for the FQDN returned in the DHCP ACK?

Get the configuration script file wpad.dat

At this point the client has all the information needed to start the downloading of the wpad.dat configuration script file. This download happens in frame 7 - 19 in the Ethereal trace. This is a normal HTTP GET request. A snapshot of what exactly is downloaded is shown in the figure below:

If you look at the HTTP headers in the HTTP response, you will see a parameter Cache-Control as highlighted in the figure above. The value of this parameter is max-age=3000 what means that the downloaded wpad.dat file has a time-to-live of 50 minutes in the Internet Explorer cache. After that time the cached wpad.dat file is no longer valid and will be flushed from the Internet Explorer cache. You can monitor the content of the Internet Explorer cache at the location C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files.

Another important piece of information in the configuration script file wpad.dat is the FQDN of the ISA server and the TCP port number of the Web Proxy listener. This is also highlighted in the above figure. Because the client has already resolved the FQDN of the ISA server in frame 5 - 6 when resolving the FQDN returned in the DHCP Option 252, we will not see again this action in the Ethereal trace. The client needs just to lookup the local DNS resolver cache.

Get the web page requested by the user

Now that the client has completed the automatic discovery process, the user request can finally be sent by the client to the Web Proxy listener on ISA server for further processing. This is shown in frame 20 and following in the Ethereal trace:

No Time      Source         Destination     Proto Info
20 13.382720 192.168.22.100 192.168.22.1    TCP   1304 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
21 13.391508 192.168.22.1   192.168.22.100  TCP   8080 > 1304 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
22 13.391540 192.168.22.100 192.168.22.1    TCP   1304 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
23 13.391954 192.168.22.100 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
24 13.394582 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 407 Proxy Authentication Required 

As you can see in the above excerpt of the Ethereal trace, the client makes a connection to the ISA server on TCP port 8080, that is the Web Proxy listener port configured on the ISA server.

4.2. DNS WPAD Alias

In this scenario the Automatically detect settings is checked and the Use automatic configuration script is unchecked in Internet Explorer. The DNS hostname wpad is a CNAME of isa.intranet.splab.net and no DHCP Option 252 is configured. Here is the corresponding Ethereal trace:

No Time      Source         Destination     Proto Info
 1  0.000000 192.168.22.100 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0x8d0d16ce
 2  0.008601 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0x8d0d16ce
 3  4.779755 192.168.22.100 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0x8d0d16ce
 4  4.785769 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0x8d0d16ce
---------------------------------------------------------------------------------------------
 5 13.273400 192.168.22.100 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 6 13.278922 192.168.22.2   192.168.22.100  DNS   Standard query response CNAME isa.intranet.splab.net A 192.168.22.1
---------------------------------------------------------------------------------------------------------
 7 13.283118 192.168.22.100 192.168.22.1    TCP   1267 > http [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
 8 13.298734 192.168.22.1   192.168.22.100  TCP   http > 1267 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
 9 13.298760 192.168.22.100 192.168.22.1    TCP   1267 > http [ACK] Seq=1 Ack=1 Win=65535 Len=0
10 13.299129 192.168.22.100 192.168.22.1    HTTP  GET /wpad.dat HTTP/1.1
11 13.302236 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 200 OK (application/x-ns-proxy-autoconfig)
12 13.302258 192.168.22.1   192.168.22.100  HTTP  Continuation or non-HTTP traffic
13 13.302277 192.168.22.100 192.168.22.1    TCP   1267 > http [ACK] Seq=115 Ack=2921 Win=65535 Len=0
14 13.308849 192.168.22.1   192.168.22.100  HTTP  Continuation or non-HTTP traffic
15 13.499129 192.168.22.100 192.168.22.1    TCP   1267 > http [ACK] Seq=115 Ack=2936 Win=65520 Len=0
16 13.509106 192.168.22.1   192.168.22.100  TCP   http > 1267 [FIN, ACK] Seq=2936 Ack=115 Win=65421 Len=0
17 13.509206 192.168.22.100 192.168.22.1    TCP   1267 > http [ACK] Seq=115 Ack=2937 Win=65520 Len=0
18 13.509509 192.168.22.100 192.168.22.1    TCP   1267 > http [FIN, ACK] Seq=115 Ack=2937 Win=65520 Len=0
19 13.511300 192.168.22.1   192.168.22.100  TCP   http > 1267 [ACK] Seq=2937 Ack=116 Win=65421 Len=0
-----------------------------------------------------------------------------------------------------
20 13.528225 192.168.22.100 192.168.22.2    DNS   Standard query A isa.intranet.splab.net
21 13.529485 192.168.22.2   192.168.22.100  DNS   Standard query response A 192.168.22.1
------------------------------------------------------------------------------------------------------
22 13.532855 192.168.22.100 192.168.22.1    TCP   1268 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
23 13.539081 192.168.22.1   192.168.22.100  TCP   8080 > 1268 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
24 13.539106 192.168.22.100 192.168.22.1    TCP   1268 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
25 13.539545 192.168.22.100 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
26 13.541505 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 407 Proxy Authentication Required 

To facilitate the analysis of the above Ethereal trace, I have splitted up the trace into five parts:

  • Frame 1 - 4: Check the DHCP Option 252.
  • Frame 5 - 6: Check the DNS Alias WPAD.
  • Frame 7 - 19: Get the configuration script file wpad.dat.
  • Frame 20 - 21: Resolve the FQDN returned in the configuration script file.
  • Frame 22 and later: Get the web page requested by the user.

Check the DHCP Option 252

In frame 1 the client request  in a DHCP INFORM packet a number of parameters from the DHCP server. Take note that this request is sent as a limited broadcast (destination IP address is 255.255.255.255). One of the parameters requested is Option 252 as shown in the figure below:

In frame 2 the DHCP server responds in a DHCP ACK packet with the requested parameters as long as they are configured on the DHCP server. Because in this scenario no DHCP Option 252 is configured on the DHCP server,  the DHCP server will not respond with a parameter Option 252 as shown in the figure below:

Because no DHCP Option 252 is returned by the DHCP server, you would expect that the client would try now the DNS Alias WPAD method. However, we see in frame 3 and 4 again a DHCP INFORM and DHCP ACK packet with exactly the same information. Moreover, frame 4 is sent by the client about a 4 seconds after the receiving of frame 2. This is happening for all Windows operating systems tested (Windows 2000 and above). So, the question is why does the client send after 4 seconds again a DHCP INFORM packet to request Option 252 although the answer was already given in frame 2?

Check the DNS Alias WPAD

Once the client has completed the DHCP part of the automatic discovery process and has detected that no DHCP Option 252 is returned by the DHCP server, the client needs to switch to the DNS Alias WPAD method for completing the automatic discovery process. Because we have flushed the local DNS resolver cache before starting the automatic discovery process, we can see the DNS request in frame 5 and the DNS response in frame 6 in the Ethereal trace. Although this DNS transaction completes perfectly well, you can find something very strange in the Ethereal trace:

No Time      Source         Destination     Proto Info
 4  4.785769 192.168.22.2   192.168.22.100  DHCP  DHCP ACK - Transaction ID 0x8d0d16ce
---------------------------------------------------------------------------------------------------------------------
 5 13.273400 192.168.22.100 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 6 13.278922 192.168.22.2   192.168.22.100  DNS   Standard query response CNAME isa.intranet.splab.net A 192.168.22.1

If you look very carefully to the timestamps in the Ethereal trace, then you should see that the DNS request in frame 5 is sent about a 9 seconds after the DHCP response has been received in frame 4. Again, the question is why does the client wait another 9 seconds before the client starts the DNS resolving for the the DNS Alias WPAD?

Get the configuration script file wpad.dat

At this point the client has all the information needed to start the downloading of the wpad.dat configuration script file. This download happens in frame 7 - 19 in the Ethereal trace. This is a normal HTTP GET request. A snapshot of what exactly is downloaded is shown in the figure below:

If you look at the HTTP headers in the HTTP response, you will see a parameter Cache-Control as highlighted in the figure above. The value of this parameter is max-age=3000 what means that the downloaded wpad.dat file has a time-to-live of 50 minutes in the Internet Explorer cache. After that time the cached wpad.dat file is no longer valid and will be flushed from the Internet Explorer cache. You can monitor the content of the Internet Explorer cache at the location C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files

Resolve the FQDN returned in the configuration script file

Another important piece of information in the configuration script file wpad.dat is the FQDN of the ISA server and the TCP port number of the Web Proxy listener. This is also highlighted in the figure above. Because the client has never resolved before the FQDN of the ISA server and therefore that information is not yet in the local DNS resolver cache, we can see the DNS resolving in the Ethereal trace in frame 20 - 21:

No Time      Source         Destination     Proto Info
20 13.528225 192.168.22.100 192.168.22.2    DNS   Standard query A isa.intranet.splab.net
21 13.529485 192.168.22.2   192.168.22.100  DNS   Standard query response A 192.168.22.1

Get the web page requested by the user

Now that the client has completed the automatic discovery process, the user request can finally be sent by the client to the Web Proxy listener on ISA server for further processing. This is shown in frame 22 and following in the Ethereal trace:

No Time      Source         Destination     Proto Info
22 13.532855 192.168.22.100 192.168.22.1    TCP   1268 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
23 13.539081 192.168.22.1   192.168.22.100  TCP   8080 > 1268 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
24 13.539106 192.168.22.100 192.168.22.1    TCP   1268 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
25 13.539545 192.168.22.100 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
26 13.541505 192.168.22.1   192.168.22.100  HTTP  HTTP/1.1 407 Proxy Authentication Required 

As you can see in the above excerpt of the Ethereal trace, the client makes a connection to the ISA server on TCP port 8080, that is the Web Proxy listener port configured on the ISA server.

4.3. Configuration Script

In this scenario the Automatically detect settings and the Use automatic configuration script are both checked in Internet Explorer. The Use automatic configuration script points to http://isa.intranet.splab.net:8080/array.dll?Get.Routing.Script. No DHCP Option 252 and DNS WPAD Alias are configured. Here is the corresponding Ethereal trace:

No Time      Source         Destination     Proto Info
 1  0.000000 192.168.22.118 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0x6265bf8
 2  0.003173 192.168.22.2   192.168.22.118  DHCP  DHCP ACK - Transaction ID 0x6265bf8
 3  5.058464 192.168.22.118 255.255.255.255 DHCP  DHCP Inform - Transaction ID 0x6265bf8
 4  5.059915 192.168.22.2   192.168.22.118  DHCP  DHCP ACK - Transaction ID 0x6265bf8
---------------------------------------------------------------------------------------------------------------------
 5 14.136890 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 6 14.141653 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
 7 14.141792 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 8 14.151555 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
 9 14.151713 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.splab.net
10 14.161649 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
11 14.162034 192.168.22.118 192.168.22.2    NBNS  Name query NB WPAD<00>
12 14.171614 192.168.22.2   192.168.22.118  NBNS  Name query response
13 14.191673 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>
14 14.952899 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>
15 15.704025 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>
---------------------------------------------------------------------------------------------------------------------
16 16.457322 192.168.22.118 192.168.22.2    DNS   Standard query A ISA.intranet.splab.net
17 16.465098 192.168.22.2   192.168.22.118  DNS   Standard query response A 192.168.22.1
---------------------------------------------------------------------------------------------------------------------
18 16.469153 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
19 16.475163 192.168.22.1   192.168.22.118  TCP   8080 > 1492 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
20 16.475202 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
21 16.475455 192.168.22.118 192.168.22.1    HTTP  GET /array.dll?Get.Routing.Script HTTP/1.1
22 16.479337 192.168.22.1   192.168.22.118  HTTP  HTTP/1.1 200 OK (application/x-ns-proxy-autoconfig)
23 16.479371 192.168.22.1   192.168.22.118  HTTP  Continuation or non-HTTP traffic
24 16.479398 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [ACK] Seq=150 Ack=2921 Win=65535 Len=0
25 16.485172 192.168.22.1   192.168.22.118  HTTP  Continuation or non-HTTP traffic
26 16.665549 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [ACK] Seq=150 Ack=2936 Win=65520 Len=0
27 16.675468 192.168.22.1   192.168.22.118  TCP   8080 > 1492 [FIN, ACK] Seq=2936 Ack=150 Win=65386 Len=0
28 16.675528 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [ACK] Seq=150 Ack=2937 Win=65520 Len=0
29 16.675793 192.168.22.118 192.168.22.1    TCP   1492 > 8080 [FIN, ACK] Seq=150 Ack=2937 Win=65520 Len=0
30 16.678151 192.168.22.1   192.168.22.118  TCP   8080 > 1492 [ACK] Seq=2937 Ack=151 Win=65386 Len=0
---------------------------------------------------------------------------------------------------------------------
31 16.692220 192.168.22.118 192.168.22.1    TCP   1493 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
32 16.695580 192.168.22.1   192.168.22.118  TCP   8080 > 1493 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
33 16.695613 192.168.22.118 192.168.22.1    TCP   1493 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
34 16.697202 192.168.22.118 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
35 16.699193 192.168.22.1   192.168.22.118  HTTP  HTTP/1.1 407 Proxy Authentication Required 

To facilitate the analysis of the above Ethereal trace, I have splitted up the trace into five parts:

  • Frame 1 - 4: Check the DHCP Option 252.
  • Frame 5 - 15: Check the DNS Alias WPAD.
  • Frame 16 - 17: Resolve the FQDN specified in the Automatic Configuration Script.
  • Frame 18 - 30: Get the configuration script file.
  • Frame 31 and later: Get the web page requested by the user.

Check the DHCP Option 252

In frame 1 the client request  in a DHCP INFORM packet a number of parameters from the DHCP server. Take note that this request is sent as a limited broadcast (destination IP address is 255.255.255.255). One of the parameters requested is Option 252 as shown in the figure below:

In frame 2 the DHCP server responds in a DHCP ACK packet with the requested parameters as long as they are configured on the DHCP server. Because in this scenario no DHCP Option 252 is configured on the DHCP server,  the DHCP server will not respond with a parameter Option 252 as shown in the figure below:

Because no DHCP Option 252 is returned by the DHCP server, you would expect that the client would try now the DNS Alias WPAD method. However, we see in frame 3 and 4 again a DHCP INFORM and DHCP ACK packet with exactly the same information. Moreover, frame 4 is sent by the client about a 5 seconds after the receiving of frame 2. This is happening for all Windows operating systems tested (Windows 2000 and above). So, the question is why does the client send after 5 seconds again a DHCP INFORM packet to request Option 252 although the answer was already given in frame 2?

Check the DNS Alias WPAD

Once the client has completed the DHCP part of the automatic discovery process and has detected that no DHCP Option 252 is returned by the DHCP server, the client needs to switch to the DNS Alias WPAD method for completing the automatic discovery process. Because we have flushed the local DNS resolver cache before starting the automatic discovery process, we can see the DNS request in frame 5. Because the DNS Alias WPAD is not configured in the DNS server, you will see the usual Windows name resolving behaviour for an unknown hostname including a number of Netbios name queries as last resort. This is shown in the following excerpt of the Ethereal trace:

No Time      Source         Destination     Proto Info
 5 14.136890 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 6 14.141653 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
 7 14.141792 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net
 8 14.151555 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
 9 14.151713 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.splab.net
10 14.161649 192.168.22.2   192.168.22.118  DNS   Standard query response, No such name
11 14.162034 192.168.22.118 192.168.22.2    NBNS  Name query NB WPAD<00>
12 14.171614 192.168.22.2   192.168.22.118  NBNS  Name query response
13 14.191673 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>
14 14.952899 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>
15 15.704025 192.168.22.118 192.168.22.255  NBNS  Name query NB WPAD<00>

As in the previous scenario's, you can find something very strange in the Ethereal trace:

No Time      Source         Destination     Proto Info
 4  5.059915 192.168.22.2   192.168.22.118  DHCP  DHCP ACK - Transaction ID 0x6265bf8
---------------------------------------------------------------------------------------------------------------------
 5 14.136890 192.168.22.118 192.168.22.2    DNS   Standard query A wpad.intranet.splab.net

If you look very carefully to the timestamps in the Ethereal trace, then you should see that the DNS request in frame 5 is sent about a 9 seconds after the DHCP response has been received in frame 4. Again, the question is why does the client wait another 9 seconds before the client starts the DNS resolving for the the DNS Alias WPAD?

Resolve the FQDN specified in the Automatic Configuration Script

At this point the automatic discovery process has terminated and no useful information has been detetected whatsoever. Therefore, the client (in this case Internet Explorer) will try the next automatic configuration option Use automatic configuration script that is configured with the URL http://isa.intranet.splab.net:8080/array.dll?Get.Routing.Script. Take note that if the option Automatically detect settings would not have been checked, then the first part in the Ethereal trace would have been the current frames 16 - 17 as shown below:

No Time      Source         Destination     Proto Info
16 16.457322 192.168.22.118 192.168.22.2    DNS   Standard query A ISA.intranet.splab.net
17 16.465098 192.168.22.2   192.168.22.118  DNS   Standard query response A 192.168.22.1

Get the configuration script file

The client has now all the information needed to start the downloading of the configuration script file. This download happens in frame 18 - 30 in the Ethereal trace. This is a normal HTTP GET request. A snapshot of what exactly is downloaded is shown in the figure below:

If you look at the HTTP headers in the HTTP response, you will see a parameter Cache-Control as highlighted in the figure above. The value of this parameter is max-age=3000 what means that the downloaded wpad.dat file has a time-to-live of 50 minutes in the Internet Explorer cache. After that time the cached wpad.dat file is no longer valid and will be flushed from the Internet Explorer cache. You can monitor the content of the Internet Explorer cache at the location C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files.

Another important piece of information in the configuration script file is the FQDN of the ISA server and the TCP port number of the Web Proxy listener. This is also highlighted in the figure above. Because the client has already resolved the FQDN of the ISA server in frame 16 - 17 when resolving the FQDN configured in the Use automatic configuration script, we will not see again this action in the Ethereal trace. The client needs just to lookup the local DNS resolver cache.

Get the web page requested by the user

Now that the client has downloaded the configuration script file, the user request can finally be sent by the client to the Web Proxy listener on ISA server for further processing. This is shown in frame 31 and following in the Ethereal trace:

No Time      Source         Destination     Proto Info
31 16.692220 192.168.22.118 192.168.22.1    TCP   1493 > 8080 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
32 16.695580 192.168.22.1   192.168.22.118  TCP   8080 > 1493 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
33 16.695613 192.168.22.118 192.168.22.1    TCP   1493 > 8080 [ACK] Seq=1 Ack=1 Win=65535 Len=0
34 16.697202 192.168.22.118 192.168.22.1    HTTP  GET http://www.pouseele.be/ HTTP/1.1
35 16.699193 192.168.22.1   192.168.22.118  HTTP  HTTP/1.1 407 Proxy Authentication Required  

As you can see in the above excerpt of the Ethereal trace, the client makes a connection to the ISA server on TCP port 8080, that is the Web Proxy listener port configured on the ISA server.

4.4. Firewall Client 2004

As already said before, whether the DHCP Option 252 or the DNS WPAD Alias is used, the discovery process is exactly the same for Internet Explorer and the Firewall client. However, the configuration script file downloaded is called wspad.dat. Of course the content of that file is quite different as shown in the figure below:

If you look at the HTTP headers in the HTTP response, you will see a parameter Cache-Control as highlighted in the figure above. The value of this parameter is max-age=0 what means that the downloaded wspad.dat file should never be cached by the client. However, if you look somewhat further down into the wspad.dat file then you should see in the [Common] section a parameter Configuration Refresh Time. This parameter tells the Firewall client to reload the wspad.dat file every 6 hours from the ISA server. Of course you can enforce an immediate configuration refresh by clicking the Detect Now button in the Firewall client (General tab).

An interesting feature of the Firewall client 2004 is the capability to configure the LAN connection settings of Internet Explorer. These settings are centrally configured on the ISA server 2004 (Configuration > Networks > Properties > Firewall Client) and downloaded by the Firewall client 2004 as part of the wspad.dat configuration file. In the above figure you can find these configuration settings in the [Common] section of the wspad.dat file. From our experience we know that these configuration settings are automatically applied to Internet Explorer in the following cases:

  • immediately after installing the Firewall client.
  • whenever the Firewall client becomes active. That means when logging on to Windows or when the Firewall client changes from the disabled to the enabled state.
  • whenever the Firewall client configuration file wspad.dat is refreshed, either by pushing the Detect Now button in the Firewall client (General tab) or every 6 hours (Configuration Refresh Time).
  • whenever you push the Configure Now button in the Firewall client (Web Browser tab).

Needless to say that this can be a very handy feature to automatically configure Internet Explorer with the correct LAN connection settings.

5. Round-up

When you enable in Internet Explorer the options Automatically detect settings, Use automatic configuration script and Proxy server, then the processing order is as follows:

  • DHCP Option 252 (Automatically detect settings)
  • DNS WPAD Alias (Automatically detect settings)
  • Configuration script (Use automatic configuration script)
  • Manual configuration (Proxy server)

Note: the DHCP Option 252  is only supported for all users in Windows XP SP2 and Windows 2003 SP1 or later. The DNS WPAD Alias is supported for all users in Windows 2000 or later.

For the Firewall Client 2004 the available options are only Automatically detect ISA Server or Manual select ISA Server. Moreover these options are mutual exclusive.

Another difference between the Firewall client 2004 and Internet Explorer is that in Internet Explorer the configuration file (wpad.dat or array.dll?Get.Routing.Script) is cached with a default time-to-live of 50 minutes. That means that any new Web Browser session will first check the local cache for the configuration file. Therefore, Internet Explorer will only download the configuration file from the ISA server when that object has expired from or is not available in the local cache. This is something you must keep in mind when you want to take a network monitor trace of the Internet Explorer startup behaviour.

One of the biggest mysteries found in exploring the different automatic configuration options is the fact that excessive delays occure in the DHCP part of the automatic discovery process. We have measured delays up to 15 seconds regardless if the DHCP server was configured with Option 252 or not. This happens as well for the Firewall client 2004 as for Internet Explorer. However, for the Firewall client these delays are not so visible for the end user because they normally occure only during the Windows logon process and are therefore somewhat concealed. For Internet Explorer this is not the case because they will occure when the first Web page is requested.

To complicate the matter, neither the Firewall client nor Internet Explorer will always do a full automatic discovery process, that means including the DHCP part. The reason for it is that the DHCP parameters returned by the DHCP server are cached on an adapter base at the client. From our observations we know that a DHCP INFORM will be sent when:

  • The TCP/IP settings changes on the adapter (e.g. new IP address). Mobile users are here a case in point or when you have to work with a rather short DHCP lease.
  • A new user logs on to Windows. This scenario is typical seen in a computer classroom environment.
  • The setting Automatically detect changes from off to on.

Therefore we think that to mitigate the frustrations for the end user it is probably wise to avoid as much as possible a full automatic discovery process by not enabling Automatically detect settings in Internet Explorer. Whenever possible, let the Firewall client 2004 automatically configure Internet Explorer to use a configuration script or use a Group Policy to accomplish the same. This implies of course that those hosts are managed clients. For unmanaged clients there is no optimal solution. Either they configure Internet Explorer manually for Automatically detect settings and accept the possible delays or they configure Internet Explorer manually for Use automatic configuration script (e.g. http://wpad/wpad.dat). In both cases, make sure that the primary DNS suffix is set correctly.

6. Conclusion

In this article we have explored in great detail how the ISA Server 2004 Web Proxy and Firewall Client Automatic Configuration really works from a client point of view. With that knowledge you should be able to decide which method is the most appropriate for your specific environment. We have also discovered excessive delays in the DHCP part of the automatic discovery process and that can be considered as a bug. We have logged a ticket with Microsoft PSS for this issue. If we got a fix for it, we will post a message in the related message board topic.

I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=35;t=000166 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks!  – Stefaan.

See Also


The Author — Stefaan Pouseele

Stefaan Pouseele is a network engineer, working for Cevi NV in Belgium. On October 1, 2002 he received for the first time the prestigious Microsoft Most Valuable Professional (MVP) Award 2003 for his contribution to the ISA Server online community. To this very day, his award period was extended to 2008.

Advertisement

Featured Links