Microsoft Internet Security
and Acceleration Server 2000 SharePoint Portal
Server Deployment Kit
Chapter 10
Configuring Web Proxy Clients for Direct Access to
Intranet Resources

Martin Grasdal
Dr. Thomas W Shinder
December
2003
Table of Contents
Overview
of ISA Server 2000 Clients
Name
Resolution for ISA Server 2000 Clients.
Step-by-Step
How To: Manually Configuring Web Proxy Clients for Direct Access
Step-by-Step
How To: Automating the Delivery of Local Domain Table Information to Web Proxy
Clients
Configuring
Local Domain Table on ISA Server 2000.
Configuring
Web Proxy Clients with Location of Configuration Script
SharePoint
Portal Server 2003 Web Proxy Client Configuration
Step-by-Step
How To: Configuring Web Proxy Client
Settings for the SharePoint Search Service
Step-by-Step
How To: Configuring Web Proxy Client Settings for SharePoint Web Parts
Step-by-Step
How To: Controlling Outbound HTTP(S) Access for Web Proxy and SecureNAT Clients
Configuring
Outbound Web Requests Listener
Configuring
HTTP Redirector for Unauthenticated SecureNAT Client Access
Results
of Configuration Change
Implementing Web Proxy clients allows you to exercise greater control over outbound access and to provide more detailed information in the Web Proxy logs. Web Proxy clients can be configured so that they will bypass the Web Proxy service whenever they need to contact local hosts that use FQDNs or IP addresses. By being able to bypass the Web Proxy service for any local host, you can reduce the consumption of unnecessary resources on the ISA Server. As a pre-requisite for bypassing the Web Proxy service, it is important that DNS name resolution is working properly on the internal network. SharePoint Portal Server 2003 has its own Web Proxy client configurations that are separate from the Web Proxy client configurations in Internet Explorer. It is possible to use separate Web Proxy configurations for the SharePoint search service and Web Parts, such that, for example, the search service uses a Web Proxy client configuration and Web Parts use a SecureNAT configuration.
This chapter provided an overview of Web Proxy clients and
shows you via step by step instructions on how to configure Web Proxy clients
for both Internet Explorer and SharePoint Portal Server 2003.
Web Proxy and
Firewall clients communicate with the ISA Server 2000 firewall whenever they
need access to external resources. These requests are mediated by the Web Proxy
and Firewall services on the ISA Server 2000 firewall. However, what resources
are considered external and what resources are considered internal is dependent
on the type of request and the ISA Server 2000 firewall and client
configuration.
For example, if a
Web Proxy client issues a request using a name that contains periods (.), it
will potentially send the request through the firewall, even if the resource is
located on the internal network and the Web Proxy client can contact a DNS
server capable of resolving the name to an internal IP address. Although the
client may be able to connect to the internal resource when it is looped back
through the ISA Server 2000 firewall, such a configuration needlessly consumes
firewall resources.
This behavior can
be prevented by configuring the ISA Server 2000 firewall and client
configurations properly so that a request for a particular FQDN is forwarded
directly to the internal network Web server and not mediated by the ISA Server
2000 Web Proxy service. We want to
configure the ISA Server 2000 firewall and internal network clients so that all
requests for internal resources bypass the ISA Server 2000 firewall and are
forwarded directly to the internal network server.
This chapter
explains how Web Proxy clients work and interact with the ISA Server. It further explains how to configure the ISA
Server and Web Proxy clients to bypass the ISA Server when connecting to
internal resources. Also, this chapter
covers details for configuring the Web Proxy settings for use by SharePoint
services, in particular the search service and Web Parts. Finally, the chapter provides a demonstration
of using Web Proxy client and ISA Server configuration to control outbound
HTTP(S) access by user name and group membership.
There are three client types that can be used with ISA Server 2000: SecureNAT, Firewall, and Web Proxy client.
·
A SecureNAT client is one configured with the
internal interface of the ISA Server 2000 firewall as its default gateway (in
more complex environments, the default gateway of the SecureNAT client is a
router interface that will route packets to the internal interface of the ISA
Server 2000 firewall.) A SecureNAT
client does not pass any authentication information to the firewall, and
therefore outbound access can only controlled by client IP address and not user
name. Furthermore, for a SecureNAT
client to use a particular protocol, an isa2d Protocol Definition must exist
for it.
·
A Firewall client is one on which the ISA Server
2000 Firewall client software is installed. The Firewall client operates on TCP
and UDP protocols. When a Firewall
client needs access to an external resource, it will use a special version of
the Winsock DLL to enable the mediation of the Winsock connection through ISA
Server 2000 firewall. The Firewall client works in conjunction with the Local
Address Table and Local Domain Table configurations and determines what version
of the Winsock DLL to use when accessing either an internal or external network
resource. A key advantage of the Firewall client is that it passes
authentication information to the ISA Server 2000 firewall, so that user and
group names can be logged and used to enforce outbound access policies. Another
advantage of the Firewall client is that, in contrast to a SecureNAT client, it
can use almost any TCP or UDP protocol, even if a protocol definition does not
exist for it.
·
A Web Proxy client is one that sends HTTP,
HTTPS, and FTP (when using the Web browser) requests for external resources to
an ISA Server 2000 firewall that is configured in the proxy settings of the Web
browser. The ISA Server processes requests
for Web Proxy clients through the Web Proxy service. When a Web Proxy client makes a request for
an external Web based resource, it establishes a session with the ISA Server
2000 firewall, which in turn establishes another session with the external resource.
The Web Proxy client does not communicate directly with the external resource;
the connection is proxied through the Web proxy service. An advantage of Web Proxy clients is that
they can take advantage of the ISA Web Proxy cache for faster access to Web
resources on the external network. Furthermore, Web Proxy clients can pass
authentication information to the ISA Server 2000 firewall for logging and
access control.
It is important to note that a single computer can be configured as a SecureNAT, Firewall, and Web Proxy client. It is the type of request that determines the client type in use at any particular moment. For example, when a client configured as both a SecureNAT and Web Proxy client makes a request for an HTTP resource, it will use the Web Proxy client configuration to communicate with the ISA Server 2000 firewall. However, if the same client makes a request for an external SMTP resource, it will make the request as a SecureNAT client.
The HTTP Redirector application filter extends the performance advantages of the Web Proxy cache to firewall and SecureNAT clients. By default, the HTTP Redirector filter passes any request for Web resources through the Web Proxy service. Therefore, Firewall and SecureNAT client requests for Web resources are processed through the Web Proxy service, even if these clients are not configured as Web Proxy clients.
The advantage of using the HTTP Redirector filter is that is allows SecureNAT and Firewall client computers that do not have their browsers configured to take advantage of the Web Proxy cache. The drawback is that SecureNAT clients cannot take full advantage of the Web Proxy client functionality, such as user/group based access control.
When Firewall and SecureNAT client Web requests are redirected to the Web Proxy service, any authentication credentials they may be using are not passed along with the request. This issue isn’t so important for the SecureNAT client because the SecureNAT client never sends credentials to the ISA Server 2000 firewall. However, the Firewall client always sends user credentials to the ISA Server 2000 firewall. This behavior has consequences if we have configured the Outgoing Web Requests listener to require authentication. If this is the case, the connection will fail regardless of any Protocol Rule or Site and Content Rule that is configured to allow anonymous access. You must either configure the SecureNAT and Firewall clients as Web Proxy clients, remove the requirement to authenticate outgoing Web requests, or make changes to the HTTP redirector to avoid this problem (we will look at this last point later on in this chapter).
SecureNAT clients are always responsible for performing name resolution to connect to both internal and external resources. In contrast, Firewall and Web Proxy clients perform name only if the requested resource is considered to be local. If the requested resource is considered to be external, the ISA Server 2000 firewall performs DNS resolution on behalf of the client.
When the ISA Server 2000 firewall performs DNS name resolution on behalf of a client, it will cache the result for a default period of 6 hours. This 6 hour TTL overrides the value specified in the TTL for the Resource Record returned by the DNS server. While extending or overriding the TTL of a resource record returned by the DNS server can optimize name resolution traffic, it also means that a significant problems can arise when IP addresses change more quickly than every six hours.
Whether resources are considered local or remote is a matter of definition and configuration. On an ISA Server 2000 firewall, the Local Address Table (LAT) defines the local network IDs. Firewall clients periodically connect to the ISA Server 2000 firewall and download the msplat.txt file that contains the LAT.
When a user enters an IP address to connect to a resource, this file is consulted to determine whether or not the connection should go through the ISA Server. However, when the user enters an FQDN (or a name that contains periods) to connect to a resource, the Firewall client consults Local Domain Table (LDT) entries in the mspclnt.ini file, which the client downloads at regular intervals from the ISA Server, to determine whether the request should be processed by the ISA Server. As with LAT entries, LDT entries are configured on the ISA Server 2000 firewall.
Web Proxy clients do not use the same mechanisms as Firewall clients to determine what resources are internal and what resources are external. Web Proxy clients do not have access to the msplat.txt or mspclnt.ini file to determine whether resources are internal or external. By default, a Web Proxy client communicates directly with a resource when a user enters an unqualified (single label) name that does not contain periods, in the browser.
For example, a user wants to connect to an internal SharePoint web site with a FQDN of sps.internal.net. If the user enters http://sps in the browser configured as a Web Proxy client, the resource will be treated as internal. However, if the user enters http://sps.internal.net in the browser, the Web Proxy client considers this an external request and sends it to the ISA Server 2000 firewall. In complex environments where a number of different domain namespaces are used on the internal network and FQDNs are used to connect to internal resources, it is necessary to ensure that requests for these resources are not processed by the ISA Server 2000 firewall.
Web Proxy clients must either be manually configured with IP addresses and domains that are considered local or be configured to use the autoconfiguration script. The autoconfiguration script contains LDT settings for local resources. Web Proxy clients connect directly to internal resources when the list of internal network domains is included in the autoconfiguration script.
Figure 1 Internet Explorer Web Proxy Settings

Figure 2 Configuring Exceptions List for Direct Access

In the example above, the exceptions list is configured so that whenever a user enters a request to connect to Web servers in the example.com or contoso.com domains, it will connect directly to these Web servers and bypass the Web Proxy service on the ISA Server 2000 firewall. Notice that an IP address or collection of IP addresses can be configured as internal resources.
While it may be appropriate to configure Web Proxy clients manually in small environments, it is not practical to manually configure Web Proxy clients in large or dynamic environments. Web Proxy clients can be configured to receive the Local Domain Table (LDT) information via the autoconfiguration script in order to automate delivery of direct access information. The autoconfiguration script is a JScript file. Automating the delivery of LDT information to Web Proxy clients requires configuration changes on both the ISA Server 2000 firewall and the Web Proxy client.
LDT settings are configured on the ISA Server 2000 firewall and delivered to clients via the autoconfiguration script. The default location of the autoconfiguration script is http://Computer_name/array.dll?Get.Routing.Script and is downloaded every time a Web Proxy client browser is opened. The first step in automating the delivery of this information is to configure the LDT on the ISA Server 2000 firewall computer.
Figure 3 Client Configuration Node

Figure 4 Direct Access Tab

Figure 5 Add/Edit Server LDT Entry

Figure 6 LDT Entries

The next step is to configure the Web Proxy clients to download the autoconfiguration script that contains these entries. There are two ways to do this: manually enter the location of the autoconfiguration script in the Web Proxy client configuration or configure the Web Proxy client to use automatic discovery to download the script.
Perform the following steps to configure the Web Proxy client browser to automatically download the autoconfiguration script:
Figure 7 Automatic Configuration of Web Proxy Clients

3. Click OK.
Another way for the client to receive the script is to configure the Web Proxy client to Automatically detect settings. When this setting is configured, Web Proxy clients can query a DHCP or DNS server to find the location of the wpad.dat file that contains the Web Proxy client settings. Only Windows 2000 and higher DHCP clients can make use of a special DHCP option containing information on the location of the wpad.dat file. Also, autodiscovery works only for Internet Explorer 5.0 or higher.
By default, the ISA Server does not enable the wpad.dat file for delivery to clients that are configured to use autodiscovery. Perform the following steps to enable publication of the wpad.dat file on the ISA Server,
Figure 8 Auto Discovery Tab

After you have enabled publishing of the wpad.dat file, you will have to decide whether you want clients to locate the file by means of a DHCP option (option 252) or a DNS entry. The advantage of configuring DNS to provide wpad information to the Web Proxy clients is that all users can take advantage of the autoconfiguration script. Only administrators can use the DHCP option to obtain the autoconfiguration script location via DHCP.
The figure below shows part of the contents of the Wpad.dat file after the LDT has been configured to bypass the proxy server for the example.com and the 10.0.0.0/24 network.
Figure 9 Contents of Wpad.dat file
Note:
If you are using URLScan 2.5 as part of Feature Pack 1 for ISA Server 2000, you
should be aware that by default URLScan blocks files with multiple periods and a .dat extension. This
means that URLScan will potentially block the delivery of the wpad.dat or the
routing script file from the ISA Server.
The URLScan log files will indicate whether either of these files is
being blocked. However, this should not
be a problem if the LAT is configured properly, as URLScan only analyzes
connections from untrusted networks. Please see Chapter 4, Configuring URLScan 2.5 to
Protect Published SharePoint Web Sites, for details on resolving
this issue.
In configuring your Web Proxy clients for direct access in your environment, you should also keep in mind that the SharePoint Portal Server can also be configured as Web Proxy client for communication with the ISA Server. And, just as with Internet Explorer, the SharePoint Portal Server has to be configured to communicate appropriately with local and remote hosts. The next section discusses Web Proxy client issues for the SharePoint Portal Server.
One of the powerful features of SharePoint Portal Server is its ability to crawl content from external content sources, such as Web sites, other SharePoint Web sites, and Exchange Servers, and index it. When the SharePoint search service crawls external content for the purpose of indexing it, it interacts with the ISA Server 2000 Web Proxy service.
Furthermore, SharePoint Server can make use of Web parts that are located in Microsoft’s online Web part gallery. These Web parts may make use of a data retrieval service (Web service) that allows the retrieval and manipulation of data from and against remote data source through the Simple Object Access Protocol (SOAP) and XML. Access to remote Web parts and Web services also occurs by default through the Web Proxy service of the ISA Server 2000 firewall.
At a minimum, the published SharePoint Server is set up as a SecureNAT client. (It is not recommended that the Firewall client be installed on published servers.) However, if the SharePoint Server is crawling external content or using Web parts from remote online galleries, it may be desirable to configure the SharePoint server as a Web Proxy service.
For example, if the security policies of the organization require that all outbound HTTP(S) access be controlled and logged according to user name, Protocol and/or Site and Content rules be put in place that allow outbound access according to user name or group membership. Because SecureNAT clients can not pass authentication information to the Web Proxy service, the published SharePoint server must be set up as a Web Proxy client.
The Web Proxy settings for the SharePoint search service and Web parts are separate from the Web Proxy client settings in Internet Explorer. The Web Proxy client settings for the search service and Web parts are separate and distinct. This means that potentially 3 different Web Proxy client settings will have to be configured on the SharePoint server to enable it to interact properly with the local and remote Web resources.
The following sections discuss and demonstrate the configuration of the Web Proxy client settings for the SharePoint search service and Web parts.
When SharePoint Portal Server 2003 crawls external content to create full-text indexes, it does so in the security context of the default content access account. This account must have at least read permissions to the crawled data. Furthermore, the account must have permissions to access the Web Proxy service on the ISA Server 2000 firewall. The SharePoint server will try to authenticate by first using NTLM and then basic authentication.
By default, the SharePoint search service uses the Web Proxy client settings in Internet Explorer of the default content access account. Depending on the account used, the configuration of Internet Explorer for the account, and the configuration of ISA Server 2000 firewall, no additional configuration of the Web Proxy settings for the search service may be necessary.
However, in many circumstances it will be both necessary and desirable to change the default access account and the Web Proxy client settings for the search service. For example, if no default content access account has been specified, SharePoint server will use the anonymous account to crawl external data.
To change the default access account and Web Proxy client settings for the search service on the SharePoint Server,
Figure 10 SharePoint Central Administration Page

Figure 11 Manage Search Settings Page

Figure 12 Configure Server Farm Account Settings Page

This completes the configuration of the default access account and the proxy server settings. You should note that you can override the default access account for crawling content by specifying a different account in site restriction and site path rules for individual content sources. However, configuring accounts to override the default access accounts for specific content sources is beyond the scope of this document. Please refer to the SharePoint Server Help files for more information on configuring accounts to crawl specific content sources.
Web Parts are an important feature of SharePoint Portal 2003 Server. A Web Part is a modular, reusable, and customizable object that has a single function or purpose and forms the basis for Web part pages. A Web Part consists of two files: description file and an assemble file. The description file has a .dwp extension and contains XML code specifying property names, settings, and a reference to its companion assembly .dll file. The assembly file is an ASP.NET dynamic link library (.dll) file that contains code to perform the function of the Web Part. Multiple Web Parts can be connected on Web Part pages to display and manipulate information from related but dispersed content sources.
In some cases, it may be desirable to use Web Parts provided by external sources. For example, Microsoft provides an online gallery of MSNBC Web Parts that provides information on news, weather, sports, and financial markets. If you wish to use Web Parts from external network clients, it may be necessary to configure Web Proxy client settings to access these Web parts.
The Web Proxy client settings for Web Parts are separate from the settings for Internet Explorer and the SharePoint search service. The figure below shows a typical error message that occurs when the Web Proxy client settings are absent or misconfigured for a given ISA Server 2000 firewall configuration. Note that while it is possible to use a Web Part that has previously been imported (the MSNBC Weather Web Part), it is not possible to search the online gallery for additional Web Parts.
Figure 13 Failure to Open Online Gallery as a Result of Web Proxy Client Settings

To allow access to the online gallery, it is necessary to create Web Proxy client settings for the Web Part. There is no graphical interface for doing this, as is the case with other Web Proxy client settings. To enable Web Proxy client settings for Web parts, you must modify the XML-based web.config file that is found in the root directory of the virtual server(s). Specifically, you must add a system.net element that contains the proxy settings.
The system.net element is responsible proxy settings, authentication modules, connection management, and Internet request modules. The default configuration for the system.net element is listed below:
<configuration>
<system.net>
<authenticationModules>
<add type =
"System.Net.DigestClient" />
<add type =
"System.Net.NegotiateClient" />
<add type =
"System.Net.KerberosClient" />
<add type =
"System.Net.NtlmClient" />
<add type =
"System.Net.BasicClient" />
</authenticationModules>
<connectionManagement>
<add address = "*"
maxconnection = "2" />
</connectionManagement>
<defaultProxy>
<proxy
usesystemdefault = "true"
bypassonlocal = "true"
/>
</defaultProxy>
<webRequestModules>
<add prefix = "http"
type =
"System.Net.HttpRequestCreator"
/>
<add prefix = "https"
type =
"System.Net.HttpRequestCreator"
/>
<add prefix = "file"
type =
"System.Net.FileWebRequestCreator"
/>
</webRequestModules>
</system.net>
</configuration>
To configure the Web Proxy client settings, it is necessary to specify settings in the defaultProxy element. This element consists of 3 child elements, the <proxy> element to specify the proxy server, the <bypasslist> element to list domain names and IP addresses that should bypass the proxy server, and the <module> element to add additional proxy modules.
The following is an example usage of the <defaultProxy> element:
<system.net>
<defaultProxy>
<proxy
proxyaddress="http://172.16.1.1:8080" bypassonlocal =
"true"/>
<bypasslist>
<add
address="[a-z]+\.example\.com" />
<add address="192.\.168\..*"
/>
</bypasslist>
</defaultProxy>
</system.net>
In the above example, the Web service uses 172.16.1.1:8080 as the proxy server and bypass the proxy server for any local addresses, hosts in the example.com domain, and IP address beginning with 192.168.
Note:
Be very careful about making modifications to the Web.config file. In particular, you should make a backup of
the file before modify it. Also, XML is
very sensitive to syntax, so double check the entries you make.
Perform the following steps to modify the <defaultProxy> element:
The figure below shows a modified web.config file.
Figure 14 Web.config file with System.net Element

Note that no authentication settings have been configured. In instances where the Web proxy service requires authentication for outgoing web requests, this particular configuration will not work. Specifically, the error message that gets returned is this:
The request
failed with HTTP status 407: Proxy Authentication Required (The ISA Server
requires authorization to fulfill the request. Access to the Web Proxy service
is denied.).
However, ASP.NET is a powerful and flexible platform, and it should be possible (in principle) to create custom code to reply to the 407 status message returned by the ISA Server. A good starting point for further investigation of this issue is .Net Framework Web site on the Microsoft MSDN Web site.
Note:
A workaround for the authentication issue is to create an access rule for the
Sharepoint Portal Server. Do not
configure the Outgoing Web Requests listener to require authentication. Then
create a Site and Content Rule and Protocol Rule that allows outbound access to
HTTP/HTTPS for a client address set that contains the IP address of the
Sharepoint Server.
You should be aware that when you do not configure the <defaultProxy> element, access to remote Web Parts occurs in the context of a SecureNAT client. This fact can be leveraged to provide access to remote Web parts, even when authentication is required for Web Proxy clients. The next section shows you a possible way to leverage this kind of configuration.
Warning:
Web Parts contain code. Therefore, you should make sure that Web Parts come
only from trusted sources.
One of the advantages of using Web Proxy clients is that it is possible to control and log access through the Web Proxy service by username or group membership. This section shows you how to control outbound HTTP(S) for both Web Proxy and SecureNAT clients.
The scenario is as follows:
· Whenever possible, we want to limit outbound HTTP(S) access to authenticated users.
· When it is not possible to authenticate outbound HTTP(S) requests, we want to limit outbound HTTP(S) access to specific client address sets. For example, if we have no mechanism for authenticating outbound access for Web Parts, we want to ensure that these requests will go through the ISA Server 2000 firewall.
· We want to limit outbound access for a small number of other protocols (DNS and SMTP) according to client address sets.
There are a number of ways to limit outbound access through the ISA Server 2000 firewall. For example, we could use Site and Content Rules or Protocol Rules. In this demonstration, we are will use Protocol Rules and the Outbound Web Requests Listener to limit outbound access.
The general steps for creating this configuration are as follows:
One way to force authentication and obtain user information in the Web Proxy logs is to force authentication at the Outgoing Web Requests listener to.
Perform the following steps to force authentication at the Outgoing Web Requests listener: