URLScan 2.5 on ISA Server 2000 enhances the defensive capabilities of ISA Server 2000. Installing URLScan on ISA Server 2000 firewalls prevents suspicious HTTP requests from entering the internal network and mitigates threats to internal web servers. URLScan is highly configurable through the URLScan.ini file, which can be easily customized to meet a wide range of security and accessibility requirements. Consequently, when URLScan is installed on ISA Server, it plays a crucial role in a comprehensive defense-in-depth strategy that is designed to protect mission-critical applications such as SharePoint Portal Server 2003. This document explains how URLScan for ISA Server 2000 works and how to configure URLScan to protect your SharePoint Portal Server 2003 Web sites.

URLScan 2.5 is a powerful tool for protecting Web sites from invalid or deliberately malformed HTTP requests that can exploit potential weaknesses on Web sites. This document discusses how to install and configure URLScan 2.5 on ISA Server 2000 to protect SharePoint Portal Server 2003 and other web sites.

Because the configuration of URLScan will be, to some extent, unique to each environment, this document attempts to provide guidelines for implementing URLScan configurations, rather than providing a step by step set of configurations for given environments. The first part of this document provides an overview of URLScan and the attacks it prevents. The next part provides step-by-step instructions for installing and configuring URLScan on ISA Server. The last part provides information and guidance for troubleshooting and fine tuning URLScan configurations.

Overview of URLScan 2.5

URLScan 2.5 for ISA Server installs as a Web application filter that inspects HTTP traffic for invalid or suspicious requests and allows or denies these requests on the basis of configurable settings. The advantage of using URLScan 2.5 on ISA Server 2000 is that all HTTP traffic can be statefully inspected at the application layer and suspicious traffic blocked before it reaches the protected network.

As an example of suspicious traffic that URLScan 2.5 can protect against, imagine an attacker who creates and sends an HTTP request that appends a number of “..” characters in combination with “\” or “/” characters to a request for a virtual directory. Further imagine that the requested virtual directory is one likely to contain executable files or permit files to be executed. Such an exploit is illustrated by the following log entry of an attempted Nimda attack:

GET /scripts/..\xc1\x9c../winnt/system32/cmd.exe?/c+dir

This invalid request represents a directory traversal attack in which the attacker attempts to break out from the Web root directory into the file system and execute a command. When Web servers are protected against directory traversal attacks by disallowing HTTP requests that have too many dots or slashes, attackers may attempt to get around this protection by substituting Unicode characters for the slashes (this vulnerability is known as the Unicode translation vulnerability). Attackers can also send excessively long URLs in order to exploit potential vulnerabilities in unchecked buffers (this is a buffer overflow attack). Hackers may also attempt attacks that exploit functionality of HTTP 1.1 verbs (also known as HTTP methods or commands), such as PUT or DELETE, to alter files in virtual directories.

To defend against these and similar attacks, one should ensure that IIS has as small an attack surface as possible by applying the most recent security patches. You should also implement the suggestions and procedures outlined in the security configuration guidelines document, and/or upgrade to Windows 2003 and IIS 6.0, which provides significantly enhanced default security over previous versions of IIS.

You can significantly security for your published Web sites, such as installing URLScan 2.5 on an ISA Server 2000 firewall to prevent invalid or suspect URLs from being forwarded to internal Web servers in the first place. This is true even when those servers are using IIS 6.0 and have been hardened against HTTP-request exploits using the IIS version of URLScan and other methods. Please see the TechNet document on URLScan 2.5 for IIS for a comparison of the features of URLScan 2.5 and the built-in security features of IIS 6.0.

URLScan 2.5 for ISA Server 2000 is part of ISA Server 2000 Feature Pack 1, which is available as a free download. URLScan 2.5 installs as a high priority Web application filter on ISA Server. It is similar in functionality to URLScan 2.5 for Web servers, which is available as part of the IIS Lockdown tool or as a separate download.

However, some entries in the URLScan.ini file have no effect on URLScan processing on ISA Server 2000. For example, because URLScan 2.5 for ISA Server installs as a Web application filter that causes processing to occur immediately in the order specified in ISA Management console, enabling the Allowlatescanning option in the URLScan.ini has no effect. Also, because ISA Server 2000 Destination Sets used in combination with Web Publishing rules already mitigate directory traversal attacks, some options in the URLScan.ini will be superfluous. That said, these options should be configured as part of a comprehensive defense-in-depth strategy. Furthermore, URLScan 2.5 provides protection against directory traversal attacks for Web sites that are published through Server Publishing rules and lack this native level of protection afforded by Web Publishing rules.

URLScan 2.5 is configured through entries in a URLScan.ini file. Two different versions of the URLScan.ini are included in URLScan 2.5 for ISA Server 2000: URLScan_owa.ini and URLScan_iis.ini. These files are used as templates for the URLScan.ini file settings applied to publishing Outlook Web Access and Web sites respectively. The main difference between these two template files is their level of protection.

The settings in the URLScan_iis.ini file are more restrictive than the settings in the URLScan_owa.ini file. In particular, the URLScan_owa.ini allows a greater range of HTTP 1.1 verbs (methods) to support the WebDav methods required for Outlook Web Access. Using the URLScan_iis.ini file as the template for the URLScan.ini file causes Outlook Web Access to break. However, either file can be used as a template for the URLScan.ini file to secure a published SharePoint Web site. If you are publishing only a SharePoint Web site, you can use the more restrictive URLScan_iis.ini file as the template and relax or tighten its settings as necessary in your environment.

Note:
There can only be only one URLScan 2.5 configuration for all the published Web servers on the ISA Server. If different published Web sites have different security requirements, you will have to use a more general and relaxed URLScan configuration on the ISA Server and install more specific configurations of URLScan on the Web servers themselves.

You will likely need to customize the URLScan file, depending on your security requirements, to ensure both security and functionality of the published SharePoint Web site because there is no specific version of a URLScan.ini file for SharePoint portal services. However, the good news is SharePoint Portal Server 2003 runs on IIS 6.0 and is built on the .NET Framework. IIS 6.0 has much greater built-in security than previous versions of IIS. Consequently, IIS 6.0 provides a much smaller attack surface. This means a restrictive URLScan configuration is less likely to break a SharePoint Portal Server 2003 Web site, while still enabling users to perform sophisticated tasks.

Unlike the previous version of SharePoint Server, SharePoint Portal Server 2003 does not require WebDAV support on the Web server hosting the SharePoint site. One improvement of IIS 6.0 over IIS 5.0 is that you can “turn off” WebDAV, something not possible in IIS 5.0. Even without WebDAV installed on the Web server, users can connect to Web folders and manipulate files in these folders just as they could with the previous version of SharePoint server. Interestingly, this means users can still add, modify, and delete files in a SharePoint Web folder, even though URLScan may be blocking HTTP 1.1 and WebDAV verbs commonly used for these purposes, such as PUT or DELETE. Furthermore, an examination of the IIS log files reveal only a limited number of HTTP methods, primarily GET and POST, are used by external clients to perform tasks on SharePoint Server.

Tip:
To help determine optimal URLScan configurations, you should test external clients accessing the SharePoint site through ISA Server and then examine the IIS logs.

URLScan 2.5 Settings

The URLScan.ini file contains a number of sections where various settings are configured. These sections and the various setting appropriate to each are summarized below. It should be noted that a number of these settings have no effect when used in combination with the built-in security of IIS 6.0. That is, the built-in security of IIS 6.0 already protects against many of the same attacks as URLScan 2.5. However, the goal here is to prevent invalid HTTP requests from being forwarded to the internal network in the first place.

Note:
Improper settings in the URLScan.ini file will cause problems with published SharePoint and other Web sites. It is therefore important that you configure and verify appropriate URLScan.ini settings in a test lab before deploying URLScan on a production ISA Server.

The following is a detailed description and explanation of the URLscan.ini file settings.

[Options] Section

This [Options] section contains most of the primary settings for URLScan functionality. Entries in this section have the format OptionName=OptionValue. Below are the various URLScan options and their default values:

· AllowDotInPath=0
When this setting is 0, URLs containing more than one period (.) are not allowed. This protects against certain HTTP-request exploits that attempt to fool the Web server into processing a request for an otherwise blocked file extension. For example, the HTTP header contains a request for something like /NotAllowedExtension.asp/AllowedExtension.htm. This might be interpreted as a request for a file with an allowed .htm extension, which is passed to the PATH_INFO variable, when in fact it is a request for a blocked file extension (in this case, a .config extension). If URLScan is configured to block .asp files, but allow .htm files, this URL will be allowed if AllowDotInPath is set to 1. If this value is set to 0, directory names in the request path must not contain any periods, and the URL must not contain more than one period. It should be noted that URLScan always recognizes the first .exe, .dll, or .com extensions it sees in the request path.

· AllowHighBitCharacters=0
When this setting is 0, characters outside of the standard ASCII character set are not allowed. This mitigates weaknesses such as the Unicode translation vulnerability whereby hackers substitute Unicode characters for slashes in order to initiate a directory traversal attack. Disallowing high bit characters will limit request paths to those that use only ASCII characters and will consequently limit the number of allowable language-specific character sets.

· AlternateServerName=(not specified by default)
This setting is used as a masquerading/obscurity technique of somewhat limited value to impersonate another Web server type in the HTTP Server header responses. For example, the setting could be used to give the impression that the IIS Web server was an Apache Web server. The functionality of this setting depends on the setting for the RemoveServerHeader option.

· RemoveServerHeader=0
When this setting is 1, URLScan will remove the HTTP Server header from outgoing responses.

· EnableLogging=1
When this setting is 1, URLScan will log rejected URLs along with the reason for rejection in a file named URLScan.log that is created in the same directory containing the URLScan.dll file.

· LoggingDirectory=(not specified by default)
Specifies an alternate directory for the collection of log files. By default, URLScan log files are created in the same directory where the URLScan.dll is located.

· LogLongURLs=0
When this setting is 0, only the first 1024 bits of a URL are logged. When set to 1, a maximum of 128 kilobytes for each URL are logged.

· PerDayLogging=1
When set to 1, URLScan creates a new log file each day.

· PerProcessLogging=0
When set to 1, URLScan creates a separate log file for each process, appending the process ID to the log file name.

· NormalizeURLBeforeScan=1
A raw URL request may contain numbers preceded by a % sign to encode special characters, such as “%20” to indicate a space. When this setting is 1, which is the more secure setting, URLScan decodes (normalizes) the raw URL before processing it.

· VerifyNormalization=1
When this setting is 1, URLScan performs a second normalization on the decoded URL to mitigate double-encoding attacks that work by encoding the % sign.

· RejectResponseURL=(not specified by default)
Use of this setting allows the redirection of a rejected URL to an alternate location to create a customized message that is sent to the client providing the reason the URL was blocked. When no path is specified, the default location for the response is /<Rejected-By-URLScan>, which causes a 404 error to be sent to the client and which shows up on the IIS/ISA logs as failed request for /<Rejected-By-URLScan>. A key advantage of this setting is that it allows IIS or the Web Proxy Logs to record more detailed information. A useful feature of URLScan is that you can specify a path to an ASP file that generates the custom response, such as /path/customresponse.asp, even when URLScan is configure to otherwise block ASP files. When the path is set to a special value of /~*, URLScan will allow the requests to be processed, but will log them in the URLScan.log file. This is useful for testing purposes.

URLScan provides a number of server variables that can be used to create the custom response for the client:

o HTTP_URLSCAN_STATUS_HEADER. Provides the reason for rejection.

o HTTP_URLSCAN_ORIGINAL_VERB. Specifies the HTTP verb (PUT, POST, DELETE, GET, OPTIONS, CONNECT, etc.,) that caused the rejection

o HTTP_URLSCAN_ORIGINAL_URL. Specifies the rejected URL.

· UseAllowVerbs=1
When set to 1, URLScan will allow only URLs that contain verbs explicitly permitted in the [AllowVerbs] section of the file. When set to 0, URLScan will deny URLs on the basis of verb explicitly denied in the [DenyVerbs] section.

· UseAllowExtensions=0
Works like the UseAllowVerbs section. Used to allow or deny URLs on the basis of extensions of requested files listed in the [AllowExtensions] or [DenyExtensions] sections.

· UseFastPathReject=1
When set to 1, URLScan ignores the setting of RejectResponseURL and immediately sends an error message to the client browser without any further processing. The requested URL is still recorded in the Web Proxy log and the reason for the rejection in the URLScan.log on the ISA Server. The advantage of this setting is faster processing of rejected URLs.

[AllowVerbs] and [DenyVerbs] Sections

These sections are used to accept or deny HTTP traffic on the basis of verbs (methods) specified in the HTTP header. When UseAllowVerbs is set to 1, URLScan will accept traffic that contains only the verbs explicitly listed in the [AllowVerbs] sections. When UseAllowVerbs is set to 0, URL scan will reject traffic on the basis of the presence of verbs specified in [DenyVerbs] section.

Depending on the Web applications you wish to publish, the URLScan.ini file may have to allow both HTTP 1.1 and WebDAV verbs. HTTP 1.1 verbs include GET, POST, PUT, HEAD, DELETE, and OPTIONS. WebDAV verbs are an extension of HTTP 1.1 verbs. Some of these verbs are COPY, LOCK, MOVE, PROPFIND, PROPPATCH, SEARCH, and others. Consequently, administrators need to take care when configuring settings in either the [AllowVerbs] or [DenyVerbs] section to ensure URLScan is not configured so restrictively that it interferes with the functionality of the Web application.

[AllowExtensions] and [DenyExtensions] Sections

These sections are used to accept or deny HTTP requests on the basis of requests for files that have particular extensions. When UseAllowExtensions is set to 1, URLScan will accept requests for files that correspond to explicit settings in the [AllowExtensions] section.

SharePoint Portal Server 2003 is based on the .NET Framework. One consequence of this is that there is a greater range of file extensions that have to be taken into consideration when configuring URLScan. For example, an [AllowExtensions] section, if used, should include .aspx and .ashx extensions. The [DenyExtensions] section, if used, should possibly include .asax, .ascx, .config, .dll, .resx, xsd, xsx, and other extensions. The Microsoft Knowledge Base article, HOW TO: Configure URLScan to Protect ASP.NET Web Applications, provides a good starting point for learning more about this topic.

[DenyHeaders] Section

This section is used to deny specific HTTP headers in the request. This section is used primarily to prevent HTTP headers that could potentially invoke WebDAV from being forwarded to the Web server. For example, the URLScan_iis.ini template prevents the Translate, If, and Lock-Token headers from being forwarded to the internal network. But, this section could be used to deny any header, including non-standard headers.

[DenyURLSequence] Section

This section is used to deny URLs that contain specific strings. A primary use of this section is to prevent directory traversal attacks by specifying strings such as “..”, “\”, “/.”, “%”, and “&”. Note that entries in this section are processed after normalization so that URLs containing a % character will be allowed as long as they do not contain the % character after normalization.

[RequestLimits] Section

This section is used to set maximum lengths for specific parts of incoming requests. There are three settings that can be assigned values:

· MaxAllowedContentLength
Specifies the maximum value for the Content-Length. The default value is approximately 30 MBs, as reported by the URLScan initialization log file, and can be reduced significantly depending on the size of files that are uploaded to the Web server. It should be noted that a value specified here is not effective in limiting content sizes of a chunked-transfer encoded POST request. Chunked-transfer encoding can be disabled through URLScan by denying the Transfer-Encoding header in the [DenyHeaders] section.

· MaxURL
Specifies the maximum allowable length of the URL, excluding query strings that may be present. This value should be set to one representing the longest possible URL on the site.

· MaxQueryString.
Specifies the maximum allowable length of a query string in the URL. The default is 4 KB and should be reduced to reflect a more reasonable value.

A useful feature of this section is that it can also be used to restrict the length of any header by prepending the “Max-“ setting to the header name plus some value. For example, specifying “Max-Content-Type=100” and “Max-Referrer=100” will limit the length of both the Content-Type and Referrer headers to 100 characters. A value of 0 will cause URLScan not to enforce any limits on the header size. (To deny headers completely, use the [DenyHeaders] section.) Also, header sizes have to be explicitly listed here in order for URLScan to enforce size limits on them.

Step-by-Step: Installing Feature Pack 1 and URLScan 2.5

This section describes steps for installing Feature Pack 1 and then installing and configuring URLScan 2.5 on ISA Server 2000. You should note at the outset that these instructions only show how to set up a basic, working configuration of URLScan 2.5 for published SharePoint Portal Sites and do not necessarily represent a configuration providing the greatest functionality with the highest level of security.

Achieving the maximum level of security requires extensive prior experimentation in a test lab to determine optimal settings for a customized configuration that is predicated on the specific security requirements of your environment. In fact, because settings in URLScan can disable access to a published SharePoint Web site, you should test URLScan settings in a test lab before deploying it on a production ISA Server.

Step-by-Step Background Information

The test lab used to demonstrate these step-by-step instructions has the following configuration:

· Internal Network.
The internal network uses the 172.16.1.0/24 network ID. The default gateway for the network is 172.16.1.1, which is the internal IP address of the ISA Server.

· External Network
The external network uses a 192.168.100.0/24 network ID.

· Internal DNS and Active Directory Namespace
Internal.net is used as the Active Directory and DNS namespace for the internal network.

· Active Directory.
A Windows 2003 Active Directory domain controller named Ad1.internal.net is used to provide directory and DNS services. DNS is set up with root hints and forwarding to support resolution to the external network and the Internet. The IP address of the domain controller is 172.16.1.10.

· External DNS Namespace.
External.net is used as the DNS namespace for external clients connecting to resources published through ISA Server. The DNS zone files for external.net are located on the external network. A single host record points to external IP address of ISA Server to resolve the Fully Qualified Domain Name (FQDN) extranet.external.net for access to extranet SPS web site.

· SharePoint Portal Server 2003 Configuration.
SharePoint Portal Server 2003 installed with a co-located WMSDE database is set up on a Windows 2003 computer named Sps.internal.net.

· Virtual Web Site Configuration.
IIS 6.0 was installed on the Windows 2003 server as per the SharePoint Portal Server 2003 prerequisites found in the SharePoint Portal Server 2003 help files. An additional virtual Web site was set up on the SPS computer to support both intranet and extranet access to the same SPS configuration and content SQL databases. (Please see the SharePoint Portal Server 2003 help files for instructions on creating this configuration.) The intranet virtual web site is located at 172.16.1.11 and is configured to use Windows Integrated Authentication only. The extranet virtual web site is located at 172.16.1.12 and is configured to use Basic Authentication only. No DNS Resource Records resolve to the 172.16.1.12 address.

· ISA Server Configuration.
A stand-alone ISA Server 2000 with Service Pack 1 has been set up in integrated mode (Firewall and Web Proxy service) on a Windows 2000 member server with Service Pack 4. The basic configuration of ISA Server is as follows:

External NIC configuration:

IP address: 192.168.100.22/24

Default Gateway: 192.168.100.254/24

NetBIOS disabled

Registration of external IP address in Dynamic DNS zone disabled

DNS server: None

Internal NIC configuration:

IP address: 172.16.1.1/24

Default Gateway: None

NetBIOS enabled

Registration of IP address in Dynamic DNS zone enabled

DNS server: 172.16.1.10

ISA configuration details:

Local Address Table: 172.16.1.0 – 172.16.1.255

Packet Filtering: Enabled

Site and Content Rule: Default

Protocol Rule: Single rule added to provide access to all protocols

Destination Sets: Single Destination Set configured to support Web Publishing rule. Destination set points to extranet.external.net as FQDN.

Incoming Web Listener: Enabled with default settings. No authentication required.

Web Publishing Rule: Single Web publishing rule added to publish SPS extranet SharePoint virtual server (internal IP address = 172.16.1.12/24).

Client Configuration: All clients configured as S-NAT clients.

Installing and Configuring URLScan 2.5 for ISA Server 2000

URLScan 2.5 for ISA Server 2000 installs separately from Feature Pack 1. To install URLScan for ISA Server 2000, you must first install Service Pack 1 for ISA Server and then install Feature Pack 1. You can find both Service Pack 1 and Feature Pack 1 at the Microsoft ISA Server 2000 Web site. Feature Pack 1 is not available as a single download, and you must download multiple files to use all the features of Feature Pack 1. In particular, you must download URLScan 2.5 for ISA Server 2000 as a separate executable file.

Before installing Feature Pack 1, you should back up your ISA Server. Please consult the ISA Server and the Windows 2003 help files for procedures for backing up the ISA Server configuration and the Windows 2003 System State. Assuming you have installed Service Pack 1 for ISA Server and have downloaded and expanded the files for Feature Pack 1, the procedure for installing Feature Pack 1 and URLScan is as follows:

Click Start, then click Run, type cmd in the Run dialog box, and press Enter to open up a command prompt.
Type cd\<path-to-featurepack1-files-directory>, and press Enter.
To install Feature Pack 1, from the command prompt, type isafp1.exe, and press Enter. Feature Pack 1 prompts you to agree to the End User License Agreement (EULA) and then installs without requiring additional information.

To install URLScan 2.5, from the same command prompt, type isafp1ur.exe, and press Enter. You will be asked to confirm if you wish to continue with the installation of URLScan 2.5, as in Figure 1 below.

Figure 1 URLScan 2.5 Installation Alert

Click Yes on the Microsoft ISA Server 2000 Update Setup prompt. You will be presented with the option to use either the IIS or the Outlook Web Access URLScan template .ini file, as in Figure 2 below.

Figure 2 URLScan 2.5 Configuration File Selection

Click Yes on the Microsoft ISA Server 2000 Update Setup prompt to choose the OWA configuration. (If you are not publishing an OWA site, you can click No instead. This will result in a much more restrictive configuration, but external SharePoint users will still be able to perform many tasks.)

This completes the basic installation of Feature Pack 1 with URLScan 2.5. At this point, you will want to verify the installation and configuration of URLScan. This will require that you verify the installation of URLScan 2.5 on the ISA Server, examine the URLScan 2.5 installation log file, and examine the URLScan.ini file.

To verify installation of URLScan on ISA Server 2000:

Open the ISA Server Management MMC console, expand the Extensions node, and click on Web Filters. Note the green circle with white arrow pointing up. This visual cue indicates the filter is enabled.
To view the properties of the URL Scan filter, click on the URLScan Filter in the MMC content pane with the right mouse button, and select Properties from the context menu, as in Figure 3 below.

Figure 3 URLScan Filter Properties

The URLScan 2.5 log and URLScan.ini files are located in the installation directory of ISA Server, along with the URLScan.dll file that creates the URLScan Web filter. To view these files, navigate to the installation directory of the ISA Server 2000 executables (by default, found in the \Program Files\ISA Server folder) and open them with Notepad. The URLScan log files use the following naming convention: URLscan.<mmddyy>.log.

Upon installation, the first URLScan log file is created showing the details of the URLScan initialization. The initialization information summarizes the URLScan restrictions, as in the following sample log.

[11-20-2003 - 13:07:10] ---------------- Initializing UrlScan.log ----------------

[11-20-2003 - 13:07:10] -- Filter initialization time: [11-20-2003 - 13:07:10] --

[11-20-2003 - 13:07:10] ---------------- UrlScan.dll Initializing ----------------

[11-20-2003 - 13:07:10] URLs will be normalized before analysis.

[11-20-2003 - 13:07:10] URL normalization will be verified.

[11-20-2003 - 13:07:10] URLs may contain OEM, international and UTF-8 characters.

[11-20-2003 - 13:07:10] Requests with Content-Length exceeding 30000000 will be rejected.

[11-20-2003 - 13:07:10] Requests with URL length exceeding 260 will be rejected.

[11-20-2003 - 13:07:10] Requests with Query String length exceeding 4096 will be rejected.

[11-20-2003 - 13:07:10] Only the following verbs will be allowed (case sensitive):

[11-20-2003 - 13:07:10] 'GET'

[11-20-2003 - 13:07:10] 'POST'

[11-20-2003 - 13:07:10] 'PROPFIND'

[11-20-2003 - 13:07:10] 'PROPPATCH'

[11-20-2003 - 13:07:10] 'BPROPPATCH'

[11-20-2003 - 13:07:10] 'MKCOL'

[11-20-2003 - 13:07:10] 'DELETE'

[11-20-2003 - 13:07:10] 'BDELETE'

[11-20-2003 - 13:07:10] 'BCOPY'

[11-20-2003 - 13:07:10] 'MOVE'

[11-20-2003 - 13:07:10] 'SUBSCRIBE'

[11-20-2003 - 13:07:10] 'BMOVE'

[11-20-2003 - 13:07:10] 'POLL'

[11-20-2003 - 13:07:10] 'SEARCH'

[11-20-2003 - 13:07:10] Requests for following extensions will be rejected:

[11-20-2003 - 13:07:10] '.exe'

[11-20-2003 - 13:07:10] '.bat'

[11-20-2003 - 13:07:10] '.cmd'

[11-20-2003 - 13:07:10] '.com'

[11-20-2003 - 13:07:10] '.htw'

[11-20-2003 - 13:07:10] '.ida'

[11-20-2003 - 13:07:10] '.idq'

[11-20-2003 - 13:07:10] '.htr'

[11-20-2003 - 13:07:10] '.idc'

[11-20-2003 - 13:07:10] '.shtm'

[11-20-2003 - 13:07:10] '.shtml'

[11-20-2003 - 13:07:10] '.stm'

[11-20-2003 - 13:07:10] '.printer'

[11-20-2003 - 13:07:10] '.ini'

[11-20-2003 - 13:07:10] '.log'

[11-20-2003 - 13:07:10] '.pol'

[11-20-2003 - 13:07:10] '.dat'

[11-20-2003 - 13:07:10] Requests containing the following character sequences will be rejected:

[11-20-2003 - 13:07:10] '..'

[11-20-2003 - 13:07:10] './'

[11-20-2003 - 13:07:10] '\'

[11-20-2003 - 13:07:10] '%'

[11-20-2003 - 13:07:10] '&'

[11-20-2003 - 13:10:45] ---------------- UrlScan.dll Terminating -----------------

Using the OWA URLScan 2.5 configuration file ensures external users are able to connect to both the published SharePoint Portal site and Outlook Web Access sites without requiring much further configuration. However, this configuration file was not designed specifically for SharePoint Portal Services and ASP.NET. You should keep in mind that there are some important differences between Exchange Server and SharePoint Portal Server. For example, Exchange Server 2000 and 2003 uses the Web Storage System (WSS) to store information, whereas SharePoint Portal Server 2003 uses SQL Server to store information.

While both Exchange and SharePoint Server may provide similar functional access to content through a Web browser or Web folders, they may not use or require an identical set of HTTP verbs to provide that access. SharePoint Portal Server 2003 requires fewer verbs to accomplish similar actions as Exchange Server. However, SharePoint Portal Server 2001, which was also based on the Exchange WSS, uses the same set of HTTP 1.1 and WebDAV verbs as Exchange Server, plus an additional HTTP verb, INVOKE.

At the time of this writing, specific documentation on the verbs used by SharePoint Portal Server 2003 is not available. However, additional documentation on SharePoint Portal Server 2003 is being continuously added to the Microsoft Web site, and readers of this document are encouraged to check the Microsoft Web site frequently for newly added and relevant documentation.

You should also note that this OWA configuration file does not prevent requests for file types specific to ASP.NET. Some ASP.NET file types, such as .vb, .config, and others should never be forwarded to external clients. Consequently, to provide additional security, you will wish to add these sensitive ASP.NET file types to the [DenyExtensions] list in the URLScan.ini file.

Once you have enabled URLScan, you can test external access to the SharePoint site. Figure 4 below shows an authentication request for connecting to the SharePoint site through the ISA Server with URLScan enabled:

Figure 4 Connecting to SharePoint Extranet Site

Figure 5 below shows the home page of the extranet SharePoint extranet site. Notice that a Web part from the Microsoft Web part gallery has been added to this page and that the web part is functioning correctly through the ISA Server firewall.

Figure 5 SharePoint Site Accessed Through ISA Server with URLScan Enabled

Either the IIS and the OWA URLscan configuration file allow a high degree of functionality for external users trying to perform tasks in the SharePoint Site. It is possible to navigate throughout the site, view file properties, perform searches, perform multiple file uploads, etc. However, not all features were tested, such as Presence, for this demonstration. Furthermore, Explorer View of Document Libraries do not work.

Further testing reveals the URLScan configuration was not at fault: Netmon captures of traffic reveal that the Explorer View requires NetBIOS ports to be opened on the firewall (specifically TCP Ports 135 and 139, plus others required for RPC connectivity). In the absence of a specific RPC application filter installed on ISA Server for this purpose, it is not recommend that you open these ports for security reasons.

Troubleshooting and Fine Tuning URLScan 2.5

Owing to the complexity of multiple components, including the URLScan configuration file, the configuration of the internal network, the ISA Server 2000 configuration, and of corporate security requirements, incorrect configuration of the URLScan configuration file can cause a myriad of problems that may not at first appear to be related to the URLScan configuration.

However, most problems will likely be related to overly restrictive settings for HTTP verbs and file extensions. To effectively troubleshoot problems related to URLScan, you should be familiar with the URLScan log files, IIS and ISA log files, and protocol analysis tools, such as Netmon. Additionally, you should try to become familiar with the details of the HTTP 1.1 and WebDAV protocols, specifically the verbs and header types used by these protocols. (A number of links to information on these topics is provided later on in this document.)

Tip:
By default, the URLScan setting, UseFastPathReject, is set to 1. This means that very limited information is sent to the IIS or Web Proxy logs when requests are rejected. To record more detailed information in these logs, you should set this configuration to 0 and configure the RejectResponseURL setting.

You should note that when URLScan rejects a request, it sends an altered (safe) version of the request to either IIS or the Web Proxy service for processing so that it can be logged. Specifically, URLScan substitutes GET for the HTTP verb, sets the content-length to zero, substitutes a blank string for any denied headers, and changes the URL to that specified by the RejectResponseURL setting. The originally requested URL is appended as a query string to the altered request that URLScan sends to IIS. If you are using IIS or ISA logs to troubleshoot problems with URL scan, you will want to configure logging so that query strings are recorded in order to capture as much detail as possible.

If users experience difficulties after you implement URLScan, it is a good idea to first check the URLScan log files to see if any problems show up there. Below is a sample from a URLScan log file, which has been edited to improve readability.

[11-23-2003 - 03:00:14] ---------------- Initializing UrlScan.log ----------------

[11-23-2003 - 03:00:14] -- Filter initialization time: [11-20-2003 - 13:10:54] --

[11-23-2003 - 03:00:14] Client at 192.168.100.50: URL contains extension '.dat', which is disallowed. Request will be rejected. Site Instance='*****', Raw URL='/wpad.dat'

[11-23-2003 - 03:00:26] Client at 192.168.100.50: URL contains extension '.dat', which is disallowed. Request will be rejected. Site Instance='*****', Raw URL='/wpad.dat'

[11-23-2003 - 14:19:38] Client at 200.214.190.60: Sent verb 'CONNECT', which is not specifically allowed. Request will be rejected.

[11-23-2003 - 22:14:50] Client at 192.168.100.67: Sent verb 'OPTIONS', which is not specifically allowed. Request will be rejected.

This log file shows a potential problem with the configuration of the URLScan.ini file that could cause havoc on a network. Note that the log file shows that requests for the wpad.dat file are being rejected. The wpad.dat file is used for the auto-configuration of Web Proxy clients through an auto-discovery process. Typically, clients discover the location of the wpad.dat through either a DHCP option or a DNS Resource Record. If Web browsers have been configured to use this feature to discover the Web proxy settings, and if these clients are not also configured as S-NAT clients, they will no longer be able to browse the Internet through the ISA Server. To correct this situation, you must either allow the .dat extension in the URLScan.ini file or explicitly specify the Web proxy settings in Web browser settings, either manually or through a Group Policy.

Note:
By default, both the IIS and OWA configuration files for URLScan block requests for .dat file types. This includes requests from clients on the internal network. You therefore must take this setting into consideration if you are using Web proxy auto-discovery in your environment. It is relatively safe to remove the exclusion for the .dat file extension. A properly ISA server will prevent the wpad.dat file from being forwarded to the Internet because there is no Web or server publishing rule that would allow this to occur. To verify that the wpad.dat file is being delivered correctly to internal clients, enter http://<isaservername>/wpad.dat in the browser and open the file with Notepad. To ensure that the wpad.dat file is not being forwarded to the Internet, perform a similar operation from an external client. You should receive a 404 error.

By default, URLScan.ini rejects the OPTIONS verb. A primary use of this verb is to determine what verbs (methods) the Web server supports or allows. (Blocking the OPTIONS verb is a good thing.) To determine what HTTP verbs need to be allowed in the URLScan.ini file, it is a good idea to analyze the traffic passing between clients and applications using a protocol analysis tool, such as Network Monitor, on your internal network.

Network Monitor includes a number of powerful features that assist in the analysis of network traffic for this purpose. For example, it is possible to filter output of the capture to show only the traffic of interest. Figure 6 below demonstrates a Network Monitor filter configuration to show only HTTP requests that contain the OPTIONS verb and the replies to those requests.

Figure 6 Configuring Netmon Filters

Once you have configured Network Monitor display filters, it is a relatively easy matter to focus on the traffic of interest. Figure 7 below shows the reply from a SharePoint server to a query for supported verbs.

Figure 7 Network Monitor Capture Showing Supported HTTP Verbs on SharePoint Server

There are a couple of things to note about this trace. First of all, almost all the verbs (methods) listed here are either standard HTTP 1.1 or Web Distributed Author Versioning (WebDAV) verbs. The exception is the GETLIB verb, which appears to be a custom extension, specific either to .NET or to SharePoint server. A thorough explanation of these methods and others can be found in a number of places, such as the Microsoft MSDN Web site, RFC 2616 for HTTP 1.1, and RFC 2518 for WebDAV.

The second thing to note is that a number of verbs in the Allow response header (seen in the details pane of the Network Monitor graphic above) do not have a corresponding entry in the URLScan OWA configuration file. Depending on your environment, it may be necessary to modify the [AllowVerbs] section of the URLScan.ini file.

Finally, and perhaps most importantly, the fact that these verbs are supported by the Web server does not mean external clients need to use them to perform operations in the SharePoint Site. To determine what verbs and file extensions you will wish to allow or reject for your Web applications and Web sites will require more extensive research.

When the IIS URLScan configuration file is used as the template for URLScan, external client activity generates relatively few rejections. For example, if you use the IIS configuration file as a template, you may sometimes encounter the following in your URLScan log files:

[11-25-2003 - 09:15:57] Client at 192.168.100.67: Sent verb 'PROPFIND', which is not specifically allowed. Request will be rejected.

[11-25-2003 - 09:15:57] Client at 192.168.100.67: Sent verb 'OPTIONS', which is not specifically allowed. Request will be rejected.

In this instance, rejections occurred because the external client attempted to use the Explorer View of a Document Library. External clients accessing the SharePoint Web site through a firewall normally can not use the Explorer View because this requires allowing inbound access for RPC and NetBIOS communication. (It is not recommended that you allow inbound access to these protocols.) The point here is that, even though URLScan is configured very restrictively to allow only the HEAD, GET, and POST verbs, URLScan encountered only very little traffic that it was configured to reject during the period when the external client performed a number of advanced operations using the SharePoint Web site.

Summary

URLScan 2.5 on ISA Server 2000 enhances the defensive capabilities of ISA Server 2000. Installing URLScan on ISA Server prevents suspicious HTTP requests from entering the internal network and mitigates threats to internal web servers. URLScan is highly configurable through the URLScan.ini file, which can be easily customized to meet a wide range of security and accessibility requirements. Consequently, when URLScan is installed on ISA Server, it plays a crucial role in a comprehensive defense-in-depth strategy that is designed to protect mission-critical applications such as SharePoint Portal Server 2003.