Publishing Remote Desktop Web Connection Sites with the ISA Firewall Part 3: Testing and Troubleshooting

by [Published on 31 Jan. 2006 / Last Updated on 20 May 2013]

In part one of this three part series on publishing remote desktop Web connection sites, we went over the details on how the process works and how the process does not work. In part two of the series we went over the step by step details on how to publish the remote desktop connection Web site and RDP servers. In this, part 3 and the last part of the article series, we’ll test the configuration and then go into a deep discussion on troubleshooting issues you might run into when publishing Web sites and RDP servers.

Publishing Remote Desktop Web Connection Sites with the ISA Firewall
Part 3: Testing and Troubleshooting
By Thomas W Shinder MD, MVP

Have Questions about the article? 
Ask at: http://tinyurl.com/8vdwj

If you would like to read the other articles in this series please go here:


Test the solution

Now we’re ready to test the solution. However, before you make your test, make sure you have configured name resolution so that external clients resolve the name used on the Web site certificate bound to the Web listener to the IP address used for the Web listener. In a production environment you will create a Host (A) record on your public DNS servers. In a test environment, you can create a HOSTS file entry so that the name resolves to the IP address on the external interface of the ISA firewall.

In this example, we’ve created a HOSTS file entry for tsweb.msfirewall.org so that it resolves to the IP address used on the Web listener.

Perform the following steps on the client machine to test the configuration:

  1. Open Internet Explorer on the client machine located on an external network. Enter into the address bar https://tsweb.msfirewall.org and press ENTER.
  2. Enter valid user credentials into the Connect to dialog box and click OK.


Figure 1

  1. If you’re running Windows XP Service Pack 2, then you’ll find the ActiveX control is blocked the first time you hit the site. Click OK in the dialog box informing you that the ActiveX control could not be installed.
  2. Click the Information Bar in Internet Explorer to install the ActiveX control. Click the Install button in the Internet Explorer – Security Warning dialog box.
  3. After the ActiveX control is installed the Remote Desktop Web Connection appears. Enter the server name in the Server text box. In this example, we’ll use the name tsweb.msfirewall.org as the server name. Note that this name resolves to the IP address used on the RDP Server Publishing Rule’s listener. Also remember that this name may or may not be the actual name of the server on the corporate network. It doesn’t matter what the server’s actual name is, what matters is that the name you enter in the Server text box resolves to an IP address that will get the connection to the RDP Server Publishing Rule’s listener address. Put a checkmark in the Send logon information for this connection checkbox, and then enter the User name and Domain in the text boxes. Click Connect.


Figure 2

  1. The terminal services log on dialog box appears in a full screen session. The User name and Domain are automatically entered. Enter the password and click OK.


Figure 3

  1. At this point you will enter the terminal services session. Everything worked!

Troubleshooting the Solution

Even though the procedures seem straightforward, there are several common troubleshooting issues that come up time and again on the ISAserver.org message boards, mailing list, and newsgroups. Some of these issues are related to ISA firewall configuration errors and some of them have nothing to do with the ISA firewall configuration. We’ll talk about both these types of problems in this section.

Common troubleshooting issues include:

  • RDP Service on the RDP Server is not Started
  • External Interface of the ISA Firewall uses a Private IP Address
  • Authentication with the ISA Firewall Fails
  • Certificate Dialog Box Appears when Client Connects

RDP Service on the RDP Server is not Started

Imagine this scenario: you’ve gone through this article and done everything I’ve told you to do. You have a good understanding of the ISA firewall’s Web and Server Publishing models, Windows networking, and the TCP/IP protocol suite. Everything seems to be in place but when you try to connect to the remote desktop Web connection Web site, the connection attempts repeatedly fail.

You look at the log files and the results don’t make a lot of sense. It appears that the SSL connections are accepted, but you’re really not sure what to look for so you shake your head and wonder what questions you should ask yourself to come up with a solution.

The first question you ask yourself is “I wonder what would happen if I used the native Remote Desktop Connection client to connect to the RDP site, without going through the Remote Desktop Web connection?”  You fire up the RDP client software and connect to the IP address of the RDP Server Publishing Rule listener. You end up seeing what appears in the figure below.


Figure 4

When you look at the ISA firewall’s log files for entries related to this connection attempt, you’ll see something like that in the figure 5. Notice that nowhere in these log file entries is there any information regarding the connection being denied. All you see in the Action column are entries for Initiated Connection and Closed Connection.

This should lead you to two conclusions:

  • There is no rule denying the connection
  • There is a rule allowing the connection

If those are the two conclusions you came up with, then you’re absolutely right. There is a rule allowing the connection, the RDP Server Server Publishing Rule. There is no rule denying the connection.


Figure 5

So what’s up? What’s causing the failed connection? The key is the entry in the Result Code column. Here you see that Result Code for the Closed Connection entries is 0x80074e21. If you look up this result code you’ll see that it means that either the client or the server closed the connection with a TCP RST. OK, so what is a TCP RST?

A TCP RST is a TCP segment (notice that TCP messages are segments, NOT packets, which is a common misuse of the language among network admins) which indicates something is wrong with the connection. A RST is sent when:

  • A SYN arrives for a non-existent server.
  • TCP wants to abort the connection.
  • A segment is received for which a connection doesn’t exist.

The problem in our case is related to the first reason. The “non-existent server” is the absence of the RDP listener being enabled on the terminal server or remote desktop connection server (like Windows XP). When you enable terminal services listener on the terminal server, or enable remote connections to a published remote desktop server (for example, Windows XP Professional or Windows Server 2003 Admin RDP) everything will work as expected.

For your future reference, Tables 1 and 2 list firewall service error codes that help you solve problems using the information in the Result Code field in the ISA firewall’s log files. This information was obtained from the MSDN Web site over at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/isasdk/isa/error_codes.asp

Symbolic name

Hexidecimal ID

Message text

FWX_E_TERMINATING

0xC0040001

The object is shutting down.

FWX_E_INVALID_ARG

0xC0040002

The argument is invalid.

FWX_E_ALREADY_IN_BLOCKING_OP

0xC0040003

The blocking operation is already started.

FWX_E_NOT_IN_BLOCKING_OP

0xC0040004

There is no blocking operation to be ended.

FWX_E_FILTER_NOT_REGISTERED

0xC0040005

The filter is not registered.

FWX_E_ALREADY_EXISTS

0x800700B7

The object cannot be created because an object with the same name already exists.

FWX_E_BUFFERFULL

0xC0040007

Not all the data was appended to the buffer object because the buffer was full.

FWX_E_ALREADY_EMULATED

0xC0040009

The connection is already emulated by another filter.

FWX_E_BAD_CONTEXT

0xC004000A

The method was not called while handling any of the supported events.

FWX_E_NOT_SUPPORTED

0xC004000B

Modifying this property is not allowed for this session.

FWX_E_NOT_AUTHENTICATED

0xC004000C

The action cannot be performed because the session is not authenticated.

FWX_E_POLICY_RULES_DENIED

0xC004000D

The policy rules do not allow the user request.

FWX_E_MIME_NEEDED

0xC004000E

The MIME type is required.

FWX_E_MUST_USE_DS

0xC004000F

FWX_E_NOT_EMULATED

0xC0040010

The connection is not emulated.

FWX_E_IS_BUSY

0xC0040011

FWX_E_NETWORK_RULES_DENIED

0xC0040012

FWX_E_FRAGMENT_PACKET_DROPPED

0xC0040013

FWX_E_FWE_SPOOFING_PACKET_DROPPED

0xC0040014

FWX_E_TCPIPDROP_PACKET_DROPPED

0xC0040015

FWX_E_NO_BACKLOG_PACKET_DROPPED

0xC0040016

FWX_E_TCP_NOT_SYN_PACKET_DROPPED

0xC0040017

A non-SYN packet was dropped because it was sent by a source that does not have an established connection with the ISA Server computer.

FWX_E_BAD_LENGTH_PACKET_DROPPED

0xC0040018

FWX_E_PING_OF_DEATH_PACKET_DROPPED

0xC0040019

FWX_E_OUT_OF_BAND_PACKET_DROPPED

0xC004001A

FWX_E_IP_HALF_SCAN_PACKET_DROPPED

0xC004001B

FWX_E_LAND_ATTACK_DROPPED

0xC004001C

FWX_E_UDP_BOMB_DROPPED

0xC004001D

FWX_E_FULLDENY_DROPPED

0xC004001E

FWX_E_IPOPTIONS_DROPPED

0xC004001F

FWX_E_UNCOMPLETED_CONNECTION_REQUEST

0xC0040020

An attempt to log on to the VPN server was rejected during the authentication phase because the authentication data was not received in a timely manner. The client session was disconnected.

FWX_E_CONNECTION_REQUEST_REJECTED

0xC0040021

An attempt to log on to the VPN server was rejected during the authentication phase. The client session was disconnected.

FWX_E_VALIDATE_QUARANTINE_FAILED

0xC0040022

The VPN quarantine settings could not be validated. The client session was disconnected.

FWX_E_VPN_CONNECTIONS_LIMIT_EXCEEDED

0xC0040023

The VPN client connection limit was exceeded. The client session was disconnected.

FWX_E_OUT_OF_RESOURCES

0xC0040024

FWX_E_BROADCAST_PACKET_DROPPED

0xC0040025

FWX_E_UNKNOWN_ADAPTER_DROPPED

0xC0040026

FWX_E_ICMP_ERROR_PACKET_DROPPED

0xC0040027

FWX_E_INVALID_PROTCOL_PACKET_DROPPED

0xC0040028

FWX_E_PORT_ZERO_PACKET_DROPPED

0xC0040029

FWX_E_SYN_ATTACK_START

0xC004002A

ISA Server detected a SYN attack.

FWX_E_SYN_ATTACK_END

0xC004002B

ISA Server is no longer experiencing a SYN attack.

FWX_E_INVALID_DHCP_OFFER

0xC004002C

FWX_E_UNREACHABLE_ADDRESS

0xC004002D

FWX_E_ADDRESS_NOT_ALLOWED

0xC004002E

FWX_E_IPSEC_NO_ROUTE_DROPPED

0xC004002F

FWX_E_OUTBOUND_PATH_THROUGH_DROPPED

0xC0040030

FWX_E_BAD_TCP_CHECKSUM_DROPPED

0xC0040031

FWX_E_VPN_USER_MAPPING_FAILED

0xC0040032

An attempt to map a VPN client to a Windows user failed. The client session was disconnected.

FWX_E_RULE_QUOTA_EXCEEDED_DROPPED

0xC0040033

A connection was rejected because the maximum number of connections that can be created for a rule during one second was exceeded.

FWX_E_SEQ_ACK_MISMATCH

0xC0040034

A TCP packet was rejected because it has an invalid sequence number or an invalid acknowledgement number.

Table 1: Firewall Service Error Codes

Symbolic name

Hexidecimal ID

Description

WSA_RWS_GRACEFUL_SHUTDOWN or FWX_E_GRACEFUL_SHUTDOWN

0x80074E20

A connection was gracefully closed in an orderly shutdown process with a three-way FIN-initiated handshake.

WSA_RWS_ABORTIVE_SHUTDOWN or FWX_E_ABORTIVE_SHUTDOWN

0x80074E21

A connection was abortively closed after one of the peers sent a RST segment.

WSA_RWS_QUOTA or FWX_E_RULE_QUOTA_EXCEEDED_DROPPED

0x80074E23

A connection was refused because a quota set in a rule was exceeded.

WSA_RWS_CONNECTION_KILLED or FWX_E_CONNECTION_KILLED

0x80074E24

ISA Server killed a connection.

WSA_RWS_TIMEOUT or FWX_E_TIMEOUT

0x80074E25

A connection was terminated because it was idle for more than the timeout period, or the timeout on an incompleted action expired.

WSA_RWS_ADMIN_TERMINATE or FWX_E_ADMIN_TERMINATE

0x80074E26

A connetion was terminated from ISA Server Management, during shutdown, or when a VPN client was disconnected.

Table 2: Additional run-time codes that may be returned by the Firewall service and may appear as result codes in ISA Server logs

Have Questions about the article? 
Ask at: http://tinyurl.com/8vdwj

External Interface of the ISA Firewall uses a Private IP Address

A remarkably common problem relates to deployments where there is a NAT device in front of the ISA firewall. The NAT device could be a simple NAT device like a broadband “router”, or it can be a simple stateful packet inspection firewall, such as a PIX or Sonicwall. What the broadband “router” and the simple stateful packet inspection devices have in common is that they both are responsible for the Internet connection and perform network address translation (in any of its forms) between its LAN and WAN interfaces.

When there is a NAT device in front of the ISA firewall, the ISA firewall’s external interface will have one or more private IP addresses bound to it. Connections cannot be routed over the Internet to a private IP address. That’s why all incoming connections to resources located on your internal network must be directed to the public address that is bound to the external interface of your NAT device.

Figure 6 shows a common troubleshooting situation. The NAT device has a public address assigned to its WAN interface and a private address bound to its LAN interface and performs NAT on communications moving across its interfaces. The ISA firewall has private addresses on both its internal and external interfaces. The ISA firewall also performs NAT across its internal and external interface (however, you do have the option to route between those interfaces, which is very helpful in SBS 2003 scenarios where there is a NAT/VPN device in front of the ISA firewall and a site to site VPN connection between that NAT/VPN device and another device.)

As we spoke about earlier in this article, the external client must be able to query a public DNS server to resolve the names it uses to connect to the remote desktop connection Web site and the RDP Server Publishing Rule. I’ve said several times in this article that the public DNS server must be able to resolve the names the remote desktop Web connection client uses to access the Web site and RDP Server Publishing Rule listener to the IP addresses used by the Web listener and the RDP Server Publishing Rule’s listener.

The problem I have when discussing ISA firewall scenarios is that I assume you have a public address bound to the external interface of the ISA firewall. When you have a public address bound to the external interface of the ISA firewall, then my comments about requiring that the names the external client uses to connect to the Web and RDP sites resolving to the IP addresses on the external interface of the ISA firewall is correct. However, if you have a NAT device in front of the ISA firewall, then this advice is not correct, because external client cannot connect to private IP addresses.

In order to keep things simple (mostly simple for me), I always make the assumption that the ISA firewall has a public address on its external interface or if you have a NAT device in front of the ISA firewall, I assume that you understand the implications of that configuration. However, making these assumptions are for my convenience and not necessarily reflective of the overall knowledge base of the ISA firewall community.

The figure below shows a common misconfiguration that leads to failure for both the Web connection and the RDP connection. The user has configured the DNS server with a Host (A) record that resolves the name used to connect to the Web site and the RDP site to the IP address bound to the external interface of the ISA firewall. This cannot work, since the IP address on the external interface is a private IP address and is not directly accessible through the Internet.


Figure 6

The correct configuration is shown in the figure below. The public DNS server has a Host (A) record that resolves the name used to connect to the Web and RDP sites using the public IP address bound to the external interface of the NAT device located in front of the ISA firewall. There is the only way it can work. You never used use the external address on the ISA firewall when a private address is used on the external interface.


Figure 7

Now, getting the DNS entry right is only half the battle. The other half is to configure the NAT device to forward the required connection requests to the correct IP addresses on the external interface of the ISA firewall. This is commonly referred to as port forwarding. For example, in the scenario discussed in this article you would need to forward incoming connections to TCP port 80 and TCP port 3389 to the correct IP addresses on the external interface of the ISA firewall.

In this scenario it’s fairly simple, because there is a single IP address bound to the external interface of the ISA firewall and we’re only publishing a single RDP server. It gets slightly more complex (but not terribly so) when you need to publish multiple RDP servers, because you will need multiple IP addresses bound to the WAN interface of the NAT device and multiple private addresses bound to the external interface of the ISA firewall.

One last thing related to this scenario is the default gateway setting on the external interface of the ISA firewall. The default gateway on the external interface of the ISA firewall must be the LAN IP addresses assigned to the NAT device located in front of the ISA firewall.

Authentication with the ISA Firewall Fails

If there’s one thing that I’ve been remiss in explaining in my books and articles is how the ISA firewall’s pre-authentication features requires an authentication provider. As you probably already know, the ISA firewall includes a pre-authentication feature that allows the ISA firewall to authenticate incoming connections mediated by the Web proxy filter before allowing the connection to be forwarded to the Web site on your internal network. The is a powerful security feature, since external attackers aren’t able to leverage an anonymous connection to attack your corporate Web servers.

However, in order for the ISA firewall to authenticate a user, it must have an authentication provider. For Web Publishing Rules, you can use the following authentication providers:

  • The local SAM database maintained on the ISA firewall itself
  • The internal network Active Directory user database
  • RADIUS authentication against a RADIUS compliant user directory

Keeping the user database on the ISA firewall is not a recommended configuration, as the management overhead of synchronizing passwords can be overwhelming. Also, there is a remote, although infinitesimal possibility that that ISA firewall will be “owned” and keeping the directory on the ISA firewall potentially exposes all of them to the attacker.

The preferred configuration for an ISA firewall used to control both inbound and outbound access is Active Directory authentication. In this case, the ISA firewall must be a domain member to authenticate users against the Active Directory. This is a secure configuration and in the overall security evaluation, a domain member ISA firewall is more secure than a non-domain member ISA firewall. There are many reasons for this being true and I’ll publish a comprehensive article in the near future highlighting the security considerations for domain membership and non-domain membership and prove that domain membership for ISA firewall’s controlling inbound and outbound access is the most secure configuration.

When the ISA firewall is deployed to allow only inbound access, then RADIUS authentication is the preferred method. This brings up an ISA firewall best practice. If your budget allows for it, you should deploy ISA firewalls in a role-based configuration. When I am allowed the appropriate budget, I separate my ISA firewalls into the following roles:

  • Secure Remote Access VPN server and gateway
  • Inbound access firewall for Web and server publishing
  • Outbound access firewall for SecureNAT, Web proxy and Firewall clients

I don’t think I’ve done an article or described in any of our books the rationale for such a role-based deployment, but I’ll put it on my list, as it’s an interesting and exceptionally powerful security and performance solution.

But back to the RADIUS authentication provider. When the ISA firewall is deployed to allow only inbound connections via Web Publishing Rules and Server Publishing Rules, then RADIUS authentication is often the best solution, as the ISA firewall is positioned well outside the security perimeter of the authentication providers. For this reason, and others, I prefer to the inbound access only ISA firewall to use RADIUS authentication and not make the ISA firewall a domain member.

BOTTOM LINE:
When configuring the ISA firewall to perform pre-authentication, think about what type of authentication provider you want to use. In single ISA firewall deployments where the ISA firewall is responsible for both inbound and outbound access control, always make the ISA firewall a domain member and use Windows Integrated (Active Directory) authentication. When the ISA firewall is providing only inbound access control, then configure the ISA firewall to use RADIUS authentication.

Certificate Dialog Box Appears when Client Connects

You have two certificate options when deploying secure SSL connections to the ISA firewall:

  • Commercial certificates
  • Private certificates

Commercial certificates must be purchased from a commercial certificate authority. An example of such a certificate authority is Verisign. Private certificates are those you generated yourself, or have someone else generate for you, but which are not part of a commercial certificate authority chain.

The advantage of private certificates is that you can generate them yourself. The disadvantage of private certificates is that no one trusts those certificates out of the box. If you have an Active Directory domain, you can automatically configure all the domain member computers in that domain to trust the certificates you generate. For computers that are not members of the domain, you must do some “out of band” configuration to make the those computers trust your certificates.

This points out the advantage of using commercial certificates. Most Windows computers trust most of the commercial certificate authority’s right out of the box. You don’t need to add the commercial certificate authority’s CA certificate to your machine’s Trusted Root Certification Authorities machine certificate store because Microsoft has already put those into your store and updates them on the periodic basis for you.

If you have an Active Directory domain and configure Group Policy to automatically deliver your CA certificates to all domain members and if you only allow domain member computers to connect to your published resources over the Internet, then everything will be fine. Users will never know any difference and their access to your SSL secured sites will be transparent.

However, if you have computers that are not domain members, or any computer that doesn’t have your CA’s certificate installed in its Trusted Root Certification Authorities machine certificate store, then users connecting to your SSL secured Web site will be confronted with a dialog box informing them that their machine doesn’t trust the certificate presented by the Web site. Of course, the typical user has no idea why this appears and either clicks past this dialog box or calls the Help Desk asking what to do.

The solution to the problem is to install the CA certificate into the Trusted Root Certification Authorities machine certificate store on all machines that will connect to your secure Web site. There are several ways to do this, including using the Web enrollment site, Active Directory Group Policy, and the Certificates MMC snap-in. PKI deployment and provisioning is far beyond the scope of this article, but you might want to check out the ISA Server 2000 VPN Deployment Kit at http://isaserver.org/articles/isa2000vpndeploymentkit.html for some great information on setting up different certificate authorities and how to deploy certificates to your clients.

Have Questions about the article? 
Ask at: http://tinyurl.com/8vdwj

Summary

In this article we finished up the series on how to publish remote desktop Web connection sites by testing the solution and going over some troubleshooting issues. I hope you learned something about Web and Server Publishing during this series and I’d be very happy to hear from you about things I can do to make this information even more useful to you. Send me a note at tshinder@isaserver.org and I’ll get back to you ASAP. Thanks! –Tom.

If you would like to read the other articles in this series please go here:

Advertisement

Featured Links