I get a lot of requests for troubleshooting problems with ISA firewalls that stop running for no apparent reason. I don't mean that they blue screen or anything like that. Instead, they just seem to stop accepting new connections. Existing connections seem to work fine but new connections are a no go. The problem is often due to an exhaustion of connection resources. The challenge is to figure out what is causing an exhaustion of these connection resources.
If you had to line up a list of "typical suspects", the one that should always be on the top of your list is DNS. DNS queries, in my experience, as the type of connection that most commonly leads to this kind of problem. Often it's due to large amounts of spam coming into your network and your SMTP servers trying to send NDRs back to the non-existent spammer domains. But there can be another reason for a DNS query related resource exhaustion problem.
As Yuri Diogenes points out in his article over at the ISA/TMG Firewall Team blog over at https://blogs.technet.com/isablog/archive/2009/01/12/isa-server-2006-stops-answering-requests.aspx, the ISA/TMG firewall itself can also be the cause of the DNS query resource exhaustion issue. When you use Domain Name Sets and URL Sets to control outbound access, the firewall has to do reverse DNS lookups to make sure that IP addresses that match the names in the domain name set or URL set aren't getting past the firewall. The issue becomes an important one when you apply such a restriction on a rule that applies to all protocols and all traffic moving through the firewall. Check out the article for details.
When troubleshooting the problem, make sure to take a look at your backlogged packets counter, and you should also look at your pending DNS name resolution counter too.
Thomas W Shinder, M.D., MCSE
Sr. Consultant / Technical Writer