How to send your IIS server into a frenzy with one line of HTML code? I didn't think it was possible until a few days ago we were stung with a persistent denial of service at work. This is what the event log showed on every instance of outage:
Event Type: Warning
Event Source: W3SVC-WP
Event Category: None
Event ID: 2262
Date: 10/23/2008
Time: 8:32:51 PM
User: N/A
Computer: WEB
Description:
ISAPI
'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll'
reported itself as unhealthy for the following reason:
'Deadlock detected'.
I have always been frustrated with IIS event logs and there's one proof of that. OK, I know there was a problem, but can this be more specific? which process? which application pool? Which page? This is hardly helpful. I believe new Failed Request Tracing feature in IIS7 was designed to help with just that. Countless debugging hours would be saved if one could quickly identify a misbehaving page. Googling this entry lead me to the http://support.microsoft.com/kb/821268 KB article and that sent me on a wrong path for some time before realizing that I was on a fruitless chase.
Finally, After 6 hours of struggling with the server (Window Server 2003, IIS6, CLR 2, FCL 3.5) and slicing and dicing and moving various applications to different pools, I found the offending ASPX pages. It didn't take long for my colleagues to discover an HTML anomaly in those files. And here it is in a generic format:
<p param1="" param2="" param3="" param4="" param5=""
param6="" param7="" param8="" param9="" param10="" param11=""
param12="" param13="" param14="" param15="" param16="" param17=""
param18="" param19="" param20="" param21="" param22="" param23=""
param24="" param25.="" />
Notice that last parameter with a trailing period (.)? that's the culprit right there, an HTML tag with a long list of attributes and a punctuation mark in one or more of the ending parameters. It had crept into our pages via a poorly made web editing product.
To test for yourself, just drop the line into an ASPX file and browse to the page and watch the CPU crank up and eventually the server refuse to serve pages. Remove the period or reduce the parameters and the page will display fine. I know, seems hard to believe. I didn't believe it either at first. But the results were the same on IIS6 and IIS7 on various platforms.
My hunch is that this tag throws the ASP.NET engine into a regular expressions frenzy as it tries to construct the page elements and compile the page into a DLL application. Now this HTML syntax might look weird but I don't think it's illegal and it certainly shouldn't cause a denial of service. Certainly IIS or browsers have no issue with HTML files containing such tags.
While at first I had feared serious security risks, this issue has a limited risk factor. In order for an attacker to exploit this issue, he would need access to the page sources. It's not something that can be injected in or remotely scripted into the page. Of course someone with a shared hosting service, could potentially take down the entire server and all the sites along with it.
So if you run into such a problem on your IIS web site, you might want to check your source files for these types of HTML tags. And by the way, Microsoft has already been advised of this issue and they have indicated a fix will be incorporated in the next ASP.NET release.
Give a few Sats