We have been running into almost daily hangs of w3wp
processes on 6 different farm machines (all 2003, various
hardware specs). This wouldn't be horribly, except these
processes cannot be killed (tried recycling app pool,
restarting IIS, killing w3wp.exe from taskmgr). A system
restart attempt leaves the machine stuck in an
inaccessible state, and it has to be power cycled to get
back up.
There is no ASP in use, strictly an ISAPI filter that
monitors/logs/limits access to hosted files.
The ISAPI filter that didn't seem to be a problem with
Windows 2000.
By the time we notice this, there are usually 20-30 w3wp
processes running (there are normally 3), and none of them
can be stopped.
When this happens, I stop sending traffic to the server
(from the load balancing appliance), yet CPU use is still
around 50% (dual CPU machine), and system/service host is
what's generating that usage.
An IISState attempt returns an error 5:
Unable to examine process id 5224, Win32 error 5
WaitForEvent failed, 0x80070005
Again, nothing we do can seem to kill these processes,
leaving the machine unusable until a power cycling.
I've tried various settings on the application pool.
Restarting after n requests, not restarting, limiting
memory (which isn't a problem), etc. The problem seems to
be that the w3wp process hangs, and refuses to die.
IISDump returned the output below (the bottom portion of
the results).
Is this a problem with the ISAPI filter, or our
configuration?
Thanks for any help anyone can offer.
0:000> ~*kv50
. 0 Id: 7f0.1048 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr Args to Child
0006f9a8 77f4303b 77e4905d 0000004c 00000000
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\ADVAPI32.dll -
WARNING: Stack unwind information not available. Following
frames may be wrong.
0006fa14 77db51f1 0000004c 0006fad0 00000216 ntdll!
ZwReadFile+0xc
0006fa40 77db5297 0000004c 0006fad0 00000216 ADVAPI32!
LsaLookupPrivilegeName+0xce
0006fab0 77dfa7f1 0000004c 0006fad0 00000216 ADVAPI32!
LsaLookupPrivilegeName+0x174
0006fb4c 0006fb98 77f45a34 77f89458 ffffffff ADVAPI32!
StartServiceCtrlDispatcherA+0x91
0006f750 00000000 00000000 00000000 00000000 0x6fb98
1 Id: 7f0.1288 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr Args to Child
0083fe18 77f43741 77e41817 00000044 00000000
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\kernel32.dll -
WARNING: Stack unwind information not available. Following
frames may be wrong.
0083fe8c 77e4168f 00000044 ffffffff 00000000 ntdll!
NtWaitForSingleObject+0xc
0083ffb8 77e4a990 002627d0 00000000 00000000 kernel32!
WaitForSingleObject+0xf
0083ffec 00000000 01002cbe 002627d0 00000000 kernel32!
FlsSetValue+0x779
2 Id: 7f0.1738 Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr Args to Child
0087fcc4 77f43741 77e41817 000000a0 00000000
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
0087fd38 77e4168f 000000a0 ffffffff 00000000 ntdll!
NtWaitForSingleObject+0xc
77e41680 74ff0c24 00e80c24 c2000001 408b0008 kernel32!
WaitForSingleObject+0xf
74ff006a 00000000 00000000 00000000 00000000 0x74ff0c24
3 Id: 7f0.1294 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr Args to Child
00a7ff9c 77f4262b 77f6b5b2 00000001 00a7ffb0
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
00a7ffb8 77e4a990 00000000 00000000 00000000 ntdll!
NtDelayExecution+0xc
00a7ffec 00000000 77f6b56d 00000000 00000000 kernel32!
FlsSetValue+0x779
4 Id: 7f0.1334 Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr Args to Child
00b8feb0 77f4372d 77e41bfa 00000002 00b8ff00
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
00b8ff5c 77e4b0e4 00000002 00b8ff9c 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
00b8ffb4 00b8ffec 77e4a990 00000000 00000000 kernel32!
WaitForMultipleObjects+0x17
00000000 00000000 00000000 00000000 00000000 0xb8ffec
5 Id: 7f0.dc4 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr Args to Child
0162feac 77f4372d 77e41bfa 00000002 0162fefc
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
0162ff58 77e4b0e4 00000002 00be9650 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32
\inetsrv\metadata.dll -
0162ffb0 620b4390 00000000 77e4a990 00be9648 kernel32!
WaitForMultipleObjects+0x17
0162ffec 00000000 620b437a 00be9648 00000000 metadata!
CBIN::Clone+0x3a40
6 Id: 7f0.10a4 Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr Args to Child
0172fbc0 77f4372d 77e41bfa 00000002 0172fc10
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\USER32.dll -
WARNING: Stack unwind information not available. Following
frames may be wrong.
0172fc6c 77d076f5 00000002 0172fc94 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
0172fcc8 77d077f5 00000001 00c2132c ffffffff USER32!
GetLastInputInfo+0xe4
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32
\inetsrv\INFOCOMM.dll -
0172fce4 643f5723 00000001 00c2132c 00000000 USER32!
MsgWaitForMultipleObjects+0x1d
643e5f60 56524553 3a454349 6174533a 65537472 INFOCOMM!
IIS_SERVICE::StartServiceOperation+0x22f
5f534949 00000000 00000000 00000000 00000000 0x56524553
7 Id: 7f0.838 Suspend: 1 Teb: 7ffae000 Unfrozen
ChildEBP RetAddr Args to Child
017efbc0 77f4372d 77e41bfa 00000002 017efc10
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
017efc6c 77d076f5 00000002 017efc94 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
017efcc8 77d077f5 00000001 00c1c4a4 ffffffff USER32!
GetLastInputInfo+0xe4
017efce4 643f5723 00000001 00c1c4a4 00000000 USER32!
MsgWaitForMultipleObjects+0x1d
643e5f60 56524553 3a454349 6174533a 65537472 INFOCOMM!
IIS_SERVICE::StartServiceOperation+0x22f
5f534949 00000000 00000000 00000000 00000000 0x56524553
8 Id: 7f0.11a8 Suspend: 1 Teb: 7ffad000 Unfrozen
ChildEBP RetAddr Args to Child
01a6feac 77f4372d 77e41bfa 00000003 01a6fefc
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
01a6ff58 77e4b0e4 00000003 00123690 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\inetsrv\IISCFG.DLL -
01a6ffac 648c983d 00000000 00000000 77e4a990 kernel32!
WaitForMultipleObjects+0x17
01a6ffec 00000000 648c9808 001218e0 00000000 IISCFG!
DllGetSimpleObjectByIDEx+0xea2e
9 Id: 7f0.1138 Suspend: 1 Teb: 7ffac000 Unfrozen
ChildEBP RetAddr Args to Child
01b6ff50 77f430c7 77e430bc 000004a8 01b6ffac
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\inetsrv\ISATQ.dll -
WARNING: Stack unwind information not available. Following
frames may be wrong.
01b6ff80 63ec7235 000004a8 01b6ffb8 01b6ffac ntdll!
ZwRemoveIoCompletion+0xc
77f6e0ff e877f924 fffd78e9 00e745c6 20243d83 ISATQ!
AtqInitialize+0x1255
a8682c6a 00000000 00000000 00000000 00000000 0xe877f924
10 Id: 7f0.1550 Suspend: 1 Teb: 7ffab000 Unfrozen
ChildEBP RetAddr Args to Child
01baff50 77f430c7 77e430bc 000004a8 01baffac
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
01baff80 63ec7235 000004a8 01baffb8 01baffac ntdll!
ZwRemoveIoCompletion+0xc
77f6e0ff e877f924 fffd78e9 00e745c6 20243d83 ISATQ!
AtqInitialize+0x1255
a8682c6a 00000000 00000000 00000000 00000000 0xe877f924
11 Id: 7f0.e88 Suspend: 1 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr Args to Child
01eafe20 77f4313f 77c57b25 000004fc 01eaff7c
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
*** ERROR: Symbol file could not be found. Defaulted to
export symbols for C:\WINDOWS\system32\RPCRT4.dll -
WARNING: Stack unwind information not available. Following
frames may be wrong.
01eaff8c 77c69659 77c695a1 00124d18 00000000 ntdll!
NtReplyWaitReceivePortEx+0xc
01eaffb0 77c69687 00086cf8 77e4a990 0012c998 RPCRT4!
I_RpcBindingIsClientLocal+0x799
01eaffec 00000000 77c69670 0012c998 00000000 RPCRT4!
I_RpcBindingIsClientLocal+0x7c7
12 Id: 7f0.1578 Suspend: 1 Te
: 7ffa8000 Unfrozen
ChildEBP RetAddr Args to Child
01eefeb4 77f430c7 77e430bc 00000058 01eeff0c
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
01eefee4 77c5df31 00000058 01eeff1c 01eeff0c ntdll!
ZwRemoveIoCompletion+0xc
01eeff20 77c5ddd8 00015f90 01eeff74 01eeff78 RPCRT4!
I_RpcTransConnectionAllocatePacket+0x4e1
01eeff8c 77c580f9 77c695a1 00087b20 00000000 RPCRT4!
I_RpcTransConnectionAllocatePacket+0x388
01eeffb0 77c69687 00086cf8 77e4a990 0012dde0 RPCRT4!
NdrGetTypeFlags+0x329
01eeffec 00000000 77c69670 0012dde0 00000000 RPCRT4!
I_RpcBindingIsClientLocal+0x7c7
13 Id: 7f0.109c Suspend: 1 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr Args to Child
016efec8 77f4372d 77e41bfa 00000002 016eff18
SharedUserData!SystemCallStub+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following
frames may be wrong.
016eff74 77e4b0e4 00000002 016effac 00000000 ntdll!
ZwWaitForMultipleObjects+0xc
016effb8 77e4a990 00000000 00000000 00000000 kernel32!
WaitForMultipleObjects+0x17
016effec 0000