I'm running into an issue with our IIS install where the service just
stops responding. I've run iisstate during one of these hangs which had
the following output:
-----------------------------------------------------------------------------------------------------------------------------------
Starting new log output
IISState version 3.3.1
Mon Mar 13 15:39:47 2006
OS = Windows 2003 Server
Executable: inetinfo.exe
PID = 4376
Note: Thread times are formatted as HH:MM:SS.ms
***********************
Thread ID: 0
System Thread ID: 1454
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0006f9a8 77f4303b SharedUserData!SystemCallStub+0x4
01 0006f9ac 77e4905d ntdll!NtReadFile+0xc
02 0006fa14 77db51f1 kernel32!ReadFile+0x16c
03 0006fa40 77db5297 ADVAPI32!ScGetPipeInput+0x28
04 0006fab0 77dfa7f1 ADVAPI32!ScDispatcherLoop+0x4c
05 0006fcec 01002655 ADVAPI32!StartServiceCtrlDispatcherA+0x91
06 0006fe1c 010027ea inetinfo!StartDispatchTable+0x214
07 0006ff44 01003160 inetinfo!main+0x104
08 0006ffc0 77e4f38c inetinfo!mainCRTStartup+0x12f
09 0006fff0 00000000 kernel32!BaseProcessStart+0x23
Thread ID: 1
System Thread ID: fdc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0083fe18 77f43741 SharedUserData!SystemCallStub+0x4
01 0083fe1c 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0083fe8c 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0083fe9c 01002cf9 kernel32!WaitForSingleObject+0xf
04 0083ffb8 77e4a990 inetinfo!W3SVCThreadEntry+0x3b
05 0083ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 2
System Thread ID: d3c
Kernel Time: 0:0:0.78
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 0087fcc4 77f43741 SharedUserData!SystemCallStub+0x4
01 0087fcc8 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0087fd38 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0087fd48 649f24ac kernel32!WaitForSingleObject+0xf
04 0087fd70 010023b6 iisadmin!ServiceEntry+0x214
05 0087ffa8 77db571b inetinfo!InetinfoStartService+0x2a6
06 0087ffb8 77e4a990 ADVAPI32!ScSvcctrlThreadA+0xe
07 0087ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 3
System Thread ID: 52c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00a7ff9c 77f4262b SharedUserData!SystemCallStub+0x4
01 00a7ffa0 77f6b5b2 ntdll!NtDelayExecution+0xc
02 00a7ffb8 77e4a990 ntdll!RtlpTimerThread+0x45
03 00a7ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 4
System Thread ID: 1290
Kernel Time: 0:0:0.0
User Time: 0:0:0.31
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00b8feb0 77f4372d SharedUserData!SystemCallStub+0x4
01 00b8feb4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 00b8ff5c 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 00b8ff74 6e0b377a kernel32!WaitForMultipleObjects+0x17
04 00b8ffa0 6e0b6012 COADMIN!NOTIFY_CONTEXT::GetNextContext+0x68
05 00b8ffb8 77e4a990 COADMIN!NOTIFY_CONTEXT::NotifyThreadProc+0x62
06 00b8ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 5
System Thread ID: 1700
Kernel Time: 0:0:0.78
User Time: 0:0:0.62
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 016efbc0 77f4372d SharedUserData!SystemCallStub+0x4
01 016efbc4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 016efc6c 77d076f5 kernel32!WaitForMultipleObjectsEx+0x11a
03 016efcc8 77d077f5 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 016efce4 643f5723 USER32!MsgWaitForMultipleObjects+0x1d
05 016efd30 6b77305d INFOCOMM!IIS_SERVICE::StartServiceOperation+0x22f
06 016efd70 010023b6 SMTPSVC!ServiceEntry+0x129
07 016effa8 77db571b inetinfo!InetinfoStartService+0x2a6
08 016effb8 77e4a990 ADVAPI32!ScSvcctrlThreadA+0xe
09 016effec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 6
System Thread ID: 128c
Kernel Time: 0:0:0.78
User Time: 0:0:0.15
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 01a2ff50 77f430c7 SharedUserData!SystemCallStub+0x4
01 01a2ff54 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01a2ff80 63ec7235 kernel32!GetQueuedCompletionStatus+0x27
03 01a2ffb8 77e4a990 ISATQ!AtqPoolThread+0x40
04 01a2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 7
System Thread ID: 15dc
Kernel Time: 0:0:0.156
User Time: 0:0:0.31
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 01a6ff50 77f430c7 SharedUserData!SystemCallStub+0x4
01 01a6ff54 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01a6ff80 63ec7235 kernel32!GetQueuedCompletionStatus+0x27
03 01a6ffb8 77e4a990 ISATQ!AtqPoolThread+0x40
04 01a6ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 8
System Thread ID: 1728
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 01e6fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 01e6fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 01e6ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 01e6ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 01e6ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 01e6ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 01e6ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 9
System Thread ID: 1b8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 01eafeb4 77f430c7 SharedUserData!SystemCallStub+0x4
01 01eafeb8 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01eafee4 77c80bd1 kernel32!GetQueuedCompletionStatus+0x27
03 01eaff20 77c80a78 RPCRT4!COMMON_ProcessCalls+0x9f
04 01eaff8c 77c58159 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x115
05 01eaff90 77c60771 RPCRT4!ProcessIOEventsWrapper+0x9
06 01eaffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
07 01eaffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
08 01eaffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 10
System Thread ID: 1650
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01eefec8 77f4372d SharedUserData!SystemCallStub+0x4
01 01eefecc 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 01eeff74 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 01eeff8c 69532430 kernel32!WaitForMultipleObjects+0x17
04 01eeffb8 77e4a990 exstrace!RegNotifyThread+0x68
05 01eeffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 11
System Thread ID: 8d8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01f2fecc 77f4372d SharedUserData!SystemCallStub+0x4
01 01f2fed0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 01f2ff78 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 01f2ff90 695319c0 kernel32!WaitForMultipleObjects+0x17
04 01f2ffb8 77e4a990 exstrace!WriteTraceThread+0x2f
05 01f2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 12
System Thread ID: 2c4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0212ff18 77f43741 SharedUserData!SystemCallStub+0x4
01 0212ff1c 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0212ff8c 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0212ff9c 01738673 kernel32!WaitForSingleObject+0xf
04 0212ffb8 77e4a990 FCACHDLL!CScheduleThread::ScheduleThread+0x60
05 0212ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 13
System Thread ID: 150c
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 0226fec0 77f4372d SharedUserData!SystemCallStub+0x4
01 0226fec4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 0226ff6c 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 0226ff84 6b78b2d6 kernel32!WaitForMultipleObjects+0x17
04 0226ffb8 77e4a990 SMTPSVC!TcpRegNotifyThread+0xdc
05 0226ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 14
System Thread ID: d10
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 022aff20 77f43741 SharedUserData!SystemCallStub+0x4
01 022aff24 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 022aff94 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 022affa4 6b78b17a kernel32!WaitForSingleObject+0xf
04 022affb8 77e4a990 SMTPSVC!FreeLibThread+0x2c
05 022affec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 15
System Thread ID: 1c4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 02a2fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 02a2fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 02a2ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 02a2ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 02a2ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 02a2ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 02a2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 16
System Thread ID: 428
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 02a8fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 02a8fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 02a8ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 02a8ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 02a8ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 02a8ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 02a8ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 17
System Thread ID: 4b0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 02acff10 77f43741 SharedUserData!SystemCallStub+0x4
01 02acff14 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 02acff84 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 02acff94 029ad064 kernel32!WaitForSingleObject+0xf
04 02acffb8 77e4a990
aqueue!CSMTP_RETRY_HANDLER::RetryThreadRoutine+0xc1
05 02acffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 18
System Thread ID: 224
Kernel Time: 0:0:0.15
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 02b4fe84 77f4372d SharedUserData!SystemCallStub+0x4
01 02b4fe88 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 02b4ff30 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 02b4ff48 0298c648 kernel32!WaitForMultipleObjects+0x17
04 02b4ffa4 6b77e8ae aqueue!CConnMgr::GetNextConnection+0x1e1
05 02b4ffb8 77e4a990 SMTPSVC!PERSIST_QUEUE::QueueThreadRoutine+0x21
06 02b4ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 19
System Thread ID: 17e8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 02b8fe38 77f43741 SharedUserData!SystemCallStub+0x4
01 02b8fe3c 71b23ac3 ntdll!ZwWaitForSingleObject+0xc
02 02b8fe78 71b239d1 mswsock!SockWaitForSingleObject+0x19b
03 02b8ff3c 71c016c9 mswsock!WSPSelect+0x229
04 02b8ff8c 63ec4696 WS2_32!select+0xb9
05 02b8ffb4 63ec4700 ISATQ!ATQ_BMON_SET::BmonThreadFunc+0x22
06 02b8ffb8 77e4a990 ISATQ!BmonThreadFunc+0x9
07 02b8ffc4 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 20
System Thread ID: 17bc
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 02c2fcec 77f4372d SharedUserData!SystemCallStub+0x4
01 02c2fcf0 77f75297 ntdll!NtWaitForMultipleObjects+0xc
02 02c2ffb8 77e4a990 ntdll!RtlpWaitThread+0x158
03 02c2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 21
System Thread ID: d9c
Kernel Time: 0:0:0.375
User Time: 0:0:0.328
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 00abfe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00abfe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00abff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00abff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00abffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00abffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00abffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 22
System Thread ID: ffc
Kernel Time: 0:0:0.328
User Time: 0:0:0.453
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 00f8fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00f8fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00f8ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00f8ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00f8ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00f8ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 23
System Thread ID: 16b8
Kernel Time: 0:0:0.359
User Time: 0:0:0.390
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 00f4fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00f4fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00f4ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00f4ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00f4ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00f4ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 24
System Thread ID: fd4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00ecff70 77f430c7 SharedUserData!SystemCallStub+0x4
01 00ecff74 77f7e6ae ntdll!ZwRemoveIoCompletion+0xc
02 00ecffb8 77e4a990 ntdll!RtlpWorkerThread+0x3b
03 00ecffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 25
System Thread ID: a34
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00f0ff70 77f430c7 SharedUserData!SystemCallStub+0x4
01 00f0ff74 77f7e6ae ntdll!ZwRemoveIoCompletion+0xc
02 00f0ffb8 77e4a990 ntdll!RtlpWorkerThread+0x3b
03 00f0ffec 00000000 kernel32!BaseThreadStart+0x34
*****
Dump name is formatted as: PID-Timestamp.dmp
Creating C:\iisstate\output\4376-1142278788.dmp - mini user dump
*****
Closing open log file C:\iisstate\output\IISState-4376.log
Finished
-------------------------------------------------------------------------------------------------------
I'm wondering if anyone could suggest where I go from here?
Thanks in advance...