Hi all,
A few hours ago, our main web server started locking, with 100% CPU
taken up by the aspnet_wp process.
Restarting IIS stops the lock, but anything from 10 seconds to 10
minutes later, the 100%CPU problem is back again - so our sites have
been down now for 4 hours; and our phone's jamming with tech support
calls.
Although IIS State locked up with 100% CPU the first time we ran it,
after another go we managed to get this dump, below.
Please, if there's anyone who can shed some light on the cause of the
problem, it would be really appreciated.
Kind regards,
Paul McGillivray
Opened log file 'F:\Admin\iisstate\output\IISState-2288.log'
***********************
Starting new log output
IISState version 3.3.1
Mon Apr 02 14:50:13 2007
OS = Windows 2000
Executable: aspnet_wp.exe
PID = 2288
Note: Thread times are formatted as HH:MM:SS.ms
***********************
Thread ID: 0
System Thread ID: d00
Kernel Time: 0:0:0.15
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 0012f6d8 7c59a2d1 ntdll!ZwDelayExecution+0xb
01 0012f6f8 7c59a29c KERNEL32!SleepEx+0x32
02 0012f704 00442df0 KERNEL32!Sleep+0xb
03 0012ff60 0044408d aspnet_wp!wmain+0x3a1
04 0012ffc0 7c5989a5 aspnet_wp!wmainCRTStartup+0x143
05 0012fff0 00000000 KERNEL32!BaseProcessStart+0x3d
Thread ID: 1
System Thread ID: 904
Kernel Time: 0:0:0.156
User Time: 0:0:1.109
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
succeeded
Loaded Son of Strike data table version 5 from "C:\WINNT\Microsoft.NET
\Framework\v1.1.4322\mscorwks.dll"
Thread 1
ESP EIP
Begin System Thread Information
# ChildEBP RetAddr
00 00bfff54 7c585433 ntdll!NtRemoveIoCompletion+0xb
01 00bfff80 792e9b2f KERNEL32!GetQueuedCompletionStatus+0x27
02 00bfffb4 7c57b396 mscorwks!ThreadpoolMgr::CompletionPortThreadStart
+0x49
03 00bfffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 2
System Thread ID: cd0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 2
Not a managed thread.
Begin System Thread Information
# ChildEBP RetAddr
00 00cfff2c 7c59a2d1 ntdll!ZwDelayExecution+0xb
01 00cfff4c 7c59a29c KERNEL32!SleepEx+0x32
02 00cfff58 792e9ebe KERNEL32!Sleep+0xb
03 00cfffb4 7c57b396 mscorwks!ThreadpoolMgr::GateThreadStart+0x54
04 00cfffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 3
System Thread ID: d04
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00dfff3c 7c59a072 ntdll!ZwWaitForSingleObject+0xb
01 00dfff64 7c57b3e9 KERNEL32!WaitForSingleObjectEx+0x71
02 00dfff74 00443020 KERNEL32!WaitForSingleObject+0xf
03 00dfff80 7c34940f aspnet_wp!DoPingThread+0x10
04 00dfffb4 7c57b396 MSVCR71!_endthread+0xaa
05 00dfffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 4
System Thread ID: 880
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 4
Not a managed thread.
Begin System Thread Information
# ChildEBP RetAddr
00 04f8fecc 7c59a23d ntdll!ZwWaitForMultipleObjects+0xb
01 04f8ff1c 7c59a150 KERNEL32!WaitForMultipleObjectsEx+0xea
02 04f8ff34 791d25d5 KERNEL32!WaitForMultipleObjects+0x17
03 04f8ff9c 791d254c mscorwks!DebuggerRCThread::MainLoop+0x90
04 04f8ffac 791d4d50 mscorwks!DebuggerRCThread::ThreadProc+0x68
05 04f8ffb4 7c57b396 mscorwks!DebuggerRCThread::ThreadProcStatic+0xb
06 04f8ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 5
System Thread ID: 718
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 5
ESP EIP
Begin System Thread Information
# ChildEBP RetAddr
00 0712ff28 7c59a072 ntdll!ZwWaitForSingleObject+0xb
01 0712ff50 7c57b3e9 KERNEL32!WaitForSingleObjectEx+0x71
02 0712ff60 791bbe7a KERNEL32!WaitForSingleObject+0xf
03 0712ff70 791bbea4 mscorwks!WaitForFinalizerEvent+0xde
04 0712ffb4 7c57b396 mscorwks!GCHeap::FinalizerThreadStart+0x96
05 0712ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 6
System Thread ID: b5c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 0723fe24 77d595d9 ntdll!ZwReplyWaitReceivePortEx+0xb
01 0723ff74 77d58e4a RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x74
02 0723ff78 77d3aeed RPCRT4!RecvLotsaCallsWrapper+0x9
03 0723ffa8 77d37de8 RPCRT4!BaseCachedThreadRoutine+0x4f
04 0723ffb4 7c57b396 RPCRT4!ThreadStartRoutine+0x18
05 0723ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 7
System Thread ID: 464
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0733ff5c 7c59a2d1 ntdll!ZwDelayExecution+0xb
01 0733ff7c 7c59a29c KERNEL32!SleepEx+0x32
02 0733ff88 7ce89acc KERNEL32!Sleep+0xb
03 0733ff90 7ce85ba7 ole32!CROIDTable::WorkerThreadLoop+0xc
04 0733ffa8 7ce85aff ole32!CRpcThread::WorkerLoop+0x22
05 0733ffb4 7c57b396 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a
06 0733ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 8
System Thread ID: 4a0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 8
Not a managed thread.
Begin System Thread Information
# ChildEBP RetAddr
00 076dff80 7c59a2d1 ntdll!ZwDelayExecution+0xb
01 076dffa0 792ea483 KERNEL32!SleepEx+0x32
02 076dffb4 7c57b396 mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03 076dffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 9
System Thread ID: d18
Kernel Time: 0:0:0.140
User Time: 0:0:0.812
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 9
ESP EIP
Begin System Thread Information
# ChildEBP RetAddr
00 077eff54 7c585433 ntdll!NtRemoveIoCompletion+0xb
01 077eff80 792e9b2f KERNEL32!GetQueuedCompletionStatus+0x27
02 077effb4 7c57b396 mscorwks!ThreadpoolMgr::CompletionPortThreadStart
+0x49
03 077effec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 10
System Thread ID: 854
Kernel Time: 0:0:0.31
User Time: 0:0:0.15
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 10
ESP EIP
Begin System Thread Information
# ChildEBP RetAddr
00 0794ff58 7c59a072 ntdll!ZwWaitForSingleObject+0xb
01 0794ff80 7c57b3e9 KERNEL32!WaitForSingleObjectEx+0x71
02 0794ff90 792e91e0 KERNEL32!WaitForSingleObject+0xf
03 0794ffb4 7c57b396 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x3a
04 0794ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 11
System Thread ID: d1c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 11
Not a managed thread.
Begin System Thread Information
# ChildEBP RetAddr
00 0877ff54 7c585433 ntdll!NtRemoveIoCompletion+0xb
01 0877ff80 792e9b2f KERNEL32!GetQueuedCompletionStatus+0x27
02 0877ffb4 7c57b396 mscorwks!ThreadpoolMgr::CompletionPortThreadStart
+0x49
03 0877ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 12
System Thread ID: b70
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** ERROR: Symbol file could not be found. Defaulted to export
symbols for C:\WINNT\Microsoft.NET\Framework\v1.1.4322\Gdiplus.dll -
Thread Type: Other
# ChildEBP RetAddr
00 0909fea0 7c59a23d ntdll!ZwWaitForMultipleObjects+0xb
01 0909fef0 77e4169f KERNEL32!WaitForMultipleObjectsEx+0xea
02 0909ff4c 77e41706 USER32!__ClientExtTextOutW+0x3f
03 0909ff68 70d0e2a6 USER32!__ClientGetTextExtentPointW+0x48
WARNING: Stack unwind information not available. Following frames may
be wrong.
04 00000001 00000000 Gdiplus!GdipCreateFontFamilyFromName+0x223
*****
Dump name is formatted as: PID-Timestamp.dmp
Creating F:\Admin\iisstate\output\2288-1175521827.dmp - mini user dump
*****
Closing open log file F:\Admin\iisstate\output\IISState-2288.log