Hi all.
I have a problem with our website.
It is currently running on IIS 6. The site is mainly constructed around a
bunch of DLL's. Now, everything seems to work smoothly. As soon as more than
(roughly) 20 users are on one of the servers, the Current Anonymous Users
begins to rise. I know this is a strong pointer to possible hangups of the
DLL. So, to see which one it is, and why it happens, I ran IISState against
the main DLLHOST.exe, and pulled the DMP file through DebugDiag with a crash
analyser.
I get the following output. To be honest, I have NO idea what is happening,
and why this is causing a slowdown on the site.
Can anyone give a pointer to where to search?
Raymond
Analysis Summary
Type Description Recommendation
Warning Detected a possible critical section related
problem in 456-1129118709.dmp
Lock at PBP!DllUnregisterServer+ff2c0 is Transitioning
Impact analysis
11,54% of threads blocked
(Threads 10 11 12)
The following functions are involved in the root cause
PBP+25ea
PBP+27a1
The following modules are involved in the root cause
C:\SecureDir\PBP.dll from Inhouse development The
following vendors were identified for follow up based on root cause analysis
Inhouse development
Please follow up with the vendors identified above
Analysis Details
Table Of Contents
456-1129118709.dmp
Top 5 threads by CPU time
Locked critical section report
Thread report
Report for 456-1129118709.dmp
Type of Analysis Performed Hang Analysis
Machine Name MARKTPLAATS1
Operating System Windows Server 2003
Number Of Processors 4
Process ID 456
Process Image C:\WINDOWS\system32\dllhost.exe
System Up-Time 7 day(s) 23:41:28
Process Up-Time 0 day(s) 02:49:04
Top 5 Threads by CPU time
Note - Times include both user mode and kernel mode for each
thread Thread ID: 9 Total CPU Time: 0 day(s) 00:04:06.625 Entry
Point for Thread: msvcrt!_endthreadex+26
Thread ID: 10 Total CPU Time: 0 day(s) 00:03:16.296
Entry Point for Thread: msvcrt!_endthreadex+26
Thread ID: 11 Total CPU Time: 0 day(s) 00:02:16.875
Entry Point for Thread: msvcrt!_endthreadex+26
Thread ID: 12 Total CPU Time: 0 day(s) 00:00:41.265
Entry Point for Thread: msvcrt!_endthreadex+26
Thread ID: 13 Total CPU Time: 0 day(s) 00:00:34.406
Entry Point for Thread: msvcrt!_endthreadex+26
Locked critical section report
Critical Section PBP!DllUnregisterServer+ff2c0
Lock State Transitioning
Lock Count 2
Recursion Count 0
Entry Count 76561553
Contention Count 76560923
Spin Count 0
Thread report
Thread 0 - System ID 2672
Entry point dllhost!WinMainCRTStartup
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.15
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
ole32!CSurrogateProcessActivator::WaitForSurrogateTimeout+49
ole32!CoRegisterSurrogateEx+1a4
dllhost!WinMain+da
dllhost!WinMainCRTStartup+182
kernel32!BaseProcessStart+23
Thread 1 - System ID 2684
Entry point ole32!CRpcThreadCache::RpcWorkerThreadEntry
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
user32!NtUserGetMessage+c
ole32!CDllHost::STAWorkerLoop+70
ole32!CDllHost::WorkerThread+c6
ole32!DLLHostThreadEntry+9
ole32!CRpcThread::WorkerLoop+1e
ole32!_imp__InstallApplication <PERF> (ole32+0x0)
ole32!CRpcThreadCache::RpcWorkerThreadEntry+1f
kernel32!BaseThreadStart+34
Thread 2 - System ID 1676
Entry point ntdll!RtlpTimerThread
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
ntdll!RtlpTimerThread+45
kernel32!BaseThreadStart+34
Thread 3 - System ID 476
Entry point ntdll!RtlpWorkerThread
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34
Thread 4 - System ID 3976
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
comsvcs!WORK_QUEUE::WorkerLoop+66
comsvcs!WORK_QUEUE::ThreadLoop+15
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 5 - System ID 4620
Entry point comsvcs!PostData
Create time 12-10-2005 11:16:05
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
kernel32!Sleep+b
comsvcs!PostData+1db
kernel32!BaseThreadStart+34
Thread 6 - System ID 7468
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+25
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 7 - System ID 6596
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
comsvcs!CSTAThreadPool::KillThreadControlLoop+21
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 8 - System ID 3888
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:6.765
Time spent in kernel mode 0 Days 0:0:4.109
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 9 - System ID 5104
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:3:36.562
Time spent in kernel mode 0 Days 0:0:30.62
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 10 - System ID 5036
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:2:33.296
Time spent in kernel mode 0 Days 0:0:43.0
This thread is waiting on critical section
PBP!DllUnregisterServer+ff2c0 which is transitioning owners (no current
owner)
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
ntdll!RtlpWaitForCriticalSection+126
ntdll!RtlEnterCriticalSection+46
PBP+25ea
PBP+2c00
PBP!DllUnregisterServer+86f4e
PBP!DllUnregisterServer+8d542
PBP!DllUnregisterServer+8f457
PBP!DllUnregisterServer+c25ff
PBP!DllUnregisterServer+c2e63
PBP!DllUnregisterServer+f3728
rpcrt4!Invoke+30
rpcrt4!NdrStubCall2+229
rpcrt4!CStdStubBuffer_Invoke+3f
oleaut32!CUnivStubWrapper::Invoke+c3
ole32!SyncStubInvoke+33
ole32!StubInvoke+a5
ole32!CCtxComChnl::ContextInvoke+e3
ole32!MTAInvoke+18
ole32!STAInvoke+56
ole32!AppInvoke+7c
ole32!ComInvokeWithLockAndIPID+2be
ole32!ComInvoke+9d
ole32!ThreadDispatch+1f
ole32!ThreadWndProc+fc
user32!InternalCallWinProc+1b
user32!UserCallWinProcCheckWow+151
user32!DispatchMessageWorker+327
user32!DispatchMessageW+b
comsvcs!CSTAQueueLessMessageWork::DoWork+45
comsvcs!CSTAThread::DoWork+14
comsvcs!CSTAThread::ProcessQueueWork+30
comsvcs!CSTAThread::WorkerLoop+17a
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 11 - System ID 6256
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:1:9.31
Time spent in kernel mode 0 Days 0:1:7.843
This thread is waiting on critical section
PBP!DllUnregisterServer+ff2c0 which is transitioning owners (no current
owner)
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
ntdll!RtlpWaitForCriticalSection+126
ntdll!RtlEnterCriticalSection+46
PBP+27a1
PBP+2c2b
PBP!DllUnregisterServer+7695d
PBP!DllUnregisterServer+778d8
PBP!DllUnregisterServer+778d8
PBP!DllUnregisterServer+778d8
PBP!DllUnregisterServer+778d8
PBP!DllUnregisterServer+77dc3
PBP!DllUnregisterServer+6a900
PBP!DllUnregisterServer+6b9ab
PBP!DllUnregisterServer+6bb2d
PBP!DllUnregisterServer+6ba45
PBP!DllUnregisterServer+6bb2d
PBP!DllUnregisterServer+6ba45
PBP!DllUnregisterServer+6bb2d
PBP!DllUnregisterServer+702c2
PBP!DllUnregisterServer+70558
PBP!DllUnregisterServer+7139e
PBP!DllUnregisterServer+74168
PBP!DllUnregisterServer+74b85
PBP!DllUnregisterServer+ee717
rpcrt4!Invoke+30
rpcrt4!NdrStubCall2+229
rpcrt4!CStdStubBuffer_Invoke+3f
oleaut32!CUnivStubWrapper::Invoke+c3
ole32!SyncStubInvoke+33
ole32!StubInvoke+a5
ole32!CCtxComChnl::ContextInvoke+e3
ole32!MTAInvoke+18
ole32!STAInvoke+56
ole32!AppInvoke+7c
ole32!ComInvokeWithLockAndIPID+2be
ole32!ComInvoke+9d
ole32!ThreadDispatch+1f
ole32!ThreadWndProc+fc
user32!InternalCallWinProc+1b
user32!UserCallWinProcCheckWow+151
user32!DispatchMessageWorker+327
user32!DispatchMessageW+b
comsvcs!CSTAQueueLessMessageWork::DoWork+45
comsvcs!CSTAThread::DoWork+14
comsvcs!CSTAThread::ProcessQueueWork+30
comsvcs!CSTAThread::WorkerLoop+17a
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 12 - System ID 7080
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:17.406
Time spent in kernel mode 0 Days 0:0:23.859
This thread is waiting on critical section
PBP!DllUnregisterServer+ff2c0 which is transitioning owners (no current
owner)
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
ntdll!RtlpWaitForCriticalSection+126
ntdll!RtlEnterCriticalSection+46
PBP+25ea
PBP+2c00
PBP!DllUnregisterServer+86f4e
PBP!DllUnregisterServer+8d542
PBP!DllUnregisterServer+8f3a1
PBP!DllUnregisterServer+75ef7
PBP!DllUnregisterServer+7792a
PBP!DllUnregisterServer+75321
PBP!DllUnregisterServer+76a0c
PBP!DllUnregisterServer+77dc3
PBP!DllUnregisterServer+6a900
PBP!DllUnregisterServer+71128
PBP!DllUnregisterServer+7445a
PBP!DllUnregisterServer+74b85
PBP!DllUnregisterServer+74bd6
PBP!DllUnregisterServer+ee717
rpcrt4!Invoke+30
rpcrt4!NdrStubCall2+229
rpcrt4!CStdStubBuffer_Invoke+3f
oleaut32!CUnivStubWrapper::Invoke+c3
ole32!SyncStubInvoke+33
ole32!StubInvoke+a5
ole32!CCtxComChnl::ContextInvoke+e3
ole32!MTAInvoke+18
ole32!STAInvoke+56
ole32!AppInvoke+7c
ole32!ComInvokeWithLockAndIPID+2be
ole32!ComInvoke+9d
ole32!ThreadDispatch+1f
ole32!ThreadWndProc+fc
user32!InternalCallWinProc+1b
user32!UserCallWinProcCheckWow+151
user32!DispatchMessageWorker+327
user32!DispatchMessageW+b
comsvcs!CSTAQueueLessMessageWork::DoWork+45
comsvcs!CSTAThread::DoWork+14
comsvcs!CSTAThread::ProcessQueueWork+30
comsvcs!CSTAThread::WorkerLoop+17a
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 13 - System ID 152
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:16.515
Time spent in kernel mode 0 Days 0:0:17.890
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 14 - System ID 3512
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:3.109
Time spent in kernel mode 0 Days 0:0:3.187
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 15 - System ID 2648
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 16 - System ID 6740
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 17 - System ID 3472
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 18 - System ID 5300
Entry point msvcrt!_endthreadex+26
Create time 12-10-2005 11:16:29
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
comsvcs!CSTAThread::WorkerLoop+1e3
msvcrt!_endthreadex+95
kernel32!BaseThreadStart+34
Thread 19 - System ID 7428
Create time 00:00:00
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34
Thread 20 - System ID 8004
Entry point oledb32!ReleaseResourceThread
Create time 12-10-2005 11:20:01
Time spent in user mode 0 Days 0:0:0.31
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
oledb32!ReleaseResourceThread+54
kernel32!BaseThreadStart+34
Thread 21 - System ID 4088
Entry point oledb32!ResetResourceThread
Create time 12-10-2005 11:20:01
Time spent in user mode 0 Days 0:0:0.15
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
oledb32!ResetResourceThread+84
kernel32!BaseThreadStart+34
Thread 22 - System ID 6092
Entry point comsvcs!PingThread
Create time 12-10-2005 11:20:01
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
comsvcs!PingThread+ea
kernel32!BaseThreadStart+34
Thread 23 - System ID 3196
Create time 00:00:00
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34
Thread 24 - System ID 5252
Entry point rpcrt4!ThreadStartRoutine
Create time 12-10-2005 13:40:34
Time spent in user mode 0 Days 0:0:0.468
Time spent in kernel mode 0 Days 0:0:0.328
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34
Thread 25 - System ID 7556
Entry point ole32!CRpcThreadCache::RpcWorkerThreadEntry
Create time 12-10-2005 14:04:54
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
kernel32!Sleep+b
ole32!CROIDTable::WorkerThreadLoop+12
ole32!CRpcThread::WorkerLoop+1e
ole32!_imp__InstallApplication <PERF> (ole32+0x0)
ole32!CRpcThreadCache::RpcWorkerThreadEntry+1f
kernel32!BaseThreadStart+34
Script Summary
Script Name Status Error Code Error Source Error Description
Source Line
IISAnalysis.asp Completed