I've got a dual Xeon W2k3 box that locks up intermittently. I'd like to
debug
or crash it when it stops responding. Ctrl + double ScrollLock doesn't work.
It's not convenient to have a debugger attached all the time. I wonder if
the following will work.

- I set /debug /debugport=1394 in boot.ini and boot the box
- Once the box has locked up, I connect a 1394 cable to a laptop with WinDbg
- I use WinDbg to debug or crash the box

What are my chances to have WinDbg connected to the target machine when the
latter is stuck - ie spinning at a high IRQL?

Can anyone suggest hardware-based solutions to crash a PC with memory dump?
Like a cheap PCI card, NIC or whatever, with two wires and a red button
soldered to it? Though I am afraid the problem is in video drivers, so even
if I managed to crash the computer, I won't get the memory dump anyway.

Thanks
Pete

--

Re: Lockup debugging by Maxim

Maxim
Thu Dec 09 03:39:16 CST 2004

> Can anyone suggest hardware-based solutions to crash a PC with memory dump?

Search www.microsoft.com for CrashOnCtrlScroll.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com



Re: Lockup debugging by Doron

Doron
Thu Dec 09 10:36:18 CST 2004

if crash on control + scroll lock doesn't work, either you have not set the
setting properly (have you verified this crashes the machine when it is
running?), or the keyboard interrupt isn't running when the machine locks up
(which means spinning at high irql or a hw lockup). in these cases, getting
the debugger to break in can sometimes be impossible.

d

--
Please do not send e-mail directly to this alias. this alias is for
newsgroup purposes only.
This posting is provided "AS IS" with no warranties, and confers no rights.


"Pete Johnston" <anonymous@discussions.microsoft.com> wrote in message
news:ujPg8zY3EHA.2540@TK2MSFTNGP09.phx.gbl...
> I've got a dual Xeon W2k3 box that locks up intermittently. I'd like to
> debug
> or crash it when it stops responding. Ctrl + double ScrollLock doesn't
> work.
> It's not convenient to have a debugger attached all the time. I wonder if
> the following will work.
>
> - I set /debug /debugport=1394 in boot.ini and boot the box
> - Once the box has locked up, I connect a 1394 cable to a laptop with
> WinDbg
> - I use WinDbg to debug or crash the box
>
> What are my chances to have WinDbg connected to the target machine when
> the latter is stuck - ie spinning at a high IRQL?
>
> Can anyone suggest hardware-based solutions to crash a PC with memory
> dump? Like a cheap PCI card, NIC or whatever, with two wires and a red
> button soldered to it? Though I am afraid the problem is in video drivers,
> so even if I managed to crash the computer, I won't get the memory dump
> anyway.
>
> Thanks
> Pete
>
> --
>
>
>
>



Re: Lockup debugging by Pete

Pete
Thu Dec 09 13:29:36 CST 2004

I can crash the machine when it's running with ScrollLock, and I can break
in with a debugger. Yesterday I had WinDbg/1394 attached. The machine locks
up after being restarted from hibernation, when I insert DVD+RW into
DVD-ROM. Inserting DVD+RW into DVD writers is ok. The machine did lock up in
other circumstances, but the DVD+RW case is 100% reproducible. Neither
ScrollLock nor Ctrl+Break, SysRq, Alt+SysReq, Ctrl+C, F12 work on both
machines. The debugger cannot break in. I tried several times with no
success, with Verifier and without.

The machine seems to be spinning since the CPU temperature is going up.

Can the debugger break in when the target is spinning at DISPATCH_LEVEL? Can
Verifier detect deadlocks at higher levels, such as ISR?

What is the highest priority interrupt handler that is periodically called
on Win2k3, if any? The system clock? Maybe I can hack the handler to call
KeBugCheck if, for instance, a global variable has not been updated for
awhile?

Any other suggestions?

Thanks!
Pete

--

"Doron Holan [MS]" <doronh@nospam.microsoft.com> wrote in message
news:uoPK40g3EHA.1152@TK2MSFTNGP14.phx.gbl...
> if crash on control + scroll lock doesn't work, either you have not set
> the setting properly (have you verified this crashes the machine when it
> is running?), or the keyboard interrupt isn't running when the machine
> locks up (which means spinning at high irql or a hw lockup). in these
> cases, getting the debugger to break in can sometimes be impossible.
>
> d
>
> --
> Please do not send e-mail directly to this alias. this alias is for
> newsgroup purposes only.
> This posting is provided "AS IS" with no warranties, and confers no
> rights.
>
>
> "Pete Johnston" <anonymous@discussions.microsoft.com> wrote in message
> news:ujPg8zY3EHA.2540@TK2MSFTNGP09.phx.gbl...
>> I've got a dual Xeon W2k3 box that locks up intermittently. I'd like to
>> debug
>> or crash it when it stops responding. Ctrl + double ScrollLock doesn't
>> work.
>> It's not convenient to have a debugger attached all the time. I wonder if
>> the following will work.
>>
>> - I set /debug /debugport=1394 in boot.ini and boot the box
>> - Once the box has locked up, I connect a 1394 cable to a laptop with
>> WinDbg
>> - I use WinDbg to debug or crash the box
>>
>> What are my chances to have WinDbg connected to the target machine when
>> the latter is stuck - ie spinning at a high IRQL?
>>
>> Can anyone suggest hardware-based solutions to crash a PC with memory
>> dump? Like a cheap PCI card, NIC or whatever, with two wires and a red
>> button soldered to it? Though I am afraid the problem is in video
>> drivers, so even if I managed to crash the computer, I won't get the
>> memory dump anyway.
>>
>> Thanks
>> Pete
>>
>> --
>>
>>
>>
>>
>
>



Re: Lockup debugging by pavel_a

pavel_a
Thu Dec 09 16:37:06 CST 2004

Aha... this looks like problem in a UDF driver.
Do you have 3rd party UDF writers (DLA, Roxio Drag to disc...)?
--PA

"Pete Johnston" wrote:
> I can crash the machine when it's running with ScrollLock, and I can break
> in with a debugger. Yesterday I had WinDbg/1394 attached. The machine locks
> up after being restarted from hibernation, when I insert DVD+RW into
> DVD-ROM. Inserting DVD+RW into DVD writers is ok. The machine did lock up in
> other circumstances, but the DVD+RW case is 100% reproducible. Neither
> ScrollLock nor Ctrl+Break, SysRq, Alt+SysReq, Ctrl+C, F12 work on both
> machines. The debugger cannot break in. I tried several times with no
> success, with Verifier and without.
>
> The machine seems to be spinning since the CPU temperature is going up.
>
> Can the debugger break in when the target is spinning at DISPATCH_LEVEL? Can
> Verifier detect deadlocks at higher levels, such as ISR?
>
> What is the highest priority interrupt handler that is periodically called
> on Win2k3, if any? The system clock? Maybe I can hack the handler to call
> KeBugCheck if, for instance, a global variable has not been updated for
> awhile?
>
> Any other suggestions?
>
> Thanks!
> Pete
>
> --
>
> "Doron Holan [MS]" <doronh@nospam.microsoft.com> wrote in message
> news:uoPK40g3EHA.1152@TK2MSFTNGP14.phx.gbl...
> > if crash on control + scroll lock doesn't work, either you have not set
> > the setting properly (have you verified this crashes the machine when it
> > is running?), or the keyboard interrupt isn't running when the machine
> > locks up (which means spinning at high irql or a hw lockup). in these
> > cases, getting the debugger to break in can sometimes be impossible.
> >
> > d
> >
> > --
> > Please do not send e-mail directly to this alias. this alias is for
> > newsgroup purposes only.
> > This posting is provided "AS IS" with no warranties, and confers no
> > rights.
> >
> >
> > "Pete Johnston" <anonymous@discussions.microsoft.com> wrote in message
> > news:ujPg8zY3EHA.2540@TK2MSFTNGP09.phx.gbl...
> >> I've got a dual Xeon W2k3 box that locks up intermittently. I'd like to
> >> debug
> >> or crash it when it stops responding. Ctrl + double ScrollLock doesn't
> >> work.
> >> It's not convenient to have a debugger attached all the time. I wonder if
> >> the following will work.
> >>
> >> - I set /debug /debugport=1394 in boot.ini and boot the box
> >> - Once the box has locked up, I connect a 1394 cable to a laptop with
> >> WinDbg
> >> - I use WinDbg to debug or crash the box
> >>
> >> What are my chances to have WinDbg connected to the target machine when
> >> the latter is stuck - ie spinning at a high IRQL?
> >>
> >> Can anyone suggest hardware-based solutions to crash a PC with memory
> >> dump? Like a cheap PCI card, NIC or whatever, with two wires and a red
> >> button soldered to it? Though I am afraid the problem is in video
> >> drivers, so even if I managed to crash the computer, I won't get the
> >> memory dump anyway.
> >>
> >> Thanks
> >> Pete
> >>
> >> --
> >>
> >>
> >>
> >>
> >
> >
>
>
>

Re: Lockup debugging by Pete

Pete
Thu Dec 09 18:15:51 CST 2004

I'd like to know the exact cause of the problem. Say I disabled some driver
and the problem went away. This doesn't prove the disabled driver is faulty.

UDF writers are not supposed to service or disable interrupts. They are
expected to run at <=DispatchLevel. So if the problem is in a software
driver, ScrollLock should crash the system. I've made a simple test with a
recursively acquired spinlock - the PC locks up, but ScrollLock works.

I'm going to try the checked kernel, then hook the clock ISR.


"Pavel A." <pavel_a@NOwritemeNO.com> wrote in message
news:6A0F5C4E-E7E4-47BB-AC4C-3708DE775677@microsoft.com...
> Aha... this looks like problem in a UDF driver.
> Do you have 3rd party UDF writers (DLA, Roxio Drag to disc...)?
> --PA
>
> "Pete Johnston" wrote:
>> I can crash the machine when it's running with ScrollLock, and I can
>> break
>> in with a debugger. Yesterday I had WinDbg/1394 attached. The machine
>> locks
>> up after being restarted from hibernation, when I insert DVD+RW into
>> DVD-ROM. Inserting DVD+RW into DVD writers is ok. The machine did lock up
>> in
>> other circumstances, but the DVD+RW case is 100% reproducible. Neither
>> ScrollLock nor Ctrl+Break, SysRq, Alt+SysReq, Ctrl+C, F12 work on both
>> machines. The debugger cannot break in. I tried several times with no
>> success, with Verifier and without.
>>
>> The machine seems to be spinning since the CPU temperature is going up.
>>
>> Can the debugger break in when the target is spinning at DISPATCH_LEVEL?
>> Can
>> Verifier detect deadlocks at higher levels, such as ISR?
>>
>> What is the highest priority interrupt handler that is periodically
>> called
>> on Win2k3, if any? The system clock? Maybe I can hack the handler to call
>> KeBugCheck if, for instance, a global variable has not been updated for
>> awhile?
>>
>> Any other suggestions?
>>
>> Thanks!
>> Pete
>>
>> --
>>
>> "Doron Holan [MS]" <doronh@nospam.microsoft.com> wrote in message
>> news:uoPK40g3EHA.1152@TK2MSFTNGP14.phx.gbl...
>> > if crash on control + scroll lock doesn't work, either you have not set
>> > the setting properly (have you verified this crashes the machine when
>> > it
>> > is running?), or the keyboard interrupt isn't running when the machine
>> > locks up (which means spinning at high irql or a hw lockup). in these
>> > cases, getting the debugger to break in can sometimes be impossible.
>> >
>> > d
>> >
>> > --
>> > Please do not send e-mail directly to this alias. this alias is for
>> > newsgroup purposes only.
>> > This posting is provided "AS IS" with no warranties, and confers no
>> > rights.
>> >
>> >
>> > "Pete Johnston" <anonymous@discussions.microsoft.com> wrote in message
>> > news:ujPg8zY3EHA.2540@TK2MSFTNGP09.phx.gbl...
>> >> I've got a dual Xeon W2k3 box that locks up intermittently. I'd like
>> >> to
>> >> debug
>> >> or crash it when it stops responding. Ctrl + double ScrollLock doesn't
>> >> work.
>> >> It's not convenient to have a debugger attached all the time. I wonder
>> >> if
>> >> the following will work.
>> >>
>> >> - I set /debug /debugport=1394 in boot.ini and boot the box
>> >> - Once the box has locked up, I connect a 1394 cable to a laptop with
>> >> WinDbg
>> >> - I use WinDbg to debug or crash the box
>> >>
>> >> What are my chances to have WinDbg connected to the target machine
>> >> when
>> >> the latter is stuck - ie spinning at a high IRQL?
>> >>
>> >> Can anyone suggest hardware-based solutions to crash a PC with memory
>> >> dump? Like a cheap PCI card, NIC or whatever, with two wires and a red
>> >> button soldered to it? Though I am afraid the problem is in video
>> >> drivers, so even if I managed to crash the computer, I won't get the
>> >> memory dump anyway.
>> >>
>> >> Thanks
>> >> Pete
>> >>
>> >> --
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>>
>>
>>



Re: Lockup debugging by Mark

Mark
Thu Dec 09 18:52:35 CST 2004

Pete Johnston wrote:
> I'd like to know the exact cause of the problem. Say I disabled some driver
> and the problem went away. This doesn't prove the disabled driver is faulty.
>
> UDF writers are not supposed to service or disable interrupts. They are
> expected to run at <=DispatchLevel. So if the problem is in a software
> driver, ScrollLock should crash the system. I've made a simple test with a
> recursively acquired spinlock - the PC locks up, but ScrollLock works.
>
> I'm going to try the checked kernel, then hook the clock ISR.
>

You can run verifier as well, it might catch the culpret.
Either here or on ntdev we've discussed the merits of a 'nmi button', a
dongle of your own construction that pushes an NMI onto the processor
and, in many cases of nonresponsive hung systems, will at least get you
a crash dump or a debug session.

Of course you have to have some skill at wiring stuff.


--

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032

Re: Lockup debugging by cristalink

cristalink
Sun Dec 12 14:17:00 CST 2004

I was using wrong keywords trying to find some info about the hardware
button. Thanks to you, Mark, now I found exactly what I was looking for -
http://www.microsoft.com/whdc/system/CEC/dmpsw.mspx . I wonder if it will
work when interrupts are disabled with _asm cli. I guess yes due to N in
NMI.

As to the lock-up problem,

- checked build didn't help since it stuck resuming from hibernation.
- hooking HalpClockInterrupt didn't help, probably because the interrupts
were disabled with cli.
- verifier didn't help due to heaps of bugs in asus/nvidia drivers. The
first two bugs detected by verifier were "An IRP dispatch handler has
returned without passing down or completing this IRP, or someone forgot to
return STATUS_PENDING" in ASUS driver, and raising IRQL to the hard-coded
value of 0x1A while at IRQL=0x1D in nvidia (the drivers passed WHQL!).

The problem was with a CD-ROM connected to an old Promise Ultra66 IDE
controller. The pair did not survive the hibernation.

Cheers,
Nick

--
http://www.firestreamer.com - NTBackup to DVD and DV


"Mark Roddy" <markr@hollistech.com> wrote in message
news:uzbRKKl3EHA.2572@tk2msftngp13.phx.gbl...
> Pete Johnston wrote:
>> I'd like to know the exact cause of the problem. Say I disabled some
>> driver and the problem went away. This doesn't prove the disabled driver
>> is faulty.
>>
>> UDF writers are not supposed to service or disable interrupts. They are
>> expected to run at <=DispatchLevel. So if the problem is in a software
>> driver, ScrollLock should crash the system. I've made a simple test with
>> a recursively acquired spinlock - the PC locks up, but ScrollLock works.
>>
>> I'm going to try the checked kernel, then hook the clock ISR.
>>
>
> You can run verifier as well, it might catch the culpret.
> Either here or on ntdev we've discussed the merits of a 'nmi button', a
> dongle of your own construction that pushes an NMI onto the processor and,
> in many cases of nonresponsive hung systems, will at least get you a crash
> dump or a debug session.
>
> Of course you have to have some skill at wiring stuff.
>
>
> --
>
> =====================
> Mark Roddy DDK MVP
> Windows 2003/XP/2000 Consulting
> Hollis Technology Solutions 603-321-1032