Mutual exclusion at passive level

What's the standard way to call routines that are documented to
require passive level inside code that needs to be mutually exclusive
with some other code? A spin lock is out because it raises to DPC
level, and I suppose dispatcher and fast mutexes are out because they
raise to APC level.

This APC level thing is what bugs me. It's sort of like passive level
(preemptive) but sort of not, so maybe it's safe to call passive level
code and maybe it's not. So, does this leave that trick with fast
mutexes of combining KeEnterCriticalRegion and
ExAcquireFastMutexUnsafe, or should I use a worker thread? (which
seems kind of cumbersome)

I wrote a bunch of code that very carefully holds a lock for as short
as possible, grabs some information, does choice parts of its
operations outside, then finally grabs the lock to store the result,
but when I look at it I see it could be so much simpler if there was a
safe way to make the whole large set of operations mutually exclusive.

Doron
Fri Dec 22 19:52:17 CST 2006

events, semaphores and ERESOURCEs are all fine options to synchronize at
passive.

d

--
Please do not send e-mail directly to this alias. this alias is for
newsgroup purposes only.
This posting is provided "AS IS" with no warranties, and confers no rights.


<BubbaGump> wrote in message
news:09voo2tt02rql9usm1dsihkkbjjn7ctjqq@4ax.com...
> What's the standard way to call routines that are documented to
> require passive level inside code that needs to be mutually exclusive
> with some other code? A spin lock is out because it raises to DPC
> level, and I suppose dispatcher and fast mutexes are out because they
> raise to APC level.
>
> This APC level thing is what bugs me. It's sort of like passive level
> (preemptive) but sort of not, so maybe it's safe to call passive level
> code and maybe it's not. So, does this leave that trick with fast
> mutexes of combining KeEnterCriticalRegion and
> ExAcquireFastMutexUnsafe, or should I use a worker thread? (which
> seems kind of cumbersome)
>
> I wrote a bunch of code that very carefully holds a lock for as short
> as possible, grabs some information, does choice parts of its
> operations outside, then finally grabs the lock to store the result,
> but when I look at it I see it could be so much simpler if there was a
> safe way to make the whole large set of operations mutually exclusive.
>

BubbaGump
Sat Dec 23 09:29:21 CST 2006

Okay, but don't all those need user APCs disabled in order to prevent
a user from causing a deadlock by suspending the thread? That's not
quite the pure PASSIVE_LEVEL some routines claim.

Here's the problem. If a routine says it requires passive level, that
means it must need some sort (user, normal kernel, or special kernel)
APCs to be enabled, but the lock will need at least user APCs
disabled, and there doesn't need to be a way to satisfy this without
also disabling normal kernel APCs. How do I know whether this will
conflict with some requirement the routine has to have user or normal
APCs enabled?




On Fri, 22 Dec 2006 17:52:17 -0800, "Doron Holan [MS]"
<doronh@nospam.microsoft.com> wrote:

>events, semaphores and ERESOURCEs are all fine options to synchronize at
>passive.
>
>d

Don
Sat Dec 23 09:43:36 CST 2006


<BubbaGump> wrote in message
news:nniqo2d255j9lpc1b6bv0aeb59ikdh2dnq@4ax.com...
> Okay, but don't all those need user APCs disabled in order to prevent
> a user from causing a deadlock by suspending the thread? That's not
> quite the pure PASSIVE_LEVEL some routines claim.
>
> On Fri, 22 Dec 2006 17:52:17 -0800, "Doron Holan [MS]"
> <doronh@nospam.microsoft.com> wrote:
>
>>events, semaphores and ERESOURCEs are all fine options to synchronize at
>>passive.
>>

Nope, none of those need user APC's disabled, ERESOURCES need normal kernel
APC's disabled, but that is not APC level!


--
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

BubbaGump
Sat Dec 23 12:53:29 CST 2006

On Sat, 23 Dec 2006 10:43:36 -0500, "Don Burn" <burn@stopspam.acm.org>
wrote:

>
><BubbaGump> wrote in message
>news:nniqo2d255j9lpc1b6bv0aeb59ikdh2dnq@4ax.com...
>> Okay, but don't all those need user APCs disabled in order to prevent
>> a user from causing a deadlock by suspending the thread? That's not
>> quite the pure PASSIVE_LEVEL some routines claim.
>>
>> On Fri, 22 Dec 2006 17:52:17 -0800, "Doron Holan [MS]"
>> <doronh@nospam.microsoft.com> wrote:
>>
>>>events, semaphores and ERESOURCEs are all fine options to synchronize at
>>>passive.
>>>
>
>Nope, none of those need user APC's disabled, ERESOURCES need normal kernel
>APC's disabled, but that is not APC level!


I know they're not documented as needing user APCs disabled in order
to carry out their locking operations, but I thought disabling user
APCs while a lock is held is just a good idea. If user APCs aren't
disabled then what prevents a user from using SuspendThread while a
lock is held? (or is it not a good idea to prevent a user from using
SuspendThread while a lock is held?)


I know I asked about these APC issues a while ago, but I get confused
by the documentation when I see:

"Callers of ... must be running at IRQL <= APC_LEVEL"

but for other routines:

"Callers of ... must be running at IRQL = PASSIVE_LEVEL"

The first implies user, normal kernel, and special kernel APCs can all
be disabled.

The second seems to imply user, normal kernel, and special kernel APCs
must all be enabled, but I have trouble believing it really means all
these types of APCs. I wish the documentation was more specific.

David
Sat Dec 23 13:21:19 CST 2006

It is specific. It says what IRQL and the docs also say if APCs (user
and/or special) must be disabled. Don't try and read too much into the
docs. Sometimes that will bite you, but without the source code to the OS
you can never know for sure. Even then it would be so much code that it is
even better to have access to the author/maintainer of that code. Also try
a sequence in a simple function in a simple driver and run prefast. If you
keep the code flow simple prefast is good at finding you not grabbing the
critical section before getting an ERESOURCE.

<BubbaGump> wrote in message
news:vstqo2poa30gefckorpkgh6okkp1trc1nr@4ax.com...
> On Sat, 23 Dec 2006 10:43:36 -0500, "Don Burn" <burn@stopspam.acm.org>
> wrote:
>
>>
>><BubbaGump> wrote in message
>>news:nniqo2d255j9lpc1b6bv0aeb59ikdh2dnq@4ax.com...
>>> Okay, but don't all those need user APCs disabled in order to prevent
>>> a user from causing a deadlock by suspending the thread? That's not
>>> quite the pure PASSIVE_LEVEL some routines claim.
>>>
>>> On Fri, 22 Dec 2006 17:52:17 -0800, "Doron Holan [MS]"
>>> <doronh@nospam.microsoft.com> wrote:
>>>
>>>>events, semaphores and ERESOURCEs are all fine options to synchronize at
>>>>passive.
>>>>
>>
>>Nope, none of those need user APC's disabled, ERESOURCES need normal
>>kernel
>>APC's disabled, but that is not APC level!
>
>
> I know they're not documented as needing user APCs disabled in order
> to carry out their locking operations, but I thought disabling user
> APCs while a lock is held is just a good idea. If user APCs aren't
> disabled then what prevents a user from using SuspendThread while a
> lock is held? (or is it not a good idea to prevent a user from using
> SuspendThread while a lock is held?)
>
>
> I know I asked about these APC issues a while ago, but I get confused
> by the documentation when I see:
>
> "Callers of ... must be running at IRQL <= APC_LEVEL"
>
> but for other routines:
>
> "Callers of ... must be running at IRQL = PASSIVE_LEVEL"
>
> The first implies user, normal kernel, and special kernel APCs can all
> be disabled.
>
> The second seems to imply user, normal kernel, and special kernel APCs
> must all be enabled, but I have trouble believing it really means all
> these types of APCs. I wish the documentation was more specific.
>

BubbaGump
Sat Dec 23 16:01:19 CST 2006

The documentation is only reasonably specific to me, but I wish it was
more specific. APCs aren't always mentioned. Types of APCs only seem
to be mentioned for the routines related to locking. Most routines
only mention IRQL, but it's a bit limiting if passive level routines
really can't be called with some APCs disabled.

Read more into the docs. Sometimes not doing that will bite you
because you'll let a bug go that will blow up in random, subtle,
unexpected ways. The only downside I see to seeking understanding is
accomplishing less in the same amount of time, but I'd rather
accomplish a few really good things than a lot of sort of okay things.

I don't need all the details of the source code, just a few more, and
I'd trust them unless I found out later they were wrong. My issues
isn't with accuracy. It's with precision.

Prefast? I might run verification tools after I've found all the bugs
I can find with my own eyes and mind, not before or instead of.
Testing is a complement to design and code inspection, not a
substitute. I'd rather write code that relies on understanding and
planning rather than ignorance and trial-and-error.




On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
<Dave@yoshimuni.com> wrote:

>It is specific. It says what IRQL and the docs also say if APCs (user
>and/or special) must be disabled. Don't try and read too much into the
>docs. Sometimes that will bite you, but without the source code to the OS
>you can never know for sure. Even then it would be so much code that it is
>even better to have access to the author/maintainer of that code. Also try
>a sequence in a simple function in a simple driver and run prefast. If you
>keep the code flow simple prefast is good at finding you not grabbing the
>critical section before getting an ERESOURCE.

BubbaGump
Sat Dec 23 16:17:35 CST 2006

I noticed another case in the documentation that helps to illustrate
the ambiguity of "PASSIVE_LEVEL". I see some routines that say:

1) "Callers of ... must be running at IRQL = PASSIVE_LEVEL and
with APCs enabled."

as opposed to:

2) "Callers of ... must be running at IRQL = PASSIVE_LEVEL"

or:

3) "Callers of ... must be running at IRQL <= APC_LEVEL"


(2) is the case that's ambiguous to me. If (1) requires all APCs to
be enabled and (3) allows all APCs to be disabled, then that seems to
imply that (2) must require some APCs to be enabled and allow other
APCs to be disabled, but which APCs?

BubbaGump
Sat Dec 23 18:07:18 CST 2006

I do agree that some parts of the documentation are composed of
statements that are a bit mistaken and defy reason presented from
other sources (sometimes the documentation itself). I ignore those,
but usually it's with some reason.

For this issue, could it be that every instance of "must be running at
IRQL = PASSIVE_LEVEL" is meant to be interpreted as "must be running
at IRQL <= APC_LEVEL"? I'm interested in why the doc authors made the
subtle distinction. Is the documentation wrong because requiring all
APCs to be enabled would be an unrealistic limitation for most
routines <= DISPATCH_LEVEL? So far this is the only reason I can
find, but I don't have much faith in it yet, so I then ask which types
of APCs are important in different PASSIVE_LEVEL cases.




On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
<Dave@yoshimuni.com> wrote:

>Don't try and read too much into the
>docs.

BubbaGump
Sat Dec 23 19:48:12 CST 2006

"Reasonably" is the wrong word. I should have used "moderately".




On Sat, 23 Dec 2006 17:01:13 -0500, BubbaGump <> wrote:

>The documentation is only reasonably specific to me, but I wish it was
>more specific.

David
Sat Dec 23 19:48:10 CST 2006

I will only post one reply for the multiple messages you posted. Testing is
not a complement, but an essential element of writing code in the Windows
world. It is not possible for a human being to know enough to get it
correct all the time. Not even Microsoft can write perfect code. They have
source code access and can ask the owner of a driver stack about the rules,
but even they make mistakes. You can only use what you have and with
current documentation that is not much, but it is better than it used to be
when almost all documentation on Windows 9x and MS-DOS was from third party
books, magazine articles, etc. Most of the driver stacks have ways a
company can get help from Microsoft. MSDN has incidents that come with some
of the subscriptions and more can be purchased. Many major manufacturers of
hardware have on-site personnel at Microsoft where they have access to the
developers who write the OS interfaces for their hardware. The little guy
has big problems because the expense and isolation makes it hard to write
good drivers.

I have worked in the storage and file systems stack and now the storage
stack is almost standard with if devices follow certain specs they will be
automatically supported with in-box drivers. When working for small
companies we found it difficult to obtain good support. Some times we were
told that we could not have access to source or there was no solution to our
problem. Basically it was a polite form of the 'take it and shove it' style
answer.

All you can do is write your code the way the docs and samples indicate.
Then test it until it won't break. Don't forget driver verifier with all
options active, prefast, PC-Lint, Coverity, and anything else you can find.
Some of the IO stress tests that come with HCT 12.1 are very good for
storage and file systems drivers. Use top quality hardware for your tests
beds. Look for at least a quad processor SMP system with 8 processors being
better. Run Data Center Server with full driver verifier and even run the
partial checked build in that environment. You will not get all the
problems, but you can keep improving quality if you work at it.

Part of the problem with NT from 3.51 to LongHorn is that the initial design
was not for shrug-and-pray. That feature including power management was
added into existing code. It should have been redesigned from the ground
up, but it couldn't be done for many reasons. The pursuit of backwards
compatibilty it another problem. The MAC has dumped old hardware and OS
several times in its existance. Maybe that is an idea whose time has come
to the Windows world? Definitely not an easy decision.

<BubbaGump> wrote in message
news:kl9ro2t1tq9431m3hfa8hlmkhqj4jplroh@4ax.com...
> The documentation is only reasonably specific to me, but I wish it was
> more specific. APCs aren't always mentioned. Types of APCs only seem
> to be mentioned for the routines related to locking. Most routines
> only mention IRQL, but it's a bit limiting if passive level routines
> really can't be called with some APCs disabled.
>
> Read more into the docs. Sometimes not doing that will bite you
> because you'll let a bug go that will blow up in random, subtle,
> unexpected ways. The only downside I see to seeking understanding is
> accomplishing less in the same amount of time, but I'd rather
> accomplish a few really good things than a lot of sort of okay things.
>
> I don't need all the details of the source code, just a few more, and
> I'd trust them unless I found out later they were wrong. My issues
> isn't with accuracy. It's with precision.
>
> Prefast? I might run verification tools after I've found all the bugs
> I can find with my own eyes and mind, not before or instead of.
> Testing is a complement to design and code inspection, not a
> substitute. I'd rather write code that relies on understanding and
> planning rather than ignorance and trial-and-error.
>
>
>
>
> On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
> <Dave@yoshimuni.com> wrote:
>
>>It is specific. It says what IRQL and the docs also say if APCs (user
>>and/or special) must be disabled. Don't try and read too much into the
>>docs. Sometimes that will bite you, but without the source code to the OS
>>you can never know for sure. Even then it would be so much code that it
>>is
>>even better to have access to the author/maintainer of that code. Also
>>try
>>a sequence in a simple function in a simple driver and run prefast. If
>>you
>>keep the code flow simple prefast is good at finding you not grabbing the
>>critical section before getting an ERESOURCE.
>

BubbaGump
Sat Dec 23 20:07:21 CST 2006

On Sat, 23 Dec 2006 10:43:36 -0500, "Don Burn" <burn@stopspam.acm.org>
wrote:

>ERESOURCES need normal kernel
>APC's disabled, but that is not APC level!

Are you trying to say that just because IRQL is PASSIVE_LEVEL that
makes everything okay for routines that claim to require
"PASSIVE_LEVEL"?

If IRQL was PASSIVE_LEVEL but within a guarded region with all APCs
disabled then wouldn't this effectively be like IRQL being APC_LEVEL?
The IRQL variable inside the OS would be different, but would there be
any functional difference? Wouldn't any APCs queued not run, and
isn't the ability to have some sort of queued APCs run the reason for
being at PASSIVE_LEVEL?

BubbaGump
Sat Dec 23 23:38:11 CST 2006

There's another way to look at this. There is implementation that
holds all the details, and there is interface that only holds some of
them. The implementation and interface should be consistent with each
other, but they should also be self-consistent. You act as if I'm
asking for details about the implementation. What I'm asking for
details about is the interface. Some of the rules it presents don't
appear to be realistic. I consider that an inconsistency between the
interface and its intended purpose.

I am trying to ask questions to reconcile the inconsistency. If there
is no way to reconcile it with the resources reasonably available to
me, then the issue will drop on its own, but I won't give up without
trying.




On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
<Dave@yoshimuni.com> wrote:

>Don't try and read too much into the
>docs. Sometimes that will bite you, but without the source code to the OS
>you can never know for sure. Even then it would be so much code that it is
>even better to have access to the author/maintainer of that code.

David
Sun Dec 24 00:51:19 CST 2006

I don't think the separation of interface and implementation is all that
easy to do. With any interface there are multiple ways to interpret it and
some will choose a path that had they known the underlying implementation
they would have avoided. It is just a fact of life. Compound the problem
with multiple languages for the documents. Those translations are usually
not done by the designer or programmer who actually wrote the code. Most
programmers have to proof read the docs and may not spend enough time to
avoid possible ambiguities. I know it is hard to document something you
have been living with for several months. You get tunnel vision and assume
the docs are targeted as someone who knows as much as you do.

Let's look at little into the file system stack. When a write irp goes down
through a file system filter driver from an application what locks are
acquired by the file system, cache manager, and memory manager before the
paging IO is seen coming through the file system filter driver again? What
type of locks? What context is being used or can be used? What happens in
the storage stack and how does it affect your file system filter driver? If
you are attempting to do file specific caching by using a file system filter
driver and a storage stack driver, how can you coordinate such caching
intelligently? I know of a company that had such a product for 2000 server.

Also for the file system filter stack what problem was encountered with the
use of IoCancelFileOpen() that made it useless? You can't tell from the
docs that the implementation was fated to fail because of some minor detail.
No filter could know that there was a problem until the crash.

<BubbaGump> wrote in message
news:ba4so2p996jt5dg4k82r951ihffvqqd5l8@4ax.com...
> There's another way to look at this. There is implementation that
> holds all the details, and there is interface that only holds some of
> them. The implementation and interface should be consistent with each
> other, but they should also be self-consistent. You act as if I'm
> asking for details about the implementation. What I'm asking for
> details about is the interface. Some of the rules it presents don't
> appear to be realistic. I consider that an inconsistency between the
> interface and its intended purpose.
>
> I am trying to ask questions to reconcile the inconsistency. If there
> is no way to reconcile it with the resources reasonably available to
> me, then the issue will drop on its own, but I won't give up without
> trying.
>
>
>
>
> On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
> <Dave@yoshimuni.com> wrote:
>
>>Don't try and read too much into the
>>docs. Sometimes that will bite you, but without the source code to the OS
>>you can never know for sure. Even then it would be so much code that it
>>is
>>even better to have access to the author/maintainer of that code.
>

pavel_a
Sun Dec 24 12:39:00 CST 2006

"BubbaGump" wrote:
> I noticed another case in the documentation that helps to illustrate
> the ambiguity of "PASSIVE_LEVEL". I see some routines that say:
>
> 1) "Callers of ... must be running at IRQL = PASSIVE_LEVEL and
> with APCs enabled."
>
> as opposed to:
>
> 2) "Callers of ... must be running at IRQL = PASSIVE_LEVEL"
>
> or:
>
> 3) "Callers of ... must be running at IRQL <= APC_LEVEL"
>
>
> (2) is the case that's ambiguous to me. If (1) requires all APCs to
> be enabled and (3) allows all APCs to be disabled, then that seems to
> imply that (2) must require some APCs to be enabled and allow other
> APCs to be disabled, but which APCs?

This is probably because APC is one of ancient mechanisms that Windows
(both NT kernel and win32) got from it's DEC architecture roots.
Now, when we have threads and even thread pools, it is harder
to understand (and explain in the documentation ) why APCs exist
and what are right ways to use them.
So better don't touch them, unless you absolutely have to.
Instead, use normal threads and work items.

Regards,
--PA

BubbaGump
Sun Dec 24 12:51:38 CST 2006

Admittedly, there is one example in some code I wrote that uses a
function at a higher IRQL than the documentation explicitly mentions.
I use RtlStringCchVPrintfExA at DISPATCH_LEVEL (while holding a spin
lock). Actually, the description of RtlStringCchVPrintfExA doesn't
mention IRQL at all, but I can't imagine why string/buffer
manipulations might perform operations that can't be done at raised
IRQL. Although I did see a post once that mentioned Unicode strings
can't be manipulated at high IRQL, I hesitantly conclude that ASCII
strings will be okay. I never posted a question here to confirm this.

I just find the difference between PASSIVE_LEVEL and APC_LEVEL to be a
little more ambiguous, which is why I'm trying to ask for more
information.




On Sat, 23 Dec 2006 11:21:19 -0800, "David J. Craig"
<Dave@yoshimuni.com> wrote:

>Don't try and read too much into the
>docs.