Memory barriers and pended IRPs

I'm concerned with compiler variable caching in general and
out-of-order stores on multiprocessor systems. Recently I started
thinking about this with relation to filling in the initial values for
data that is operated on by interrupt handlers and DPCs. An IRP
contains of set of these initial values. An IRP is created by the I/O
manager (or another driver) then passed to the dispatch routine of a
driver. This probably all happens in one thread context on one CPU.
If the IRP is then pended by doing IoMarkIrpPending and returning
STATUS_PENDING from the dispatch routine, then the IRP will eventually
be completed in another thread context possibly on a different CPU.

If the IRP is completed by a worker thread, then there will have been
an implicit compiler and CPU memory barrier in the event that woke up
the thread to handle the IRP. This case should be okay.

If the IRP is completed by a DPC, then on a uniprocessor machine the
DPC may see the old values of some of the fields in the IRP that the
compiler did not yet update, and on a multiprocessor machine the DPC
may see the old values that the other CPU did not yet update. Doesn't
there need to be a memory barrier to flush any out-of-order stores
immediately before writing the device register that will initiate the
operation that will eventually cause an interrupt and the DPC handler
to run?

I get the impression that sometimes KeAcquireInterruptSpinLock and
KeSynchronizeExecution are used while programming device registers,
and the interrupt spinlock that is held would provide the necessary
barrier, but it was not clear to me this was the purpose of these
routines. Besides, does everyone even use them?

BubbaGump
Sat Oct 14 11:52:39 CDT 2006

On Sat, 14 Oct 2006 11:14:42 -0400, BubbaGump <> wrote:

>I'm concerned with compiler variable caching in general and
>out-of-order stores on multiprocessor systems. Recently I started
>thinking about this with relation to filling in the initial values for
>data that is operated on by interrupt handlers and DPCs. An IRP
>contains of set of these initial values. An IRP is created by the I/O
>manager (or another driver) then passed to the dispatch routine of a
>driver. This probably all happens in one thread context on one CPU.
>If the IRP is then pended by doing IoMarkIrpPending and returning
>STATUS_PENDING from the dispatch routine, then the IRP will eventually
>be completed in another thread context possibly on a different CPU.
>
>If the IRP is completed by a worker thread, then there will have been
>an implicit compiler and CPU memory barrier in the event that woke up
>the thread to handle the IRP. This case should be okay.
>
>If the IRP is completed by a DPC, then on a uniprocessor machine the
>DPC may see the old values of some of the fields in the IRP that the
>compiler did not yet update, and on a multiprocessor machine the DPC
>may see the old values that the other CPU did not yet update. Doesn't
>there need to be a memory barrier to flush any out-of-order stores
>immediately before writing the device register that will initiate the
>operation that will eventually cause an interrupt and the DPC handler
>to run?

I thought about this more and realized that any interrupt-driven
driver must at least perform some sort of atomic test to make sure
there is no current operation in progress before initiating a new
operation. That atomic test (grabbing a lock or doing some
interlocked operation) should provide the barrier needed to flush any
values set up before that point.

Then the only problem is values that may be set up after that locked
test is performed but before the I/O is initiated, some values
auxiliary to the operation stored in the device extension for
instance. I think these need a KeMemoryBarrier after they are set up
and before the I/O is begun.

BubbaGump
Sat Oct 14 15:10:27 CDT 2006

On Sat, 14 Oct 2006 12:52:46 -0400, BubbaGump <> wrote:

>On Sat, 14 Oct 2006 11:14:42 -0400, BubbaGump <> wrote:
>
>>I'm concerned with compiler variable caching in general and
>>out-of-order stores on multiprocessor systems. Recently I started
>>thinking about this with relation to filling in the initial values for
>>data that is operated on by interrupt handlers and DPCs. An IRP
>>contains of set of these initial values. An IRP is created by the I/O
>>manager (or another driver) then passed to the dispatch routine of a
>>driver. This probably all happens in one thread context on one CPU.
>>If the IRP is then pended by doing IoMarkIrpPending and returning
>>STATUS_PENDING from the dispatch routine, then the IRP will eventually
>>be completed in another thread context possibly on a different CPU.
>>
>>If the IRP is completed by a worker thread, then there will have been
>>an implicit compiler and CPU memory barrier in the event that woke up
>>the thread to handle the IRP. This case should be okay.
>>
>>If the IRP is completed by a DPC, then on a uniprocessor machine the
>>DPC may see the old values of some of the fields in the IRP that the
>>compiler did not yet update, and on a multiprocessor machine the DPC
>>may see the old values that the other CPU did not yet update. Doesn't
>>there need to be a memory barrier to flush any out-of-order stores
>>immediately before writing the device register that will initiate the
>>operation that will eventually cause an interrupt and the DPC handler
>>to run?
>
>I thought about this more and realized that any interrupt-driven
>driver must at least perform some sort of atomic test to make sure
>there is no current operation in progress before initiating a new
>operation. That atomic test (grabbing a lock or doing some
>interlocked operation) should provide the barrier needed to flush any
>values set up before that point.
>
>Then the only problem is values that may be set up after that locked
>test is performed but before the I/O is initiated, some values
>auxiliary to the operation stored in the device extension for
>instance. I think these need a KeMemoryBarrier after they are set up
>and before the I/O is begun.

Wait. That's not a problem either _if_ KeAcquireInterruptSpinLock or
KeSynchronizeExecution is always used while initiating the I/O, since
this locked operation would be the last step in the sequence of
operations before the interrupt could happen.

(someone chime in if any of this sounds insane)

BubbaGump
Sat Oct 14 16:22:05 CDT 2006

On Sat, 14 Oct 2006 16:10:34 -0400, BubbaGump <> wrote:

>On Sat, 14 Oct 2006 12:52:46 -0400, BubbaGump <> wrote:
>
>>On Sat, 14 Oct 2006 11:14:42 -0400, BubbaGump <> wrote:
>>
>>>I'm concerned with compiler variable caching in general and
>>>out-of-order stores on multiprocessor systems. Recently I started
>>>thinking about this with relation to filling in the initial values for
>>>data that is operated on by interrupt handlers and DPCs. An IRP
>>>contains of set of these initial values. An IRP is created by the I/O
>>>manager (or another driver) then passed to the dispatch routine of a
>>>driver. This probably all happens in one thread context on one CPU.
>>>If the IRP is then pended by doing IoMarkIrpPending and returning
>>>STATUS_PENDING from the dispatch routine, then the IRP will eventually
>>>be completed in another thread context possibly on a different CPU.
>>>
>>>If the IRP is completed by a worker thread, then there will have been
>>>an implicit compiler and CPU memory barrier in the event that woke up
>>>the thread to handle the IRP. This case should be okay.
>>>
>>>If the IRP is completed by a DPC, then on a uniprocessor machine the
>>>DPC may see the old values of some of the fields in the IRP that the
>>>compiler did not yet update, and on a multiprocessor machine the DPC
>>>may see the old values that the other CPU did not yet update. Doesn't
>>>there need to be a memory barrier to flush any out-of-order stores
>>>immediately before writing the device register that will initiate the
>>>operation that will eventually cause an interrupt and the DPC handler
>>>to run?
>>
>>I thought about this more and realized that any interrupt-driven
>>driver must at least perform some sort of atomic test to make sure
>>there is no current operation in progress before initiating a new
>>operation. That atomic test (grabbing a lock or doing some
>>interlocked operation) should provide the barrier needed to flush any
>>values set up before that point.
>>
>>Then the only problem is values that may be set up after that locked
>>test is performed but before the I/O is initiated, some values
>>auxiliary to the operation stored in the device extension for
>>instance. I think these need a KeMemoryBarrier after they are set up
>>and before the I/O is begun.
>
>Wait. That's not a problem either _if_ KeAcquireInterruptSpinLock or
>KeSynchronizeExecution is always used while initiating the I/O, since
>this locked operation would be the last step in the sequence of
>operations before the interrupt could happen.
>
>(someone chime in if any of this sounds insane)

I should really say it is still a "problem" in that it takes some of
what I think is some non-intuitive thinking to decide whether to use
KeMemoryBarrier or KeSynchronizeExecution/KeAcquireInterruptSpinLock,
and I'm bothered this hasn't jumped out at me in the driver
documentation I've seen so far.

BubbaGump
Sun Oct 15 07:59:02 CDT 2006

There's another case I'm concerned about, the one where you're not the
one completing an IRP, where you send it off to another driver and set
an IoCompletion routine. Do IoSetCompletionRoutine and
IoSetCompletionRoutineEx have memory barriers to ensure the memory
pointed to by the context parameter is flushed on the first CPU since
the IoCompletion routine might run possibly on another CPU?

BubbaGump
Sun Oct 15 08:32:33 CDT 2006

On Sun, 15 Oct 2006 08:59:03 -0400, BubbaGump <> wrote:

>There's another case I'm concerned about, the one where you're not the
>one completing an IRP, where you send it off to another driver and set
>an IoCompletion routine. Do IoSetCompletionRoutine and
>IoSetCompletionRoutineEx have memory barriers to ensure the memory
>pointed to by the context parameter is flushed on the first CPU since
>the IoCompletion routine might run possibly on another CPU?

Actually, I just realized this eventually breaks down to the case of a
single driver pending an IRP on one CPU then completing it on another.
There must be a barrier on the first CPU before the final action that
enables the possibly second CPU to complete the IRP.

(Could these blog-like posts be a waste of time because no one cares
about subtle data corruption issues that may crop up on future CPUs or
architectures?)

hs2
Sun Oct 15 14:44:33 CDT 2006

>
> (Could these blog-like posts be a waste of time because no one cares
> about subtle data corruption issues that may crop up on future CPUs or
> architectures?)
>

Oh yes - good point. And as usual you 'answered' yourself ;)
If you want to tell sth. to the world just blog it and please don't
abuse news groups.
Thank you very much and good luck,

HS2

BubbaGump
Sun Oct 15 15:48:46 CDT 2006

On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:

>>
>> (Could these blog-like posts be a waste of time because no one cares
>> about subtle data corruption issues that may crop up on future CPUs or
>> architectures?)
>>
>
>Oh yes - good point. And as usual you 'answered' yourself ;)
>If you want to tell sth. to the world just blog it and please don't
>abuse news groups.
>Thank you very much and good luck,

I don't think it's even possible to abuse newsgroups. This newsgroup
is for technical discussion, and I'm trying to check if my or other
people's understanding of different technical issues is correct, so I
post questions usually and sometimes statements and then wait for
responses to see if people agree or disagree. It's been effective in
the past. Sometimes I realize my mistakes due to feedback from other
people. Other times I realize my own mistakes.

In any case, I was looking for more technical feedback, whether you
agree or disagree with how to resolve the problem or whether there is
a problem.

BubbaGump
Sun Oct 15 15:51:32 CDT 2006

On Sun, 15 Oct 2006 16:48:46 -0400, BubbaGump <> wrote:

>I don't think it's even possible to abuse newsgroups. This newsgroup
>is for technical discussion...

(Yeah, yeah, yeah, someone will say there's been abuse discussing
opinions in the past, but judge each thread by itself. This one is
reasonably pure.)

BubbaGump
Sun Oct 15 16:21:16 CDT 2006

On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:

>>
>> (Could these blog-like posts be a waste of time because no one cares
>> about subtle data corruption issues that may crop up on future CPUs or
>> architectures?)
>>
>
>Oh yes - good point. And as usual you 'answered' yourself ;)
>If you want to tell sth. to the world just blog it and please don't
>abuse news groups.

I agree, blogs are an abuse. I shouldn't have mentioned them. Blogs,
at least the ones I'v seen, are a temple raised to worship a single
person. I have no desire to visit a web site where each day the same
person posts more of their thoughts.

Newsgroups are good because a single check for new headers will grab a
bunch of different posts from all sorts of people, and if a particular
thread isn't interesting then a decent newsreader can simply ignore
any future posts to it. You, for instance, could ignore this thread.
(Don't ignore me. I may post something you find interesting later
on.)

In any case, I'm sorry I even used the word "blog".

BubbaGump
Sun Oct 15 16:29:16 CDT 2006

On Sun, 15 Oct 2006 17:21:16 -0400, BubbaGump <> wrote:

>I agree, blogs are an abuse. I shouldn't have mentioned them. Blogs,
>at least the ones I'v seen, are a temple raised to worship a single
>person. I have no desire to visit a web site where each day the same
>person posts more of their thoughts.
>
>Newsgroups are good because a single check for new headers will grab a
>bunch of different posts from all sorts of people, and if a particular
>thread isn't interesting then a decent newsreader can simply ignore
>any future posts to it. You, for instance, could ignore this thread.
>(Don't ignore me. I may post something you find interesting later
>on.)
>
>In any case, I'm sorry I even used the word "blog".

(Okay, I just have trouble keeping threads on topic. I get
distracted.)

BubbaGump
Sun Oct 15 17:14:28 CDT 2006

On Sat, 14 Oct 2006 17:22:13 -0400, BubbaGump <> wrote:

>On Sat, 14 Oct 2006 16:10:34 -0400, BubbaGump <> wrote:
>
>>On Sat, 14 Oct 2006 12:52:46 -0400, BubbaGump <> wrote:
>>
>>>On Sat, 14 Oct 2006 11:14:42 -0400, BubbaGump <> wrote:
>>>
>>>>I'm concerned with compiler variable caching in general and
>>>>out-of-order stores on multiprocessor systems. Recently I started
>>>>thinking about this with relation to filling in the initial values for
>>>>data that is operated on by interrupt handlers and DPCs. An IRP
>>>>contains of set of these initial values. An IRP is created by the I/O
>>>>manager (or another driver) then passed to the dispatch routine of a
>>>>driver. This probably all happens in one thread context on one CPU.
>>>>If the IRP is then pended by doing IoMarkIrpPending and returning
>>>>STATUS_PENDING from the dispatch routine, then the IRP will eventually
>>>>be completed in another thread context possibly on a different CPU.
>>>>
>>>>If the IRP is completed by a worker thread, then there will have been
>>>>an implicit compiler and CPU memory barrier in the event that woke up
>>>>the thread to handle the IRP. This case should be okay.
>>>>
>>>>If the IRP is completed by a DPC, then on a uniprocessor machine the
>>>>DPC may see the old values of some of the fields in the IRP that the
>>>>compiler did not yet update, and on a multiprocessor machine the DPC
>>>>may see the old values that the other CPU did not yet update. Doesn't
>>>>there need to be a memory barrier to flush any out-of-order stores
>>>>immediately before writing the device register that will initiate the
>>>>operation that will eventually cause an interrupt and the DPC handler
>>>>to run?
>>>
>>>I thought about this more and realized that any interrupt-driven
>>>driver must at least perform some sort of atomic test to make sure
>>>there is no current operation in progress before initiating a new
>>>operation. That atomic test (grabbing a lock or doing some
>>>interlocked operation) should provide the barrier needed to flush any
>>>values set up before that point.
>>>
>>>Then the only problem is values that may be set up after that locked
>>>test is performed but before the I/O is initiated, some values
>>>auxiliary to the operation stored in the device extension for
>>>instance. I think these need a KeMemoryBarrier after they are set up
>>>and before the I/O is begun.
>>
>>Wait. That's not a problem either _if_ KeAcquireInterruptSpinLock or
>>KeSynchronizeExecution is always used while initiating the I/O, since
>>this locked operation would be the last step in the sequence of
>>operations before the interrupt could happen.
>>
>>(someone chime in if any of this sounds insane)
>
>I should really say it is still a "problem" in that it takes some of
>what I think is some non-intuitive thinking to decide whether to use
>KeMemoryBarrier or KeSynchronizeExecution/KeAcquireInterruptSpinLock,
>and I'm bothered this hasn't jumped out at me in the driver
>documentation I've seen so far.

A subtle point about using the KeMemoryBarrier alternative for this
situation that I just realized after looking at a Linux doc on memory
barriers is that there would have to be a KeMemoryBarrier both before
the final write that would cause the interrupt and a paired
KeMemoryBarrier before the data was touched again (I think it would
make sense to put this at the beginning of the interrupt handler).
Each barrier would make sure each CPU's loads would see the other
CPU's stores and that each CPU's stores would be visible to the other
CPU's loads. It's a write/read and read/write pairing.

BubbaGump
Sun Oct 15 17:42:13 CDT 2006

On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:

>>
>> (Could these blog-like posts be a waste of time because no one cares
>> about subtle data corruption issues that may crop up on future CPUs or
>> architectures?)
>>
>
>Oh yes - good point. And as usual you 'answered' yourself ;)
>If you want to tell sth. to the world just blog it and please don't
>abuse news groups.
>Thank you very much and good luck,

Also, I apologize for asking for feedback. I should know not to care
if anyone is listening. I make these posts because often when I try
to search for perspectives on these issues I don't find anything. So,
these posts are mainly for the future searches of people like me.
People like you don't matter to me.

BubbaGump
Sun Oct 15 17:43:30 CDT 2006

On Sun, 15 Oct 2006 18:42:13 -0400, BubbaGump <> wrote:

>On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:
>
>>>
>>> (Could these blog-like posts be a waste of time because no one cares
>>> about subtle data corruption issues that may crop up on future CPUs or
>>> architectures?)
>>>
>>
>>Oh yes - good point. And as usual you 'answered' yourself ;)
>>If you want to tell sth. to the world just blog it and please don't
>>abuse news groups.
>>Thank you very much and good luck,
>
>Also, I apologize for asking for feedback. I should know not to care
>if anyone is listening. I make these posts because often when I try
>to search for perspectives on these issues I don't find anything. So,
>these posts are mainly for the future searches of people like me.
>People like you don't matter to me.

(well, at least only enough to say that)

BubbaGump
Sun Oct 15 19:43:30 CDT 2006

On Sun, 15 Oct 2006 09:32:34 -0400, BubbaGump <> wrote:

>about subtle data corruption issues that may crop up on future CPUs or
>architectures?)

"Future" could be inaccurate. I'm still thinking the Pentium's
ordering rules may not be predictable, at least with respect to
speculative loads.

I think the general rule should be not to access any data that is
shared between two code sequences without the protection of a lock or
memory barriers.

Interrupts are an odd case because they aren't triggered by any
standard routine in the OS where an implicit barrier could be done.
For instance, I would think there must be a barrier when an event is
set and when a thread is created, and those both require OS routines.
Triggering an interrupt doesn't.

What an asynchronous hell modern computers are.

BubbaGump
Sun Oct 15 20:00:55 CDT 2006

On Sun, 15 Oct 2006 20:43:30 -0400, BubbaGump <> wrote:

>Interrupts are an odd case because they aren't triggered by any
>standard routine in the OS where an implicit barrier could be done.
>For instance, I would think there must be a barrier when an event is
>set and when a thread is created, and those both require OS routines.
>Triggering an interrupt doesn't.

Creation is an important point. The subtle issue here isn't the
interrupt or DPC simply accessing some data. Everyone knows shared
data must be protected by some sort of lock. The issue is the
creation of an interrupt (and the subsequent creation of a DPC) that
will access data that won't be shared with it but will be handed off
to it. This situation appears serialized as far as the programmer is
concerned, but it isn't as far as the CPU is concerned unless explicit
barriers are used.

BubbaGump
Sun Oct 15 20:17:08 CDT 2006

On Sun, 15 Oct 2006 16:48:46 -0400, BubbaGump <> wrote:

>On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:
>
>>>
>>> (Could these blog-like posts be a waste of time because no one cares
>>> about subtle data corruption issues that may crop up on future CPUs or
>>> architectures?)
>>>
>>
>>Oh yes - good point. And as usual you 'answered' yourself ;)
>>If you want to tell sth. to the world just blog it and please don't
>>abuse news groups.
>>Thank you very much and good luck,
>
>I don't think it's even possible to abuse newsgroups...

It is interesting that books can be written (on subjects such as WDM
drivers) and can be sold for money even while containing some errors.

Here in "news"-groups information can be given out and corrected for
free. http://groups.google.com/ is where I go for perspective on many
questions.

BubbaGump
Sun Oct 15 20:24:48 CDT 2006

On Sun, 15 Oct 2006 21:17:08 -0400, BubbaGump <> wrote:

>On Sun, 15 Oct 2006 16:48:46 -0400, BubbaGump <> wrote:
>
>>On Sun, 15 Oct 2006 21:44:33 +0200, hs2 <me@nospam.org> wrote:
>>
>>>>
>>>> (Could these blog-like posts be a waste of time because no one cares
>>>> about subtle data corruption issues that may crop up on future CPUs or
>>>> architectures?)
>>>>
>>>
>>>Oh yes - good point. And as usual you 'answered' yourself ;)
>>>If you want to tell sth. to the world just blog it and please don't
>>>abuse news groups.
>>>Thank you very much and good luck,
>>
>>I don't think it's even possible to abuse newsgroups...
>
>It is interesting that books can be written (on subjects such as WDM
>drivers) and can be sold for money even while containing some errors.
>
>Here in "news"-groups information can be given out and corrected for
>free. http://groups.google.com/ is where I go for perspective on many
>questions.

(The trick is to read the thread backwards, newest post first. The
newest posts are the most accurate, not accounting for unpredictable
human stupidity, but that seems to be an unavoidable aspect of this
planet with no general way to determine whether any information no
matter how official it appears can be trusted. Let the individual
take as much information as he or she desires into account and draw
conclusions for him- or herself. I speak because I am.)

BubbaGump
Sun Oct 15 20:42:44 CDT 2006

On Sun, 15 Oct 2006 21:00:56 -0400, BubbaGump <> wrote:

>On Sun, 15 Oct 2006 20:43:30 -0400, BubbaGump <> wrote:
>
>>Interrupts are an odd case because they aren't triggered by any
>>standard routine in the OS where an implicit barrier could be done.
>>For instance, I would think there must be a barrier when an event is
>>set and when a thread is created, and those both require OS routines.
>>Triggering an interrupt doesn't.
>
>Creation is an important point. The subtle issue here isn't the
>interrupt or DPC simply accessing some data. Everyone knows shared
>data must be protected by some sort of lock. The issue is the
>creation of an interrupt (and the subsequent creation of a DPC) that
>will access data that won't be shared with it but will be handed off
>to it. This situation appears serialized as far as the programmer is
>concerned, but it isn't as far as the CPU is concerned unless explicit
>barriers are used.

Of course, the creation of a DPC is actually an OS routine,
IoRequestDpc. I wonder if IoRequestDpc has a barrier. It should but
even if it does it doesn't solve the problem in a neat way.

IoRequestDpc will be done from the possibly other CPU, not from the
CPU that caused the chain of events that lead to the interrupt that in
turn queues the DPC, so the initiating CPU would still need a barrier
and so would the interrupt handler if it accessed any data set up on
the first CPU (quickly of course since it's an interrupt handler).

BubbaGump
Sun Oct 15 20:52:51 CDT 2006

On Sun, 15 Oct 2006 21:42:44 -0400, BubbaGump <> wrote:

>On Sun, 15 Oct 2006 21:00:56 -0400, BubbaGump <> wrote:
>
>>On Sun, 15 Oct 2006 20:43:30 -0400, BubbaGump <> wrote:
>>
>>>Interrupts are an odd case because they aren't triggered by any
>>>standard routine in the OS where an implicit barrier could be done.
>>>For instance, I would think there must be a barrier when an event is
>>>set and when a thread is created, and those both require OS routines.
>>>Triggering an interrupt doesn't.
>>
>>Creation is an important point. The subtle issue here isn't the
>>interrupt or DPC simply accessing some data. Everyone knows shared
>>data must be protected by some sort of lock. The issue is the
>>creation of an interrupt (and the subsequent creation of a DPC) that
>>will access data that won't be shared with it but will be handed off
>>to it. This situation appears serialized as far as the programmer is
>>concerned, but it isn't as far as the CPU is concerned unless explicit
>>barriers are used.
>
>Of course, the creation of a DPC is actually an OS routine,
>IoRequestDpc. I wonder if IoRequestDpc has a barrier. It should but
>even if it does it doesn't solve the problem in a neat way.
>
>IoRequestDpc will be done from the possibly other CPU, not from the
>CPU that caused the chain of events that lead to the interrupt that in
>turn queues the DPC, so the initiating CPU would still need a barrier
>and so would the interrupt handler if it accessed any data set up on
>the first CPU (quickly of course since it's an interrupt handler).

It might be considered that the macros (like READ_REGISTER_ULONG) that
write device registers will already have a barrier, but the ones in my
3790.1830 DDK do the barrier after the write. That should guarantee
the other CPU has visibility of all writes up to and including the
final write, but it wouldn't guarantee the order in which they would
be seen. Another barrier still needs to be before the write to make
sure it is the very last in the sequence.

(and of course the interrupt handler still needs a barrier)

Maxim
Mon Oct 16 03:42:17 CDT 2006

> I think the general rule should be not to access any data that is
> shared between two code sequences without the protection of a lock or
> memory barriers.

Correct. Spinlock is enough, and it includes a memory barrier.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com

BubbaGump
Mon Oct 16 19:35:06 CDT 2006

On Sun, 15 Oct 2006 18:14:28 -0400, BubbaGump <> wrote:

>A subtle point about using the KeMemoryBarrier alternative for this
>situation that I just realized after looking at a Linux doc on memory
>barriers is that there would have to be a KeMemoryBarrier both before
>the final write that would cause the interrupt and a paired
>KeMemoryBarrier before the data was touched again (I think it would
>make sense to put this at the beginning of the interrupt handler).

Actually, I don't think this is true anymore. The second
KeMemoryBarrier before the data is touched again shouldn't be
necessary, since any sequence of instructions that begins running or
processing input data must have an implicit barrier:

For a thread, it's got to be the act of scheduling it when an event is
set (or the atomic removal of an item from a queue).

For a DPC, it's got to be the act of scheduling it.

For an interrupt, it's got to be the act of atomically making sure no
two instances of the interrupt can run at once.

The OS should do all this. It should only be the act of triggering
such things that's a concern. Threads and DPCs are triggered by OS
routines, but interrupts aren't, which is why I still think they need
to be preceded by an explicit barrier.