PAE question about memory access

Hello,

I have some questions related to the use of PAE for access to memory beyond 4G
in 32-bit Windows drivers. I'm trying to understand how a PCI device which
supports dual address cycles works in conjunction with a device driver running
on a system with PAE enabled.

As I understand it any memory that the system provides can be either above or
below the 4G mark.

If a driver calls IoGetDmaAdapter with Dma32BitAddresses set to TRUE (and
Dma64bitAddresses set to FALSE), does this mean that when AllocateCommonBuffer
is called that it will only give memory which is physically below 4G? Or does
it mean that the memory can be anywhere, but the "logical" address it returns
will always be a 32-bit quantity? Or is there some other interpretation?

In my case, the device only wants to receive 32-bit host memory addresses
to access, but we do want the device to be able take advantage of the
upper memory, if possible.

Also, if for some reason we did want to restrict the memory for this device to
the lower 4G of memory, is there a way to specify it? We might want to
support some device types which can support DAC and some which do not in the
same driver at the same time.

Thanks,
Nathan

Mark
Tue Jun 28 06:43:50 CDT 2005

Nathan G wrote:
> Hello,
>
> I have some questions related to the use of PAE for access to memory beyond 4G
> in 32-bit Windows drivers. I'm trying to understand how a PCI device which
> supports dual address cycles works in conjunction with a device driver running
> on a system with PAE enabled.
>
> As I understand it any memory that the system provides can be either above or
> below the 4G mark.
>
> If a driver calls IoGetDmaAdapter with Dma32BitAddresses set to TRUE (and
> Dma64bitAddresses set to FALSE), does this mean that when AllocateCommonBuffer
> is called that it will only give memory which is physically below 4G? Or does
> it mean that the memory can be anywhere, but the "logical" address it returns
> will always be a 32-bit quantity? Or is there some other interpretation?

The addresses your device sees will be below 4G (otherwise your device
would not work.) The system will use map register 'bounce buffers' to
support your device, copying to or from the map register bounce buffers
and the actual user data buffer as required when you perform DMA operations.
>
> In my case, the device only wants to receive 32-bit host memory addresses
> to access, but we do want the device to be able take advantage of the
> upper memory, if possible.
>

If it does not support 64 addresses then it cannot directly access
anything above 4G, right? DMA map registers will fix up transfers from
your device to your application data buffers, copying as required,
making this all transparent to your device, your driver, and your
applications. Except of course you will be paying for the extra copy
operation.

> Also, if for some reason we did want to restrict the memory for this device to
> the lower 4G of memory, is there a way to specify it? We might want to
> support some device types which can support DAC and some which do not in the
> same driver at the same time.
>

If your device supports DAC then you indicate Dma64BitAddresses=TRUE and
you will support direct DMA transfers to any physical memory address. If
your device does not support DAC then you set Dma64BitAddresses=FALSE
and the OS will fix everything up so that the fact that your device is
limited is pretty much transparent to your driver, to the device, and to
your applications.

> Thanks,
> Nathan
>


--

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

NathanG
Tue Jun 28 10:29:12 CDT 2005

Mark,

Thanks for your reply. Just a couple of things to make certain I've got
everything straight.

>> In my case, the device only wants to receive 32-bit host memory addresses
>> to access, but we do want the device to be able take advantage of the
>> upper memory, if possible.
>
>If it does not support 64 addresses then it cannot directly access
>anything above 4G, right? DMA map registers will fix up transfers from
>your device to your application data buffers, copying as required,
>making this all transparent to your device, your driver, and your
>applications. Except of course you will be paying for the extra copy
>operation.

Some of our devices do not support 64-bit/DAC. Some of them do. Even the ones
that do are only expecting to see a 32-bit memory address from the host.

>The addresses your device sees will be below 4G (otherwise your device
>would not work.) The system will use map register 'bounce buffers' to
>support your device, copying to or from the map register bounce buffers
>and the actual user data buffer as required when you perform DMA operations.
>...
>If your device supports DAC then you indicate Dma64BitAddresses=TRUE and
>you will support direct DMA transfers to any physical memory address. If
>your device does not support DAC then you set Dma64BitAddresses=FALSE
>and the OS will fix everything up so that the fact that your device is
>limited is pretty much transparent to your driver, to the device, and to
>your applications.

Is it the case then, that whether or not Dma64BitAddresses is set to TRUE,
that AllocateCommonBuffer will always give an address value in which HighPart
is 0?

Is it the case even if Dma64BitAddresses is set to FALSE, that physical memory
beyond 4GB may be used, but it will do so inefficiently?

For those devices which do not support DAC, is there any way to tell
the system that you'd only like memory below 4GB for its buffers? Or does
one just have to make do with the inefficiencies?

Thanks,
Nathan

Mark
Tue Jun 28 11:38:59 CDT 2005

Nathan G wrote:
> Mark,
>
> Thanks for your reply. Just a couple of things to make certain I've got
> everything straight.
>
>
>>>In my case, the device only wants to receive 32-bit host memory addresses
>>>to access, but we do want the device to be able take advantage of the
>>>upper memory, if possible.
>>
>>If it does not support 64 addresses then it cannot directly access
>>anything above 4G, right? DMA map registers will fix up transfers from
>>your device to your application data buffers, copying as required,
>>making this all transparent to your device, your driver, and your
>>applications. Except of course you will be paying for the extra copy
>>operation.
>
>
> Some of our devices do not support 64-bit/DAC. Some of them do. Even the ones
> that do are only expecting to see a 32-bit memory address from the host.
>

Why not support 64bit addresses on your 64-bit/DAC devices and 32bit on
your non-64-bit/DAC devices?

>
>>The addresses your device sees will be below 4G (otherwise your device
>>would not work.) The system will use map register 'bounce buffers' to
>>support your device, copying to or from the map register bounce buffers
>>and the actual user data buffer as required when you perform DMA operations.
>>...
>>If your device supports DAC then you indicate Dma64BitAddresses=TRUE and
>>you will support direct DMA transfers to any physical memory address. If
>>your device does not support DAC then you set Dma64BitAddresses=FALSE
>>and the OS will fix everything up so that the fact that your device is
>>limited is pretty much transparent to your driver, to the device, and to
>>your applications.
>
>
> Is it the case then, that whether or not Dma64BitAddresses is set to TRUE,
> that AllocateCommonBuffer will always give an address value in which HighPart
> is 0?
>

I don't see why. It might be the case that all buffers allocated using
AllocateCommonBuffer happen to be below 4G, it might not. The guarantee
is that the buffer allocated for your device will meet the requirements
specified when you allocated a DMA adapter. So for a 64bit capable
device that would be anywhere, for a 32bit device that would be below 4G.

> Is it the case even if Dma64BitAddresses is set to FALSE, that physical memory
> beyond 4GB may be used, but it will do so inefficiently?
>

Yes. Othwerwise application direct IO would not work.

> For those devices which do not support DAC, is there any way to tell
> the system that you'd only like memory below 4GB for its buffers? Or does
> one just have to make do with the inefficiencies?

No - otherwise application direct IO would not work.

NathanG
Tue Jun 28 12:39:04 CDT 2005

>> Some of our devices do not support 64-bit/DAC. Some of them do. Even the
>> ones that do are only expecting to see a 32-bit memory address from the
>> host.
>
>Why not support 64bit addresses on your 64-bit/DAC devices and 32bit on
>your non-64-bit/DAC devices?

The 64-bit/DAC devices are already existing products which don't currently
know anything about 64-bit, even though the hardware would theoretically
support it.

So the on-board firmware only accepts 32-bit system memory addresses
from the driver. It might be possible to change that, but that would
be more of a change than just to the driver. Sounds like that would
be required for us to take proper advantage of PAE.


Btw, a sort of related question. From the documentation for IoGetDmaAdapter:

>In NumberOfMapRegisters, the caller specifies the optimal number of map
>registers it can use. On output, the I/O Manager returns the number of map
>registers it allocated. Drivers should check the returned value; there is no
>guarantee a driver will receive the same number of map registers it requested.

I don't really understand what value I would put in there. Currently
(for my 32-bit only setup) I am putting 0. When would I put non-zero?
The devices perform bus master DMA to/from the allocated buffers.

Thanks again,
Nathan

Mark
Tue Jun 28 15:54:53 CDT 2005

Nathan G wrote:
>>>Some of our devices do not support 64-bit/DAC. Some of them do. Even the
>>>ones that do are only expecting to see a 32-bit memory address from the
>>>host.
>>
>>Why not support 64bit addresses on your 64-bit/DAC devices and 32bit on
>>your non-64-bit/DAC devices?
>
>
> The 64-bit/DAC devices are already existing products which don't currently
> know anything about 64-bit, even though the hardware would theoretically
> support it.
>
> So the on-board firmware only accepts 32-bit system memory addresses
> from the driver. It might be possible to change that, but that would
> be more of a change than just to the driver. Sounds like that would
> be required for us to take proper advantage of PAE.
>
>
> Btw, a sort of related question. From the documentation for IoGetDmaAdapter:
>
>
>>In NumberOfMapRegisters, the caller specifies the optimal number of map
>>registers it can use. On output, the I/O Manager returns the number of map
>>registers it allocated. Drivers should check the returned value; there is no
>>guarantee a driver will receive the same number of map registers it requested.
>
>
> I don't really understand what value I would put in there. Currently
> (for my 32-bit only setup) I am putting 0. When would I put non-zero?
> The devices perform bus master DMA to/from the allocated buffers.
>
> Thanks again,
> Nathan
>
Actually this is an output parameter for IoGetDmaAdapter. For a 32bit
PCI S/G busmaster device on a PAE system you will get back '16' no
matter what value you use on input. The only restriction you can impose
is the MaximumLength field in the DEVICE_DESCRIPTION. For your S/G
busmaster device on a 32bit non PAE system you should get back the value
'0' indicating that you can ignore map register limits. Likewise if you
enable DAC mode in your firmware and indicate 64bit support in the
DEVICE_EXTENSION you will also get back '0' on a PAE system.

Maxim
Thu Jun 30 18:58:05 CDT 2005

> If a driver calls IoGetDmaAdapter with Dma32BitAddresses set to TRUE (and
> Dma64bitAddresses set to FALSE), does this mean that when
AllocateCommonBuffer
> is called that it will only give memory which is physically below 4G?

Yes. Such call is only for devices which are not Dual Address Cycle capable.
Otherwise, Dma64bitAddresses == TRUE.

> it mean that the memory can be anywhere, but the "logical" address it returns
> will always be a 32-bit quantity?

Can be so too, if the plaform has its own hardware address translator.

>support some device types which can support DAC and some which do not in
>the
> same driver at the same time.

Determine the device ID of your device by config space read or by
IoGetDeviceProperty in MN_START_DEVICE path.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com

Maxim
Thu Jun 30 18:59:53 CDT 2005

> For those devices which do not support DAC, is there any way to tell
> the system that you'd only like memory below 4GB for its buffers?

Dma64BitAddresses = FALSE does this.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com