accessing physical address instead of virtual address

I need to increase the performance of the driver I wrote, and to do that, I
need to write data directly to the physical address of the pci device we are
using. I have already tried, in assembly, to write data to the virtual
address of the device, which was mapped with MmMapIoSpace. It worked fine,
but when I change the virtual address with the physical address, I get the
Blue screen. The routine to write is basically

esi <- buffer
edi <- virtual add
movsd

inside a for loop (just testing).

I heard that in order to write directly to the physical address, I need to
elevate the driver to be "real kernel" level, and not only "windows kernel"
level, where I cannot access the physical memory.

Any suggestions would be greatly appreciated.

Mark
Tue May 31 20:24:10 CDT 2005

mfranco wrote:
> I need to increase the performance of the driver I wrote, and to do that, I
> need to write data directly to the physical address of the pci device we are
> using. I have already tried, in assembly, to write data to the virtual
> address of the device, which was mapped with MmMapIoSpace. It worked fine,
> but when I change the virtual address with the physical address, I get the
> Blue screen. The routine to write is basically
>
> esi <- buffer
> edi <- virtual add
> movsd
>
> inside a for loop (just testing).
>
> I heard that in order to write directly to the physical address, I need to
> elevate the driver to be "real kernel" level, and not only "windows kernel"
> level, where I cannot access the physical memory.
>
> Any suggestions would be greatly appreciated.

You don't use physical memory addresses for accessing anything from the
OS. As you discovered, using a physical address will crash or corrupt
the system. The virtual->physical address translation is performed in
hardware and its overhead is negligible. "real mode" is obsolete 16-bit
x86 architecture nonsense. Yes it exists, no you do not use it, no it
will not speed up transfers to your pci device.

You need to get a better understanding of OS fundamentals and Windows
driver programming. I suggest that you read Windows Internals (4th
edition, Russinovich and Soloman, Microsoft Press) and Programming the
Windows Driver Model (2nd edition, Wlater Oney, Microsoft Press.)

If you have access to the Windows DDK you need to read up on using PIO
mode to read and write device memory.

--

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

Calvin
Tue May 31 21:13:48 CDT 2005

You cannot access physical address if paging has been enabled which is true
for most of the generic purpose OSes. CPU treats all addresses as virtual
and the V->P translation is done automatically by the MMU and the translated
phy address is assert to A31-A0. Grab an intel pentium manual and take a
look at the "Memory Management".

--
Calvin Guan Windows DDK MVP
Staff SW Engineer, NetXtreme MINIPORT
Enterprise Network Controller Engineering
Broadcom Corporation www.broadcom.com

"mfranco" <mfranco@discussions.microsoft.com> wrote in message
news:0A4E6BF6-5CEF-40A8-8263-009A0EDF185B@microsoft.com...
>I need to increase the performance of the driver I wrote, and to do that, I
> need to write data directly to the physical address of the pci device we
> are
> using. I have already tried, in assembly, to write data to the virtual
> address of the device, which was mapped with MmMapIoSpace. It worked fine,
> but when I change the virtual address with the physical address, I get the
> Blue screen. The routine to write is basically
>
> esi <- buffer
> edi <- virtual add
> movsd
>
> inside a for loop (just testing).
>
> I heard that in order to write directly to the physical address, I need to
> elevate the driver to be "real kernel" level, and not only "windows
> kernel"
> level, where I cannot access the physical memory.
>
> Any suggestions would be greatly appreciated.

mfranco
Tue May 31 21:15:21 CDT 2005

I agree with you BUT, we ran the following test: When using virtual address,
the OS firstly copies data to system memory and then copies this data to
device memory. So we have TWO copies instead of only one, directly to device
memory. We know that because our device was set in a state that it does not
respond to READ requests. So, when we try to read data from it, it should
return nothing, invalid data. Instead, when we try to read from it, we are
getting valid data, that we just wrote to the memory.

Conclusion: we are reading data from system memory, and not from device
memory.

Anyway, a few miliseconds that we save by bypassing that first copy using
virtual address, and writing directly to device memory, would make a great
diference. Another point that comes: Just by writing the write routine in
assembly, we gained almost 1 milisecond in performance.

As we have some hardware and time limitations, I realy would like to get
everything I can from this driver, and I think that writing an assembly
routine that accesses device's physical memory directly will do the trick.

Once again, any suggestion on how to do that are more than welcome.

"Mark Roddy" wrote:

> mfranco wrote:
> > I need to increase the performance of the driver I wrote, and to do that, I
> > need to write data directly to the physical address of the pci device we are
> > using. I have already tried, in assembly, to write data to the virtual
> > address of the device, which was mapped with MmMapIoSpace. It worked fine,
> > but when I change the virtual address with the physical address, I get the
> > Blue screen. The routine to write is basically
> >
> > esi <- buffer
> > edi <- virtual add
> > movsd
> >
> > inside a for loop (just testing).
> >
> > I heard that in order to write directly to the physical address, I need to
> > elevate the driver to be "real kernel" level, and not only "windows kernel"
> > level, where I cannot access the physical memory.
> >
> > Any suggestions would be greatly appreciated.
>
> You don't use physical memory addresses for accessing anything from the
> OS. As you discovered, using a physical address will crash or corrupt
> the system. The virtual->physical address translation is performed in
> hardware and its overhead is negligible. "real mode" is obsolete 16-bit
> x86 architecture nonsense. Yes it exists, no you do not use it, no it
> will not speed up transfers to your pci device.
>
> You need to get a better understanding of OS fundamentals and Windows
> driver programming. I suggest that you read Windows Internals (4th
> edition, Russinovich and Soloman, Microsoft Press) and Programming the
> Windows Driver Model (2nd edition, Wlater Oney, Microsoft Press.)
>
> If you have access to the Windows DDK you need to read up on using PIO
> mode to read and write device memory.
>
> --
>
> =====================
> Mark Roddy DDK MVP
> Windows 2003/XP/2000 Consulting
> Hollis Technology Solutions 603-321-1032
> www.hollistech.com
>

Mark
Tue May 31 21:38:56 CDT 2005

mfranco wrote:
> I agree with you BUT, we ran the following test: When using virtual address,
> the OS firstly copies data to system memory and then copies this data to
> device memory.

Uh no. A virtual address for PCI device memory is just that: a system
virtual address for PCI device memory. There is no copy performed in
reading/writing virtual addresses, other than the actual load/store
operation itself, which I assure you, if you have done things correctly,
which I am fairly certain you have not, will result in a direct transfer
from system memory to device memeory or vice versa.

>
So we have TWO copies instead of only one, directly to device
> memory. We know that because our device was set in a state that it does not
> respond to READ requests. So, when we try to read data from it, it should
> return nothing, invalid data. Instead, when we try to read from it, we are
> getting valid data, that we just wrote to the memory.

Your experiment has some other bogus assumptions in it.

>
> Conclusion: we are reading data from system memory, and not from device
> memory.

Wrong conclusion. As the saying goes, garbage in: garbage out.

>
> Anyway, a few miliseconds that we save by bypassing that first copy using
> virtual address, and writing directly to device memory, would make a great
> diference. Another point that comes: Just by writing the write routine in
> assembly, we gained almost 1 milisecond in performance.
>
> As we have some hardware and time limitations, I realy would like to get
> everything I can from this driver, and I think that writing an assembly
> routine that accesses device's physical memory directly will do the trick.
>
> Once again, any suggestion on how to do that are more than welcome.
>

OK then. Go for it. I tried. You obviously know better. Good luck.

> "Mark Roddy" wrote:
>
>
>>mfranco wrote:
>>
>>>I need to increase the performance of the driver I wrote, and to do that, I
>>>need to write data directly to the physical address of the pci device we are
>>>using. I have already tried, in assembly, to write data to the virtual
>>>address of the device, which was mapped with MmMapIoSpace. It worked fine,
>>>but when I change the virtual address with the physical address, I get the
>>>Blue screen. The routine to write is basically
>>>
>>>esi <- buffer
>>>edi <- virtual add
>>>movsd
>>>
>>>inside a for loop (just testing).
>>>
>>>I heard that in order to write directly to the physical address, I need to
>>>elevate the driver to be "real kernel" level, and not only "windows kernel"
>>>level, where I cannot access the physical memory.
>>>
>>>Any suggestions would be greatly appreciated.
>>
>>You don't use physical memory addresses for accessing anything from the
>>OS. As you discovered, using a physical address will crash or corrupt
>>the system. The virtual->physical address translation is performed in
>>hardware and its overhead is negligible. "real mode" is obsolete 16-bit
>>x86 architecture nonsense. Yes it exists, no you do not use it, no it
>>will not speed up transfers to your pci device.
>>
>>You need to get a better understanding of OS fundamentals and Windows
>>driver programming. I suggest that you read Windows Internals (4th
>>edition, Russinovich and Soloman, Microsoft Press) and Programming the
>>Windows Driver Model (2nd edition, Wlater Oney, Microsoft Press.)
>>
>>If you have access to the Windows DDK you need to read up on using PIO
>>mode to read and write device memory.
>>
>>--
>>
>>=====================
>>Mark Roddy DDK MVP
>>Windows 2003/XP/2000 Consulting
>>Hollis Technology Solutions 603-321-1032
>>www.hollistech.com
>>


--

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

Calvin
Tue May 31 22:08:44 CDT 2005

Mark is correct as always.

OP, What do you mean by "does not respond to a READ"? Does it send a
disconnect? or just didn't claim the transaction? One thing for sure is that
you're not inserting wait state. Is the memory NonCached? Anyway, there must
something wrong here.

--
Calvin Guan Windows DDK MVP
Staff SW Engineer, NetXtreme MINIPORT
Enterprise Network Controller Engineering
Broadcom Corporation www.broadcom.com


"Mark Roddy" <markr@hollistech.com> wrote in message
news:ui0NPMlZFHA.2768@tk2msftngp13.phx.gbl...
> mfranco wrote:
>> I agree with you BUT, we ran the following test: When using virtual
>> address, the OS firstly copies data to system memory and then copies this
>> data to device memory.
>
> Uh no. A virtual address for PCI device memory is just that: a system
> virtual address for PCI device memory. There is no copy performed in
> reading/writing virtual addresses, other than the actual load/store
> operation itself, which I assure you, if you have done things correctly,
> which I am fairly certain you have not, will result in a direct transfer
> from system memory to device memeory or vice versa.
>
> >
> So we have TWO copies instead of only one, directly to device
>> memory. We know that because our device was set in a state that it does
>> not respond to READ requests. So, when we try to read data from it, it
>> should return nothing, invalid data. Instead, when we try to read from
>> it, we are getting valid data, that we just wrote to the memory.
>
> Your experiment has some other bogus assumptions in it.
>
>>
>> Conclusion: we are reading data from system memory, and not from device
>> memory.
>
> Wrong conclusion. As the saying goes, garbage in: garbage out.
>
>>
>> Anyway, a few miliseconds that we save by bypassing that first copy using
>> virtual address, and writing directly to device memory, would make a
>> great diference. Another point that comes: Just by writing the write
>> routine in assembly, we gained almost 1 milisecond in performance. As we
>> have some hardware and time limitations, I realy would like to get
>> everything I can from this driver, and I think that writing an assembly
>> routine that accesses device's physical memory directly will do the
>> trick.
>>
>> Once again, any suggestion on how to do that are more than welcome.
>>
>
> OK then. Go for it. I tried. You obviously know better. Good luck.
>
>> "Mark Roddy" wrote:
>>
>>
>>>mfranco wrote:
>>>
>>>>I need to increase the performance of the driver I wrote, and to do
>>>>that, I need to write data directly to the physical address of the pci
>>>>device we are using. I have already tried, in assembly, to write data to
>>>>the virtual address of the device, which was mapped with MmMapIoSpace.
>>>>It worked fine, but when I change the virtual address with the physical
>>>>address, I get the Blue screen. The routine to write is basically
>>>>
>>>>esi <- buffer
>>>>edi <- virtual add
>>>>movsd
>>>>
>>>>inside a for loop (just testing).
>>>>I heard that in order to write directly to the physical address, I need
>>>>to elevate the driver to be "real kernel" level, and not only "windows
>>>>kernel" level, where I cannot access the physical memory.
>>>>
>>>>Any suggestions would be greatly appreciated.
>>>
>>>You don't use physical memory addresses for accessing anything from the
>>>OS. As you discovered, using a physical address will crash or corrupt the
>>>system. The virtual->physical address translation is performed in
>>>hardware and its overhead is negligible. "real mode" is obsolete 16-bit
>>>x86 architecture nonsense. Yes it exists, no you do not use it, no it
>>>will not speed up transfers to your pci device.
>>>
>>>You need to get a better understanding of OS fundamentals and Windows
>>>driver programming. I suggest that you read Windows Internals (4th
>>>edition, Russinovich and Soloman, Microsoft Press) and Programming the
>>>Windows Driver Model (2nd edition, Wlater Oney, Microsoft Press.)
>>>
>>>If you have access to the Windows DDK you need to read up on using PIO
>>>mode to read and write device memory.
>>>
>>>--
>>>
>>>=====================
>>>Mark Roddy DDK MVP
>>>Windows 2003/XP/2000 Consulting
>>>Hollis Technology Solutions 603-321-1032
>>>www.hollistech.com
>>>
>
>
> --
>
> =====================
> Mark Roddy DDK MVP
> Windows 2003/XP/2000 Consulting
> Hollis Technology Solutions 603-321-1032
> www.hollistech.com

mfranco
Wed Jun 01 06:28:01 CDT 2005



"Calvin Guan" wrote:

> Mark is correct as always.

No arguments there :-)

> OP, What do you mean by "does not respond to a READ"? Does it send a
> disconnect? or just didn't claim the transaction? One thing for sure is that
> you're not inserting wait state.

It does not claim the transaction. We bought a driver and ran the tests with
that driver. Now we are trying to develop our own.

>Is the memory NonCached? Anyway, there must
> something wrong here.

Hummmm.... Great point there. I am currently trying all 3 types: MmCached,
MmNonCached, MmWriteCombined. MmWriteCombined, in terms of performance, works
great, but our hardware does not (yet) support out-of-order writes (as I
understand that WC does).

I will for sure follow Mark´s advice and read Walter Oney.

Thanks again guys.

+++++++++++++++++++++++++++++++++++++++++++++++++++++
> "Mark Roddy" <markr@hollistech.com> wrote in message
> news:ui0NPMlZFHA.2768@tk2msftngp13.phx.gbl...
> > mfranco wrote:
> >> I agree with you BUT, we ran the following test: When using virtual
> >> address, the OS firstly copies data to system memory and then copies this
> >> data to device memory.
> >
> > Uh no. A virtual address for PCI device memory is just that: a system
> > virtual address for PCI device memory. There is no copy performed in
> > reading/writing virtual addresses, other than the actual load/store
> > operation itself, which I assure you, if you have done things correctly,
> > which I am fairly certain you have not, will result in a direct transfer
> > from system memory to device memeory or vice versa.
> >
> > >
> > So we have TWO copies instead of only one, directly to device
> >> memory. We know that because our device was set in a state that it does
> >> not respond to READ requests. So, when we try to read data from it, it
> >> should return nothing, invalid data. Instead, when we try to read from
> >> it, we are getting valid data, that we just wrote to the memory.
> >
> > Your experiment has some other bogus assumptions in it.
> >
> >>
> >> Conclusion: we are reading data from system memory, and not from device
> >> memory.
> >
> > Wrong conclusion. As the saying goes, garbage in: garbage out.
> >
> >>
> >> Anyway, a few miliseconds that we save by bypassing that first copy using
> >> virtual address, and writing directly to device memory, would make a
> >> great diference. Another point that comes: Just by writing the write
> >> routine in assembly, we gained almost 1 milisecond in performance. As we
> >> have some hardware and time limitations, I realy would like to get
> >> everything I can from this driver, and I think that writing an assembly
> >> routine that accesses device's physical memory directly will do the
> >> trick.
> >>
> >> Once again, any suggestion on how to do that are more than welcome.
> >>
> >
> > OK then. Go for it. I tried. You obviously know better. Good luck.
> >
> >> "Mark Roddy" wrote:
> >>
> >>
> >>>mfranco wrote:
> >>>
> >>>>I need to increase the performance of the driver I wrote, and to do
> >>>>that, I need to write data directly to the physical address of the pci
> >>>>device we are using. I have already tried, in assembly, to write data to
> >>>>the virtual address of the device, which was mapped with MmMapIoSpace.
> >>>>It worked fine, but when I change the virtual address with the physical
> >>>>address, I get the Blue screen. The routine to write is basically
> >>>>
> >>>>esi <- buffer
> >>>>edi <- virtual add
> >>>>movsd
> >>>>
> >>>>inside a for loop (just testing).
> >>>>I heard that in order to write directly to the physical address, I need
> >>>>to elevate the driver to be "real kernel" level, and not only "windows
> >>>>kernel" level, where I cannot access the physical memory.
> >>>>
> >>>>Any suggestions would be greatly appreciated.
> >>>
> >>>You don't use physical memory addresses for accessing anything from the
> >>>OS. As you discovered, using a physical address will crash or corrupt the
> >>>system. The virtual->physical address translation is performed in
> >>>hardware and its overhead is negligible. "real mode" is obsolete 16-bit
> >>>x86 architecture nonsense. Yes it exists, no you do not use it, no it
> >>>will not speed up transfers to your pci device.
> >>>
> >>>You need to get a better understanding of OS fundamentals and Windows
> >>>driver programming. I suggest that you read Windows Internals (4th
> >>>edition, Russinovich and Soloman, Microsoft Press) and Programming the
> >>>Windows Driver Model (2nd edition, Wlater Oney, Microsoft Press.)
> >>>
> >>>If you have access to the Windows DDK you need to read up on using PIO
> >>>mode to read and write device memory.
> >>>
> >>>--
> >>>
> >>>=====================
> >>>Mark Roddy DDK MVP
> >>>Windows 2003/XP/2000 Consulting
> >>>Hollis Technology Solutions 603-321-1032
> >>>www.hollistech.com
> >>>
> >
> >
> > --
> >
> > =====================
> > Mark Roddy DDK MVP
> > Windows 2003/XP/2000 Consulting
> > Hollis Technology Solutions 603-321-1032
> > www.hollistech.com
>
>
>

Maxim
Thu Jun 02 01:46:04 CDT 2005

> I agree with you BUT, we ran the following test: When using virtual address,
> the OS firstly copies data to system memory and then copies this data to
> device memory.

Not so with MmMapIoSpace. The CPU just writes to the device in this case.

> memory. We know that because our device was set in a state that it does not
> respond to READ requests. So, when we try to read data from it, it should
> return nothing, invalid data. Instead, when we try to read from it, we are
> getting valid data, that we just wrote to the memory.
>
> Conclusion: we are reading data from system memory, and not from device
> memory.

Is it from the cache? You can map the device memory as noncached.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com

Maxim
Thu Jun 02 01:47:15 CDT 2005

> MmNonCached, MmWriteCombined. MmWriteCombined, in terms of performance,
works
> great, but our hardware does not (yet) support out-of-order writes (as I
> understand that WC does).

Write combining means - the chipset can combine the consecutive writes to a
large burst write. OK for video framebuffer, not OK for most device registers.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com

Pavel
Thu Jun 02 12:44:51 CDT 2005

An NDIS question... NDIS allows to specify either cached or noncached
shared memory. How to specify write combine?

--PA

"Maxim S. Shatskih" <maxim@storagecraft.com> wrote in message news:uMjOT7zZFHA.3808@TK2MSFTNGP14.phx.gbl...
> > MmNonCached, MmWriteCombined. MmWriteCombined, in terms of performance,
> works
> > great, but our hardware does not (yet) support out-of-order writes (as I
> > understand that WC does).
>
> Write combining means - the chipset can combine the consecutive writes to a
> large burst write. OK for video framebuffer, not OK for most device registers.
>
> --
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> maxim@storagecraft.com
> http://www.storagecraft.com
>
>

mfranco
Thu Jun 02 19:05:02 CDT 2005

Sorry, didn't mean out-of-order, but something like not-defined-packet-size
writes. But I was not sure about it.

Now its crystal clear ;-))

Thanks

"Maxim S. Shatskih" wrote:

> > MmNonCached, MmWriteCombined. MmWriteCombined, in terms of performance,
> works
> > great, but our hardware does not (yet) support out-of-order writes (as I
> > understand that WC does).
>
> Write combining means - the chipset can combine the consecutive writes to a
> large burst write. OK for video framebuffer, not OK for most device registers.
>
> --
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> maxim@storagecraft.com
> http://www.storagecraft.com
>
>
>