Big allocation AND per process memory!

Hi EveryOne!

0
See [
http://msdn.microsoft.com/newsgroups/managed/default.aspx?dg=microsoft.public.dotnet.languages.csharp&mid=7f32b8de-2204-47e7-9ed9-0e2f5f60c316
] what MS guy is talking about memory allocation , i wonder that they
say that process address space can be 4 GB [ 32 bit architecture ]
memory with one half for kernel and another for user. Ok! lets say that
you can just use 2 GB with any user application , fine. If i write a
kernel mode code which can access user and as well as kernel space then
it should be able to take care of 4 GB. But why Microsoft guys are just
saying NO for big allocations?

It might be possible that lack of addressing flat memory is due to
other than OS itself like TLB , HW cache , VM manager architecture
etc..

1)

Say Windows 32 bit OS for intel architecture.

0) Kernel space = 2
1) User space = 2

Total process space = 4 GB.

User mode App. = You have just 2 GB to move around , you keep you data,
routines and string for file mapping here.

Kernel mode code = Ok here you can move around 3 GB.

You can see that our system is consuming min 1 GB [PCB , stack etc.] in
both cases. So we can see that system is spoiling 4 GB [ one process]
for every four processes;-)

that is what i'm thinking.

Regards
-ali

Don
Fri Oct 14 10:55:29 CDT 2005

Where do you get the 3GB for the system. The memory regions are broken down
to be either:

2GB User/2GB Kernel - Normal
3GB User/1GB Kernel - /3GB swtich on boot

If you go with the latter, your kernel has a smaller disk cache and other
contraints, so depending on the type of application this will either help or
hurt overall performance. I don't follow your math at all.


--
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Remove StopSpam from the email to reply



"Ali" <abdulrazaq@gmail.com> wrote in message
news:1129304216.040825.305250@g49g2000cwa.googlegroups.com...
> Hi EveryOne!
>
> 0
> See [
> http://msdn.microsoft.com/newsgroups/managed/default.aspx?dg=microsoft.public.dotnet.languages.csharp&mid=7f32b8de-2204-47e7-9ed9-0e2f5f60c316
> ] what MS guy is talking about memory allocation , i wonder that they
> say that process address space can be 4 GB [ 32 bit architecture ]
> memory with one half for kernel and another for user. Ok! lets say that
> you can just use 2 GB with any user application , fine. If i write a
> kernel mode code which can access user and as well as kernel space then
> it should be able to take care of 4 GB. But why Microsoft guys are just
> saying NO for big allocations?
>
> It might be possible that lack of addressing flat memory is due to
> other than OS itself like TLB , HW cache , VM manager architecture
> etc..
>
> 1)
>
> Say Windows 32 bit OS for intel architecture.
>
> 0) Kernel space = 2
> 1) User space = 2
>
> Total process space = 4 GB.
>
> User mode App. = You have just 2 GB to move around , you keep you data,
> routines and string for file mapping here.
>
> Kernel mode code = Ok here you can move around 3 GB.
>
> You can see that our system is consuming min 1 GB [PCB , stack etc.] in
> both cases. So we can see that system is spoiling 4 GB [ one process]
> for every four processes;-)
>
> that is what i'm thinking.
>
> Regards
> -ali
>

Ali
Fri Oct 14 11:09:05 CDT 2005

Thanks Born,

1. Boot your system with /3GB (Windows NT & 2000) or /4GT (Windows
2003) switch. This way, kernel will only use 1 GB and applications can
use 3 GB.

2. If you have more physical memory installed and your hardware is
Intel with PAE support, boot Windows with /PAE switch. This way you can
use up to 64 GB memory from your application via AWE (Address Windowing
Extensions).

See this link for details.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/large_memory_support.asp


I believe that this is for allocating space for a file not for a
process itself! right? nothing math just few thoughts;-)

Thanks for post.
Regars
-ali

James
Fri Oct 14 16:23:28 CDT 2005

I'm not sure what is your question.

First, let's confine ourselves to the usual 2GB kernel space/2GB user space
setup.

Next, an allocation is in kernel space or user space. An allocation will not
span. That's just the way the OS is (I won't go into reasons).

Next, in the kernel 2GB some space is taken with executables (binaries like
the kernel, HAL, drivers), page tables, lookasides of various kinds and
plain old data.

Next, in the kernel an allocation will be satisfied only from a kind of
storage such as pageable or non-paged (there are also flavors that are
suitable for DMA, etc, etc.). These types are confined to virtual memory
ranges.

Next, some other virtual ranges just aren't used.

The end result is that a given allocation, in a freshly booted system
(nobody has grabbed memory beyond what the OS and device drivers must have),
the maximum allocation is going to be rather less than 2GB.

The short of all this is that memory comes in lots of flavors, and an
allocation always includes a flavor (maybe by default), and a flavor can be
carved out of only a certain virtual range, a range that is always less than
the 2GB that comprises all possible flavors.

Similar reasoning applies to user space.

--
James Antognini
Windows DDK and WDK Support


This posting is provided "AS IS" with no warranties, and confers no rights.



"Ali" <abdulrazaq@gmail.com> wrote in message
news:1129306145.199816.305290@g47g2000cwa.googlegroups.com...
> Thanks Born,
>
> 1. Boot your system with /3GB (Windows NT & 2000) or /4GT (Windows
> 2003) switch. This way, kernel will only use 1 GB and applications can
> use 3 GB.
>
> 2. If you have more physical memory installed and your hardware is
> Intel with PAE support, boot Windows with /PAE switch. This way you can
> use up to 64 GB memory from your application via AWE (Address Windowing
> Extensions).
>
> See this link for details.
>
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/large_memory_support.asp
>
>
> I believe that this is for allocating space for a file not for a
> process itself! right? nothing math just few thoughts;-)
>
> Thanks for post.
> Regars
> -ali
>

Ali
Sat Oct 15 09:52:00 CDT 2005

Hi James,
Thanks for such a valuable comments ,

>Next, in the kernel 2GB some space is taken with executables (binaries like the kernel, HAL, drivers), page tables, lookasides of various kinds and plain old data.
Yeah yeah i can understand that, that is the place where we keep all
process related info like PCB , I/O map and OS related objects.

>Next, in the kernel an allocation will be satisfied only from a kind of storage such as pageable or non-paged (there are also flavors that are suitable for DMA, etc, etc.). These types are confined to virtual memory ranges.
Ok here we keep the memory related stuff , our locking & unlocking
mechanisim , defined page sizes etc

>Next, some other virtual ranges just aren't used.
The end result is that a given allocation, in a freshly booted system
(nobody has grabbed memory beyond what the OS and device drivers must
have), the maximum allocation is going to be rather less than 2GB.

Do you mean that a process greater than 2GB cann't be paged in and
paged out;-)

>The short of all this is that memory comes in lots of flavors, and an allocation always includes a flavor (maybe by default), and a flavor can be carved out of only a certain virtual range, a range that is always less than the 2GB that comprises all possible flavors.
Ahha , thats the reason we can have exception for following line of
code:
"byte [] c=new byte[1300000000];" failed with OutOfMemoryException.


Regards
-ali

pavel_a
Sun Oct 16 10:00:01 CDT 2005

"Ali" wrote:
[snip]
> But why Microsoft guys are just
> saying NO for big allocations?

No, in fact the Microsoft guys created 64-bit Windows just for you
to enjoy huge allocations.

Regards,
--PA