Memory usage bug...

When I profile my application the profiler tells me that there is
roughly 4MB of memory used throughout the lifetime, which seems reasonable.

However, when I start the task manager the said memory usage is 20MB
right after starting the program and 80MB after running it. However, if
you start the program and keep the task manager open to monitor usage,
it says 20MB, however, when you minimize the program, the memory usage
falls to 2MB.

This is a known bug as far as I know
see:
http://groups.google.com/group/microsoft.public.dotnet.framework.performance/browse_frm/thread/515973d1cb381d75/cf6554802f9f3728?lnk=st&q=memory+usage+c%23&rnum=5&hl=en#cf6554802f9f3728
(watch for wrap)

However, I want to know if the memory is really being held back from the
OS until you minimize the application or is there some other explanation?

Thanks,
Nick Z.

AMercer
Tue Sep 27 06:21:04 CDT 2005

> When I profile my application the profiler tells me that there is
> roughly 4MB of memory used throughout the lifetime, which seems reasonable.
> However, when I start the task manager the said memory usage is 20MB
> right after starting the program and 80MB after running it. However, if
> you start the program and keep the task manager open to monitor usage,
> it says 20MB, however, when you minimize the program, the memory usage
> falls to 2MB.

The figure that is ballooning up and down (2mb-80mb from task manager) is the
working set size, not the amount of memory currently in use (4mb from
profiler).

> This is a known bug as far as I know

The behaviour is not a bug, it is by design.

> However, I want to know if the memory is really being held back from the
> OS until you minimize the application or is there some other explanation?

The memory is not "being held back" from the OS. In the presence of ample
physical memory, the OS does not enforce the process's working set
maximum size. So, we have poor nomenclature, but not a bug. When an
app is minimized, the OS calls Win32 API SetProcessWorkingSetSize to
minimize the working set size, and that is why the task manager reports
the big drop.

FYI, the app can call the API as well, and I have done so while chasing a
memory leak regarding failure to release unmanaged resources. So, there
is at least one development context where MS memory management is an
inconvenience. You and many others have inquired about this issue, so
there is massive confusion and concern about it. I don't know why MS did what
they did. My guess is that there is an advantage to deferring the reclamation
of memory pages - perhaps leaving pages in limbo (reclaimable but still
assigned
to a process) expidites subsequent memory allocations by the same process.

Nick
Tue Sep 27 09:23:33 CDT 2005

Thanks for the feedback. However, I'm still not quite sure how this
affects performance. Lets say the OS needs some memory, and there is
not enough of it, will the OS invoke the SetProcessWorkingSetSize on
every process untill it get enough memory or will it look at the
working set and asume that all of it is required for the process to
function?

AMercer wrote:
> > When I profile my application the profiler tells me that there is
> > roughly 4MB of memory used throughout the lifetime, which seems reasonable.
> > However, when I start the task manager the said memory usage is 20MB
> > right after starting the program and 80MB after running it. However, if
> > you start the program and keep the task manager open to monitor usage,
> > it says 20MB, however, when you minimize the program, the memory usage
> > falls to 2MB.
>
> The figure that is ballooning up and down (2mb-80mb from task manager) is the
> working set size, not the amount of memory currently in use (4mb from
> profiler).
>
> > This is a known bug as far as I know
>
> The behaviour is not a bug, it is by design.
>
> > However, I want to know if the memory is really being held back from the
> > OS until you minimize the application or is there some other explanation?
>
> The memory is not "being held back" from the OS. In the presence of ample
> physical memory, the OS does not enforce the process's working set
> maximum size. So, we have poor nomenclature, but not a bug. When an
> app is minimized, the OS calls Win32 API SetProcessWorkingSetSize to
> minimize the working set size, and that is why the task manager reports
> the big drop.
>
> FYI, the app can call the API as well, and I have done so while chasing a
> memory leak regarding failure to release unmanaged resources. So, there
> is at least one development context where MS memory management is an
> inconvenience. You and many others have inquired about this issue, so
> there is massive confusion and concern about it. I don't know why MS did what
> they did. My guess is that there is an advantage to deferring the reclamation
> of memory pages - perhaps leaving pages in limbo (reclaimable but still
> assigned
> to a process) expidites subsequent memory allocations by the same process.

Willy
Tue Sep 27 10:11:14 CDT 2005


"Nick Z." <pacemkr@gmail.com> wrote in message
news:1127831013.168327.157180@g44g2000cwa.googlegroups.com...
> Thanks for the feedback. However, I'm still not quite sure how this
> affects performance. Lets say the OS needs some memory, and there is
> not enough of it, will the OS invoke the SetProcessWorkingSetSize on
> every process untill it get enough memory or will it look at the
> working set and asume that all of it is required for the process to
> function?
>

No, the OS never calls SetProcessWorkingSetSize, this is a user API. The
memory manager will free RAM space by removing the least frequently used
pages from the processes working sets until there is enough free space to
accomodate the memory request. Note that this is done long before memory is
exhausted, the memory manager and the balance-set manager run once per
second.
I would suggest you read this
http://emea.windowsitpro.com/Article/ArticleID/3686/3686.html if you like to
know how the OS manages it's memory.

Willy.

Mehdi
Tue Sep 27 10:41:04 CDT 2005

On Tue, 27 Sep 2005 03:41:21 -0400, Nick Z. wrote:

> When I profile my application the profiler tells me that there is
> roughly 4MB of memory used throughout the lifetime, which seems reasonable.
>
> However, when I start the task manager the said memory usage is 20MB
> right after starting the program and 80MB after running it. However, if
> you start the program and keep the task manager open to monitor usage,
> it says 20MB, however, when you minimize the program, the memory usage
> falls to 2MB.

This article explains quite clearly the reasons for this behaviour:
http://getdotnetco.web101.discountasp.net/GdncStore/free/Articles/The%20Memory%20Mystery.htm
("The Memory Mystery" by Michael McIntyre)

Nick
Tue Sep 27 13:01:51 CDT 2005

Thank you very much for the link, that explained it.
I thought something similar was happening, I just didn't know exactly
what was happening and how to explain it.

Thanks,
Nick Z.

Mehdi wrote:
> On Tue, 27 Sep 2005 03:41:21 -0400, Nick Z. wrote:
>
> > When I profile my application the profiler tells me that there is
> > roughly 4MB of memory used throughout the lifetime, which seems reasonable.
> >
> > However, when I start the task manager the said memory usage is 20MB
> > right after starting the program and 80MB after running it. However, if
> > you start the program and keep the task manager open to monitor usage,
> > it says 20MB, however, when you minimize the program, the memory usage
> > falls to 2MB.
>
> This article explains quite clearly the reasons for this behaviour:
> http://getdotnetco.web101.discountasp.net/GdncStore/free/Articles/The%20Memory%20Mystery.htm
> ("The Memory Mystery" by Michael McIntyre)