Hello everyone,


I am using perfmon to watch the working set and virtual bytes, when I do a
keyword search in SourceInsight.

I found the value of working set is larger than virtual bytes when do a
search, I am confused how could working set larger than virtual bytes? I have
this confusion is virtual bytes is all things -- reserved, committed memory
-- which includes RAM (working set) and page swap file. So, virtual bytes
should be always larger than working set.

BTW: I watched that private bytes is always lower than virtual bytes when do
the search.


thanks in advance,
George

Re: working set is larger than virtual bytes? by Tom

Tom
Thu Jan 10 11:19:30 CST 2008

George wrote:
> Hello everyone,
>
>
> I am using perfmon to watch the working set and virtual bytes, when I do a
> keyword search in SourceInsight.
>
> I found the value of working set is larger than virtual bytes when do a
> search, I am confused how could working set larger than virtual bytes? I have
> this confusion is virtual bytes is all things -- reserved, committed memory
> -- which includes RAM (working set) and page swap file. So, virtual bytes
> should be always larger than working set.
>
> BTW: I watched that private bytes is always lower than virtual bytes when do
> the search.

Look at a definition of Working Set:
http://en.wikipedia.org/wiki/Working_set

It should be clear under what circumstances it can be larger than
virtual bytes.

Tom

Re: working set is larger than virtual bytes? by George

George
Thu Jan 10 22:00:47 CST 2008

Hi Tom,


I have read the link twice but can not find under what situations, working
set will be larger than virtual bytes on Windows. Could you kindly point out
please? :-)


regards,
George

"Tom Widmer [VC++ MVP]" wrote:

> George wrote:
> > Hello everyone,
> >
> >
> > I am using perfmon to watch the working set and virtual bytes, when I do a
> > keyword search in SourceInsight.
> >
> > I found the value of working set is larger than virtual bytes when do a
> > search, I am confused how could working set larger than virtual bytes? I have
> > this confusion is virtual bytes is all things -- reserved, committed memory
> > -- which includes RAM (working set) and page swap file. So, virtual bytes
> > should be always larger than working set.
> >
> > BTW: I watched that private bytes is always lower than virtual bytes when do
> > the search.
>
> Look at a definition of Working Set:
> http://en.wikipedia.org/wiki/Working_set
>
> It should be clear under what circumstances it can be larger than
> virtual bytes.
>
> Tom
>

Re: working set is larger than virtual bytes? by Tom

Tom
Fri Jan 11 07:47:11 CST 2008

George wrote:
> Hi Tom,
>
>
> I have read the link twice but can not find under what situations, working
> set will be larger than virtual bytes on Windows. Could you kindly point out
> please? :-)

I'd rather let you work it out for yourself. If you're having trouble
with that definition, Google should help you find more information to study.

Tom

Re: working set is larger than virtual bytes? by George

George
Fri Jan 11 08:51:02 CST 2008

Hi Tom,


Actually, all the information I get is, working set is the current pages the
process touched in virtual memory, this should be close to the formal
definition of working set. So, I think working set is always a group of
pages, and each page of working set could be mapped to virtual memory.

From this point of view, I have no idea why working set could be larger than
virtual memory.

Do you have any ideas? :-)


regards,
George

"Tom Widmer [VC++ MVP]" wrote:

> George wrote:
> > Hi Tom,
> >
> >
> > I have read the link twice but can not find under what situations, working
> > set will be larger than virtual bytes on Windows. Could you kindly point out
> > please? :-)
>
> I'd rather let you work it out for yourself. If you're having trouble
> with that definition, Google should help you find more information to study.
>
> Tom
>

Re: working set is larger than virtual bytes? by Tom

Tom
Tue Jan 15 08:46:52 CST 2008

George wrote:
> Hi Tom,
>
>
> Actually, all the information I get is, working set is the current pages the

What do you mean by 'current'?

> process touched in virtual memory, this should be close to the formal
> definition of working set. So, I think working set is always a group of
> pages, and each page of working set could be mapped to virtual memory.
>
> From this point of view, I have no idea why working set could be larger than
> virtual memory.
>
> Do you have any ideas? :-)

Answer my above question, and you should see it.

Tom

Re: working set is larger than virtual bytes? by George

George
Tue Jan 15 09:11:03 CST 2008

Sorry, Tom!


I can not find answers from either the wikipedia link or Google. :-)

Let me see if you or others have good ideas.


regards,
George

"Tom Widmer [VC++ MVP]" wrote:

> George wrote:
> > Hi Tom,
> >
> >
> > Actually, all the information I get is, working set is the current pages the
>
> What do you mean by 'current'?
>
> > process touched in virtual memory, this should be close to the formal
> > definition of working set. So, I think working set is always a group of
> > pages, and each page of working set could be mapped to virtual memory.
> >
> > From this point of view, I have no idea why working set could be larger than
> > virtual memory.
> >
> > Do you have any ideas? :-)
>
> Answer my above question, and you should see it.
>
> Tom
>

Re: working set is larger than virtual bytes? by Tom

Tom
Tue Jan 15 12:14:13 CST 2008

George wrote:
> Sorry, Tom!
>
>
> I can not find answers from either the wikipedia link or Google. :-)
>
> Let me see if you or others have good ideas.

Can the working set include deallocated pages?

Tom

Re: working set is larger than virtual bytes? by George

George
Thu Jan 17 01:17:00 CST 2008

Thanks Tom,


Here is my program to show the working set could be larger than vritual
bytes. Could you help to double check please? :-)

In the definition of working set, it is the set of pages from virtual memory
pages which physical memory could contain, so working set could not be larger
than virtual bytes -- it is subset relationship.

If you have any ideas why in my application working set is larger than
virtual bytes, or if there is anything wrong, please feel free to let me
know.

Some comments, In my program, I use two ways to hold file map.

1. keep all file map open and close all open handles altogether at the end;
2. close open handle each time after using it.

In (1), working set is much larger than virtual bytes, and in (2).

Here is my code.

#include <windows.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
LARGE_INTEGER start,end;
LARGE_INTEGER freq;
QueryPerformanceCounter(&start);
QueryPerformanceFrequency(&freq);

MEMORYSTATUS memstat;
void** map;
int sectionIndex = 0;
memstat.dwLength = sizeof(memstat);
GlobalMemoryStatus(&memstat);

// basic file mapping test (512 MB)
long long size = 512*1024*1024;

HANDLE mapping =
CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL);
if (mapping)
{
// create and destroy temporary views
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
const int allocSize = sysInfo.dwAllocationGranularity;

GlobalMemoryStatus(&memstat);

void *mem = new char[allocSize];
memset(mem,0x11,allocSize);

map = (void**) new char [sizeof(void*) * size / allocSize];

for (int i=0; i<10; i++)
{

sectionIndex = 0;
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
map [sectionIndex] =
MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map [sectionIndex])
{
memcpy(map [sectionIndex],mem,allocSize);
// UnmapViewOfFile(map);
}

sectionIndex++;
} // for (long long offset=0; offset<=size-allocSize; offset+=allocSize)

// close mapped files to avoid leak
for (sectionIndex = 0; sectionIndex < size/allocSize; sectionIndex++)
{
if (map [sectionIndex])
{
UnmapViewOfFile(map [sectionIndex]);
}
}

GlobalMemoryStatus(&memstat);

sectionIndex = 0;
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
map [sectionIndex] =
MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map [sectionIndex])
{
for (int t=0; t<allocSize; t++)
{
if (((char *)(map [sectionIndex]))[t]!=0x11)
{
OutputDebugString("Memory read failed\n");
}
}
}

UnmapViewOfFile(map [sectionIndex]);
}

// close mapped files to avoid leak
/*
for (sectionIndex = 0; sectionIndex < size/allocSize; sectionIndex++)
{
if (map [sectionIndex])
{
UnmapViewOfFile(map [sectionIndex]);
}
}
*/

GlobalMemoryStatus(&memstat);
} // for (int i=0; i<10; i++)

QueryPerformanceCounter(&end);

GlobalMemoryStatus(&memstat);

printf("Time %.3f\n",
double(end.QuadPart-start.QuadPart)/double(freq.QuadPart));
CloseHandle(mapping);
delete[] mem;
GlobalMemoryStatus(&memstat);
} //if (mapping)

return 0;
}


regards,
George

"Tom Widmer [VC++ MVP]" wrote:

> George wrote:
> > Sorry, Tom!
> >
> >
> > I can not find answers from either the wikipedia link or Google. :-)
> >
> > Let me see if you or others have good ideas.
>
> Can the working set include deallocated pages?
>
> Tom
>

Re: working set is larger than virtual bytes? by Tom

Tom
Mon Jan 21 11:34:19 CST 2008

George wrote:
> Thanks Tom,
>
>
> Here is my program to show the working set could be larger than vritual
> bytes. Could you help to double check please? :-)

It doesn't show the working set being larger than virtual bytes (at
least, I didn't obverse it). Note that perfmon doesn't display working
set and virtual memory usage in the same units - are you simply misusing
the tools and thinking working set usage is 10x what it actually is?

> In the definition of working set, it is the set of pages from virtual memory
> pages which physical memory could contain, so working set could not be larger
> than virtual bytes -- it is subset relationship.

From the definition of working set that I posted (the wikipedia link),
the working set could include pages that have been recently deallocated
(this may or may not be the case on Windows, I don't know). If that
holds true, then it is not a strict subset of virtual memory.

Tom

Re: working set is larger than virtual bytes? by George

George
Tue Jan 22 18:45:12 CST 2008

Thanks Tom,


I am interested in,

> least, I didn't obverse it). Note that perfmon doesn't display working
> set and virtual memory usage in the same units - are you simply misusing
> the tools and thinking working set usage is 10x what it actually is?

What do you mean not the same units?


regards,
George

"Tom Widmer [VC++ MVP]" wrote:

> George wrote:
> > Thanks Tom,
> >
> >
> > Here is my program to show the working set could be larger than vritual
> > bytes. Could you help to double check please? :-)
>
> It doesn't show the working set being larger than virtual bytes (at
> least, I didn't obverse it). Note that perfmon doesn't display working
> set and virtual memory usage in the same units - are you simply misusing
> the tools and thinking working set usage is 10x what it actually is?
>
> > In the definition of working set, it is the set of pages from virtual memory
> > pages which physical memory could contain, so working set could not be larger
> > than virtual bytes -- it is subset relationship.
>
> From the definition of working set that I posted (the wikipedia link),
> the working set could include pages that have been recently deallocated
> (this may or may not be the case on Windows, I don't know). If that
> holds true, then it is not a strict subset of virtual memory.
>
> Tom
>

Re: working set is larger than virtual bytes? by George

George
Thu Jan 24 00:54:01 CST 2008

Sorry Tim,


I do not agree with you. Here are my points and proves, please feel free to
correct me if I am wrong. :-)

1. Explanation from help of perfmon of working set,

--------------------
Working Set is the current size, in bytes, of the Working Set of this
process. The Working Set is the set of memory pages touched recently by the
threads in the process. If free memory in the computer is above a threshold,
pages are left in the Working Set of a process even if they are not in use.
When free memory falls below a threshold, pages are trimmed from Working
Sets. If they are needed they will then be soft-faulted back into the Working
Set before leaving main memory.
--------------------

as you can see working set is measured in bytes.

2.

when we set the vertical scale, we always use Properties --> Graph -->
Vertical scale, and the value we set is for all the counters in the dynamic
graph below, not for a single counter, right?

So, I do not know why you mentioned,

> The graph is always scaled, and the scale factors are different. The scale
> factor for Virtual Bytes is 0.000 001, and the scale for Working set is
> 0.000 01.

Any more description please?


regards,
George

"Tim Roberts" wrote:

> George <George@discussions.microsoft.com> wrote:
> >
> >I am interested in,
> >
> >> least, I didn't obverse it). Note that perfmon doesn't display working
> >> set and virtual memory usage in the same units - are you simply misusing
> >> the tools and thinking working set usage is 10x what it actually is?
> >
> >What do you mean not the same units?
>
> The graph is always scaled, and the scale factors are different. The scale
> factor for Virtual Bytes is 0.000 001, and the scale for Working set is
> 0.000 01. So, 6 megabytes in virtual bytes will show on the graph at 6,
> but 6 megabytes of working set will be at 60.
> --
> Tim Roberts, timr@probo.com
> Providenza & Boekelheide, Inc.
>

Re: working set is larger than virtual bytes? by George

George
Tue Jan 29 01:28:01 CST 2008

Great reply, Tim!


> The answer is that *EACH* measurement that perfmon can graph has its own
> scale factor. Percentages have a scale factor of 1, so that a value of 95%
> shows up at 95 on the graph. Working set has a scale factor of 0.000 01,
> so that 9,500,000 shows up at 95 on the graph. Virtual bytes has a scale
> factor of 0.000 001, so that 95,000,000 shows up at 95 on the graph.

How do you check the scale value for a specific counter -- like 0.000 01 for
working set and 0.000 001 for virtual bytes?

I have checked perfmon from Properties --> Data Tab --> Scale, the value of
both working set and virtual bytes are "Default".

BTW: I am using Windows Server 2003.


regards,
George

"Tim Roberts" wrote:

> George <George@discussions.microsoft.com> wrote:
> >
> >Sorry Tim,
> >
> >I do not agree with you. Here are my points and proves, please feel free to
> >correct me if I am wrong. :-)
>
> You are wrong.
>
> >1. Explanation from help of perfmon of working set,
> >...
> >as you can see working set is measured in bytes.
>
> Yes.
>
> >2.
> >
> >when we set the vertical scale, we always use Properties --> Graph -->
> >Vertical scale, and the value we set is for all the counters in the dynamic
> >graph below, not for a single counter, right?
>
> The value you are setting here is the maximum SCALED value that will be
> displayed.
>
> Think about the issue here. Some of the numbers perfmon displays are very
> small (like from 0 to 1). Some of them are percentages (0 to 100). Some
> of them are bytes per second (0 to 10,000,000). Some of them are total
> memory bytes (0 to 2,000,000,000). How do you put all of those numbers on
> a single graph with a single Y axis?
>
> The answer is that *EACH* measurement that perfmon can graph has its own
> scale factor. Percentages have a scale factor of 1, so that a value of 95%
> shows up at 95 on the graph. Working set has a scale factor of 0.000 01,
> so that 9,500,000 shows up at 95 on the graph. Virtual bytes has a scale
> factor of 0.000 001, so that 95,000,000 shows up at 95 on the graph.
>
> >So, I do not know why you mentioned,
> >
> >> The graph is always scaled, and the scale factors are different. The scale
> >> factor for Virtual Bytes is 0.000 001, and the scale for Working set is
> >> 0.000 01.
> >
> >Any more description please?
>
> Because it is true. The value 60 on the graph means different things for
> different measurements.
> --
> Tim Roberts, timr@probo.com
> Providenza & Boekelheide, Inc.
>

Re: working set is larger than virtual bytes? by George

George
Thu Jan 31 01:42:01 CST 2008

Thanks Tim!


Great! I have found that you are correct, the scale value of the scale
column in my environment is,

Virtual Bytes: 0.0000010, and Working Set: 0.0000100, and also I set
Properties --> Graph --> Scale --> Max: 10000, Min 0.

I have read your previous post, especially this one,

--------------------
Think about the issue here. Some of the numbers perfmon displays are very
small (like from 0 to 1). Some of them are percentages (0 to 100). Some
of them are bytes per second (0 to 10,000,000). Some of them are total
memory bytes (0 to 2,000,000,000). How do you put all of those numbers on
a single graph with a single Y axis?

The answer is that *EACH* measurement that perfmon can graph has its own
scale factor. Percentages have a scale factor of 1, so that a value of 95%
shows up at 95 on the graph. Working set has a scale factor of 0.000 01,
so that 9,500,000 shows up at 95 on the graph. Virtual bytes has a scale
factor of 0.000 001, so that 95,000,000 shows up at 95 on the graph.
--------------------

But I am still confused how you calculate from individual counter scale
column, graph scale value and displayed value to real counter value in bytes?
Could you give a formula please? :-)


regards,
George

"Tim Roberts" wrote:

> George <George@discussions.microsoft.com> wrote:
> >
> >How do you check the scale value for a specific counter -- like 0.000 01 for
> >working set and 0.000 001 for virtual bytes?
> >
> >I have checked perfmon from Properties --> Data Tab --> Scale, the value of
> >both working set and virtual bytes are "Default".
>
> Did you look at the graph itself when you right clicked to check the
> properties? Look at the "Scale" column in this screenshot I just took:
>
> http://www.probo.com/timr/perfmon.jpg
> --
> Tim Roberts, timr@probo.com
> Providenza & Boekelheide, Inc.
>

Re: working set is larger than virtual bytes? by George

George
Sat Feb 02 23:26:00 CST 2008

Hi Tim,


I have made some self-study. And I think to get the real value, we need to
use the following formula,

Real Value = Scale * Display Value

Is that correct?


regards,
George

"Tim Roberts" wrote:

> George <George@discussions.microsoft.com> wrote:
> >
> >But I am still confused how you calculate from individual counter scale
> >column, graph scale value and displayed value to real counter value in bytes?
> >Could you give a formula please? :-)
>
> Come on, George, this is just not that complicated! The scale only affects
> the graph. The counter number displayed in the table at the bottom of the
> graph is the raw number; the scale factor does not affect that.
> --
> Tim Roberts, timr@probo.com
> Providenza & Boekelheide, Inc.
>