Memory Problem with II6 - Windows Server 2003

Server Spec:
Windows Server 2003
SP 1
Dual 2.66 Ghz
2 GB Ram

At random times the memory usage goes way up, sometimes past 2-3 gb. And of
course the server becomes unresponsive through all services--http, ftp,
email, etc.

When I look at the Task Manager, I can see that the process that is using
the memory is w3wp.exe for one specific site.

On that site I have the Application Pool set to have 10 worker process,
recycle every 5 minutes, use a maximum of 42 mb.

What could cause it to go up so high?

Here's a screen shot:
http://www.powwows.com/ebay/server.jpg

Thanks!

David
Mon Jan 09 20:05:32 CST 2006

Use DebugDiag to triage the w3wp.exe with its "Memory Leak" option and post
its log report here for more assistance.

FYI: Your health monitoring options seem a bit extreme. Recycling every 5
minutes is a bad thing for performance (process creation triggered by
recycling is expensive CPU/Resource-wise) and may be a part of your
reliability problem. Recycling with a threshold of 42MB may also be
problematic since that is not a lot of memory and you can blow past it just
by running one ASP.Net application. And unless you can definitely explain
*why* you need a web garden of 10 worker process, you probably do not want
to use it because it can also hinder performance and reliability.

Bottom line: You have a lot of settings right now that may be causing a lot
of noise, but we will first triage the apparent "Memory Leak" and then go
from there.

http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Basics_of_IIS6_Troubleshooting.aspx
http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <Paul G@discussions.microsoft.com> wrote in message
news:3F8DD56E-ED6B-4B7D-B5AB-A7E2208F39C5@microsoft.com...
> Server Spec:
> Windows Server 2003
> SP 1
> Dual 2.66 Ghz
> 2 GB Ram
>
> At random times the memory usage goes way up, sometimes past 2-3 gb. And
> of
> course the server becomes unresponsive through all services--http, ftp,
> email, etc.
>
> When I look at the Task Manager, I can see that the process that is using
> the memory is w3wp.exe for one specific site.
>
> On that site I have the Application Pool set to have 10 worker process,
> recycle every 5 minutes, use a maximum of 42 mb.
>
> What could cause it to go up so high?
>
> Here's a screen shot:
> http://www.powwows.com/ebay/server.jpg
>
> Thanks!

PaulG
Mon Jan 09 20:47:01 CST 2006

Where do I download that from? And how do I run it?

"David Wang [Msft]" wrote:

> Use DebugDiag to triage the w3wp.exe with its "Memory Leak" option and post
> its log report here for more assistance.
>
> FYI: Your health monitoring options seem a bit extreme. Recycling every 5
> minutes is a bad thing for performance (process creation triggered by
> recycling is expensive CPU/Resource-wise) and may be a part of your
> reliability problem. Recycling with a threshold of 42MB may also be
> problematic since that is not a lot of memory and you can blow past it just
> by running one ASP.Net application. And unless you can definitely explain
> *why* you need a web garden of 10 worker process, you probably do not want
> to use it because it can also hinder performance and reliability.
>
> Bottom line: You have a lot of settings right now that may be causing a lot
> of noise, but we will first triage the apparent "Memory Leak" and then go
> from there.
>
> http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Basics_of_IIS6_Troubleshooting.aspx
> http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx
>
> --
> //David
> IIS
> http://blogs.msdn.com/David.Wang
> This posting is provided "AS IS" with no warranties, and confers no rights.
> //
>
> "Paul G" <Paul G@discussions.microsoft.com> wrote in message
> news:3F8DD56E-ED6B-4B7D-B5AB-A7E2208F39C5@microsoft.com...
> > Server Spec:
> > Windows Server 2003
> > SP 1
> > Dual 2.66 Ghz
> > 2 GB Ram
> >
> > At random times the memory usage goes way up, sometimes past 2-3 gb. And
> > of
> > course the server becomes unresponsive through all services--http, ftp,
> > email, etc.
> >
> > When I look at the Task Manager, I can see that the process that is using
> > the memory is w3wp.exe for one specific site.
> >
> > On that site I have the Application Pool set to have 10 worker process,
> > recycle every 5 minutes, use a maximum of 42 mb.
> >
> > What could cause it to go up so high?
> >
> > Here's a screen shot:
> > http://www.powwows.com/ebay/server.jpg
> >
> > Thanks!
>
>
>

Ken
Tue Jan 10 00:09:40 CST 2006

http://www.adopenstatic.com/cs/blogs/ken/archive/2005/10/05/60.aspx
How to obtain Debug Diagnostics

Cheers
Ken

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:08A33E61-9A9B-4F87-AB32-567663A5E61D@microsoft.com...
: Where do I download that from? And how do I run it?
:
: "David Wang [Msft]" wrote:
:
: > Use DebugDiag to triage the w3wp.exe with its "Memory Leak" option and
post
: > its log report here for more assistance.
: >
: > FYI: Your health monitoring options seem a bit extreme. Recycling every
5
: > minutes is a bad thing for performance (process creation triggered by
: > recycling is expensive CPU/Resource-wise) and may be a part of your
: > reliability problem. Recycling with a threshold of 42MB may also be
: > problematic since that is not a lot of memory and you can blow past it
just
: > by running one ASP.Net application. And unless you can definitely
explain
: > *why* you need a web garden of 10 worker process, you probably do not
want
: > to use it because it can also hinder performance and reliability.
: >
: > Bottom line: You have a lot of settings right now that may be causing a
lot
: > of noise, but we will first triage the apparent "Memory Leak" and then
go
: > from there.
: >
: >
http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Basics_of_IIS6_Troubleshooting.aspx
: >
http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx
: >
: > --
: > //David
: > IIS
: > http://blogs.msdn.com/David.Wang
: > This posting is provided "AS IS" with no warranties, and confers no
rights.
: > //
: >
: > "Paul G" <Paul G@discussions.microsoft.com> wrote in message
: > news:3F8DD56E-ED6B-4B7D-B5AB-A7E2208F39C5@microsoft.com...
: > > Server Spec:
: > > Windows Server 2003
: > > SP 1
: > > Dual 2.66 Ghz
: > > 2 GB Ram
: > >
: > > At random times the memory usage goes way up, sometimes past 2-3 gb.
And
: > > of
: > > course the server becomes unresponsive through all services--http,
ftp,
: > > email, etc.
: > >
: > > When I look at the Task Manager, I can see that the process that is
using
: > > the memory is w3wp.exe for one specific site.
: > >
: > > On that site I have the Application Pool set to have 10 worker
process,
: > > recycle every 5 minutes, use a maximum of 42 mb.
: > >
: > > What could cause it to go up so high?
: > >
: > > Here's a screen shot:
: > > http://www.powwows.com/ebay/server.jpg
: > >
: > > Thanks!
: >
: >
: >

PaulG
Wed Jan 11 10:01:02 CST 2006

I've downloaded the tool and loaded it on the server. I'll let you know what
it says next time it occurs.

I changed the settings to just have 3 worker processes and gave each 128 mb.
Is that a better setting?

The site is all PHP pages no .Net. But it is a high traffic site.

Thanks

PaulG
Wed Jan 11 14:31:03 CST 2006

Haven't had the memory issue today.

But even though I've set the worker processes at 3 and restarted IIS, I
still have up to 8 running.

Does IIS spawn more if needed??

David
Thu Jan 12 03:41:19 CST 2006

You set the Max number of w3wp.exe per Application Pool. Based on the number
of Application Pools which are servicing active Applications, you can expect
at most a sum of Max w3wp.exe for all active Application Pools. During
recycling, you may momentarily see one additional w3wp.exe per Application
Pool that is recycling, depending on whether you have overlapped recycling
enabled or not.

Finally, if you have orphaning enabled for debugging reasons there may be
arbitrary number of w3wp.exe stranded but that is by-design since after IIS
orphans the w3wp.exe it just hangs around for you to debug. But, this is not
default behavior.


iisapp.vbs will tell you which w3wp.exe belongs to which Application Pool as
well as PID.


FYI:
If you set the memory limit on the Application Pool to 128MB and your PHP
app does something that momentarily crosses that line, it triggers an
unnecessary process recycle which will cause your subsequent PHP request(s)
to start up slower since a new w3wp.exe must come up (and PHP has to
recompile/rewarm its caches, etc). Combined with high traffic, you can
conceivably set aggresive health-monitoring values that recycle your server
into a death spiral.

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:BFAE3EDA-D364-422A-B47A-E5CD2216E49F@microsoft.com...
> Haven't had the memory issue today.
>
> But even though I've set the worker processes at 3 and restarted IIS, I
> still have up to 8 running.
>
> Does IIS spawn more if needed??

PaulG
Thu Jan 12 08:20:02 CST 2006

Ok, so what would be a good set of configuration settings for me to start at?

That site is all PHP and MySQL. It recieves between 75-100k page views per
day.

Thanks

David
Thu Jan 12 20:34:35 CST 2006

I suggest sticking with the original default values as a starting point.

We chose those values for the majority of users who never tweak with the
server. Our expectation is that if you subsequently fiddle with those
values, you must look up information to understand how they interact and
their respective consequences... at which point the proper response is for
you to "Experiment" and figure out what works for your situation.

I'm not certain what happened in the past to cause you to select the
extremely aggressive and unstable set of parameters (Web Garden of 10,
recycle every 5 minutes, and Memory Threshold of 42MB). For what exact
reason do you Web Garden, do you actually want to recycle every 5 minutes,
and why such a low Memory Threshold.

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:6629328E-CA0D-4162-B5F8-45B93D43512D@microsoft.com...
> Ok, so what would be a good set of configuration settings for me to start
> at?
>
> That site is all PHP and MySQL. It recieves between 75-100k page views
> per
> day.
>
> Thanks

PaulG
Thu Jan 12 20:56:02 CST 2006

I changed them because of the memory issue I was having.

I set it at 5 minutes because when the memory issued occured I needed a way
for the server to restart the app pool more quickly. With the default it
would be a really long time before the server would be responsive again. And
since this is a remote server, I couldn't even remote desktop in to stop the
processes.

I set the memory low because I was using 10 worker processes and didn't want
to overload the ram.

So that being said, should I still stay with the defaults?

David
Fri Jan 13 05:05:13 CST 2006

I think you should start with the defaults and make one modification
specifically to deal with your large-memory consumption.

To me, if the issue is large-memory consumption, it makes sense to just set
IIS's memory-limit heuristic to some large value which if reached means that
too much memory was happening... but definitely not so small that you
false-trigger the recycle. Because as soon as the heuristic is triggered,
IIS automatically shuts the old w3wp.exe down and bring up a new w3wp.exe
immediately, as necessary.

Thus, setting periodic restart to 5 minutes is not so good because:
1. not only is it restarting the app pool LESS quickly than the actual
memory heuristic which caused the recycle (you are merely guaranteeing a
recycle every 5 minutes; the heuristic-triggered recycle will guarantee it
immediately)
2. it also forces your applications to only have 5 minutes worth of data
caching -- which for a synchronous application like PHP and MySQL, you are
working against their attempts to improve performance through caching and
could cause additional work to be done, which may increase the possibility
of high memory usage... which triggers the recycle more frequently and goes
down a death spiral.

Also, can you explain WHY you are using Web Garden.

Finally, I think you need to investigate WHY you are getting the
large-memory consumption issue... because health monitoring is meant to buy
you time to hobble along to find and fix the issue. You should not depend on
it as a long-term crutch because it simply removes one way to buy you more
time in the future.

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:95490A5E-A6C5-4662-86E0-C04FFD491475@microsoft.com...
>I changed them because of the memory issue I was having.
>
> I set it at 5 minutes because when the memory issued occured I needed a
> way
> for the server to restart the app pool more quickly. With the default it
> would be a really long time before the server would be responsive again.
> And
> since this is a remote server, I couldn't even remote desktop in to stop
> the
> processes.
>
> I set the memory low because I was using 10 worker processes and didn't
> want
> to overload the ram.
>
> So that being said, should I still stay with the defaults?

PaulG
Fri Jan 13 10:04:02 CST 2006

I went to a web garden because pages were loading slow. Because of all the
PHP processing and queries to MySQL I hoped to increase the speed of
processing by having multiple worker processes. Is that not the case?

I've set things back to default settings.

And I'm still trying to caputure the info when I have the memory problem.

Thanks

PaulG
Fri Jan 13 10:11:03 CST 2006

Couple of other questions.

First I'm trying to use the debug tool for memory leak. But if the process
restarts every 10 minutes I get a new PID. The rules track by the PID so it
makes it difficult to catch it. Any suggestions?

And what should I being seeing for page faults? Should the numbers be high?
For MySQL it is over 1 million.

Thanks

PaulG
Fri Jan 13 10:13:04 CST 2006

Couple of other questions.

First how do I use the debug tool. I turned it on for memory leak. And
that associates the rule with a PID. But when the worker process restarts it
gets a new number right?

Second how high should page faults be? For MySQL it is over 1 million.

Thanks

David
Sun Jan 15 05:30:38 CST 2006

For memory leak, you generally want to have the leak happen for a while, and
then you want to run a tool like Debug Diag on the process to look through
its allocated memory and examine the leaked data to look for the culprit.
This means that you don't want IIS to also be tracking the leak and
recycling the process once it pasts a certain threshold, PRIOR to you
looking through the memory of the process leaking memory.

Page Faults aren't exactly bad; it is fragmented memory address space or
leaked memory that is bad because they both reduce the number and size of
memory allocations that you can make.

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:FA28B8F4-E58A-4307-8F72-98A96ACC4083@microsoft.com...
> Couple of other questions.
>
> First I'm trying to use the debug tool for memory leak. But if the
> process
> restarts every 10 minutes I get a new PID. The rules track by the PID so
> it
> makes it difficult to catch it. Any suggestions?
>
> And what should I being seeing for page faults? Should the numbers be
> high?
> For MySQL it is over 1 million.
>
> Thanks

PaulG
Thu Jan 19 13:36:02 CST 2006

I haven't had a memory problem since I reset the configurations. Pages were
loading slow so I increased the worker process to 3.

There are times when the CPU 0 stays at 100% for 10 minutes. And it is
w3wp.exe for this app pool showing the most cpu time.

What now?

David
Thu Jan 19 20:50:11 CST 2006

Attach a debugger onto that w3wp.exe when it is at 100%, determine what code
is currently executing and chewing up CPU cycles, and fix that code to not
chew up CPU.

DebugDiag or IISState can help with this.
http://www.microsoft.com/windowsserver2003/iis/diagnostictools/default.mspx

> Pages were loading slow so I increased the
> worker process to 3.

Not certain how this actually helps, given my prior explanation. The right
way to tune performance is to identify and remove bottlenecks inherent in
the code causing performance issue. Changing configuration simply masks the
inherent issue under some circumstances, but other situations can always
come up and re-expose the issue.

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:255EA266-0FCD-4830-AC4D-E48584F21414@microsoft.com...
>I haven't had a memory problem since I reset the configurations. Pages
>were
> loading slow so I increased the worker process to 3.
>
> There are times when the CPU 0 stays at 100% for 10 minutes. And it is
> w3wp.exe for this app pool showing the most cpu time.
>
> What now?

PaulG
Tue Jan 24 15:28:51 CST 2006

I've been watching it closely. And it isn't 1 process causing it to go to
100%. There are a few things that go up above 20 or 30% to make it get to
100%.

So I don't know if it is one particular site at this point or just a high
load. What things can I do at this point?

PaulG
Tue Jan 24 15:30:26 CST 2006

The largest cpu utilization is by mysqld-nt.exe. It jumps to 70% at times.

But everything else seems to be staying around 10-30.

Thanks

David
Wed Feb 01 04:24:30 CST 2006

If your problem is with mysql under "load", then you need to engage that
product's support community to help.

Consider that PHP/MySQL is not exactly high-performance stuff. Does not use
asynchronous IO to exploit inherent performance advantages of Windows...

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:8595A080-A0E5-4F40-AA56-C12A85B8AB0C@microsoft.com...
> The largest cpu utilization is by mysqld-nt.exe. It jumps to 70% at
> times.
>
> But everything else seems to be staying around 10-30.
>
> Thanks

PaulG
Mon Feb 13 14:38:16 CST 2006

"David Wang [Msft]" wrote:

> If your problem is with mysql under "load", then you need to engage that
> product's support community to help.
>
> Consider that PHP/MySQL is not exactly high-performance stuff. Does not use
> asynchronous IO to exploit inherent performance advantages of Windows...
>
> --
> //David
> IIS
> http://blogs.msdn.com/David.Wang
> This posting is provided "AS IS" with no warranties, and confers no rights.
> //
>
> "Paul G" <PaulG@discussions.microsoft.com> wrote in message
> news:8595A080-A0E5-4F40-AA56-C12A85B8AB0C@microsoft.com...
> > The largest cpu utilization is by mysqld-nt.exe. It jumps to 70% at
> > times.
> >
> > But everything else seems to be staying around 10-30.
> >
> > Thanks
>
>
>

I appreciate your help so far. But I really don't want to get into a
discussion about whether open source products are better than other ones.
There are some very large applications running on MySQL and PHP--much larger
than my little site will ever be. So I don't think I'm hitting limits of PHP
or MySQL.

I've gotten a report from the debug tool. Can you look at it? How should I
send it to you?

Thanks

David
Tue Feb 21 22:59:20 CST 2006

Please post the report to the newsgroup so that the community can help and
more people benefit.

I am just saying that if your solution uses PHP/MySQL, then you really want
to obtain support from that community. Especially if you believe you have
not reached its limits; maybe you misconfigured it - best people to tell you
that would be PHP/MySQL community and probably not me. :-)

--
//David
IIS
http://blogs.msdn.com/David.Wang
This posting is provided "AS IS" with no warranties, and confers no rights.
//

"Paul G" <PaulG@discussions.microsoft.com> wrote in message
news:DA6BCD89-2760-412D-B128-4F91E3398C72@microsoft.com...
> "David Wang [Msft]" wrote:
>
>> If your problem is with mysql under "load", then you need to engage that
>> product's support community to help.
>>
>> Consider that PHP/MySQL is not exactly high-performance stuff. Does not
>> use
>> asynchronous IO to exploit inherent performance advantages of Windows...
>>
>> --
>> //David
>> IIS
>> http://blogs.msdn.com/David.Wang
>> This posting is provided "AS IS" with no warranties, and confers no
>> rights.
>> //
>>
>> "Paul G" <PaulG@discussions.microsoft.com> wrote in message
>> news:8595A080-A0E5-4F40-AA56-C12A85B8AB0C@microsoft.com...
>> > The largest cpu utilization is by mysqld-nt.exe. It jumps to 70% at
>> > times.
>> >
>> > But everything else seems to be staying around 10-30.
>> >
>> > Thanks
>>
>>
>>
>
> I appreciate your help so far. But I really don't want to get into a
> discussion about whether open source products are better than other ones.
> There are some very large applications running on MySQL and PHP--much
> larger
> than my little site will ever be. So I don't think I'm hitting limits of
> PHP
> or MySQL.
>
> I've gotten a report from the debug tool. Can you look at it? How should
> I
> send it to you?
>
> Thanks

PaulG
Wed Apr 12 11:10:02 CDT 2006

It is still happening. Here's the report from debugdiag:

http://www.powwows.com/temp/leak.htm

Thanks