Troubleshooting High Anonymous User Traffic

On occasion, usually once a week, we have some process that is generating a
spike in anonymous users. During normal operations the current anonymous
user count is less than 10. When the spike occurs the count will jump from 0
to over 50 and then continues to climb in a matter of a milliseconds. Once
this spike reaches around 80-100 the server becomes unresponsive. By
unresponsive meaning that I cannot access it through a terminal service
connection (remote desktop) and have to physically restart the server. After
the restart all traffic is processed as normal until the spike occurs again.

This issue of anonymous users was originally tied to custom DLLs that were
not compiled correctly. They were not compiled with the unattended execution
or retained in memory options selected. Once these were recompiled the
anonymous user counts were more stable and the frequency of restarting the
IIS server was reduced.

We are currently logging the performance counter for Web
Service(_Total)\Current Anonymous Users to a file to identify when the counts
are increasing but have not been able to isolate the root cause.

Normal
"02/01/2005 12:40:43.108","1"
"02/01/2005 12:40:44.108","2"
"02/01/2005 12:40:45.108","2"
"02/01/2005 12:40:46.108","2"

Trouble
"02/01/2005 12:41:08.171","22"
"02/01/2005 12:41:13.047","25"
"02/01/2005 12:41:17.562","35"
"02/01/2005 12:41:21.781","40"
"02/01/2005 12:41:27.438","52"
"02/01/2005 12:41:36.250","55"
"02/01/2005 12:41:42.813","59"
"02/01/2005 12:41:45.657","61"

Reboot required
"02/01/2005 12:43:25.003","77"
"02/01/2005 12:43:27.691"," "
"02/01/2005 12:43:28.691"," "
"02/01/2005 12:43:29.691"," "
"02/01/2005 12:43:37.191","104"
"02/01/2005 12:43:42.222","97"

I am looking for a way to identify what is causing the sudden spike in
anonymous users and any suggestions on what to troubleshoot. Once the rogue
process is identified I can begin working out a permanent solution.

Environment Details:
Server1 -> ISA Server
Windows 2000 Advanced Server SP4

Server1 directs traffic to a load balanced Application Center cluster with
Server2 and Server3. The website is not .NET and contains asp and custom
DLLs.

Server2 and Server3 ->
Windows 2000 Advanced Server SP4
Application Center 2000 SP2

Sparky
Fri Aug 05 11:27:19 CDT 2005

Seems to me like you should check the logs. (Or enable them, then check
when it happens again.)

"smilden" <smilden@discussions.microsoft.com> wrote in message
news:93885B65-BC07-4BE8-AE60-9A7E0AB21C4F@microsoft.com...
> On occasion, usually once a week, we have some process that is generating
> a
> spike in anonymous users. During normal operations the current anonymous
> user count is less than 10. When the spike occurs the count will jump
> from 0
> to over 50 and then continues to climb in a matter of a milliseconds.
> Once
> this spike reaches around 80-100 the server becomes unresponsive. By
> unresponsive meaning that I cannot access it through a terminal service
> connection (remote desktop) and have to physically restart the server.
> After
> the restart all traffic is processed as normal until the spike occurs
> again.
>
> This issue of anonymous users was originally tied to custom DLLs that were
> not compiled correctly. They were not compiled with the unattended
> execution
> or retained in memory options selected. Once these were recompiled the
> anonymous user counts were more stable and the frequency of restarting the
> IIS server was reduced.
>
> We are currently logging the performance counter for Web
> Service(_Total)\Current Anonymous Users to a file to identify when the
> counts
> are increasing but have not been able to isolate the root cause.
>
> Normal
> "02/01/2005 12:40:43.108","1"
> "02/01/2005 12:40:44.108","2"
> "02/01/2005 12:40:45.108","2"
> "02/01/2005 12:40:46.108","2"
>
> Trouble
> "02/01/2005 12:41:08.171","22"
> "02/01/2005 12:41:13.047","25"
> "02/01/2005 12:41:17.562","35"
> "02/01/2005 12:41:21.781","40"
> "02/01/2005 12:41:27.438","52"
> "02/01/2005 12:41:36.250","55"
> "02/01/2005 12:41:42.813","59"
> "02/01/2005 12:41:45.657","61"
>
> Reboot required
> "02/01/2005 12:43:25.003","77"
> "02/01/2005 12:43:27.691"," "
> "02/01/2005 12:43:28.691"," "
> "02/01/2005 12:43:29.691"," "
> "02/01/2005 12:43:37.191","104"
> "02/01/2005 12:43:42.222","97"
>
> I am looking for a way to identify what is causing the sudden spike in
> anonymous users and any suggestions on what to troubleshoot. Once the
> rogue
> process is identified I can begin working out a permanent solution.
>
> Environment Details:
> Server1 -> ISA Server
> Windows 2000 Advanced Server SP4
>
> Server1 directs traffic to a load balanced Application Center cluster with
> Server2 and Server3. The website is not .NET and contains asp and custom
> DLLs.
>
> Server2 and Server3 ->
> Windows 2000 Advanced Server SP4
> Application Center 2000 SP2
>
>
>

smilden
Fri Aug 05 11:58:12 CDT 2005

I have logging enabled on the IIS servers and using the W3C Extended Log File
Format.

In the past I have tried to sift through the log to identify the requests
causing the spike in anonymous users but have been unsuccessful.

Is there a different format I should be using for capturing the log
information? Are there any tell tell signs that I can be looking for?

"Sparky Polastri" wrote:

> Seems to me like you should check the logs. (Or enable them, then check
> when it happens again.)
>
> "smilden" <smilden@discussions.microsoft.com> wrote in message
> news:93885B65-BC07-4BE8-AE60-9A7E0AB21C4F@microsoft.com...
> > On occasion, usually once a week, we have some process that is generating
> > a
> > spike in anonymous users. During normal operations the current anonymous
> > user count is less than 10. When the spike occurs the count will jump
> > from 0
> > to over 50 and then continues to climb in a matter of a milliseconds.
> > Once
> > this spike reaches around 80-100 the server becomes unresponsive. By
> > unresponsive meaning that I cannot access it through a terminal service
> > connection (remote desktop) and have to physically restart the server.
> > After
> > the restart all traffic is processed as normal until the spike occurs
> > again.
> >
> > This issue of anonymous users was originally tied to custom DLLs that were
> > not compiled correctly. They were not compiled with the unattended
> > execution
> > or retained in memory options selected. Once these were recompiled the
> > anonymous user counts were more stable and the frequency of restarting the
> > IIS server was reduced.
> >
> > We are currently logging the performance counter for Web
> > Service(_Total)\Current Anonymous Users to a file to identify when the
> > counts
> > are increasing but have not been able to isolate the root cause.
> >
> > Normal
> > "02/01/2005 12:40:43.108","1"
> > "02/01/2005 12:40:44.108","2"
> > "02/01/2005 12:40:45.108","2"
> > "02/01/2005 12:40:46.108","2"
> >
> > Trouble
> > "02/01/2005 12:41:08.171","22"
> > "02/01/2005 12:41:13.047","25"
> > "02/01/2005 12:41:17.562","35"
> > "02/01/2005 12:41:21.781","40"
> > "02/01/2005 12:41:27.438","52"
> > "02/01/2005 12:41:36.250","55"
> > "02/01/2005 12:41:42.813","59"
> > "02/01/2005 12:41:45.657","61"
> >
> > Reboot required
> > "02/01/2005 12:43:25.003","77"
> > "02/01/2005 12:43:27.691"," "
> > "02/01/2005 12:43:28.691"," "
> > "02/01/2005 12:43:29.691"," "
> > "02/01/2005 12:43:37.191","104"
> > "02/01/2005 12:43:42.222","97"
> >
> > I am looking for a way to identify what is causing the sudden spike in
> > anonymous users and any suggestions on what to troubleshoot. Once the
> > rogue
> > process is identified I can begin working out a permanent solution.
> >
> > Environment Details:
> > Server1 -> ISA Server
> > Windows 2000 Advanced Server SP4
> >
> > Server1 directs traffic to a load balanced Application Center cluster with
> > Server2 and Server3. The website is not .NET and contains asp and custom
> > DLLs.
> >
> > Server2 and Server3 ->
> > Windows 2000 Advanced Server SP4
> > Application Center 2000 SP2
> >
> >
> >
>
>
>

Sparky
Mon Aug 08 14:18:08 CDT 2005

Well, one would think making sure you understand the timestamp (some logs
use GMT) and then looking in the time when the problems occur?

If the requests are coming from the outside, and are unusual traffic you
should see it in the log.

Also get URLscan installed and configured (good idea anyway for security
reasons) and look in its logs too.

I am sure someone can help figure out what a log entry is from/doing if you
copy and paste one.