David
Tue Jul 29 13:23:23 CDT 2008
Since Linux lacks File Notification Change capabilities (i.e.
asynchronously detect when a file has been created/modified/deleted on
the filesystem), it means the remote system must have one thread
rapidly polling that directory looking for new files, which it
triggers as work request and produce the response file. This polling
is synchronous and thus a real bottleneck at the remote side.
Samba allows multi-user, concurrent access to files. However, it needs
to be used that way by the application -- which does not appear to be
the case.
Yes, given the design of your system, your web server will pretty much
sit idle as it waits single-threaded for the response from the remote
server via this file-system remote invocation method -- while the
remote server has high CPU from constantly polling the file-system.
Remember, performance perception by the end-user is the sum of all
parts -- just because your web server has low CPU utilization hardly
means that it is the problem nor should you try to load the web server
CPU more.
You will have to make clear to your management that:
1. This design is BAD
2. Without changing the design on both local ASP app and remote app,
performance cannot be improved
Sorry, they can apply all sorts of pressure, but given the software's
design, they are asking the impossible.
To help you make clear the problems with the design, here's an
analogy. Suppose you are sending a piece of mail and want delivery
confirmation of that piece of mail. The efficient way to do it would
be for you to drop off the piece of mail with delivery confirmation
attached, and when the postman delivers the mail, he detaches the
delivery confirmation and mails it back.
The way your apps design works is -- you drop off the piece of mail
with delivery confirmation attached and then WAIT at the Post Office
counter until the postman delivers the mail to the recipient. The
Recipient goes to his PO Box once every second to check if any such
mail has been received, and takes another second to detach and process
the delivery confirmation and mail it back, which eventually makes it
back to that Post Office and they grab it out of pre-sorting and hand
it back to you.
Clearly, NO ONE waits at the Post Office counter waiting for delivery
confirmation, and NO ONE polls his PO Box every second of every day to
check for mail and return delivery confirmation, but that is EXACTLY
what is designed within your local and Remote Application...
To further the analogy -- suppose you run your own Mail Delivery
business (sitting on top of the USPS -- I know, a silly thing, but
bear with me), so you cannot afford to wait at such a Post Office for
delivery confirmation to return back to your clients. So you hire a
pool of 5 worker couriers whose sole jobs is to courier the mail to
the Post Office, wait for Delivery Confirmation to return, and then
courier that confirmation back to your business.
Now, if SIX customers simultaneously come in to send mail with your
business, the first five get their mail couriered, but the sixth
customer has to sit in the queue, waiting for one of the worker
couriers to finish. And the other five customers also have to sit
around waiting for the delivery confirmation to be returned. And in
this wacky world, everyone sends mail to the exact same fellow and his
PO Box.
Clearly, customers would NOT like such a system where they are sending
mail and have to sit around waiting for the delivery confirmation. And
even worse, everyone is waiting on the same fellow and his PO Box
polling.
All the Internet has done is make the mail and delivery confirmation
transport be "instaneous", say one second to deliver between any
destination. So, how long does your customer sit waiting for their
mail delivery confirmation? They have to wait in the queue for your
couriers to dispatch to the Post Office. Your Couriers have to wait at
the Post Office for the other fellow polling his PO Box every second
to get the mail, each time taking one second of processing time and
only returning ONE Delivery Confirmation. And when the Delivery
Confirmation is returned, it is couriered back to your customers'
hands.
Such a system would be fine if there's only one customer at a time,
but if there are six simultaneous customers, one is guaranteed to wait
in a queue. And since the PO Box fellow only processes one mail at a
time and takes one second to process each mail delivery, here's the
amount of time that it takes for all six customers to get their
delivery confirmation:
1st customer: 3 seconds (1 second sending, 1 second receiving, 1
second processing)
2nd customer: 4 seconds (1 second sending, 1 second receiving, 1
second processing, 1 second waiting on PO Box fellow to finish
processing 1st customer's confirmation)
3rd customer: 5 seconds (1 second sending, 1 second receiving, 1
second processing, 1 second waiting on PO Box fellow to finish
processing 1st customer's confirmation, 1 second waiting on PO Box
fellow to finish processing 2nd customer's confirmation)
4th customer: 6 seconds (1 second sending, 1 second receiving, 1
second processing, 1 second waiting on PO Box fellow to finish
processing 1st customer's confirmation, 1 second waiting on PO Box
fellow to finish processing 2nd customer's confirmation, 1 second
waiting on PO Box fellow to finish processing 3rd customer's
confirmation)
5th customer: 7 seconds (1 second sending, 1 second receiving, 1
second processing, 1 second waiting on PO Box fellow to finish
processing 1st customer's confirmation, 1 second waiting on PO Box
fellow to finish processing 2nd customer's confirmation, 1 second
waiting on PO Box fellow to finish processing 3rd customer's
confirmation, 1 second waiting on PO Box fellow to finish processing
4th customer's confirmation)
6th customer: 9 seconds (3 seconds waiting in queue, 1 second sending,
1 second receiving, 1 second processing, 1 second waiting on PO Box
fellow to finish processing 3rd customer's confirmation, 1 second
waiting on PO Box fellow to finish processing 4th customer's
confirmation, 1 second waiting on PO Box fellow to finish processing
5th customer's confirmation)
As you can see, this system is seriously bottlenecked at the pace of
processing by the remote app. And you cannot improve the performance
by adding threads to ASP (i.e. adding more worker couriers) because
the work still gets stuck at the remote end being processed once a
second. And if you have more incoming work than ASP threads, the
remainder will queue and take longer. And you cannot question whether
your workers are working hard (i.e. high CPU) if the system has people
idling and waiting for results all the time.
So, given this analogy, how do you propose to reduce the waiting time
of your customers awaiting delivery confirmation of their mails?
//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//
On Jul 29, 9:27=A0am, JamieC <jcondli...@gmail.com> wrote:
> Hi David,
>
> Many thanks for your help!
>
> I couldn't agree more that using filesystem as a communication
> transport is a terrible, terrible idea. Unfortunately I am under
> pressure to 'fix' the system without any changes being made to the
> remote location, so I have work with the interface I have.
>
> You mention concurrent access to the file system - this is another
> area that the application has an interesting solution to. The file
> exchange mechanism is used by the system when a user completes a
> transaction on the web site:
>
> - user submits form
> - file is created
> - 302 redirect to new ASP script
> - app spins (typically a response is received in a couple of seconds
> but this can be much longer)
> - response file is picked up
> - app displays contents of response file (as well as recording details
> locally in the database).
>
> The file name of the files generated is dictated by a bigint field in
> a one row database table which is incremented every time a file is
> generated by the system (thus another bottleneck!). Are you suggesting
> that the Samba server at the other end may restrict the number of
> concurrent users able to create files, even if they are creating
> different files? Also, when the ASP script connects to the Samba
> server, will each of these instances manifest as a seperate user?
>
> Assuming the spin time is an average of 3 seconds, is it possible to
> estimate the load that would use up the available worker threads in
> the pool? Is this kind of period actually a big difference to the
> typical < 1 second execution of an ASP page?
>
> I have a slight problem in that the web servers do not actually appear
> to max out the CPU - in fact although they spike to 100% otherwise it
> remains pretty low. Is it possible for these application-wide
> performance issues to be caused by the file exchange mechanism even
> without maxing the processor?
>
> Many thanks for your suggestions on tools for diagnosing the problem.
> I'm certainly going to give IISDiag a try next time we come under
> heavy load.
>
> Again, I realise that I'm being excrutiatingly vauge - my knowledge of
> the system is not yet as extensive as I'd like - so any help
> appreciated!
>
> Thanks,
>
> Jamie
>
> PS - FWIW, the perl script used for the pause is:
>
> <script language=3D"perlscript" runat=3D"server">
> =A0 =A0 =A0 =A0 sub pause
> =A0 =A0 =A0 =A0 {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 my $del =3D shift(@_);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sleep($del);
> =A0 =A0 =A0 =A0 }
> </script>
>
> And appears to be called from the ASP VBScript in the manner of a
> usual VBScript subroutine, eg. pauseIt(3).
>
> On Jul 26, 12:34=A0am, David Wang <w3.4...@gmail.com> wrote:
>
>
>
> > Performance problems have nothing to do with ASP or SMB shares
> > specifically -- if you are talking about throughput performance, then
> > you are interested in latency (i.e. something that takes a long time
> > to execute).
>
> > How long is the perl script going to block execution? Because it chews
> > up a worker thread when it does a synchronous blocking operation like
> > that, and actions like that will almost ALWAYS kill performance of the
> > entire system that uses the same worker thread pool. All ASP apps in
> > the same process use the same thread pool (no, it makes no sense to
> > have a per-application thread pool for the purposes of isolation --
> > all it takes is one thread in the process to take up 100% CPU and
> > there's nothing the other threads can do about it).
>
> > FYI: Using filesystem as a means of remote-communication is almost
> > never going to scale. If its usage does not allow concurrent access,
> > it means that while IIS can allow 10 concurrent users across the
> > cluster to access the same web page, 9 users will wait for 1 user to
> > first finish this remote communication protocol using SMB before they
> > can proceed in a single-file fashion. That is just single-threaded
> > design which will just CHOKE under any load and never be scalable for
> > a web application no matter what you do.
>
> > And no, the solution is NOT to increase the number of threads in the
> > pool to some massive number to avoid thread starvation and increased
> > latency (which often lead to timeouts) -- increasing concurrency with
> > threads only helps to a certain point (depends on many, many factors),
> > at which point the cost of maintaining concurrency (i.e. context
> > switching) overwhelms the perception of thread availability.
>
> > In short, you have identified a very critical bottleneck in this
> > application that will affect its scalability. You just need some proof
> > that the page running the Perl script is the cause of your issue --
> > you can use a tool like Debug Diagnostics 1.1 or IISDiag to capture a
> > memory dump during times of poor performance to see what ASP page is
> > causing it -- and deduce if it is related to the Perl script or not.
> > And in general, you cannot fix a performance flaw in the application's
> > design by configuration. You can only mask it, and only under certain
> > load and conditions.
>
> > For health monitoring, you want to monitor things that your
> > application actually impacts. There's no standard set; it always has
> > to be customized and account for the application's specifics.
>
> > //David
http://w3-4u.blogspot.comhttp://blogs.msdn.com/David.Wang
> > //
>
> > On Jul 25, 5:22=A0am, JamieC <jcondli...@gmail.com> wrote:
>
> > > Hi,
>
> > > I have recently inherited a very large, very old ASP application. The
> > > app is on a cluster of web servers, that also talk to a database
> > > server running SQL Server 2005.
>
> > > Among other things, the app has a page that accesses a remote file
> > > share (Samba, actually) down a 1mb private circuit. The page also
> > > appears to load in a perl script that blocks execution while it waits
> > > for the machine with the share to read in the file that's been
> > > written, and write a 'response' file - which the script then reads in=
.
>
> > > During periods of intense load, the entire application seems to
> > > perform poorly, with page requests timing out or taking a long time t=
o
> > > return data. This behaviour is exhibited by all ASP pages, not just
> > > ones that perform the file exchange or access the database. Our
> > > monitoring shows that the web servers do occasionally peak at 100% CP=
U
> > > usage, but it is not constant.
>
> > > My question is whether the performance of all ASP pages in a given
> > > application could be affected by having an ASP page that is working
> > > with SMB shares, or taking a long time to execute? If so, what is the
> > > bottleneck? Pooled worker threads? Outgoing SMB ports? I was also
> > > wondering what people might recommend for monitoring the health of
> > > this system to attempt to identify the cause of the problem.
>
> > > I realise I have been pretty vague, so many thanks in advance for any
> > > help people may offer, as well as suggetions about how I can provide
> > > more information about the issue.
>
> > > Cheers,
>
> > > Jamie- Hide quoted text -
>
> - Show quoted text -