Running out of Sockets

I am working on a .Net Web Service which will be used by thousands of users
at a time. It does lot of file operations (Creating, deleting, enumerating,
reading attributes etc) and Ldap queries to AD using .Net APIs. Under stress
test, service machine runs out of Sockets and we start seeing weird errors. I
created MaxUserPort entry in registry on service machine toa llow 65,500 max
no. of sockets to be created. It runs fine now. It could be a problem later
during more stress in production. Is there a way in .Net to minimize socket
usage during file and ldap AD operations? Is there a way i can force the
machine to recycle the sockets so it never runs out of socket.

Thanks,
Salim Ansari

Jani
Thu Jul 12 05:35:38 CDT 2007

Hi Salim,

> I am working on a .Net Web Service which will be used by thousands of
> users
> at a time. It does lot of file operations (Creating, deleting,
> enumerating,
> reading attributes etc) and Ldap queries to AD using .Net APIs. Under
> stress
> test, service machine runs out of Sockets.

If you need to support thousands of users (incoming connections) and your
applications opens many further connections (outgoing) for each incoming
connection, then you could quite easily run out of sockets especially if the
connections you make take a second of more to complete. TCP protocol uses a
16-bit port number, so you can have no more than about 65500 connections
open at any given time.

However, that should be plenty for most applications. If you need to support
thousands of users, then there are three things I can suggest (not in any
special order):

1. Conserve the sockets for example with a pool, or, make the use of your
outgoing sockets sequential instead of firing all queries at once.

2. Make the outgoing connections faster or eliminate them altogether for
example with a cache, or by improving performance on the other end-point.

3. If the above won't work or can't be easily done, then load balance the
queries into one or more server(s). There are appliances available that can
show a single IP address to the outside world, but still maintain a farm of
servers behind it. If you are having thousands of users, I believe the costs
for an additional server + the said appliance are easily justified. I'd
estimate they could be in the range of $5,000-$10,000.

Hope this gives food for thought. There's only so much software alone can
do.

--
Regards,

Mr. Jani Järvinen
C# MVP
Helsinki, Finland
janij@removethis.dystopia.fi
http://www.saunalahti.fi/janij/

Salim
Thu Jul 12 08:50:03 CDT 2007

Thanks for the reply. Problem is my service can run in different user context
depending on the client domain. So when i am running in different user
contexts, ldap connection caching is not working and ADSI is creating new
connections to ldap for every request i guess.

Is there a way to use connection pooling from ldap while running in
different context?

Thanks,
Salim

"Jani Järvinen [MVP]" wrote:

> Hi Salim,
>
> > I am working on a .Net Web Service which will be used by thousands of
> > users
> > at a time. It does lot of file operations (Creating, deleting,
> > enumerating,
> > reading attributes etc) and Ldap queries to AD using .Net APIs. Under
> > stress
> > test, service machine runs out of Sockets.
>
> If you need to support thousands of users (incoming connections) and your
> applications opens many further connections (outgoing) for each incoming
> connection, then you could quite easily run out of sockets especially if the
> connections you make take a second of more to complete. TCP protocol uses a
> 16-bit port number, so you can have no more than about 65500 connections
> open at any given time.
>
> However, that should be plenty for most applications. If you need to support
> thousands of users, then there are three things I can suggest (not in any
> special order):
>
> 1. Conserve the sockets for example with a pool, or, make the use of your
> outgoing sockets sequential instead of firing all queries at once.
>
> 2. Make the outgoing connections faster or eliminate them altogether for
> example with a cache, or by improving performance on the other end-point.
>
> 3. If the above won't work or can't be easily done, then load balance the
> queries into one or more server(s). There are appliances available that can
> show a single IP address to the outside world, but still maintain a farm of
> servers behind it. If you are having thousands of users, I believe the costs
> for an additional server + the said appliance are easily justified. I'd
> estimate they could be in the range of $5,000-$10,000.
>
> Hope this gives food for thought. There's only so much software alone can
> do.
>
> --
> Regards,
>
> Mr. Jani Järvinen
> C# MVP
> Helsinki, Finland
> janij@removethis.dystopia.fi
> http://www.saunalahti.fi/janij/
>
>
>

Chris
Thu Jul 12 12:11:25 CDT 2007

If you're bumping into these limits, you're going to need multiple IP
Addresses.Recycling of ports is something that can happen, but in boundry
cases - where it may take several minutes for the O/S to close a TCP
connection - you could still exhaust things.

We we did our big scalability tests, we had 4 network cards, each with 10 IP
Addresses bound to them. For incoming connections, we relied on basic
client-side DNS randomization. That is, we assigned 10 IP Addresses to
"host1", 10 to "host2' , 10 to "host3", etc. When the client picks an IP for
"host1", it's supposed to randomize which of the 10 addresses it uses. All
the standard DNS lookup tools will take care of this.

Now, for LDAP queries using .Net API's, if you're opening and closing
sockets that frequently, you've got bigger problems. You really should build
some sort of LDAP Connection Pool, which allows you to re-use LDAP
connections. You don't want to setup and tear-down an LDAP connection for
each query. Given the load this introduces, it's a big scalability hit.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

"Salim" <Salim@discussions.microsoft.com> wrote in message
news:050B3C04-591E-4596-B5ED-B772D2CF5838@microsoft.com...
>I am working on a .Net Web Service which will be used by thousands of users
> at a time. It does lot of file operations (Creating, deleting,
> enumerating,
> reading attributes etc) and Ldap queries to AD using .Net APIs. Under
> stress
> test, service machine runs out of Sockets and we start seeing weird
> errors. I
> created MaxUserPort entry in registry on service machine toa llow 65,500
> max
> no. of sockets to be created. It runs fine now. It could be a problem
> later
> during more stress in production. Is there a way in .Net to minimize
> socket
> usage during file and ldap AD operations? Is there a way i can force the
> machine to recycle the sockets so it never runs out of socket.
>
> Thanks,
> Salim Ansari

Jani
Fri Jul 13 02:07:30 CDT 2007

Hi Salim,

> Is there a way to use connection pooling from ldap while running in
> different context?

No, I'm afraid you would need to develop such a pool yourself. If scaling
out your application into two or more servers isn't an option, I believe
pooling would be your best option to solve your problem.

Hope this helps.

--
Regards,

Mr. Jani Järvinen
C# MVP
Helsinki, Finland
janij@removethis.dystopia.fi
http://www.saunalahti.fi/janij/

Chris
Fri Jul 13 12:09:15 CDT 2007

"Salim" <Salim@discussions.microsoft.com> wrote:
> Is there a way to use connection pooling from ldap while running in
> different context?

Developing a connection pool is pretty easy:
http://www.coversant.com/Default.aspx?tabid=88&EntryID=36

(That particular pattern, is the only time I've ever had a chance to put
Resurrection to use. Fun stuff. )

You do need to be VERY aware of the threading requirements of the object
you're creating pools with. If I remember correctly, the various
System.DirectoryServices classes are not thread safe in the least, and you
really need to error on the side of extreme caution.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins