Using a message queue as a buffer

I'm developing a server application which will, basically, receive data from
some TCP/IP clients and store them into a database; it's being developed
using .NET 2.0/C#.
The size of each data packet will be small (a few bytes), but the
application need to be able to scale well and handle hundreds of clients,
with some readings/second for each of them.

The application will have two main set of worker threads, with a buffer
between them (a classic producer-consumer model); some threads will acquire
data from the clients and store them in the buffer, while some other will
get data from the buffer and actually perform the database queries; this
structure will reduce the workload on socket reading and allow more socket
reads to be done quickly, instead of having to wait for a DB query to run
after each read; it also will help in the (unlucky) case the DB server can't
be accessed for a while.

Now, the question. Someone suggested me to use a message queue as my buffer,
due to its easy of programming and its automatic handling of thread
synchronization; also, there seems to be an added bonus: if my server stops
or crashes for some reason without being able to store all the buffered data
into the DB, the same queue will be available upon restarting, with all
queued messages being still there.

This seems to be a really good suggestion, but I have a couple of concerns
about this.

First one: performance. Since MSMQ is a general-purpose system, it will
surely have some overhead when confronted to an in-memory buffer; it also
stores its data on disk, wich will introduce even more overhead. How does
MSMQ behave when used for this purpose? If its performance penalty is
significantly low compared to the time needed for a DB query, I don't care
much about it; but if it's significantly high, this can be an issue. Also,
how does it behave in terms of disk storage? Does it create log files which
will need to be purged periodically (like SQL Server do)?

Second one: data format. My program will acquire many kinds of different
data from clients: integers, floating point numbers, text, even images;
sometimes, a single block of data will be composed of combinations of these
(i.e. two floating point numbers, or three integers and some text, and so
on). These blocks of data will be managed as single objects by the
application, and will need to be passed from socket-reading threads to
DB-writing threads; this is quite easy to do with an in-memory buffer, since
objects (and pointers!) are just the same across different threads in the
same program; but what will happen when sending one of these objects to a
queue and receiving it? Will it still be valid, even if internally it
includes, say, a string or an image (which are pointers to other objects)?

Thanks for any help/suggestion :-)


Massimo

sloan
Wed Oct 04 13:40:18 CDT 2006

Here are some links.

http://www.eggheadcafe.com/articles/20041204.asp

http://www.codeproject.com/csharp/survivingpoisonmessages.asp

The queue would be good for what you're doing.

Performance is good. You can test it, but compared to db connectivity, the
queue is small drops of water in the bucket.

You need to research transactional Queues. That will answer some of your
pros and con's questions.

On your second question.
If the object is serializable, then you'll be fine. Look at Peter's
article. You'll basically drop any assembly.dll's you need into the /bin
directory of the WindowsService used to monitor the Queue.


This can help also:
http://msmvps.com/blogs/manoj/archive/2005/10/16/70979.aspx


This book
http://www.apress.com/book/supplementDownload.html?bID=279&sID=2257

has "framework" code for creating queue logic. Its 1.1. I don't know if
there is a 2.0 version of the book/code yet






"Massimo" <barone@mclink.it> wrote in message
news:%23QpjM$95GHA.3452@TK2MSFTNGP05.phx.gbl...
> I'm developing a server application which will, basically, receive data
from
> some TCP/IP clients and store them into a database; it's being developed
> using .NET 2.0/C#.
> The size of each data packet will be small (a few bytes), but the
> application need to be able to scale well and handle hundreds of clients,
> with some readings/second for each of them.
>
> The application will have two main set of worker threads, with a buffer
> between them (a classic producer-consumer model); some threads will
acquire
> data from the clients and store them in the buffer, while some other will
> get data from the buffer and actually perform the database queries; this
> structure will reduce the workload on socket reading and allow more socket
> reads to be done quickly, instead of having to wait for a DB query to run
> after each read; it also will help in the (unlucky) case the DB server
can't
> be accessed for a while.
>
> Now, the question. Someone suggested me to use a message queue as my
buffer,
> due to its easy of programming and its automatic handling of thread
> synchronization; also, there seems to be an added bonus: if my server
stops
> or crashes for some reason without being able to store all the buffered
data
> into the DB, the same queue will be available upon restarting, with all
> queued messages being still there.
>
> This seems to be a really good suggestion, but I have a couple of concerns
> about this.
>
> First one: performance. Since MSMQ is a general-purpose system, it will
> surely have some overhead when confronted to an in-memory buffer; it also
> stores its data on disk, wich will introduce even more overhead. How does
> MSMQ behave when used for this purpose? If its performance penalty is
> significantly low compared to the time needed for a DB query, I don't care
> much about it; but if it's significantly high, this can be an issue. Also,
> how does it behave in terms of disk storage? Does it create log files
which
> will need to be purged periodically (like SQL Server do)?
>
> Second one: data format. My program will acquire many kinds of different
> data from clients: integers, floating point numbers, text, even images;
> sometimes, a single block of data will be composed of combinations of
these
> (i.e. two floating point numbers, or three integers and some text, and so
> on). These blocks of data will be managed as single objects by the
> application, and will need to be passed from socket-reading threads to
> DB-writing threads; this is quite easy to do with an in-memory buffer,
since
> objects (and pointers!) are just the same across different threads in the
> same program; but what will happen when sending one of these objects to a
> queue and receiving it? Will it still be valid, even if internally it
> includes, say, a string or an image (which are pointers to other objects)?
>
> Thanks for any help/suggestion :-)
>
>
> Massimo
>

Massimo
Wed Oct 04 14:31:09 CDT 2006

"sloan" <sloan@ipass.net> ha scritto nel messaggio
news:%23dkvpS%235GHA.3808@TK2MSFTNGP06.phx.gbl...

> Here are some links.

Thanks for the link, many good points.

> The queue would be good for what you're doing.

I was under the same assumption :-)

> Performance is good. You can test it, but compared to db connectivity,
> the
> queue is small drops of water in the bucket.

That's precisely the answer I was looking for.

What about disk storage? Do you know anything about its architecture and how
it behaves when processing tons of small messages? SQL Server (and any other
transaction-based storage system) would build *huge* log files here.


> You need to research transactional Queues. That will answer some of your
> pros and con's questions.

Yes, I was also wondering what to do if a DB query fails after the message
is removed from the queue. Howewer, since I don't particularly care about
message odering (each data block has its own timestamp), I think I can just
send the message back to the queue.

> If the object is serializable, then you'll be fine.

It should, since I'm using only framework-provided data types.
Will the queue sending throw an exception if an unserializable object is
sent?

> Look at Peter's
> article. You'll basically drop any assembly.dll's you need into the /bin
> directory of the WindowsService used to monitor the Queue.

In this scenario, the producer and the consumer are all threads of a single
application; this should avoid any issue between them.
What about the queue itself? Does MSMQ need to be told (someway) what kind
of objects it's processing?

> This book
> http://www.apress.com/book/supplementDownload.html?bID=279&sID=2257
>
> has "framework" code for creating queue logic. Its 1.1. I don't know if
> there is a 2.0 version of the book/code yet

.NET has a whole namespace of messaging objects (System.Messaging); basic
use is quite easy. Maybe many of these details have already been addressed
by the framework developers.


Massimo

sloan
Wed Oct 04 15:06:10 CDT 2006

See inline:


"Massimo" <barone@mclink.it> wrote in message
news:uUC4iu%235GHA.2288@TK2MSFTNGP05.phx.gbl...
> "sloan" <sloan@ipass.net> ha scritto nel messaggio
> news:%23dkvpS%235GHA.3808@TK2MSFTNGP06.phx.gbl...
>
> > Here are some links.
>
> Thanks for the link, many good points.
>
> > The queue would be good for what you're doing.
>
> I was under the same assumption :-)
>
> > Performance is good. You can test it, but compared to db connectivity,
> > the
> > queue is small drops of water in the bucket.
>
> That's precisely the answer I was looking for.
>
> What about disk storage? Do you know anything about its architecture and
how
> it behaves when processing tons of small messages? SQL Server (and any
other
> transaction-based storage system) would build *huge* log files here.
>

I don't know the specifics of this one. But I don't think there are alot of
transactional logs.

I testout my Queue at 5000-10000 items. I pounce it with .Send()'s and it
handles it.
I don't envision ever having more than 1000 items in there, so the
5000-10000 test was good enough for me.
I default to Transactional Queues in my coding practices.
Using Peter's startup work, I created an email sending ICommand object.
I slammed the Queue with 8000 messages, 8 threads sending 1000 each. I
think that took about 90 seconds.
The queue took it. And my WindowsService processed them all. I think it
took about 2 hours to send them all out.
It is the way to go to queue up work, instead of overload failure.


>
> > You need to research transactional Queues. That will answer some of
your
> > pros and con's questions.
>
> Yes, I was also wondering what to do if a DB query fails after the message
> is removed from the queue. Howewer, since I don't particularly care about
> message odering (each data block has its own timestamp), I think I can
just
> send the message back to the queue.
>
> > If the object is serializable, then you'll be fine.
>
> It should, since I'm using only framework-provided data types.
> Will the queue sending throw an exception if an unserializable object is
> sent?

Do a search on BinaryFormatter and msmq.
There is also an xmlformatter and an ActiveX formater (if you put objects in
from VB6 )
The WindowsService that tries to deserialize the object will fail. Esp if
you use the ICommand type thing that Peter uses.

>
> > Look at Peter's
> > article. You'll basically drop any assembly.dll's you need into the
/bin
> > directory of the WindowsService used to monitor the Queue.
>
> In this scenario, the producer and the consumer are all threads of a
single
> application; this should avoid any issue between them.
> What about the queue itself? Does MSMQ need to be told (someway) what kind
> of objects it's processing?

Peter uses everything as an ICommand. That's why his approach is very clean
and good OO.
The queue is just an object holder. It doesn't care.
The thing putting messages in.... and the thing (windows service) reading
things out needs to know what it handling, aka, what kind of objects its
dealing with.



>
> > This book
> > http://www.apress.com/book/supplementDownload.html?bID=279&sID=2257
> >
> > has "framework" code for creating queue logic. Its 1.1. I don't know
if
> > there is a 2.0 version of the book/code yet
>
> .NET has a whole namespace of messaging objects (System.Messaging); basic
> use is quite easy. Maybe many of these details have already been addressed
> by the framework developers.

Yeah. My approach to this framework idea has been:
Use the Factory Design Pattern, and it handles the Transactional or
NonTransactional queue decisions for me.
That's where taking the time to create a generic framework object can help,
among others.



>
>
> Massimo
>

Massimo
Thu Oct 05 12:06:33 CDT 2006

"sloan" <sloan@ipass.net> ha scritto nel messaggio
news:eVuOoC$5GHA.3644@TK2MSFTNGP03.phx.gbl...

> I don't know the specifics of this one. But I don't think there are alot
> of
> transactional logs.
>
> I testout my Queue at 5000-10000 items. I pounce it with .Send()'s and it
> handles it.
> I don't envision ever having more than 1000 items in there, so the
> 5000-10000 test was good enough for me.
> I default to Transactional Queues in my coding practices.
> Using Peter's startup work, I created an email sending ICommand object.
> I slammed the Queue with 8000 messages, 8 threads sending 1000 each. I
> think that took about 90 seconds.
> The queue took it. And my WindowsService processed them all. I think it
> took about 2 hours to send them all out.
> It is the way to go to queue up work, instead of overload failure.

Ok, so it seems this can scale quite well. Good.

>> Will the queue sending throw an exception if an unserializable object is
>> sent?
>
> Do a search on BinaryFormatter and msmq.
> There is also an xmlformatter and an ActiveX formater (if you put objects
> in
> from VB6 )
> The WindowsService that tries to deserialize the object will fail. Esp if
> you use the ICommand type thing that Peter uses.

I'm not using any complex architecture here, just sending a couple of
objects containing data to be inserted into the DB; these classes are just
data structures, there's no code in them. And the producer and the consumer
will be threads of the same process, or maybe (I'm still thinking about it)
two processes using the same assembly DLLs which define the data objects.
The only thing I don't know is how, when and why exactly an object becomes
not serializable...

> Peter uses everything as an ICommand. That's why his approach is very
> clean
> and good OO.

But it's just too complex for my needs :-)

> The queue is just an object holder. It doesn't care.
> The thing putting messages in.... and the thing (windows service) reading
> things out needs to know what it handling, aka, what kind of objects its
> dealing with.

They will.


Massimo

sloan
Thu Oct 05 13:39:32 CDT 2006



At the most basic level

[Serializable]
public class MySerializableClass

Just mark it with this attribute.

See here for more info:
http://www.ondotnet.com/pub/a/dotnet/2002/08/26/serialization.html

If you using vb.net.... there is an additional gotcha.
VB.NET does not allow the application the <NonSerialized> attribute to
events (you can in C# by using the Field: modifier).
As a result, there is no simple way of telling the runtime not to serialize
the event fields.
This results in serializing objects that you didn't expect, resulting in a
larger stream.
If the object handling the events is not Serializable, then the
serialization process will throw an exception.
( See http://www.codeproject.com/vb/net/serializevbclasses.asp for more info
and workaround )


But if you simply have some int, strings or basic objects, then just mark it
as [Serializable] as that should be it.

..

Sounds like you're off to the races....................






"Massimo" <barone@mclink.it> wrote in message
news:uFToZCK6GHA.3452@TK2MSFTNGP05.phx.gbl...
> "sloan" <sloan@ipass.net> ha scritto nel messaggio
> news:eVuOoC$5GHA.3644@TK2MSFTNGP03.phx.gbl...
>
> > I don't know the specifics of this one. But I don't think there are
alot
> > of
> > transactional logs.
> >
> > I testout my Queue at 5000-10000 items. I pounce it with .Send()'s and
it
> > handles it.
> > I don't envision ever having more than 1000 items in there, so the
> > 5000-10000 test was good enough for me.
> > I default to Transactional Queues in my coding practices.
> > Using Peter's startup work, I created an email sending ICommand object.
> > I slammed the Queue with 8000 messages, 8 threads sending 1000 each. I
> > think that took about 90 seconds.
> > The queue took it. And my WindowsService processed them all. I think
it
> > took about 2 hours to send them all out.
> > It is the way to go to queue up work, instead of overload failure.
>
> Ok, so it seems this can scale quite well. Good.
>
> >> Will the queue sending throw an exception if an unserializable object
is
> >> sent?
> >
> > Do a search on BinaryFormatter and msmq.
> > There is also an xmlformatter and an ActiveX formater (if you put
objects
> > in
> > from VB6 )
> > The WindowsService that tries to deserialize the object will fail. Esp
if
> > you use the ICommand type thing that Peter uses.
>
> I'm not using any complex architecture here, just sending a couple of
> objects containing data to be inserted into the DB; these classes are just
> data structures, there's no code in them. And the producer and the
consumer
> will be threads of the same process, or maybe (I'm still thinking about
it)
> two processes using the same assembly DLLs which define the data objects.
> The only thing I don't know is how, when and why exactly an object becomes
> not serializable...
>
> > Peter uses everything as an ICommand. That's why his approach is very
> > clean
> > and good OO.
>
> But it's just too complex for my needs :-)
>
> > The queue is just an object holder. It doesn't care.
> > The thing putting messages in.... and the thing (windows service)
reading
> > things out needs to know what it handling, aka, what kind of objects its
> > dealing with.
>
> They will.
>
>
> Massimo
>

Massimo
Thu Oct 05 13:57:21 CDT 2006

"sloan" <sloan@ipass.net> ha scritto nel messaggio
news:Owt952K6GHA.4692@TK2MSFTNGP04.phx.gbl...

> [Serializable]
> public class MySerializableClass
>
> Just mark it with this attribute.

[...]

> But if you simply have some int, strings or basic objects, then just mark
> it
> as [Serializable] as that should be it.

Ok, thanks.


Massimo