I developed an application that automates dwonloading of a web page
hundreds of times. I first developed it by using http web request, but
then I read up that HTTP Web Request has some overhead that I thought I
could eliminate if I re-wrote it with sockets. However, what I have
found is that sockets actually works significantly slower (I get about
1/2 to 2/3 of the performance I get with HTTPWebRequest).
Now, the web server sends a close command on every request so even if I
post a Keep-Alive request I have to create and open a new socket for
every request, but I assume the HTTPRequest class must do the same
thing under the covers. Below is my Socket Send Method (very basic).
Can anyone explain this strange disparity in performance?

public string Send(string strRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

string strRetPage = null;

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[256];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
strRetPage = strRetPage + ASCII.GetString(RecvBytes, 0, bytes);
} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return strRetPage;

}

Re: Socket Class Slower Then HTTP Web Request by Feroze

Feroze
Thu Mar 24 13:45:31 CST 2005

The best thing to do is to do a network sniff of the app running with
sockets code and compare it to the app running with HttpWebRequest code.

--
feroze

-----------------
This posting is provided as-is. It offers no warranties and assigns no
rights.

See http://weblogs.asp.net/feroze_daud for System.Net related posts.
----------------

<mwieder@gmail.com> wrote in message
news:1111675798.366757.54250@g14g2000cwa.googlegroups.com...
> I developed an application that automates dwonloading of a web page
> hundreds of times. I first developed it by using http web request, but
> then I read up that HTTP Web Request has some overhead that I thought I
> could eliminate if I re-wrote it with sockets. However, what I have
> found is that sockets actually works significantly slower (I get about
> 1/2 to 2/3 of the performance I get with HTTPWebRequest).
> Now, the web server sends a close command on every request so even if I
> post a Keep-Alive request I have to create and open a new socket for
> every request, but I assume the HTTPRequest class must do the same
> thing under the covers. Below is my Socket Send Method (very basic).
> Can anyone explain this strange disparity in performance?
>
> public string Send(string strRequest)
> {
> m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
> ProtocolType.Tcp);
>
> m_sock.Connect(ipe);
>
> string strRetPage = null;
>
> Encoding ASCII = Encoding.ASCII;
>
> Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
> Byte[] RecvBytes = new Byte[256];
> m_sock.Send(ByteGet, ByteGet.Length, 0);
> Int32 bytes;
>
> do
> {
> bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
> strRetPage = strRetPage + ASCII.GetString(RecvBytes, 0, bytes);
> } while (bytes > 0);
>
> m_sock.Shutdown(SocketShutdown.Both);
> m_sock.Close();
>
> return strRetPage;
>
> }
>



Re: Socket Class Slower Then HTTP Web Request by Joerg

Joerg
Fri Mar 25 04:24:18 CST 2005

mwieder@gmail.com wrote:

> I developed an application that automates dwonloading of a web page
> hundreds of times. I first developed it by using http web request,
> but then I read up that HTTP Web Request has some overhead that I
> thought I could eliminate if I re-wrote it with sockets.

<rant>
Isn't it funny how people run off and reinvent the wheel because of
rumors :-/
</rant>

> However,
> what I have found is that sockets actually works significantly slower
> (I get about 1/2 to 2/3 of the performance I get with HTTPWebRequest).
> Now, the web server sends a close command on every request so even if
> I post a Keep-Alive request I have to create and open a new socket for
> every request, but I assume the HTTPRequest class must do the same
> thing under the covers.

Did you use a network sniffer to verify that? Creating a new TCP
connection will create some significant overhead.

> Below is my Socket Send Method (very basic).
> Can anyone explain this strange disparity in performance?
>
> public string Send(string strRequest)
> {
> m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
> ProtocolType.Tcp);
>
> m_sock.Connect(ipe);
>
> string strRetPage = null;
>
> Encoding ASCII = Encoding.ASCII;
>
> Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
> Byte[] RecvBytes = new Byte[256];
> m_sock.Send(ByteGet, ByteGet.Length, 0);
> Int32 bytes;
>
> do
> {
> bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
> strRetPage = strRetPage + ASCII.GetString(RecvBytes, 0, bytes);
> } while (bytes > 0);
>
> m_sock.Shutdown(SocketShutdown.Both);
> m_sock.Close();
>
> return strRetPage;
>
> }

You're using a rather small read buffer and decode the incoming bytes
on each iteration (and use ASCIIEncoding, which breaks all non-US-ASCII
content).

You're also using naive string concatenation ("+"), which will create a
lot of temporary objects. Use a MemoryStream or a StringBuilder to
buffer the incoming data.


Cheers,
--
http://www.joergjooss.de
mailto:news-reply@joergjooss.de

Re: Socket Class Slower Then HTTP Web Request by mwieder

mwieder
Fri Mar 25 09:02:13 CST 2005

the netwrok sniff returns the same thing for both. The time difference
is due to the work being done on my machine in between requests. I
also suspect the issue is creating and deleting the socket every time,
but because the server forces closure of the socket (attempts at re-use
confirm this) I must open a new socket to connect. I would think the
HTTPWebRequest class must do teh same thing, I just can't understand
why it is slower...


Re: Socket Class Slower Then HTTP Web Request by mwieder

mwieder
Fri Mar 25 09:31:10 CST 2005

Hi - thanks for the suggestions - I understood everything you suggested
(see new code below) except the comment on the ASCIIEncoding - please
change th code below to illustarte what you are suggesting there.
thanks!

public string Send(string HttpRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[16384];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

StringBuilder sbDataReceived = new StringBuilder();

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
sbDataReceived.Append(ASCII.GetString(RecvBytes, 0, bytes));

} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return sbDataReceived.ToString();

}


Re: Socket Class Slower Then HTTP Web Request by Feroze

Feroze
Fri Mar 25 17:40:08 CST 2005

Look at this post, which will show you what the suggested way of downloading
is:

http://blogs.msdn.com/feroze_daud/archive/2004/03/30/104440.aspx

--
feroze

-----------------
This posting is provided as-is. It offers no warranties and assigns no
rights.

See http://weblogs.asp.net/feroze_daud for System.Net related posts.
----------------

<mwieder@gmail.com> wrote in message
news:1111764670.352222.307260@f14g2000cwb.googlegroups.com...
> Hi - thanks for the suggestions - I understood everything you suggested
> (see new code below) except the comment on the ASCIIEncoding - please
> change th code below to illustarte what you are suggesting there.
> thanks!
>
> public string Send(string HttpRequest)
> {
> m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
> ProtocolType.Tcp);
>
> m_sock.Connect(ipe);
>
> Encoding ASCII = Encoding.ASCII;
>
> Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
> Byte[] RecvBytes = new Byte[16384];
> m_sock.Send(ByteGet, ByteGet.Length, 0);
> Int32 bytes;
>
> StringBuilder sbDataReceived = new StringBuilder();
>
> do
> {
> bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
> sbDataReceived.Append(ASCII.GetString(RecvBytes, 0, bytes));
>
> } while (bytes > 0);
>
> m_sock.Shutdown(SocketShutdown.Both);
> m_sock.Close();
>
> return sbDataReceived.ToString();
>
> }
>



Re: Socket Class Slower Then HTTP Web Request by Joerg

Joerg
Sat Mar 26 02:38:24 CST 2005

mwieder@gmail.com wrote:

> Hi - thanks for the suggestions - I understood everything you
> suggested (see new code below) except the comment on the
> ASCIIEncoding - please change th code below to illustarte what you
> are suggesting there. thanks!
>
> public string Send(string HttpRequest)
> {
> m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
> ProtocolType.Tcp);
>
> m_sock.Connect(ipe);
>
> Encoding ASCII = Encoding.ASCII;
>
> Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
> Byte[] RecvBytes = new Byte[16384];
> m_sock.Send(ByteGet, ByteGet.Length, 0);
> Int32 bytes;
>
> StringBuilder sbDataReceived = new StringBuilder();
>
> do
> {
> bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
> sbDataReceived.Append(ASCII.GetString(RecvBytes, 0, bytes));
>
> } while (bytes > 0);
>
> m_sock.Shutdown(SocketShutdown.Both);
> m_sock.Close();
>
> return sbDataReceived.ToString();
>
> }

See Feroze's post for a version that uses a MemoryStream for buffering
and decodes the whole buffer later.

Here's another one, simpler version without characters set detection:
http://tinyurl.com/6aedc

Cheers,

--
http://www.joergjooss.de
mailto:news-reply@joergjooss.de

Re: Socket Class Slower Then HTTP Web Request by mwieder

mwieder
Mon Mar 28 08:48:49 CST 2005

Thanks for your help. Hopefully this optomizes the process as much as
can be done.


Re: Socket Class Slower Then HTTP Web Request by Joerg

Joerg
Mon Mar 28 10:45:36 CST 2005

mwieder@gmail.com wrote:

> Thanks for your help. Hopefully this optomizes the process as much as
> can be done.

No probs. I still recommend sticking to HttpWebRequest, though. There's
more to a proper HTTP implementation that sending a

GET /path/page.html HTTP/1.1
Host: www.foo.org

over a TCP socket.

Cheers,
--
http://www.joergjooss.de
mailto:news-reply@joergjooss.de