NDISPROT read and speed problems.

Hi all,

I have to develop an 1 Gbps connection between our front-end board and
PC. I found that the easiest and fastest way (especially form board
development point of view) will be to use the raw Ethernet frames.
I found that NDISPROT example from DDK could almost directly be used to
send and receive raw frames. I've made some tests - just connect two PC
with onboard gigabit NICs - and got problems.

First is that the speed of the connection is about 350 Mbps and 150 Mbps
for 9k frames and 1.5k frames respectively, which is event not close to
1 Gbps. Is that problem because of the NDISPROT driver itself or the
NICs (NVIDIA nForce + Intel PRO 1000 MT) are not good enough?

Second problem is when I sending frames from the first NIC and receive
on the second no all frames are read out. Now I solve that problem just
sending another small confirmation frame back, but it is no the way
which I like. I have to mentioned that if on the receiver side I look to
"Local Area Connection Status" the number of received frames is ok, so
all have reached the receiver PC even without that confirmation frames.

Regards,
Bartek

Stephan
Thu May 18 04:42:08 CDT 2006

Not sure what you actually want to achieve. Is it just that you want to
stress the GigE wire with maximum load?

In this case you can use network sniffer tools such as Ethereal, which
allow to both receive (and display) as well as also send frames to the
wire.

AFAIK, the NDISPROT approach is a quite simple one and implies some
overhead. So there is a good chance you cannot get to 1 Gbps overall
throughput.

Also the PCI bus plays a mojor role here. Take care you do not install
your cards behind some PCI bridge. Also both the card and machine
should support 64-bit and/or 66 MHz. Note that both 64-bit and 66 MHz
are optional and a NIC capable of 64-bit/66 MHz usually also runs in an
"old" 32-bit/33 MHz PCI slot.

IIRC, you can usually get tools like TTCP (see e.g.
http://www.pcausa.com/Utilities/pcattcp.htm) to fully load a GigE wire
when using the UDP option (TCP implies too much overhead).

Stephan
---
misiu wrote:
> Hi all,
>
> I have to develop an 1 Gbps connection between our front-end board and
> PC. I found that the easiest and fastest way (especially form board
> development point of view) will be to use the raw Ethernet frames.
> I found that NDISPROT example from DDK could almost directly be used to
> send and receive raw frames. I've made some tests - just connect two PC
> with onboard gigabit NICs - and got problems.
>
> First is that the speed of the connection is about 350 Mbps and 150 Mbps
> for 9k frames and 1.5k frames respectively, which is event not close to
> 1 Gbps. Is that problem because of the NDISPROT driver itself or the
> NICs (NVIDIA nForce + Intel PRO 1000 MT) are not good enough?
>
> Second problem is when I sending frames from the first NIC and receive
> on the second no all frames are read out. Now I solve that problem just
> sending another small confirmation frame back, but it is no the way
> which I like. I have to mentioned that if on the receiver side I look to
> "Local Area Connection Status" the number of received frames is ok, so
> all have reached the receiver PC even without that confirmation frames.
>
> Regards,
> Bartek

misiu
Thu May 18 05:31:50 CDT 2006


> Not sure what you actually want to achieve. Is it just that you want to
> stress the GigE wire with maximum load?
>
> In this case you can use network sniffer tools such as Ethereal, which
> allow to both receive (and display) as well as also send frames to the
> wire.

I would like to transfer the data from front-end board to the PC with
the speed close to 1 Gbps. The board has FPGA with ready to use MAC via
which I will send the data using raw frames (it is much easier than
develop TCP or UDP protocol). That board will be directly connected to
the PC on which I have to receive all the packets, store and analyze
them. For that porpoise I need my own program not a network sniffer.
Before starting to use that front-end bard I would like to have some
experience with the PC so I decided to make some tests and on the sander
side I put the other PC instead of the real readout board.

>
> AFAIK, the NDISPROT approach is a quite simple one and implies some
> overhead. So there is a good chance you cannot get to 1 Gbps overall
> throughput.

Any idea what I can use instead of the NDISPROT or how to improve that
simple driver, at this level I could not develop my own NDIS driver.

> Also the PCI bus plays a mojor role here. Take care you do not install
> your cards behind some PCI bridge. Also both the card and machine
> should support 64-bit and/or 66 MHz. Note that both 64-bit and 66 MHz
> are optional and a NIC capable of 64-bit/66 MHz usually also runs in an
> "old" 32-bit/33 MHz PCI slot.

The cards are build-in onboard PCIe cards not PCI.

> IIRC, you can usually get tools like TTCP (see e.g.
> http://www.pcausa.com/Utilities/pcattcp.htm) to fully load a GigE wire
> when using the UDP option (TCP implies too much overhead).
>
Is it clear that fully load of 1 Gbps for TCP implies too much overhead?

Bartek

Stephan
Thu May 18 07:06:06 CDT 2006

misiu wrote:
[..]
> Any idea what I can use instead of the NDISPROT or how to improve that
> simple driver, at this level I could not develop my own NDIS driver.

NDISPROT copies the data from a received NDIS_PACKET to the buffer
supplied in the IOCTL (IRP). See ndisprotServiceReads().

So it is noticeable that all data, i.e. 1 Gbps = ~125 MByte/s, needs to
be copied by the CPU in ndisprotServiceReads().

Maybe you could move your packet checking code to NDISPROT instead of
checking the packet contents in your application.

> The cards are build-in onboard PCIe cards not PCI.

Ok, PCIe should not be the bottleneck.

> Is it clear that fully load of 1 Gbps for TCP implies too much overhead?

Well, it depends on what "too much" means. Modern NICs usually have
built-in support for TCP (Checksum Offload etc.), so the CPU overhead
is reduced to a minimum. Still TCP needs to acknowledge frames
("handshake") and if only one packet is lost, a retransmission is
initiated for one or more packets.

Since packet loss is only detected after some ten milliseconds, these
ten milliseconds cost you a lot of performance (since all traffic stops
for this period of time).

Stephan

misiu
Thu May 18 07:31:45 CDT 2006

Stephan Wolf [MVP] wrote:
>
> Maybe you could move your packet checking code to NDISPROT instead of
> checking the packet contents in your application.
>

But I need all of these packets to store on disks and maybe analyze them
online, so I think the application level is much easier (for me).

What about the problem with reading that not all packets which I've sent
I've not received?

Regards,
Bartek

Stephan
Thu May 18 09:27:19 CDT 2006

misiu wrote:
> What about the problem with reading that not all packets which I've sent
> I've not received?

Several possible reasons:

- Frame gets corrupted on the wire (very unlikely)

- Receiving NIC runs out of resources (i.e. receive buffers)

- In the latter case, the receiving miniport usually sets
NDIS_STATUS_RESOURCES, but NdisProtReceivePacket() could also run out
of receive buffers (i.e. ndisprotAllocateReceivePacket() fails)

- NdisProt discards received packets due to your app. does not pick
packets fast enough; see comment in ndisprotQueueReceivePacket():

"Queue up a received packet on the open context structure.
If the queue size goes beyond a water mark, discard a packet
at the head of the queue."

If you also need to write all packet data to disk - well, ok, but think
about how fast you can write data to disk, i.e. is your disk capable of
writing 125 MByte/s?

Hmm, just a quick thought: Maybe you should actually write data to disk
from NDISPROT directly. That should work for a couple of seconds at
least. Then start your app. and analyse the packets in the file written
by NDISPROT.

The advantage is that you don't need to copy packets, but write the
original data from the NDIS_PACKET (that is, the NDIS_BUFFER chain) to
disk. Aso you save the IOCTL overhead.

Stephan