TCP Sockets - Limiting one write at a time is harming performance(at least on windows)

[Leon Finker]

After your suggestion, I tried 1,000 connections (on loopback and remotely)
that are concurrently bombarding server with data and have not encountered
single short write. I also tried single connection and sent 300 megabytes
array at once. This also completed the send notification as single 300
megabytes array and not as short writes.  This is on Vista. I'm not prepared
to give any numbers because I do not have required hardware or the OS
versions to do proper testing.

-----Original Message-----
From: Alexander Libman [mailto:libman at terabit.com.au] 
Sent: Sunday, June 07, 2009 1:18 PM
To: Leon Finker; Alan.Bateman at Sun.COM
Cc: nio-dev at openjdk.java.net
Subject: RE: TCP Sockets - Limiting one write at a time is harming
performance(at least on windows)

Leon,
How many connections did you use in your tests?
I assume, having 1000+ connections, short writes will occur on regular
basis.

What is the overall performance on your tests bytes/sec and operations/sec?

> -----Original Message-----
> From: nio-dev-bounces at openjdk.java.net
> [mailto:nio-dev-bounces at openjdk.java.net]On Behalf Of Leon Finker
> Sent: Friday, June 05, 2009 10:53 AM
> To: Alan.Bateman at Sun.COM
> Cc: nio-dev at openjdk.java.net
> Subject: RE: TCP Sockets - Limiting one write at a time is
> harming performance(at least on windows)
>
>
> >You may be right that Windows doesn't do this with
> >asynchronous I/O, at least for when buffering is
> >disabled (which is what I think you are doing).
> >Is this specified anywhere?
>
> I did disable send buffering. I did not find this
> publicly documented by Microsoft.
> I only know that windows API and .NET does not enforce
> this. Also, it says the following in MSDN for WSASend:
> "If you are using I/O completion ports, be aware that
> the order of calls made to WSASend is also the order
> in which the buffers are populated. WSASend should
> not be called on the same socket simultaneously from
> different threads, because it can result in an
> unpredictable buffer order." They don't say that
> one has to wait for completion of WSASends before
> scheduling next ones (as long as it's not done
> concurrently). Although, short-writes will make this
> pointless if they are to happen regularly on stream
> connections. If they only happen in IOCP under low
> resources conditions, then it's probably advantageous
> to cleanup some of the connections. This is what's not
> documented, but only observed in practice.
>
> >What size are the messages and have you
> >disabled Nagle by any chance? Also, what is the
> >size of the send buffer that you are comparing with?
>
> I was testing with message sizes of 100, 2048, 4096 bytes.
> I tried disabling Nagle, and it only affected testing of small
> message sizes (100 bytes). Made throughput much slower.
> I tried send buffer size of 0, 8192, 512K.
>
>
> -----Original Message-----
> From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM]
> Sent: Thursday, June 04, 2009 8:25 PM
> To: Leon Finker
> Cc: nio-dev at openjdk.java.net
> Subject: Re: TCP Sockets - Limiting one write at a time is harming
> performance (at least on windows)
>
> Leon Finker wrote:
> > Yes, short-writes will cause a problem, but on windows with
> IOCP/overlapped
> > model short-writes are not observed in practice (unless resources are
> > depleted; in that case connection probably should be closed/cleaned up
> > anyway). Not sure about other platforms.
> >
> > Have you ever seen (with latest jdk build) that
> ByteBuffer#hasRemaining is
> > true in write's completed() on windows under stress (more
> likely to happen
> > on pre-Vista due to easier non-paged pool resource depletion)?
> Even though
> > Platform SDK IOCP sample does handle this case, I have not seen it being
> > handled in other production frameworks. Anyhow, something to ponder on.
> The
> > data throughput is definitely much better.
> >
> J1 is this week so a belated reply.
>
> Short-writes are "normal" when there is congestion and insufficient
> space in the socket's send buffer. Even Windows does this with blocking
> and non-blocking I/O. You may be right that Windows doesn't do this with
> asynchronous I/O, at least for when buffering is disabled (which is what
> I think you are doing). Is this specified anywhere? In any case, the
> 2x-3x results are kinda surprising. I recall the previous version of
> your tests completely saturated the network so I'm not sure how the
> throughput is increased. What size are the messages and have you
> disabled Nagle by any chance? Also, what is the size of the send buffer
> that you are comparing with?
>
> -Alan.
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.339 / Virus Database: 270.12.54/2157 - Release Date:
> 06/05/09 17:55:00
>