TCP Sockets - Limiting one write at a time is harming performance(at least on windows)

[Alexander Libman]

Leon,
How many connections did you use in your tests?
I assume, having 1000+ connections, short writes will occur on regular
basis.

What is the overall performance on your tests bytes/sec and operations/sec?

> -----Original Message-----
> From: nio-dev-bounces at openjdk.java.net
> [mailto:nio-dev-bounces at openjdk.java.net]On Behalf Of Leon Finker
> Sent: Friday, June 05, 2009 10:53 AM
> To: Alan.Bateman at Sun.COM
> Cc: nio-dev at openjdk.java.net
> Subject: RE: TCP Sockets - Limiting one write at a time is
> harming performance(at least on windows)
>
>
> >You may be right that Windows doesn't do this with
> >asynchronous I/O, at least for when buffering is
> >disabled (which is what I think you are doing).
> >Is this specified anywhere?
>
> I did disable send buffering. I did not find this
> publicly documented by Microsoft.
> I only know that windows API and .NET does not enforce
> this. Also, it says the following in MSDN for WSASend:
> "If you are using I/O completion ports, be aware that
> the order of calls made to WSASend is also the order
> in which the buffers are populated. WSASend should
> not be called on the same socket simultaneously from
> different threads, because it can result in an
> unpredictable buffer order." They don't say that
> one has to wait for completion of WSASends before
> scheduling next ones (as long as it's not done
> concurrently). Although, short-writes will make this
> pointless if they are to happen regularly on stream
> connections. If they only happen in IOCP under low
> resources conditions, then it's probably advantageous
> to cleanup some of the connections. This is what's not
> documented, but only observed in practice.
>
> >What size are the messages and have you
> >disabled Nagle by any chance? Also, what is the
> >size of the send buffer that you are comparing with?
>
> I was testing with message sizes of 100, 2048, 4096 bytes.
> I tried disabling Nagle, and it only affected testing of small
> message sizes (100 bytes). Made throughput much slower.
> I tried send buffer size of 0, 8192, 512K.
>
>
> -----Original Message-----
> From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM]
> Sent: Thursday, June 04, 2009 8:25 PM
> To: Leon Finker
> Cc: nio-dev at openjdk.java.net
> Subject: Re: TCP Sockets - Limiting one write at a time is harming
> performance (at least on windows)
>
> Leon Finker wrote:
> > Yes, short-writes will cause a problem, but on windows with
> IOCP/overlapped
> > model short-writes are not observed in practice (unless resources are
> > depleted; in that case connection probably should be closed/cleaned up
> > anyway). Not sure about other platforms.
> >
> > Have you ever seen (with latest jdk build) that
> ByteBuffer#hasRemaining is
> > true in write's completed() on windows under stress (more
> likely to happen
> > on pre-Vista due to easier non-paged pool resource depletion)?
> Even though
> > Platform SDK IOCP sample does handle this case, I have not seen it being
> > handled in other production frameworks. Anyhow, something to ponder on.
> The
> > data throughput is definitely much better.
> >
> J1 is this week so a belated reply.
>
> Short-writes are "normal" when there is congestion and insufficient
> space in the socket's send buffer. Even Windows does this with blocking
> and non-blocking I/O. You may be right that Windows doesn't do this with
> asynchronous I/O, at least for when buffering is disabled (which is what
> I think you are doing). Is this specified anywhere? In any case, the
> 2x-3x results are kinda surprising. I recall the previous version of
> your tests completely saturated the network so I'm not sure how the
> throughput is increased. What size are the messages and have you
> disabled Nagle by any chance? Also, what is the size of the send buffer
> that you are comparing with?
>
> -Alan.
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.339 / Virus Database: 270.12.54/2157 - Release Date:
> 06/05/09 17:55:00
>