TCP Sockets - Limiting one write at a time is harming performance (at least on windows)

[Leon Finker]

>You may be right that Windows doesn't do this with 
>asynchronous I/O, at least for when buffering is 
>disabled (which is what I think you are doing). 
>Is this specified anywhere?

I did disable send buffering. I did not find this 
publicly documented by Microsoft.
I only know that windows API and .NET does not enforce 
this. Also, it says the following in MSDN for WSASend: 
"If you are using I/O completion ports, be aware that 
the order of calls made to WSASend is also the order 
in which the buffers are populated. WSASend should 
not be called on the same socket simultaneously from 
different threads, because it can result in an 
unpredictable buffer order." They don't say that 
one has to wait for completion of WSASends before
scheduling next ones (as long as it's not done 
concurrently). Although, short-writes will make this 
pointless if they are to happen regularly on stream 
connections. If they only happen in IOCP under low 
resources conditions, then it's probably advantageous 
to cleanup some of the connections. This is what's not 
documented, but only observed in practice.

>What size are the messages and have you 
>disabled Nagle by any chance? Also, what is the 
>size of the send buffer that you are comparing with?

I was testing with message sizes of 100, 2048, 4096 bytes.
I tried disabling Nagle, and it only affected testing of small 
message sizes (100 bytes). Made throughput much slower. 
I tried send buffer size of 0, 8192, 512K. 


-----Original Message-----
From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] 
Sent: Thursday, June 04, 2009 8:25 PM
To: Leon Finker
Cc: nio-dev at openjdk.java.net
Subject: Re: TCP Sockets - Limiting one write at a time is harming
performance (at least on windows)

Leon Finker wrote:
> Yes, short-writes will cause a problem, but on windows with
IOCP/overlapped
> model short-writes are not observed in practice (unless resources are
> depleted; in that case connection probably should be closed/cleaned up
> anyway). Not sure about other platforms. 
>
> Have you ever seen (with latest jdk build) that ByteBuffer#hasRemaining is
> true in write's completed() on windows under stress (more likely to happen
> on pre-Vista due to easier non-paged pool resource depletion)? Even though
> Platform SDK IOCP sample does handle this case, I have not seen it being
> handled in other production frameworks. Anyhow, something to ponder on.
The
> data throughput is definitely much better.
>   
J1 is this week so a belated reply.

Short-writes are "normal" when there is congestion and insufficient 
space in the socket's send buffer. Even Windows does this with blocking 
and non-blocking I/O. You may be right that Windows doesn't do this with 
asynchronous I/O, at least for when buffering is disabled (which is what 
I think you are doing). Is this specified anywhere? In any case, the 
2x-3x results are kinda surprising. I recall the previous version of 
your tests completely saturated the network so I'm not sure how the 
throughput is increased. What size are the messages and have you 
disabled Nagle by any chance? Also, what is the size of the send buffer 
that you are comparing with?

-Alan.