[sctp-dev] Unexpected SocketException due to latency of SCTP ShutdownNotification

[Danny]

Chris,

1) I confirm : java.net.SocketException: Socket is shutdown for writing
2) I catch the SocketException separately but DO NOT check the message 
text. The catch code is as such that whatever socket level exception 
occurs it needs to do the same thing.
    The reason is that when such a trhow occurs you cannot rely on the 
socket being 100% operational. Furthermore my code is Selector driven 
and a lot of problems are avoided because the SelectionKey will not be 
selected for SEND when the conditions are not as such that send may 
occur which eliminates "would block" alike  throws.

I don't know if a specialized exception is at order. Personally i would 
go for a documentation addition, because there are a lot more pitfalls 
of this kind. Just think about the scenario of a non-blocking send where 
the shutdown for writing would occur in between the call to send() and 
the effective sending due to thread swapping. In that case the problem 
we discuss here would become visible via a SendFailedNotification in the 
notificationHandler (if any!). Furthermore, depending on the underlying 
send buffer size more then one such notification may occur for a same 
channel.

Danny


Christopher Hegarty - Sun Microsystems Ireland wrote:
> Danny,
>
> To confirm, you are receiving an exception like:
>
>   java.net.SocketException: Socket is shutdown for writing
>     at sun.nio.ch.SctpChannelImpl.send0(Native Method)
>     at 
> sun.nio.ch.SctpChannelImpl.sendFromNativeBuffer(SctpChannelImpl.java:1027) 
>
>     at sun.nio.ch.SctpChannelImpl.send(SctpChannelImpl.java:987)
>     at sun.nio.ch.SctpChannelImpl.send(SctpChannelImpl.java:967)
>     ....
>
> You are of course correct, due to the event driven nature of the SCTP 
> stack there is a window of opportunity that one thread processing 
> notifications will not receive and process a shutdown notification in 
> time to prevent another thread from trying to perform an operation, 
> like send.
>
> It may be a good idea to document this scenario in the send javadoc. 
> SocketException is a specialized IOException throw to indicate that 
> there is a problem creating/accessing a socket. I guess you are using 
> the specific Exception message to determine this type of failure?
>
> -Chris.
>
>
> On 06/02/2010 10:50, Danny wrote:
>> Hi,
>>
>> I the process of using the SCTP client in combination with a Selector,
>> to make it asynchronous, i received throws from the Socket layer by
>> calling the SCTP Client send() method.
>> This problem didn't occur during standard debugging but during stress
>> testing and was therefor some harder to research but i found the problem
>> and would suggest and addition to the documentation of the send() method
>> to solve it.
>>
>> The exception thrown was a generic "SocketException" which is due to a
>> latency problem between the activity of the underlying SCTP stack (in my
>> case lksctp on fedora) and the proliferation of notifications to the
>> application level. The SocketException should be added explicitly to the
>> send documentation because it is unexpected and one has the impression
>> that all Socket related throws are caught by the SCTP API and that the
>> application is abstracted from them.
>>
>> This is how it happens :
>>
>> In heavy stress SCTP client outbounds connections to an SCTP Server,
>> resulting in SCTP Client inbounds connections to be accepted by the
>> Server using a Selector, are set-up. The test is done with only the
>> local host address.
>> The problem is consistent and now that i know what it is i can reproduce
>> it with a single connection, no need for stress.
>>
>> Once the connection is setup data can be exchanged. The test randomly
>> decides whether it is the inbound or outbound client that will call SCTP
>> shutdown() to disconnect the connection. The channel however is NOT
>> closed() so that new connections can be made using the same channel. The
>> result is as expected. The one that calls shutdown() receives the
>> AssociationChangedNotification(SHUTDOWN) and the other peer received the
>> ShutdownNotification. The problem occurs always with the client that did
>> receive the ShutdownNotification if it calls the send() method just in
>> between the moment that lksctp lib already acted upon that SHUTDOWN
>> message by disabling the socket for send and only keeps it open for
>> receive (graceful shutdown), and before the ShutdownNotification reaches
>> the application level so that the application knows i may not send 
>> anymore.
>>
>> At first one would think that this is not possible because using a
>> Selector, the Selector would never trigger the SEND to allow the
>> application to send because it is in sync with the underlying provider.
>> And indeed, there is no problem there. However, a Selector based
>> implementation, when it has nothing to send, disables it's interest in
>> receiving SEND triggers from the Selector, in order to avoid continues
>> SEND triggers from the Selector while there is nothing to send, and one
>> only re-enables the interest when a next message becomes available and
>> is send in order to receive a SEND trigger when the message is
>> effectively send at which point it can check if it has more data to send
>> or should disable it's interest for SEND triggers again.
>> I think we all are kind of forced into that scenarion when using
>> Selectors in a cpu optimized way.
>>
>> As a consequence, once in a while, the send() method is not called due
>> to a SEND trigger from the Selector but by class code in order to
>> restart the sending cycle. Therefor it becomes possible that send() is
>> called in between
>> the acting upon a SCTP SHUTDOWN message by the lksctp lib and the
>> arrival of the ShutdownNotification at the application level where
>> actions should be taken to avoid the application from further sending 
>> data.
>>
>> Because from where i stand, and after having a look inside the lksctp
>> source code, there is no way to avoid this it would be a good idea to
>> solves this by documenting, or at least mention the possibility of a
>> SocketException throw in the
>> list of throws of the send() method. In my application i now simple
>> catch that exception specifically and give it the appropriate treatment.
>>
>>
>> Greetz,
>> Danny
>
>