[sctp-dev] SendFailedNotification and fragmentation

Wed Feb 10 07:11:36 PST 2010

Danny,

I understand your comments and suggestions here, and I agree in 
principle with what you are saying. When developing this API and 
implementation we were greatly constrained by the limitations of the 
native socket API extensions. Also, the fact that the native API is not 
finalized and the various implementations support different versions of 
the draft really caused headaches. Solaris is still on version 10 of the 
draft API. For these reasons we did not put any sophisticated error 
handling for send failed, or other exotic features.

Looking at the latest socket extensions draft [1], it looks like they 
added support (since around version 18) similar to what you are 
suggesting ( or at least part of it ). I think if we could expose this 
information through the Java API it should greatly help you efforts, 
right? see SCTP_DATA_NOT_FRAG, SCTP_DATA_LAST_FRAG, etc.

I don't believe that this draft version has been adopted by the main 
stream implementations yet. At least not lksctp or Solaris. Have you had 
a chance to look at this?

That being said, I think that returning the MessageInfo in the 
SendFailedNotification would be a useful addition. To confirm: you 
require such information like ppid, ordered, complete, ttl, right? Since 
the other members of MessageInfo are already exposed to the notification 
handler directly.

-Chris.

[1] http://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-21#section-5.3.5

On 09/02/2010 17:46, Danny wrote:
> Chris,
>
> As an intro I'd like to synthesize from other SCTP related posts who's
> content will be partiality relevant to the problem explained below (for
> those reading only this post) that:
> - SCTP_EXPLICIT_CONFIRM is support at the SCTP API level, waiting for
> lksctp to support SCTP_EXPLICIT_EOR, but a work-around can be used.
> - the other fragmentation related socket options (SCTP_DISABLE_FRAGMENTS
> and SCTP_FRAGMENT_INTERLEAVE) are already operational and functioning
> since earlier lksctp versions
> - it is confirmed that situations exist in which send() throws
> SocketExceptions when they occur between SHUTDOWN processing at lksctp
> level and before ShutdownNotification at application level
>
> Furthermore :
> - if they occur after the first part of a non-blocking send() returning
> !0 (not 0) and before the effective send would take place on the wire a
> SendFailureNotification would occur in stead of a SocketException throw
> - the SendFailedNotification object does not have the MessageInfo, as
> provided to the send() method, associated with a buffer. It provides
> address, association, stream, error and buffer which don't lead to the
> related MessageInfo.
>
> Statement :
>
> Given the above, and based on effective working code, i state that it is
> possible to implement an asynchronous (non-blocking) class that offers
> transparent fragmentation and that can for 100% guarantee that no
> message loss will occur between the moment where an application calls a
> send() method and the moment that the related data is effectively send
> (what wasn't send is returned to the application =100% recovery). There
> is however one situation of which i think, or fail to see, that any
> implementation that uses the fragmentation features of SCTP will today
> and in the future not be able to recover from or provide such a 100%
> guarantee, resulting in data loss, unless the interface is adapted some.
>
> Problem:
>
> The specific case is when SendFailedNotification must be handled. These
> notification will occur extremely rarely when there is data in the send
> buffer while suddenly the socket becomes unfit to write. In that case
> one, or more, SendFailedNotifications can be triggered for a same
> channel/stream combination depending on how many non-blocking send()'s
> have been absorbed into the underlying send buffer, because the buffer
> was still sufficiently large to take the message(s) in. This multiple
> pending send messages situation is equally the case if a Selector is
> used because the SelectionKey will keep getting selected as long as
> there is free space in the buffer, unless the application cancels SEND
> interest because it has nothing to send anymore.
>
> A robust implementation would need to recover inside a
> SendFailureNotification. This is easy and straight forward if the buffer
> contains a complete message (not fragmented) because one just has to
> bounce it back to the sender with whatever means have been implemented
> to do so, but that information is not available at
> SendFailureNotification time because it is part of the MessageInfo that
> is not provided. However, even worse, if the buffer is a fragment, and
> given the fact that in a good implementation the sender should be
> fragmentation unaware, the implementation should salvage to complete
> message, in order to conform to the SCTP contract on message boundary
> guarantee, and then bounce that complete message, and not only that
> fragment, to the sender. This means that the implementation must keep
> the complete message until the last part is send because relying only on
> consecutive SendFailureNotification doesn't guarantee that the complete
> message can be re-composed because one doesn't know on what fragment the
> first failure occurs and if the rest of the message is already in the
> send buffer or not. More, last messages must be kept on a per-stream
> basis because the buffer can contain messages for different streams (the
> buffers are socket level not stream level) from an application viewpoint
> at the moment that the connection or the channel unexpectedly closes.
> And as if all this isn't already enough, there could be more then one
> fragmented message for a same stream in the buffer which would imply
> that the implementation must keep more then the last message per stream.
>
> Suggestion:
>
> All this brings me to the point to say that, in my opinion and current
> state of knowledge, there is a need for some local means (send side)
> that makes it possible for code that receives SendFailureNotifications
> to access the MessageInfo that was associated with the buffer in the
> send() method and preferably with an extra field on the MessageInfo, in
> good java tradition an object, that the implementation can use to
> associate data. This object should/may not be send to the peer, it is a
> local thing in order to stay conform with the SCTP specs and compatible
> with other SCTP stacks.
>
> Unfortunately this is only halve the solution because an implementation
> that needs to keep messages until they are effectively send on the wire
> due to the fact that until that moment a SendFailureNotifcation may
> still occur for that message, needs some mechanism to be informed of the
> sending in order to do the housekeeping of these kept messages. As far
> as i know there is no mechanism that allows to know this at the SCTP API
> level (=one that includes the channel/stream number/MessageInfo).
> A possible solution could be an overload of the send method taking an
> extra argument to provide a notification handler (same type as the one
> used for the receive() statement) and add a notification type (eg:
> SendReadyNotification or something). If the caller provides the
> notification handle object to the send() method then the handler is
> called with the SendReadyNotification containing the MessageInfo
> (already second parameter of the send() method at this time). If the
> caller does not provide the the notification handler then everything
> stays as it is today.
>
> I would like feedback on this because i am under the impression that is
> is going to be a generic problem, and therefore a generic need, for the
> SCTP interface if it is going to be used with the SCTP fragmentation
> features as intended to avoid application level (in message or higher
> protocol) involvement. I think there may be consensus that the SCTP
> fragmentation was intended to perform fragmentations without the need to
> know what is in the data buffers (audio, video, text,etc) and without
> the need to understand how to recombine it.
>
> Thanks,
> Danny