Differentiate synchronous and asynchronous error of an AIO operation

[Zhong Yu]

Alan and John, I do not want to turn a synchronous event into an
asynchronous one. I still want the event to be processed promptly by
the same thread - but by exception/return, not by calling the
completion handler.

I'm with you on the idea that it's bad to dispatch a synchronous event
to a new thread. However my reason is not performance, but semantics.
If we do that, there's no way to know whether an event was sync or
async. Maybe that doesn't matter to most developers, but for those who
do care, we are depriving them of this information.

Setting maxCompletionHandlersOnStack=1 will turn all sync events into
async, that is not the behavior I'm looking for. I want to keep sync
events sync.

Actually, since maxCompletionHandlersOnStack has to be a small number,
It won't be rare that the max is reached, and the next sync event
becomes async. For example, a channel has 64K bytes ready for
immediate read; the memory conscious application uses a 1K buffer to
drain it. The first 16 reads occur on the same thread, with nested
frames. (There will be 16N call stacks, and N is not small in typical
Java apps) The 17th read however begin on a new thread.

If instead the sync event is returned to the caller, there won't be
nested calls; the caller will have an iteration of arbitrary length,
on the same thread and with zero increase of call stack.

The API could look like this
    Integer read(buffer, handler) throws SyncError
or
    void read(buffer, syncHandler, asyncHandler)

If we keep the current API, it would be nice to be provided a utility
method to determine whether the event is sync or async

    void read(buffer, new CompletionHandler(){
        completed()
            if( eventIsSync() )
                S1
            else
                S2
        failed()
            if( eventIsSync() )
                S3
            else
                S4

Finally, in real world applications, does it really matter that an
event is sync or async? What would real code look like? I admit my
previous concurrency arguments are too abstract and may be much more
general than reality. I suspect that an AIO operation is usually the
last call

    lock
    ...
    operation( handler );
    // nothing in between
    unlock

If this pattern is consistently followed by a developer, the handlers
don't have to worry about sync/async, all event processing executions
are serialized, and the order is what it appears to be in source code.

We'll have to see some real code examples to judge. I'll port a
selector based HTTP server to AIO, and see how the experience is like.

Zhong Yu