Comment on fibers and async library interfaces

[Ron Pressler]

Message passing and other communication/synchronization mechanisms are orthogonal. I see no reason why the same
selective blocking as Erlang couldn’t be implemented (indeed, it was in Quasar).

In any event, there is nothing stopping you from using async APIs if you find the style preferable. As to libraries offering
async APIs or not, it’s up to them and their user’s requirements.

Ron


On May 24, 2019 at 8:26:57 PM, Nils Henrik Lorentzen (nils.lorentzen at gmail.com) wrote:

I think the issue is when passing messages though a state machine message passing framework (or probably any such completely async message passing), fibers will indeed "block" the processing for the duration of the call to the fiber since one cannot process other events in the mean time. It would swap out the whole call stack of the message passing framework basically, and put message processing on hold until result comes back. 

"Instead of trying to recast fibers as state machines, think of how you’d program if creating and blocking threads were essentially free."

For a telco server I would still model it as statemachines since they need to be able to handle events from multiple sources (call teardown from each end, no media, call transfer request, mute, unmute, call on hold, disconnect from web console, etc) all occuring simultaneously or in any order, thus it just does not follow a blocking pattern by nature, one can't block on all of them. State machines model these cases well since reactive and one can easier verify by code that cases are covered and give errors for cases that are not.

Lightweight threads are indeed a nice way to handle and increase throughput for the case when the fiber initiator is calling out to perform a number of tasks.
For telco that is opposite, the outside world pounds and ongoing call or conference with events in any order, and the event processor just has to handle those reactively, it can't decide order.

So for message passing case here I think the problem is not lack of asynchronousity in execution of lightweight threads, but that they no matter what are blocking by nature as the path of execution is swapped out for some time till a response comes back, meanwhile eg. in a phone call lots of other events may have occured but have not been duly processed from the incoming message queue. One could perhaps spawn another fiber to re-trigger message queue processing if one knows that state machine action might block but that seems very hackish.

Erlang has selective receive on the process mailbox which helps with all this, it was made for telco but I a not sure how to do that with Java fibers since one would have to wait for multiple fibers. Not saying it can't be done, but I certainly don't see it as a good fit.

State machines/message passing is no matter what a tried and tested pattern and I don't think one should unnecessarily narrow down the possibilities for how to design an application. Imagine an employer that has a number of people that are very well versed in async programming and state machines, they would have to retrain to another paradigm that might not even be the best fit to express the problem domain.

Regards,
Nils Henrik Lorentzen


On Fri, May 24, 2019 at 8:43 PM Ron Pressler <ron.pressler at oracle.com> wrote:
But the underlying (JDK core-library) IO used by the driver is fiber-friendly, i.e. you use it with a synchronous API, but it’s entirely non blocking.
You can schedule as many fibers as you want onto as few kernel threads as you want (even a single one).

I don’t see anything of what you said being hampered by fibers; if anything, they assist. 
And if you happen to find async APIs easier to understand, it’s easy to expose (fiber) blocking code as an async APIs than vice-versa.

I’ve found that people who know async well often find it harder to understand lightweight threads than those who don’t.
Instead of trying to recast fibers as state machines, think of how you’d program if creating and blocking threads were essentially free.

Ron


On May 24, 2019 at 7:29:18 PM, Nils Henrik Lorentzen (nils.lorentzen at gmail.com) wrote:


After thinking about how it had to be implemented under the hood, I figured too it had to be async/non-blocking in the implementation (otherwise it would have to spawn a thread to do I/O...).
Also documented on wiki

The implementation of the networking APIs in the java.net and java.nio.channels  packages have as been updated so that fibers doing blocking I/O operations park, rather than block in a system call, when a socket is not ready for I/O. When a socket is not ready for I/O it is registered with a background multiplexer thread. The fiber is then unpacked when the socket is ready for I/O. These same blocking I/O are also updated to support cancellation. If a fiber is cancelled while in a blocking I/O operation then it will abort with an IOException.

"In what cases will fibers not suffice?"

One probably said the same about threads back in the day too :)  The unknown unknowns thing.

Async message passing can be one. What I mean by pure async server (very poorly explained) is for example a telecom server where everything is solely based on messages passed asynchronously between state machines. There is no blocking I/O invoked at all, I wrote such a thing many years ago (in C++ though)

 - a single mainloop using select() or epoll()
 - every implemented telco protocol adapter registered similar to nonblock NIO selectors, but with method to detect complete-length inbound protocol message and to decode such message.
 - all communication in business logic (like call menus - "press 1 for xyz") on non-blocking message queues.
 - no blocking receive either (unlike Erlang).

Nature of async I/O allowed for the whole system could do call setup, teardown, business logic and media conferencing all to run on one single thread, with smooth media mixing. Shows the power of async processing.
One could still configure on the side how many threads, what protocol ran in which thread, etc. , separates processing completely from threading model.

Not a bad programming model, statemachines do well for reasoning about concurrent events from multiple sources, as long as one avoids "state explosion" problem (which fibers admittedly could help avoid, but purely reactive state machines are easy to reason about as all state transitions are expressed in the code as state tables, easy to see if one missed a state/signal combination).

Now at long last to the issue

In this case, I'd probably want to integrate any protocol as completely async.
If this was Java and I wanted to integrate say JDBC into this, I'd might want to have an async JDBC driver where I could get the selector, add it to the main loop of the server and then perform queries async from statemachines, all of it happening on the same thread. So setting the statemachine to QUERYING state and to QUERIED or similar after received response.

The issue of using fibers here and call a blocking API is that the statemachine would not be able to process other events while the query to DB was going on, that call chain would get swapped out til DB returned response, no matter what.

Now one could perhaps work around it by spawning threads instead, but I am not quite sure how it would work out. A model where all processing is done by message passing statemachines or protocol adapters is very simple and clean, once used to the idea.

There are probably other such cases that one has not thought about, they tend to turn up after a while...

The point here is that if main network entry points are implemented async (eg. JDBC drivers) then one can easily implement reactive/statemachine patterns on top of them while also easily add a synchronous API. The other way around would require spawning threads, which seems needless given that one could have the driver be async directly.

Thus as I see it now at least, one way (sync driver) has a risk of somewhat shutting out or make more difficult perhaps rare but quite doable implementation patterns, the other allows for both.
If driver is already async I/O, fiber has to do nothing particular but for sync, yield when it hits the object.wait() that waits for completion of async JDBC (if I have understood fibers correctly).
 
Regards
Nils Henrik Lorentzen


On Fri, May 24, 2019 at 6:56 PM Ron Pressler <ron.pressler at oracle.com> wrote:
There are two different issues here: async IO and async APIs (or programming style).

Fibers do async IO automatically given a synchronous API. I don’t think I understand your concern about deemphasizing async APIs.
Is a server that uses blocking APIs with fibers considered a “pure async server” or not? If not, why not? 
Under the covers, only async IO is used. This is the same as in Erlang and Go.

You also write that "The opposite requires threads to be spawned and that defeats the purpose of async for scalability/throughput”,
but the whole point of lightweight threads is that spawning (and blocking) them is cheap so that it does harm scalability/throughput.

In what cases will fibers not suffice?

Ron


On May 24, 2019 at 10:11:32 AM, Nils Henrik Lorentzen (nils.lorentzen at gmail.com) wrote:

Hi,


I am a longtime Java developer just becoming aware of project Loom and its
lightweight threads. It seems like an idea well worth implementing in the
core JVM/libraries for easier making scalable server applications.



From reading the proposal at
https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html, I do have a
few concerns though, if I have understood fibers correctly.


Not subscribed to the list so sending this as food for thought.


These are mainly related to that there probably will be corner cases where
one has to write async code, and fibers just will not suffice. These cases
might be not be known yet but will surface in the future.


What is of concern is the statement "In addition to making concurrent
applications simpler and/or more scalable, this will make life easier for
library authors, as there will no longer be a need to provide both
synchronous and asynchronous APIs for a different simplicity/performance
tradeoff."


I understand this is written with the best intentions, who wouldn't want to
make life easier for library writers, and I have no intention to criticise
the author on this.

What I am wary of here is that this might discourage providing async APIs,
even at the low level, which will then make it way more difficult to write
pure async servers if need be. Or one just prefers that way of programming
(better logs solve much of the no-proper-stacktrace issues, and better log
capabilities are also a plus in prodution deployed systems)


Consider JDBC as an example, one is now at long last working on providing
async JDBC drivers that can be useful for high throughput processing and
reactive/async apps.


When it comes to network communication, similarily to what the proposal
states that async/await can be easily implemented by continuations, so can
a synchronous network driver API easily be made on top of an asynchronous
driver. The opposite requires threads to be spawned and that defeats the
purpose of async for scalability/throughput.

Keep in mind that even for request/response protocols, the base
communication is always async by nature. There is no blocking operation on
an ethernet card :) Thus async operation on top of a sync driver means
async network => sync API => thread to simulate asynchronousity => async
application, which is a long chain for something that was asynchronous in
the first place.


I would argue that for essential drivers (especially proprietary ones like
JDBC), one should always implement an async API at the base using NIO and
then just have a generic sync wrapper on top.


Async driver at the core does imply either spawning a thread in the driver
for its own select() mainloop or an API for integrating NIO Selectors into
another mainloop (eg. of an application server) but should be manageable.


An example of this architecture is Erlang. From what I can tell, socket
communication is non-blocking and done via message passing between
processes. The trick (and elegance) of Erlang is that it has a "selective
receive for messages" and from what I can tell, 'receive' is pretty much
the only place in all of Erlang that it would suspend lightweight threads
(probably a setjmp()/longjmp() libc call at that place in its VM).


For an async network driver in Java would be the blocking API doing
Object.wait()/notify() for threads. For suspend in fibers, the underlying
sync/async wrapper implementation could continue the fiber when there is
input (or on writeability for writes).



Just raising a flag here a bit because even if it is not such now, it could
become a classic case of group think where async becomes discouraged, and
then at some point one figures one needs it anyways. Except that all APIs
have adopted synchronous functioning and it would be even more difficult to
convince someone provide async network drivers as they would argue that
fibers should solve it, so no need for it.

Lightweight threads have a bright future but hopefully not at the expense
of tried and proven patterns for high throughput servers :)


Kind regards,

Nils Henrik Lorentzen