JDBC Next questions regarding j.u.c.Flow integration

[Rossen Stoyanchev]

hi Douglas,

First of all thank you for engaging with the community.

> ​The current version of the API needs back pressure in two places, to
limit the rate at which Operations are created/submitted and to limit the
rate at which rows are processed.

What's lacking from the above statement is the application perspective. As
a consumer of row data I would expect the ability to apply back pressure
against the source and get only as many rows as requested. In a scenario
where a web application consumes rows of data with back pressure that
propagates all the way to the HTTP connection, if the HTTP connection is
too slow, back pressure can be applied against the producer of the data to
slow down. This is not an imaginary scenario. You can run such applications
today with Reactive Streams as the glue from HTTP servers through to
reactive NoSQL data drivers.

> The Java Class Library Team strongly encouraged us to use
CompletableFuture. While they did not discourage use of Flow they did not
emphasize it. One of the design goals of the API is to integrate well with
Java SE and CompletableFuture seemed to do that better than Flow.

Indeed there is a conundrum here. Reactive Streams provides a crucial
mechanism (summarized on Flow.java and also reactive-streams.org) but as an
API it is very low level and ideal for direct use in applications. It would
result in the callback hell that your presentation mentions. At the same
time CompletableFuture has the nice continuations style API that
applications would want to use but does not provide back pressure and it
only represents a single result while here we're talking about modelling
rows of data.

Doug Lea summarized the various async API options in Java in the initial
Flow announcement [1]:

CompletableFuture/CompletionStage best supports
continuation-style programming on futures, and java.util.stream best
supports (multi-stage, possibly-parallel) "pull" style operations on
the elements of collections. Until now, one missing category was
"push" style operations on items as they become available from an
active source.""

JDBC fits that last category and here is him again later on defending the
decision to include Reactive Streams contracts in the JDK [2]. His argument
was that those interfaces are needed in the JDK itself for socket and other
I/O APIs that will be designed in the future. I wonder what Doug Lea would
advise us here? It would be great for him to weigh in.

Those of us in the community who have been down this road of designing
reactive APIs understand that there is a solution to the conundrum even if
it wasn't obvious to us either in the beginning. The preferred approach
would be to expose a Reactive Streams API in order to support pub-sub with
back pressure. Then layer a declarative API on top of use in applications.
It's trivial to to CompletableFuture for something out of the box. Keep in
mind also that a Flow-based API would be immediately usable from RxJava,
Reactor, and other Reactive Streams libraries. In other words there will be
great value for a very large audience and the choice of Flow will not lead
to callback hell. The other way around, an API is designed around
CompletableFuture, would mean that reactive pressure has to be built on
top. The other difference is that CompletableFuture does not allow for
deferred initiation of an operation like Reactive Streams does and also the
Stream API. So while CompleteableFuture is the best choice for an
application-level API based on what's available in the Java class library,
it is not the best choice to work with and the JDK does have Flow now to
serve as an alternative to be considered at least.

For an important decision like that, personally I would prefer a more
explicit answer from the Java Class Library Team rather than the passive
"they did not emphasize Flow so much". What is their take on why Flow was
added to the JDK if not for these kinds of pub-sub scenarios with latency
such as Async JDBC and the HTTP/2 client?

Regards,
Rossen

[1]
http://cs.oswego.edu/pipermail/concurrency-interest/2015-January/013641.html
[2]
http://cs.oswego.edu/pipermail/concurrency-interest/2015-January/013690.html