JDBC Next questions regarding j.u.c.Flow integration

[Douglas Surber]

There have been a lot of comments wrt reactive streams and Flow. I have a few general comments.

1) the PublisherOperation in the public version of the API should not have been included. It was created as part of an ultimately failing approach to using Flow. I thought I had created it in a separate branch but apparently not. It has been removed and will not be present in the next upload.

2) The goal of the API is to be asynchronous. It is not a goal to use reactive streams. That is not to say it won’t use them where appropriate but it is not a goal.

3) The Java Class Library Team strongly encouraged us to use CompletableFuture. While they did not discourage use of Flow they did not emphasize it. One of the design goals of the API is to integrate well with Java SE and CompletableFuture seemed to do that better than Flow.

4) We have made several attempts to use Flow more extensively but none have worked out. While it is easy enough to give a hand waving description of a way to use Flow, the devil is in the details. We would be happy to better integrate with Flow but at this point we are out of ideas. If the community thinks Flow is important the best way to help that happen is by producing a detailed proposal in the form of a change set to the existing code. The proposal would have to follow the goals and design principles in the 2016 and 2017 JavaOne presentations and integrate well with the existing API and the rest of Java SE and EE.

5) That said, one of the design principles for the async API is that one way to do something is enough. There are a thousand ways to solve almost any programming problem. An API can always be extended to provide tools to solve a problem in yet another way. For v1 at least the EG intends to keep the async API as small as possible. Adding a second way to do something that is already possible will require a very convincing use case.

6) The current version of the API needs back pressure in two places, to limit the rate at which Operations are created/submitted and to limit the rate at which rows are processed. It (mis)uses Flow to meet those needs. This certainly could stand improvement.

Douglas



> On Oct 6, 2017, at 7:28 AM, Dávid Karnok <akarnokd at gmail.com> wrote:
> 
> Hi. I have a couple of questions/suggestions about how the proposal uses
> j.u.c.Flow.
> 
> As I understand, the proposed underlying non-blocking IO uses
> CompletableFutures, which are one shot async sources unlike Flow. Is this
> due to async NIO being based around CompletableFutures?
> 
> I haven't delved too deeply into NIO but I'd think a Flow-based NIO could
> be possible (and perhaps lower overhead) - I understand such thing is out
> of scope here but from an API perspective, would it be possible to have as
> less ties to CompletableFutures as possible? For example, have most API
> Flow-based and the underlying implementation can use CompletableFutures as
> sources of notifications?
> 
> The OperationGroup.operationPublisher(Flow.Publisher<Operation>
> publisher) Javadoc has this statement:
> 
> * Any {@link Operation} passed to {@link Flow.Subscriber#onNext} must
> be created by
> 
> * this {@link OperationGroup}. If it is not {@link
> Flow.Subscriber#onNext} throws
> * {@link IllegalArgumentException}
> 
> 
> The Reactive-Streams spec forbids throwing from onNext. The only option, in
> general, for "invalid" onNext is to cancel the Subscription. I don't know
> how such error could be propagated back to the user other than adding
> failure state/signal to the returned OperationGroup.
> 
> I think Row.isLast shouldn't return Future because one can only block or
> spin on an non-completed future. On the conceptional level, why would the
> row's consumer want to know this information asynchronously? In the Flow
> API, onComplete will be eventually called which indicates there are no more
> data coming. On the practical level, isLast poses a concurrency problem
> because if the consumer may call isLast for multiple items, it will have
> multiple outstanding CompletableFutures one of which may signal true while
> the processing of the last row is in progress. Now the consumer's
> implementor has to work out the concurrency implications of that.
> 
> -- 
> Best regards,
> David Karnok