java.sql2 DynamicMultiOperation with interlaced exceptions

[Douglas Surber]

Lukas,

Thanks for your comments. My reaction however is that this is an example of trying to shoehorn the database access problem into a reactive stream solution. For example bind variables are absolutely not a stream of any kind. They are best considered a map. However rather that require construction of a Map the API allows passing each key/value pair separately. This is acceptable as the number of key/value pairs, the number of parameters, is part of the code (at least for the use cases we are targeting). Representing bind values as a stream is inappropriate. Sure lots of these concepts can be represented as publishing and subscribing, but that doesn’t make it the best idea.

The two places where back pressure are needed are controlling the rate at which the client submits Operations and controlling the rate at which the implementation produces rows. In a large majority of cases neither of these matter. Only in a very, very few cases will a client produce enough Operations fast enough that controlling the rate is of benefit. I would be perfectly happy if the API did not include any support for back pressure for Operation submission. A very few of the target use cases might have a query or two that produce enough rows that back pressure would be valuable. But even in those cases the overwhelming majority of queries would produce such a limited number of rows that there would be no benefit to back pressure. So only a small fraction of the queries in the target use cases would benefit from back pressure.

Multi-result SQL could be represented as a stream of results. This is a SQL use case that I have no experience with. I would guess that the number of results is generally small. While any one result might produce a lot of rows (the second case in the previous paragraph) the number of results is probably small and the benefit of back pressure on the results (not the rows) is minimal.

I’m no fan of CompletableFuture. In all honesty I dislike it quite a bit for reasons that would distract from this discussion. The Class Library Team strongly encouraged us to use CompletableFuture. I recall there was a reason not to use CompletionStage but I don’t remember what it was. I’ll review that decision. On the face of it I’d prefer CompletionStage for the reasons you give and others.

Java 10 is not 18.3. The old release cycle was nominally every 18 months and actually more like every two years. 18-24 months after Java 9 would be 19.3 or 19.9.

I would suggest that you not think about this as a Flow API for database access. That is not what it is. It is an async API for database access. It may use Flow where appropriate but there is no specific goal to use Flow.

Douglas


> On Oct 12, 2017, at 2:50 PM, Lukas Eder <lukas.eder at gmail.com> wrote:
> 
> Hi Douglas
> 
> 2017-10-09 18:54 GMT+02:00 Douglas Surber <douglas.surber at oracle.com <mailto:douglas.surber at oracle.com>>:
> It’s not clear to me why you would think Operation does too much. 
> 
> I feel that the type hierarchy exposes too much coupling between Operation subtypes that aren't strictly related. Unfortunately, I currently cannot explain myself better without having played with the API more in depth...
>  
> Operation only has three methods and one convenience method. An Operation encapsulates a database action and how the result of that action is processed. No Operation subclass does any more. Yes, full specification of the action can be complex, e.g. the various Parameterized subclasses, and in the case of StaticMultiOperation and DynamicMultiOperation the result processing is complex. But every method on Operation either adds to the specification of the action or to the processing of the result.
> 
> It would be possible to split Operation into Action and Result components. Something like
> 
>   conn.parameterizedAction(selectSql)
>     .set(arg, value, type)
>     .rowResult()
>     .initialValue( () -> new ArrayList() )
>     .rowAggregator( (p, r) -> { p.add(r.get(column, javaType); return p; } )
>     .submit();
> 
> I don’t see the value. Negotiating the path of fooAction/barResult doesn’t seem to add anything.
> 
> I'm more than willing to discuss the concept, although I don't completely understand your example yet, because again, I don't fully understand what initialValue() or rowAggregator() do. Perhaps that's already quite API-draft-specific
> 
> I do like the split between Action and Result, though. After all, this is what the Flow API does as well. From the client perspective, an Action corresponds to Publisher and a Result corresponds to a Subscriber, whereas from the driver / server perspective, things are inversed: The Action is hooked to a Subscriber whereas the Result is hooked to a Publisher
> 
> Ultimately, all the different types of Operations are just a combination of:
> 
> - Client "publishing" SQL statements, bind variables, result specifications (e.g. OUT parameter registrations)
> - Client "subscribing" to the outcome of the above
> - Server "subscribing" to client requests
> - Server "publishing" results from the database (including interlaced update counts / result sets / errors / warnings / server output, as discussed here [1])
> 
> In addition to the above, should the new API embrace vendor-specific messaging (e.g. Oracle AQ), we could take this one step further:
> 
> - Client "publishes" AQ messages by enqueuing them
> - Client "subscribes" to AQ queues, receiving messages
> - Server "subscribes" to client messages
> - Server "publishes" AQ messages to clients
> 
> With all this in mind, I think the Operation abstraction might just be the wrong abstraction if you want to include the Flow API, but you don't seem to want that as you've mentioned several times.
>  
> As I responded to Jens, PublisherOperation is a mistake and should not have been uploaded. This is not a Flow based API.
> 
> Yes, I've read that thread as well, and I'll be reading through the one where Konrad and Viktor got involved soon. While I understand the sentiment that it could be a mistake in the current API's context (either the API is completely reactive, or not at all), I'm inclined to think that the Flow API would have been a good thing.
> 
> If it is not, may I suggest to completely remove all references to Flow, including:
> 
> - Connection.onSubscribe()
> - Row -> Flow.Subscription dependency
> 
> Also, the DynamicMultiOperation.resultHandler() callback-based API to me seems more in the line of thought of a Flow based API...
>  
> It is a CompletableFuture based API. There is a need for back pressure in two places and we have tried, perhaps unfortunately, to shoehorn Flow into those places. For those that want a Flow based API this is not that API. We explored that problem space and did not come up with anything that met our goals. Not to say that it isn’t possible, but we did not succeed. 
> 
> Just to get this right, those places are:
> 
> - Bind variables
> - Results (through Submission), including things like Lobs
> 
> Correct?
> 
> What I'm having trouble understanding here is: The CompletableFuture API is primarly an API that runs Java logic on the ForkJoinPool by default, or on some other Executor on demand.
> 
> I have always been a bit surprised by the excitement about this API, because it seems to be of quite limited use, and very tightly coupled to the ForkJoinPool. For instance, if a custom Executor is passed to any method, the resulting CompletableFuture will not "remember" this Executor but yet again default to the ForkJoinPool. Yet, the "heavy work" doesn't happen in the client, it happens on the server. I'm afraid that a CompletableFuture based API will put blocking burden on the ForkJoinPool (which is mainly used for things like client side parallel stream processing), saturating it while waiting for server responses.
> 
> Perhaps, CompletionStage might be better suited, then, because it does not impose any such implementation?
>  
> If the community wants an async database access API based on Flow in the Java 10 equivalent release, someone else is going to have to develop a fairly complete initial draft. The EG needs to make progress on what we have to have any hope of inclusion in the Java 10 equivalent release.
> 
> If Java 10 = Java 18.3, due in 5-6 months, then I'm not too optimistic that the asynchronous JDBC API will stabilise by then. To me, JDBC seems like one of the most important Java APIs. Design time here is definitely time well spent, so I'm not in a hurry to get this Flow based API quickly :)
> 
> Lukas 
> 
> [1]: http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2017-October/000098.html <http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2017-October/000098.html>