SPI vs Third party dependency

[Douglas Surber]

> In other words, backpressure is an integral part of this model in order to allow the queues which mediate between threads to be bounded.
— https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.1/README.md <https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.1/README.md>

We have tried to use reactive streams (Flow) where appropriate. With the inspiration of the discussion in this list I think I have developed Flow based mechanisms for Operation creation and row processing. Those will be included in the next upload to the sandbox. The one other place where back pressure is required is interacting with LOBs. The current API is more primitive than Flow. I think it would be straightforward to implement a Flow API on top of the one proposed.

And this is my approach toward a reactive stream API. Back pressure is important in a few places in the API. In those places the API must support back pressure either through Flow or through some other mechanism. If someone wants a fully Flow based API the hooks are in place to support one. If one does not require a fully Flow based API then there is no extra weight in the spec (and the implementations) trying to cram something that does not require it into a Flow shaped box.

If you want a 100% Flow based database access library, by all means write one. If there is something about the spec that prevents you from doing so, please let us know. Ideally include a concrete suggestion. We will do our best to accommodate you.

Douglas


> On Oct 25, 2017, at 2:39 AM, Viktor Klang <v at lightbend.com> wrote:
> 
> Hi Douglas!
> 
> Answers inline.
> 
> On Tue, Oct 17, 2017 at 8:29 PM, Douglas Surber <douglas.surber at oracle.com <mailto:douglas.surber at oracle.com>> wrote:
> It is not the lack of implementations of Flow.X that is the problem. It is the lack of a support library that manipulates streams.
> 
> Fortunately, in this case, there's nothing which prevents the JDK to incorporate more support for that in the existing j.u.stream package.
>  
> 
> If you look at the example code in the sketch David Karnok referenced in the Example of a Flow-based JDBC API thread you will see that the code makes extensive use of a reactive stream library. Writing the same example using the same sketch API but without a reactive stream library is impractical. There is (currently) no reactive stream library in Java SE so it would be impractical to use the sketch API in Java SE. Doing so would require some non standard reactive stream library. This fails to meet one of the goals of this project which is integration with Java SE, not integration with Java SE plus some random other library.
> 
> While I sympathize with the argument, the point of Flow/RS is *interoperability*. There's already ongoing work to add Flow ot other parts of Java, and given that the more things support it, the network effect is exponential.
> 
> I assume that the goal of this effort is not to replace Hibernate—i.e. do I need to be able to map objects to the database, do lazy-loading etc[A]?
> 
> To me, there are a couple of orthogonal requirements:
> 
> 1. Create an SPI which makes it possible to implement RDBMs access in a standardized way which does not necessitate blocking threads—because that severely impacts scalability.
>     * This is where current JDBC has a ton of issues.
>     * This is where Flow/RS/CompletionStage can really help since it provides a standardized way to integrate with the desired properties.
> 
> 2. Create an RDBMS API for library developers to build higher-level constructs/integrations which does not block any user threads—because that severely impacts scalability, and introduces the risk of deadlocks/starvation.
>     * This where Flow/RS/CompletionStage can be used to integrate with the underlying standard SPI—adding any transformation/integration without breaking the underlying properties or compromising the ability to not block user threads in the above layer.
> 
> 3. Create an application developer SQL API to interact with an RDBMS in a scalable way—allowing for many concurrent users/requests.
>      * This is where JPA/Hibernate/every ORM ever built lives. Replacing this is going to be a huge effort.
>  
> 
> IMO: If 1 and 2 is achieved then support in 3 from current solutions there is just a matter of time[B].
> 
> 
> 
> A: Disclaimer: I might have misunderstood the following, but: The purpose of JDBC is to provide an integration layer—not to provide the best possible application developer API for dealing with RDBMS:es—I hope not given that it has practically not evolved since it was created—and practically no developer deals with raw JDBC. 
> 
> B: I could be wrong, but I'm willing to make a bet on it.
>  
> 
> Douglas
> 
> > On Oct 17, 2017, at 12:29 AM, Jens Schauder <jschauder at pivotal.io <mailto:jschauder at pivotal.io>> wrote:
> >
> > Sorry if I'm a little dense but there is something I don't really
> > understand.
> >
> > The existing JDBC, as well as the upcoming new version, is what I consider
> > an SPI: a set of interfaces plus rules how they should behave, which won't
> > get implemented in the JDK itself. Instead, other parties (in this case
> > probably database vendors) will provide implementations.
> >
> > In my eyes, the Flow API as well is such an SPI. It is intended to get
> > implemented by third parties.
> >
> > Yet you don't want to depend on it although it seems to fit the
> > requirements quite well, especially "Maximum throughput with low and stable
> > latency"
> >
> > So my question is: what is the fundamental difference between the Flow API
> > and the new async JDBC that makes it ok for one to depend on third-party
> > implementations and the other not?
> >
> > Also, if this difference is relevant beyond the scope of JDBC the question
> > arises how/if the Flow API is supposed to be used at all in the JDK?
> > Although that question might be better suited for a different mailing list?
> >
> > Just to make sure: I'm not trying to change anybody's mind. I'm just trying
> > to understand the underlying principles for those decisions.
> >
> > Regards,
> > Jens Schauder
> 
> 
> 
> 
> -- 
> Cheers,
> √
> 
> ——————
> Viktor Klang
> Deputy CTO
> Lightbend, Inc.
> 
>