Example of a Flow-based JDBC API

[Viktor Klang]

Answers in line

On Wed, Oct 25, 2017 at 4:06 PM, Douglas Surber <douglas.surber at oracle.com>
wrote:

> Answers in line.
>
> Douglas
>
> On Oct 25, 2017, at 2:07 AM, Viktor Klang <v at lightbend.com> wrote:
>
> Hi Douglas,
>
> thanks for your patience, the past 2 weeks have been crazy.
>
> On Mon, Oct 16, 2017 at 6:56 PM, Douglas Surber <douglas.surber at oracle.com
> > wrote:
>
>> The target app is a large scale app running many instances in parallel
>> against a large database system, probably geographically distributed.
>> Maximum throughput with low and stable latency.
>>
>> The goals for the implementation are
>>
>>  - vendor neutral
>>  - avoid features that reduce scalability
>>  - every feature must balance user benefit vs performance cost
>>  - every feature must balance user benefit vs code to implement
>>
>> The goals for the user API are
>>
>>  - asynchronous. No user thread blocks
>>
>
> This is fairly simple with current JDBC though, just put all blocking ops
> in another threadpool.
> I'd recommend that the goal be set higher.
>
>
> It is out of scope for a Java standard to specify details of how that
> standard is implemented. The EG hopes and the user community would
> certainly expect any implementation of the spec to be as deeply
> asynchronous as possible. Oracle has developed an internal prototype for
> the Oracle Database  that does not block internally.
>

Well, it is definitely important that swapping out an implementation does
not change the semantics of my program, is it not?


>
>
>
>>  - plays well with Java SE
>>
>
> What does this mean in practice (i.e. how can this be evaluated?).
>
>
>  - Use what Java SE provides
>  - Provide what Java SE uses.
>
>
>
>>  - minimal. Every feature must be important to the target app. 80/20 rule
>>
> What is "the target app" in this?
>
>
> See the first line of previous reply above.
>

Ah, missed that one, thanks. Is it available somewhere? (i.e. if it doesn't
already exist, it would probably be a good idea to have that as an actual,
runnable, test case.)


>
>
>
>>  - consistent. No special cases
>>
>
> So all operations on primitives require boxing?
>
>
> In the current version. The only places auto-boxing appears are Result
> gets and Parameter sets. Hotspot can do a lot to optimize these. And at
> least from an implementers POV this decision reduces the amount of code in
> these areas by a factor of seven. Code that doesn’t exist doesn’t have bugs.
>

I assume the plan is to have JMH(-like) benches to verify this?


>
>
>
>>  - transparent. Principle of least surprise
>>
>
> Principle of least surprise is good.
>
>
>>  - ready by Java 10/19.x
>>
>
> That's a great goal. How many implementations of "drivers" are required to
> make this pass the bar?
>
>
> No implementations are required, only the API being included in Java 19.x.
>

Isn't there a real risk that it will be hard to vet the implementability
without at the very least one implementation?


>
>
>
>>  - independent of java.sql and javax.sql
>>
>
> Completely independent?
>
>
> Yes, completely independent. I have removed the few dependencies that were
> present in the sandbox version.
>

Ok, cool.


>
>
>
>>
>> We haven’t discussed TCK or testing.
>>
>
> Then I'd suggest doing that early—a specification tends to evolve
> alongside its canonical interpretation (TCK).
>
>
> Agreed.
>

>
>
>>
>> We made the following design choices. They aren’t requirements. But
>> having made those choices consistency requires that we be rigorous in
>> abiding by them.
>>
>>  - Rigorous use of types
>>  - Builder pattern
>>  - Fluent API
>>  - Immutable after initialization
>>  - One way to do something is enough
>>  - Avoid SQL processing
>>  - Avoid callback hell
>>
>
> Sounds like a great set of design choices!
>
>
>>
>> As we started work we looked at existing async database access APIs both
>> in Java and other languages. We looked at reactive streams but did not
>> follow that path because there was no support in Java SE. We looked at
>> java.util.stream but that blocks and cannot/will not be made non-blocking.
>> The Core Library Team encouraged us to build around CompletableFuture.
>> CompletableFuture seems too low level so this API provides a higher level
>> abstraction, well defined execution of Operations, to support the most
>> common use cases while still making use of CompletableFuture. (As noted
>> elsewhere replacing CompletableFuture with CompletionStage would be
>> reasonable.)
>>
>
> I think there needs to be some separation between what is intended to be
> used by "application developers" and what is for "library developers" vs
> what is for "implementors". Needs and capabilities are different amongst
> the groups.
>
>
> The goal of JDBC was to expose the underlying database. Unfortunately,
> JDBC adds a bunch of abstraction on top of the database that greatly
> complicates any implementation. These complications make drivers, bigger,
> slower, and less reliable. With ADBA we have tried to eliminate everything
> that is not strictly necessary to expose the underlying database with a
> fully async API. Application developers and library developers are of equal
> importance but with the caveat that the spec not require a “smart”
> implementation. A thin implementation will be faster and more transparent
> than an implementation that requires a neural network to implement.
>
> In all honestly, I would say that the needs of the implementer are given
> priority. If the resulting API does not provide a feature that app
> developers need that is an opportunity for library developers. If the API
> does not expose a database capability that the target app needs, that is a
> limitation in the API. A small, transparent API makes the library
> developers’ job easier. Fast implementations, which are enabled by a small,
> transparent API, make everyone happier.
>
> [Note: the above strongly suggests that POJO support is superfluous and
> should be removed.]
>

Great conversation, thank you!


>
>
>
>>
>> Douglas
>>
>>
>> > On Oct 14, 2017, at 6:07 AM, Viktor Klang <v at lightbend.com> wrote:
>> >
>> > What would help me, is to learn what the requirements for success is
>> for the following aspects:
>> >
>> > * Implementor SPI—ergonomics, performance, scalability
>> > * User API—ergonomics, performance, scalability
>> > * Verification of implementations—TCK for implementations
>> > * Verification of usage—how to write tests for JDBC 2
>> >
>>
>>
>
>
> --
> Cheers,
> √
>
> ——————
> *Viktor Klang*
> Deputy CTO
> Lightbend, Inc.
>
>
>


-- 
Cheers,
√

——————
*Viktor Klang*
Deputy CTO
Lightbend, Inc.