From douglas.surber at oracle.com  Mon Nov  6 22:06:14 2017
From: douglas.surber at oracle.com (Douglas Surber)
Date: Mon, 6 Nov 2017 14:06:14 -0800
Subject: Updated ADBA (Async DataBase Access) source
Message-ID: <B529C66D-EB98-414B-BB2C-F5F6ED76D368@oracle.com>

Lance uploaded the latest update of ADBA to the JDK sandbox.

  http://hg.openjdk.java.net/jdk/sandbox/file/b49fd5dbaab6/src/java.sql/share/classes/java/sql2

Changes:

 - moved into module java.sql.async (not reflected in the sandbox yet)
 - Removed references to java.sql
 - Implementation of SqlException and DataSourceFactory
 - New Transaction model
 - Flow.Processor for Operation submission and Row processing
 - Replace Future and CompletableFuture with CompletionStage
 - Replace initialValue and rowAggregator with collect(Collector)

Outstanding issues:

 - Implementation of DataSourceFactory needs work
 - Remove POJO support (open issue)
 - Remove @SqlArray and @SqlStruct (open-ish issue)
 - Improve JavaDoc on RowProcessorOperation
 - More JavaDoc
 - More text in package-info.java

Douglas

From davecramer at gmail.com  Sat Nov 11 15:10:29 2017
From: davecramer at gmail.com (Dave Cramer)
Date: Sat, 11 Nov 2017 10:10:29 -0500
Subject: ADBA license
Message-ID: <CADK3HHJc4KQ0KZaO6W3E_31hURuYe3ps_d9CzxiAvW0utfjrRQ@mail.gmail.com>

What is the license of the API? I'd like to pull this into PostgreSQL JDBC
to start working on it, but if it is GPL that will be a challenge.

Dave Cramer

From lance.andersen at oracle.com  Sun Nov 12 23:24:51 2017
From: lance.andersen at oracle.com (Lance Andersen)
Date: Sun, 12 Nov 2017 18:24:51 -0500
Subject: ADBA license
In-Reply-To: <CADK3HHJc4KQ0KZaO6W3E_31hURuYe3ps_d9CzxiAvW0utfjrRQ@mail.gmail.com>
References: <CADK3HHJc4KQ0KZaO6W3E_31hURuYe3ps_d9CzxiAvW0utfjrRQ@mail.gmail.com>
Message-ID: <66142D20-79E1-4C75-9484-9608A30C9E90@oracle.com>

Hi Dave,

It is the standard OpenJDK license.  Also, remember as the specification is not final and is subject to change. Use is subject to http://download.oracle.com/otndocs/jcp/java_se-18_3-edr-spec/license.html

Best
Lance
> On Nov 11, 2017, at 10:10 AM, Dave Cramer <davecramer at gmail.com> wrote:
> 
> What is the license of the API? I'd like to pull this into PostgreSQL JDBC
> to start working on it, but if it is GPL that will be a challenge.
> 
> Dave Cramer

 <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif> <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com <mailto:Lance.Andersen at oracle.com>




From brett.wooldridge at gmail.com  Mon Nov 13 01:20:03 2017
From: brett.wooldridge at gmail.com (Brett Wooldridge)
Date: Mon, 13 Nov 2017 10:20:03 +0900
Subject: Clarification of Connection.setNetworkTimeout() behavior
Message-ID: <CAPg6oRN3qUW2BBdhA4=rHgOiYFmDHxdWMpD-MY53SOyq7ConCQ@mail.gmail.com>

I am looking for clarification on the expected behavior of
the Connection.setNetworkTimeout() method.

I am the author of the popular connection pool, HikariCP.  One of the goals
of HikariCP is to protect against network partition events, at least with
respect to the pool's own interactions with the database.  To this end,
HikariCP makes use of *setNetworkTimeout()* quite extensively.

The fundamental pattern employed is basically this:

  int originalTimeout = conn.getNetworkTimeout();
  try {
     conn.setNetworkTimeout(executor, timeout);
     // some further interaction with Connection
  }
  finally {
     conn.setNetworkTimeout(executor, originalTimeout);
  }

We have encountered a bug with a particular JDBC driver, and in discussions
regarding a fix it has become clear that there is disagreement and/or
different understanding on two points:

1) Are the effects of setNetworkTimeout() intended to be immediate?
2) What is the role/purpose of the Executor parameter passed to
setNetworkTimeout()?


The bug in question boils down to this case:

  conn.setNetworkTimeout(executor, timeout);
  conn.close()

Here, the pool is attempting to ensure that, in the case of unacknowledged
TCP, the Connection.close() call does not hang indefinitely.

The "bug", as it were, in this particular driver, is encountered when
passing a ThreadPoolExecutor as the Executor, and this particular driver
doing something like this:

executor.execute(new Runnable() {
  public void run() {
    try {
      sock.setSocketTimeout(milliseconds); // for re-connects
    } catch (SQLException e) {
       throw new RuntimeException(e);
    }
  });
}

In our case, calling *setNetworkTimeout()* is followed by an immediately
call to *close()*.  The *close()* implementation of this driver sets sock
to null, so when the Executor gets around to executing the above Runnable
an NPE is thrown in the executor thread.

Further, because the actual setting of the socket timeout is queued into
the executor as a Runnable, the driver cannot guarantee that the timeout is
actually set upon return, and therefore a subsequent call to *close()* (or
any *other* method) is not guaranteed protection against a network
partition event.  Which rather seems to defeat the started purpose of the
*setNetworkTimeout()*:

This method is intended to address a rare but serious condition where
> network partitions can cause threads issuing JDBC calls to hang
> uninterruptedly in socket reads, until the OS TCP-TIMEOUT (typically 10
> minutes).


My interpretation is that the effects of *setNetworkTimeout()* *must* apply
to subsequent calls against the driver upon its completion, and not at
some *indeterminate
time* in the future (at the whim of the OS scheduler, Executor queue depth,
etc).

A solution proposed by one of the JDBC driver developers was to accept
*null* as a parameter, in which case they could make the action immediate,
and if an *Executor* is supplied the code would continue to behave as above.

My issues with this solution are, 1) that the JavaDoc plainly states that a
SQLException should be thrown:

if a database access error occurs, this method is called on a closed
> connection, *the **executor is null*, or the value specified for seconds is
> less than 0.


and, 2) for most users passing in a Executor, the behavior of the method
would be quite non-deterministic.

Having said that, every other driver source I have been able to find simply
*ignores* the supplied Executor completely.  Therefore, that was my
suggestion for the driver in question -- "just ignore the Executor, as
other drivers do, as you have direct access to the socket and the effect
can be immediate."

Again pointing to lack of clarity, the driver developer's objection is
based on the use of the word "will" in the JavaDoc for the executor
parameter:
executor - The Executor implementation which *will* be used by
setNetworkTimeout.
I have searched high and low for some archive of the discussions around the
original proposal for this method, and its parameters, in order to find
some clarity, but have come up short-handed.


Please, keepers of the specification, provide guidance as to these two
questions:

1) Are the effects of *setNetworkTimeout()* intended to be immediate?
2) What is the role/purpose of the *Executor* parameter passed to
*setNetworkTimeout()*?


Brett Wooldridge

From douglas.surber at oracle.com  Mon Nov 13 16:57:09 2017
From: douglas.surber at oracle.com (Douglas Surber)
Date: Mon, 13 Nov 2017 08:57:09 -0800
Subject: Clarification of Connection.setNetworkTimeout() behavior
In-Reply-To: <CAPg6oRN3qUW2BBdhA4=rHgOiYFmDHxdWMpD-MY53SOyq7ConCQ@mail.gmail.com>
References: <CAPg6oRN3qUW2BBdhA4=rHgOiYFmDHxdWMpD-MY53SOyq7ConCQ@mail.gmail.com>
Message-ID: <3AEEB111-8A12-4336-A6DE-3CDDD64376D6@oracle.com>

> 1) Are the effects of setNetworkTimeout() intended to be immediate?

The network timeout should apply to all subsequent database interactions. It does not apply to pending interactions. It is vendor dependent whether the timeout applies to the entire JDBC API method call or to each network round trip (if there are more than one) to implement the method call.

> 2) What is the role/purpose of the Executor parameter passed to
> setNetworkTimeout()?

When the timeout expires the thread that is hung will be kicked loose with an exception. This thread may not clean up the connection so it is closed properly. The Executor provides a thread that the implementation can use to do this clean up. If the hung thread does the full clean up as it throws then the Executor is not needed. 

Douglas


> On Nov 12, 2017, at 5:20 PM, Brett Wooldridge <brett.wooldridge at gmail.com> wrote:
> 
> I am looking for clarification on the expected behavior of
> the Connection.setNetworkTimeout() method.
> 
> I am the author of the popular connection pool, HikariCP.  One of the goals
> of HikariCP is to protect against network partition events, at least with
> respect to the pool's own interactions with the database.  To this end,
> HikariCP makes use of *setNetworkTimeout()* quite extensively.
> 
> The fundamental pattern employed is basically this:
> 
>  int originalTimeout = conn.getNetworkTimeout();
>  try {
>     conn.setNetworkTimeout(executor, timeout);
>     // some further interaction with Connection
>  }
>  finally {
>     conn.setNetworkTimeout(executor, originalTimeout);
>  }
> 
> We have encountered a bug with a particular JDBC driver, and in discussions
> regarding a fix it has become clear that there is disagreement and/or
> different understanding on two points:
> 
> 1) Are the effects of setNetworkTimeout() intended to be immediate?
> 2) What is the role/purpose of the Executor parameter passed to
> setNetworkTimeout()?
> 
> 
> The bug in question boils down to this case:
> 
>  conn.setNetworkTimeout(executor, timeout);
>  conn.close()
> 
> Here, the pool is attempting to ensure that, in the case of unacknowledged
> TCP, the Connection.close() call does not hang indefinitely.
> 
> The "bug", as it were, in this particular driver, is encountered when
> passing a ThreadPoolExecutor as the Executor, and this particular driver
> doing something like this:
> 
> executor.execute(new Runnable() {
>  public void run() {
>    try {
>      sock.setSocketTimeout(milliseconds); // for re-connects
>    } catch (SQLException e) {
>       throw new RuntimeException(e);
>    }
>  });
> }
> 
> In our case, calling *setNetworkTimeout()* is followed by an immediately
> call to *close()*.  The *close()* implementation of this driver sets sock
> to null, so when the Executor gets around to executing the above Runnable
> an NPE is thrown in the executor thread.
> 
> Further, because the actual setting of the socket timeout is queued into
> the executor as a Runnable, the driver cannot guarantee that the timeout is
> actually set upon return, and therefore a subsequent call to *close()* (or
> any *other* method) is not guaranteed protection against a network
> partition event.  Which rather seems to defeat the started purpose of the
> *setNetworkTimeout()*:
> 
> This method is intended to address a rare but serious condition where
>> network partitions can cause threads issuing JDBC calls to hang
>> uninterruptedly in socket reads, until the OS TCP-TIMEOUT (typically 10
>> minutes).
> 
> 
> My interpretation is that the effects of *setNetworkTimeout()* *must* apply
> to subsequent calls against the driver upon its completion, and not at
> some *indeterminate
> time* in the future (at the whim of the OS scheduler, Executor queue depth,
> etc).
> 
> A solution proposed by one of the JDBC driver developers was to accept
> *null* as a parameter, in which case they could make the action immediate,
> and if an *Executor* is supplied the code would continue to behave as above.
> 
> My issues with this solution are, 1) that the JavaDoc plainly states that a
> SQLException should be thrown:
> 
> if a database access error occurs, this method is called on a closed
>> connection, *the **executor is null*, or the value specified for seconds is
>> less than 0.
> 
> 
> and, 2) for most users passing in a Executor, the behavior of the method
> would be quite non-deterministic.
> 
> Having said that, every other driver source I have been able to find simply
> *ignores* the supplied Executor completely.  Therefore, that was my
> suggestion for the driver in question -- "just ignore the Executor, as
> other drivers do, as you have direct access to the socket and the effect
> can be immediate."
> 
> Again pointing to lack of clarity, the driver developer's objection is
> based on the use of the word "will" in the JavaDoc for the executor
> parameter:
> executor - The Executor implementation which *will* be used by
> setNetworkTimeout.
> I have searched high and low for some archive of the discussions around the
> original proposal for this method, and its parameters, in order to find
> some clarity, but have come up short-handed.
> 
> 
> Please, keepers of the specification, provide guidance as to these two
> questions:
> 
> 1) Are the effects of *setNetworkTimeout()* intended to be immediate?
> 2) What is the role/purpose of the *Executor* parameter passed to
> *setNetworkTimeout()*?
> 
> 
> Brett Wooldridge


From lance.andersen at oracle.com  Wed Nov 15 00:49:39 2017
From: lance.andersen at oracle.com (Lance Andersen)
Date: Tue, 14 Nov 2017 19:49:39 -0500
Subject: Early Draft of  javadocs 
Message-ID: <FF14FF88-38B1-4FBD-86FD-C2D89CEF7225@oracle.com>

You can find a working draft of the javadocs for the non-blocking  JDBC API at: cr.openjdk.java.net/~lancea/apidoc/

Keep in mind

- This is an early draft
- It is subject to change
- Feedback is welcome

Best
Lance

 <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif> <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com <mailto:Lance.Andersen at oracle.com>




From douglas.surber at oracle.com  Thu Nov 16 16:25:58 2017
From: douglas.surber at oracle.com (Douglas Surber)
Date: Thu, 16 Nov 2017 08:25:58 -0800
Subject: convenience methods
Message-ID: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>

ADBA includes a fair number of convenience methods. These are methods that encapsulate common code patterns into default methods. DataSource.getConnection() is an obvious example.

As default methods these are all implemented by calling other interface methods. Clearly they aren?t necessary as the app can make the same sequence of calls. On the other hand their use will result in more compact and more readable code. Also, some of them handle some not so obvious behavior so that the programmer doesn?t have to think about it in the common use case. Again DataSource.getConnection is a good example; look at Connection.connect. A third benefit and honestly the original motivation for many of them is as documentation, how to use the API.

As a general notion, do these methods carry their weight? Are the benefits they provide worth whatever additional complexity they add? 

Here is a concrete example to consider. This is a method in Connection.

  public default CompletionStage<TransactionOutcome> 
        executeAsTransaction(BiConsumer<OperationGroup, Transaction> action) {
    OperationGroup grp = this.operationGroup();
    Transaction tran = this.transaction();
    action.accept(grp, tran);
    grp.submit();
    return this.commitMaybeRollback(tran); *
  }

Clearly this method is not strictly necessary. It is implemented entirely by calling other methods present in ADBA. Any application or framework can do exactly the same. On the other hand most applications and frameworks would make exactly the same sequence of calls so including this pattern in ADBA would be convenient. Finally, this provides an example of how to use ADBA to execute a transaction. This is actually the original motivation for creating this method.

ADBA is based on some relatively primitive concepts. In some cases there is quite a conceptual distance between those primitive concepts and what an application wants to do. It is easier to specify, understand, and implement just the primitive concepts but that leaves a conceptual gap to what the app needs; how to compose the primitives into useful actions. The various convenience methods bridge this gap without adding any specification or implementation complexity. IMO these convenience methods actually reduce the complexity by adding an executable specification of how the primitives work together to support a higher level abstraction.

Should we add executeAsTransaction? What about other convenience methods? Suggestions welcome.

Douglas


* Note: commitMaybeRollback has been corrected to return CompletionStage<TransactionOutcome> which is clearly what it should do.



From lukas.eder at gmail.com  Wed Nov 22 11:28:42 2017
From: lukas.eder at gmail.com (Lukas Eder)
Date: Wed, 22 Nov 2017 12:28:42 +0100
Subject: convenience methods
In-Reply-To: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>
References: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>
Message-ID: <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>

Hi Douglas,

I think that convenience methods are really very very helpful. jOOQ has
tons of them, for instance, because that really greatly reduces boiler
plate code on the call site and makes using the API much more effective and
fun. This also helps increase adoption.

One of the main reasons why Spring's JdbcTemplate, or Apache DbUtils, or
the simpler parts of the jOOQ API exists is the fact that the (synchronous)
JDBC API doesn't have any such methods. For instance, it would make total
sense to have

    public interface Connection {
        default PreparedStatement prepareStatement(String sql, Object...
bindings) {
            PreparedStatement s = prepareStatement(sql);
            for (int i = 0; i < bindings.length; i++)
                s.setObject(i + 1, bindings[i]);
        }

        default int executeUpdate(String sql, Object... bindings) {
            try (PreparedStatement s = prepareStatement(sql, bindings)) {
                return s.executeUpdate();
            }
        }
    }

I mean, who really enjoys counting bind indexes manually and sending a very
simple static INSERT statement with 10 values to the server by setting each
bind variable individually?

This convenience stuff has been dearly missing from the very first version
of JDBC :) (granted, it would have been difficult to add, without default
methods). There are many other possible convenience methods, including:

    public interface ResultSet {
        default Object[] get() {
            Object[] result = new Object[getMetaData().getColumnCount()];

            for (int i = 0; i < result.length; i++)
                result[i] = getObject(i);

            return result;
        }

        default void forEach(Consumer<Object[]> consumer) {
            while (next())
                consumer.accept(get());
        }
    }

I could go on and on, if you're interested. In fact, consider this a
feature request! :)

Further comments inline:

2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:

> As default methods these are all implemented by calling other interface
> methods. Clearly they aren?t necessary as the app can make the same
> sequence of calls. On the other hand their use will result in more compact
> and more readable code. Also, some of them handle some not so obvious
> behavior so that the programmer doesn?t have to think about it in the
> common use case. Again DataSource.getConnection is a good example;


Yes, a very good example. No one wants to type that builder() stuff all the
time even if it is useful for the special case. The convenience API should
be there for the more general case.


> look at Connection.connect. A third benefit and honestly the original
> motivation for many of them is as documentation, how to use the API.
>

That's an interesting way to look at it - it never occurred to me but
you're absolutely right. It's a nice way to document lower level API parts.


> As a general notion, do these methods carry their weight? Are the benefits
> they provide worth whatever additional complexity they add?
>

Absolutely! Again, one of the biggest flaws of the classic JDBC API is the
verbosity it generates in client code. Now many people argue that verbosity
is Java's strength, but that's nonsense when it comes to infrastructure
logic that never changes. E.g. try-with-resources is "convenience API" that
really really helped JDBC, because the proper approach to closing all of
Connection, Statement, ResultSet in the right order and with the right
exception handling really doesn't add any value to anyone's business - au
contraire, it's too easy to get them wrong.

Languages like Kotlin don't have the try-with-resources statement, but
extension methods that can be attached to any (Auto)Closeable, which is the
same thing as *external* convenience API. Without extension methods, that's
not possible in Java, so it is the API maintainer's burden to implement
default methods.

Also, you're not alone. Stream has Stream.collect(supplier, accumulator,
combiner), which is convenience for many cases of Stream.collect(Collector).

JDK 9's Optional has Optional.stream(), which really wasn't "necessary",
but so cool as it allows for flatmapping Stream<Optional<T>> to Stream<T>
through

    stream.flatMap(Optional::stream)

instead of

    stream.filter(Optional::isPresent).map(Optional::get)

Just two examples. There are many other convenience APIs in the JDK, even
trivial ones like the ArrayList constructors. The only one that is strictly
necessary is the one that accepts a capacity. The others are there for
convenience and improved performance.

Should we add executeAsTransaction? What about other convenience methods?
> Suggestions welcome.
>

Well, since you've added collect(Collector) (still very excited about
that), how about also adding collect(supplier, accumulator, combiner) as a
convenience?

If you're positive that you're moving forward with convenience methods, I'd
be happy to look for more opportunities to contribute both in ADBA and in
JDBC (if that's something you might consider)

Thanks,
Lukas

2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:

> ADBA includes a fair number of convenience methods. These are methods that
> encapsulate common code patterns into default methods.
> DataSource.getConnection() is an obvious example.
>
> As default methods these are all implemented by calling other interface
> methods. Clearly they aren?t necessary as the app can make the same
> sequence of calls. On the other hand their use will result in more compact
> and more readable code. Also, some of them handle some not so obvious
> behavior so that the programmer doesn?t have to think about it in the
> common use case. Again DataSource.getConnection is a good example; look at
> Connection.connect. A third benefit and honestly the original motivation
> for many of them is as documentation, how to use the API.
>
> As a general notion, do these methods carry their weight? Are the benefits
> they provide worth whatever additional complexity they add?
>
> Here is a concrete example to consider. This is a method in Connection.
>
>   public default CompletionStage<TransactionOutcome>
>         executeAsTransaction(BiConsumer<OperationGroup, Transaction>
> action) {
>     OperationGroup grp = this.operationGroup();
>     Transaction tran = this.transaction();
>     action.accept(grp, tran);
>     grp.submit();
>     return this.commitMaybeRollback(tran); *
>   }
>
> Clearly this method is not strictly necessary. It is implemented entirely
> by calling other methods present in ADBA. Any application or framework can
> do exactly the same. On the other hand most applications and frameworks
> would make exactly the same sequence of calls so including this pattern in
> ADBA would be convenient. Finally, this provides an example of how to use
> ADBA to execute a transaction. This is actually the original motivation for
> creating this method.
>
> ADBA is based on some relatively primitive concepts. In some cases there
> is quite a conceptual distance between those primitive concepts and what an
> application wants to do. It is easier to specify, understand, and implement
> just the primitive concepts but that leaves a conceptual gap to what the
> app needs; how to compose the primitives into useful actions. The various
> convenience methods bridge this gap without adding any specification or
> implementation complexity. IMO these convenience methods actually reduce
> the complexity by adding an executable specification of how the primitives
> work together to support a higher level abstraction.
>
> Should we add executeAsTransaction? What about other convenience methods?
> Suggestions welcome.
>
> Douglas
>
>
> * Note: commitMaybeRollback has been corrected to return CompletionStage<TransactionOutcome>
> which is clearly what it should do.
>
>
>

From douglas.surber at oracle.com  Wed Nov 22 17:24:02 2017
From: douglas.surber at oracle.com (Douglas Surber)
Date: Wed, 22 Nov 2017 09:24:02 -0800
Subject: convenience methods
In-Reply-To: <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>
References: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>
 <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>
Message-ID: <A50FBEBB-D07F-40CD-B57B-F79395CF6BC0@oracle.com>

Lukas,

Thanks for your reply and encouragement. I?m not going to undertake changes to JDBC. My plate is full with ADBA. If you contribute specific JDBC convenience methods to this list the EG may consider them for a future release. The less work the EG and the Spec Lead have to do the more likely your contributions are to be adopted. I would recommend providing the complete method source including the JavaDoc so that all the Spec Lead has to do is paste it into the appropriate source file. That?s not to say such contributions would necessarily be adopted verbatim or even at all, but it does increase the chances of getting something.

The same recommendation applies to ADBA convenience methods. The more complete your contribution the easier it is for us to adopt them. Even just a list of desired convenience method signatures would be useful for ADBA though. We can?t think of everything.

Douglas

> On Nov 22, 2017, at 3:28 AM, Lukas Eder <lukas.eder at gmail.com> wrote:
> 
> Hi Douglas,
> 
> I think that convenience methods are really very very helpful. jOOQ has tons of them, for instance, because that really greatly reduces boiler plate code on the call site and makes using the API much more effective and fun. This also helps increase adoption.
> 
> One of the main reasons why Spring's JdbcTemplate, or Apache DbUtils, or the simpler parts of the jOOQ API exists is the fact that the (synchronous) JDBC API doesn't have any such methods. For instance, it would make total sense to have
> 
>     public interface Connection {
>         default PreparedStatement prepareStatement(String sql, Object... bindings) {
>             PreparedStatement s = prepareStatement(sql);
>             for (int i = 0; i < bindings.length; i++)
>                 s.setObject(i + 1, bindings[i]);
>         }
>     
>         default int executeUpdate(String sql, Object... bindings) {
>             try (PreparedStatement s = prepareStatement(sql, bindings)) {
>                 return s.executeUpdate();
>             }
>         }
>     }
> 
> I mean, who really enjoys counting bind indexes manually and sending a very simple static INSERT statement with 10 values to the server by setting each bind variable individually?
> 
> This convenience stuff has been dearly missing from the very first version of JDBC :) (granted, it would have been difficult to add, without default methods). There are many other possible convenience methods, including:
> 
>     public interface ResultSet {
>         default Object[] get() {
>             Object[] result = new Object[getMetaData().getColumnCount()];
>     
>             for (int i = 0; i < result.length; i++)
>                 result[i] = getObject(i);
>     
>             return result;
>         }
>     
>         default void forEach(Consumer<Object[]> consumer) {
>             while (next())
>                 consumer.accept(get());
>         }
>     }
> 
> I could go on and on, if you're interested. In fact, consider this a feature request! :)
> 
> Further comments inline:
> 
> 2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com <mailto:douglas.surber at oracle.com>>:
> As default methods these are all implemented by calling other interface methods. Clearly they aren?t necessary as the app can make the same sequence of calls. On the other hand their use will result in more compact and more readable code. Also, some of them handle some not so obvious behavior so that the programmer doesn?t have to think about it in the common use case. Again DataSource.getConnection is a good example;
> 
> Yes, a very good example. No one wants to type that builder() stuff all the time even if it is useful for the special case. The convenience API should be there for the more general case.
>  
> look at Connection.connect. A third benefit and honestly the original motivation for many of them is as documentation, how to use the API.
> 
> That's an interesting way to look at it - it never occurred to me but you're absolutely right. It's a nice way to document lower level API parts.
>  
> As a general notion, do these methods carry their weight? Are the benefits they provide worth whatever additional complexity they add?
> 
> Absolutely! Again, one of the biggest flaws of the classic JDBC API is the verbosity it generates in client code. Now many people argue that verbosity is Java's strength, but that's nonsense when it comes to infrastructure logic that never changes. E.g. try-with-resources is "convenience API" that really really helped JDBC, because the proper approach to closing all of Connection, Statement, ResultSet in the right order and with the right exception handling really doesn't add any value to anyone's business - au contraire, it's too easy to get them wrong.
> 
> Languages like Kotlin don't have the try-with-resources statement, but extension methods that can be attached to any (Auto)Closeable, which is the same thing as *external* convenience API. Without extension methods, that's not possible in Java, so it is the API maintainer's burden to implement default methods.
> 
> Also, you're not alone. Stream has Stream.collect(supplier, accumulator, combiner), which is convenience for many cases of Stream.collect(Collector).
> 
> JDK 9's Optional has Optional.stream(), which really wasn't "necessary", but so cool as it allows for flatmapping Stream<Optional<T>> to Stream<T> through 
> 
>     stream.flatMap(Optional::stream)
> 
> instead of 
> 
>     stream.filter(Optional::isPresent).map(Optional::get)
> 
> Just two examples. There are many other convenience APIs in the JDK, even trivial ones like the ArrayList constructors. The only one that is strictly necessary is the one that accepts a capacity. The others are there for convenience and improved performance.
> 
> Should we add executeAsTransaction? What about other convenience methods? Suggestions welcome.
> 
> Well, since you've added collect(Collector) (still very excited about that), how about also adding collect(supplier, accumulator, combiner) as a convenience?
> 
> If you're positive that you're moving forward with convenience methods, I'd be happy to look for more opportunities to contribute both in ADBA and in JDBC (if that's something you might consider)
> 
> Thanks,
> Lukas
> 
> 2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com <mailto:douglas.surber at oracle.com>>:
> ADBA includes a fair number of convenience methods. These are methods that encapsulate common code patterns into default methods. DataSource.getConnection() is an obvious example.
> 
> As default methods these are all implemented by calling other interface methods. Clearly they aren?t necessary as the app can make the same sequence of calls. On the other hand their use will result in more compact and more readable code. Also, some of them handle some not so obvious behavior so that the programmer doesn?t have to think about it in the common use case. Again DataSource.getConnection is a good example; look at Connection.connect. A third benefit and honestly the original motivation for many of them is as documentation, how to use the API.
> 
> As a general notion, do these methods carry their weight? Are the benefits they provide worth whatever additional complexity they add?
> 
> Here is a concrete example to consider. This is a method in Connection.
> 
>   public default CompletionStage<TransactionOutcome>
>         executeAsTransaction(BiConsumer<OperationGroup, Transaction> action) {
>     OperationGroup grp = this.operationGroup();
>     Transaction tran = this.transaction();
>     action.accept(grp, tran);
>     grp.submit();
>     return this.commitMaybeRollback(tran); *
>   }
> 
> Clearly this method is not strictly necessary. It is implemented entirely by calling other methods present in ADBA. Any application or framework can do exactly the same. On the other hand most applications and frameworks would make exactly the same sequence of calls so including this pattern in ADBA would be convenient. Finally, this provides an example of how to use ADBA to execute a transaction. This is actually the original motivation for creating this method.
> 
> ADBA is based on some relatively primitive concepts. In some cases there is quite a conceptual distance between those primitive concepts and what an application wants to do. It is easier to specify, understand, and implement just the primitive concepts but that leaves a conceptual gap to what the app needs; how to compose the primitives into useful actions. The various convenience methods bridge this gap without adding any specification or implementation complexity. IMO these convenience methods actually reduce the complexity by adding an executable specification of how the primitives work together to support a higher level abstraction.
> 
> Should we add executeAsTransaction? What about other convenience methods? Suggestions welcome.
> 
> Douglas
> 
> 
> * Note: commitMaybeRollback has been corrected to return CompletionStage<TransactionOutcome> which is clearly what it should do.
> 
> 
> 


From lukas.eder at gmail.com  Wed Nov 22 17:26:34 2017
From: lukas.eder at gmail.com (Lukas Eder)
Date: Wed, 22 Nov 2017 18:26:34 +0100
Subject: convenience methods
In-Reply-To: <A50FBEBB-D07F-40CD-B57B-F79395CF6BC0@oracle.com>
References: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>
 <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>
 <A50FBEBB-D07F-40CD-B57B-F79395CF6BC0@oracle.com>
Message-ID: <CAB4ELO4hSMyjFCOVPDS28f0oesJ3p=d5u=ghc92q19YW+fWAtw@mail.gmail.com>

2017-11-22 18:24 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:

> Lukas,
>
> Thanks for your reply and encouragement. I?m not going to undertake
> changes to JDBC. My plate is full with ADBA. If you contribute specific
> JDBC convenience methods to this list the EG may consider them for a future
> release. The less work the EG and the Spec Lead have to do the more likely
> your contributions are to be adopted. I would recommend providing the
> complete method source including the JavaDoc so that all the Spec Lead has
> to do is paste it into the appropriate source file. That?s not to say such
> contributions would necessarily be adopted verbatim or even at all, but it
> does increase the chances of getting something.
>

Makes sense. I'll look into this during the coming weeks.

The same recommendation applies to ADBA convenience methods. The more
> complete your contribution the easier it is for us to adopt them. Even just
> a list of desired convenience method signatures would be useful for ADBA
> though. We can?t think of everything.
>

Will do as well.

Thanks,
Lukas

2017-11-22 18:24 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:

> Lukas,
>
> Thanks for your reply and encouragement. I?m not going to undertake
> changes to JDBC. My plate is full with ADBA. If you contribute specific
> JDBC convenience methods to this list the EG may consider them for a future
> release. The less work the EG and the Spec Lead have to do the more likely
> your contributions are to be adopted. I would recommend providing the
> complete method source including the JavaDoc so that all the Spec Lead has
> to do is paste it into the appropriate source file. That?s not to say such
> contributions would necessarily be adopted verbatim or even at all, but it
> does increase the chances of getting something.
>
> The same recommendation applies to ADBA convenience methods. The more
> complete your contribution the easier it is for us to adopt them. Even just
> a list of desired convenience method signatures would be useful for ADBA
> though. We can?t think of everything.
>
> Douglas
>
>
> On Nov 22, 2017, at 3:28 AM, Lukas Eder <lukas.eder at gmail.com> wrote:
>
> Hi Douglas,
>
> I think that convenience methods are really very very helpful. jOOQ has
> tons of them, for instance, because that really greatly reduces boiler
> plate code on the call site and makes using the API much more effective and
> fun. This also helps increase adoption.
>
> One of the main reasons why Spring's JdbcTemplate, or Apache DbUtils, or
> the simpler parts of the jOOQ API exists is the fact that the (synchronous)
> JDBC API doesn't have any such methods. For instance, it would make total
> sense to have
>
>     public interface Connection {
>         default PreparedStatement prepareStatement(String sql, Object...
> bindings) {
>             PreparedStatement s = prepareStatement(sql);
>             for (int i = 0; i < bindings.length; i++)
>                 s.setObject(i + 1, bindings[i]);
>         }
>
>         default int executeUpdate(String sql, Object... bindings) {
>             try (PreparedStatement s = prepareStatement(sql, bindings)) {
>                 return s.executeUpdate();
>             }
>         }
>     }
>
> I mean, who really enjoys counting bind indexes manually and sending a
> very simple static INSERT statement with 10 values to the server by setting
> each bind variable individually?
>
> This convenience stuff has been dearly missing from the very first version
> of JDBC :) (granted, it would have been difficult to add, without default
> methods). There are many other possible convenience methods, including:
>
>     public interface ResultSet {
>         default Object[] get() {
>             Object[] result = new Object[getMetaData().getColumnCount()];
>
>             for (int i = 0; i < result.length; i++)
>                 result[i] = getObject(i);
>
>             return result;
>         }
>
>         default void forEach(Consumer<Object[]> consumer) {
>             while (next())
>                 consumer.accept(get());
>         }
>     }
>
> I could go on and on, if you're interested. In fact, consider this a
> feature request! :)
>
> Further comments inline:
>
> 2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:
>
>> As default methods these are all implemented by calling other interface
>> methods. Clearly they aren?t necessary as the app can make the same
>> sequence of calls. On the other hand their use will result in more compact
>> and more readable code. Also, some of them handle some not so obvious
>> behavior so that the programmer doesn?t have to think about it in the
>> common use case. Again DataSource.getConnection is a good example;
>
>
> Yes, a very good example. No one wants to type that builder() stuff all
> the time even if it is useful for the special case. The convenience API
> should be there for the more general case.
>
>
>> look at Connection.connect. A third benefit and honestly the original
>> motivation for many of them is as documentation, how to use the API.
>>
>
> That's an interesting way to look at it - it never occurred to me but
> you're absolutely right. It's a nice way to document lower level API parts.
>
>
>> As a general notion, do these methods carry their weight? Are the
>> benefits they provide worth whatever additional complexity they add?
>>
>
> Absolutely! Again, one of the biggest flaws of the classic JDBC API is the
> verbosity it generates in client code. Now many people argue that verbosity
> is Java's strength, but that's nonsense when it comes to infrastructure
> logic that never changes. E.g. try-with-resources is "convenience API" that
> really really helped JDBC, because the proper approach to closing all of
> Connection, Statement, ResultSet in the right order and with the right
> exception handling really doesn't add any value to anyone's business - au
> contraire, it's too easy to get them wrong.
>
> Languages like Kotlin don't have the try-with-resources statement, but
> extension methods that can be attached to any (Auto)Closeable, which is the
> same thing as *external* convenience API. Without extension methods, that's
> not possible in Java, so it is the API maintainer's burden to implement
> default methods.
>
> Also, you're not alone. Stream has Stream.collect(supplier, accumulator,
> combiner), which is convenience for many cases of Stream.collect(Collector).
>
> JDK 9's Optional has Optional.stream(), which really wasn't "necessary",
> but so cool as it allows for flatmapping Stream<Optional<T>> to Stream<T>
> through
>
>     stream.flatMap(Optional::stream)
>
> instead of
>
>     stream.filter(Optional::isPresent).map(Optional::get)
>
> Just two examples. There are many other convenience APIs in the JDK, even
> trivial ones like the ArrayList constructors. The only one that is strictly
> necessary is the one that accepts a capacity. The others are there for
> convenience and improved performance.
>
> Should we add executeAsTransaction? What about other convenience methods?
>> Suggestions welcome.
>>
>
> Well, since you've added collect(Collector) (still very excited about
> that), how about also adding collect(supplier, accumulator, combiner) as a
> convenience?
>
> If you're positive that you're moving forward with convenience methods,
> I'd be happy to look for more opportunities to contribute both in ADBA and
> in JDBC (if that's something you might consider)
>
> Thanks,
> Lukas
>
> 2017-11-16 17:25 GMT+01:00 Douglas Surber <douglas.surber at oracle.com>:
>
>> ADBA includes a fair number of convenience methods. These are methods
>> that encapsulate common code patterns into default methods.
>> DataSource.getConnection() is an obvious example.
>>
>> As default methods these are all implemented by calling other interface
>> methods. Clearly they aren?t necessary as the app can make the same
>> sequence of calls. On the other hand their use will result in more compact
>> and more readable code. Also, some of them handle some not so obvious
>> behavior so that the programmer doesn?t have to think about it in the
>> common use case. Again DataSource.getConnection is a good example; look at
>> Connection.connect. A third benefit and honestly the original motivation
>> for many of them is as documentation, how to use the API.
>>
>> As a general notion, do these methods carry their weight? Are the
>> benefits they provide worth whatever additional complexity they add?
>>
>> Here is a concrete example to consider. This is a method in Connection.
>>
>>   public default CompletionStage<TransactionOutcome>
>>         executeAsTransaction(BiConsumer<OperationGroup, Transaction>
>> action) {
>>     OperationGroup grp = this.operationGroup();
>>     Transaction tran = this.transaction();
>>     action.accept(grp, tran);
>>     grp.submit();
>>     return this.commitMaybeRollback(tran); *
>>   }
>>
>> Clearly this method is not strictly necessary. It is implemented entirely
>> by calling other methods present in ADBA. Any application or framework can
>> do exactly the same. On the other hand most applications and frameworks
>> would make exactly the same sequence of calls so including this pattern in
>> ADBA would be convenient. Finally, this provides an example of how to use
>> ADBA to execute a transaction. This is actually the original motivation for
>> creating this method.
>>
>> ADBA is based on some relatively primitive concepts. In some cases there
>> is quite a conceptual distance between those primitive concepts and what an
>> application wants to do. It is easier to specify, understand, and implement
>> just the primitive concepts but that leaves a conceptual gap to what the
>> app needs; how to compose the primitives into useful actions. The various
>> convenience methods bridge this gap without adding any specification or
>> implementation complexity. IMO these convenience methods actually reduce
>> the complexity by adding an executable specification of how the primitives
>> work together to support a higher level abstraction.
>>
>> Should we add executeAsTransaction? What about other convenience methods?
>> Suggestions welcome.
>>
>> Douglas
>>
>>
>> * Note: commitMaybeRollback has been corrected to return
>> CompletionStage<TransactionOutcome> which is clearly what it should do.
>>
>>
>>
>
>

From lance.andersen at oracle.com  Thu Nov 23 13:17:15 2017
From: lance.andersen at oracle.com (Lance Andersen)
Date: Thu, 23 Nov 2017 08:17:15 -0500
Subject: convenience methods
In-Reply-To: <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>
References: <6CA4FEAB-25D3-4C6A-AD9B-C2D19FD680B5@oracle.com>
 <CAB4ELO6M717d3Sf2ejVhDszt06Q94S7r49LmUrsdZm-LnJ4OLA@mail.gmail.com>
Message-ID: <1261422D-9488-47AA-8BE3-EE5FFD7251C9@oracle.com>

Hi Lukas,

If you have some specific convenience methods to suggest we can consider them
> On Nov 22, 2017, at 6:28 AM, Lukas Eder <lukas.eder at gmail.com> wrote:
> 
> Hi Douglas,
> 
> I think that convenience methods are really very very helpful. jOOQ has
> tons of them, for instance, because that really greatly reduces boiler
> plate code on the call site and makes using the API much more effective and
> fun. This also helps increase adoption.
> 
> One of the main reasons why Spring's JdbcTemplate, or Apache DbUtils, or
> the simpler parts of the jOOQ API exists is the fact that the (synchronous)
> JDBC API doesn't have any such methods. For instance, it would make total
> sense to have
> 
>    public interface Connection {
>        default PreparedStatement prepareStatement(String sql, Object...
> bindings) {
>            PreparedStatement s = prepareStatement(sql);
>            for (int i = 0; i < bindings.length; i++)
>                s.setObject(i + 1, bindings[i]);
>        }
> 
>        default int executeUpdate(String sql, Object... bindings) {
>            try (PreparedStatement s = prepareStatement(sql, bindings)) {
>                return s.executeUpdate();
>            }
>        }

The downside to this method is the PreparedStatement is closed after each execution and depending on the driver, the overhead might negate the ease of use.


But please feel free to propose methods you believe would be worthwhile and I will be happy to discuss and consider.

Best
Lance
>    }
> 
> I mean, who really enjoys counting bind indexes manually and sending a very
> simple static INSERT statement with 10 values to the server by setting each
> bind variable individually?
> 
> This convenience stuff has been dearly missing from the very first version
> of JDBC :) (granted, it would have been difficult to add, without default
> methods). There are many other possible convenience methods, including:
> 
>    public interface ResultSet {
>        default Object[] get() {
>            Object[] result = new Object[getMetaData().getColumnCount()];
> 
>            for (int i = 0; i < result.length; i++)
>                result[i] = getObject(i);
> 
>            return result;
>        }
> 
>        default void forEach(Consumer<Object[]> consumer) {
>            while (next())
>                consumer.accept(get());
>        }
>    }
> 
> I could go on and on, if you're interested. In fact, consider this a
> feature request! :)

 <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif> <http://oracle.com/us/design/oracle-email-sig-198324.gif>
 <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037
Oracle Java Engineering 
1 Network Drive 
Burlington, MA 01803
Lance.Andersen at oracle.com <mailto:Lance.Andersen at oracle.com>




From sdeleuze at pivotal.io  Thu Nov 23 15:00:45 2017
From: sdeleuze at pivotal.io (Sebastien Deleuze)
Date: Thu, 23 Nov 2017 16:00:45 +0100
Subject: CompletionStage instead of Future for Submission#cancel return type?
Message-ID: <CAF_xPQJbkSUhGUeb1avSZNgd+CE3UyQtbg2dMRtJRRpsYyhkQw@mail.gmail.com>

While reviewing ADBA API, I noticed that Submission#cancel returns Future
[1], which in practice is mostly usable in a blocking way since get() is
blocking, checking regularly isDone() is not convenient at all, and
unlike CompletionStage it does not provide a way to register a complete
callback.

Is there any specific reason why CompletionStage<Boolean> can't be used
there as return type? In addition to providing a non-blocking API, that
would be also more consistent.

I am also not sure about Submission#getCompletionStage method name. Maybe
something like Submission#getResult would be more clear than just naming it
against the return type, especially if there are 2 methods returning
CompletionStage
like in my proposal.

Any thoughts?

[1]
http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/Submission.html#cancel--

From v at lightbend.com  Fri Nov 24 10:41:45 2017
From: v at lightbend.com (Viktor Klang)
Date: Fri, 24 Nov 2017 11:41:45 +0100
Subject: CompletionStage instead of Future for Submission#cancel return
 type?
In-Reply-To: <CAF_xPQJbkSUhGUeb1avSZNgd+CE3UyQtbg2dMRtJRRpsYyhkQw@mail.gmail.com>
References: <CAF_xPQJbkSUhGUeb1avSZNgd+CE3UyQtbg2dMRtJRRpsYyhkQw@mail.gmail.com>
Message-ID: <CAPWybzn2r3EiVcp07eDe=XEysNEYczcg5pKx0hwk9VJvPfvP0w@mail.gmail.com>

I agree, Sebastien?very sensible improvements!

On Thu, Nov 23, 2017 at 4:00 PM, Sebastien Deleuze <sdeleuze at pivotal.io>
wrote:

> While reviewing ADBA API, I noticed that Submission#cancel returns Future
> [1], which in practice is mostly usable in a blocking way since get() is
> blocking, checking regularly isDone() is not convenient at all, and
> unlike CompletionStage it does not provide a way to register a complete
> callback.
>
> Is there any specific reason why CompletionStage<Boolean> can't be used
> there as return type? In addition to providing a non-blocking API, that
> would be also more consistent.
>
> I am also not sure about Submission#getCompletionStage method name. Maybe
> something like Submission#getResult would be more clear than just naming it
> against the return type, especially if there are 2 methods returning
> CompletionStage
> like in my proposal.
>
> Any thoughts?
>
> [1]
> http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/
> Submission.html#cancel--
>



-- 
Cheers,
?

??????
*Viktor Klang*
Deputy CTO
Lightbend, Inc.

From douglas.surber at oracle.com  Mon Nov 27 15:56:58 2017
From: douglas.surber at oracle.com (Douglas Surber)
Date: Mon, 27 Nov 2017 07:56:58 -0800
Subject: CompletionStage instead of Future for Submission#cancel return
 type?
In-Reply-To: <CAF_xPQJbkSUhGUeb1avSZNgd+CE3UyQtbg2dMRtJRRpsYyhkQw@mail.gmail.com>
References: <CAF_xPQJbkSUhGUeb1avSZNgd+CE3UyQtbg2dMRtJRRpsYyhkQw@mail.gmail.com>
Message-ID: <506657DC-C1D0-44C3-A154-B8CE18D013A7@oracle.com>

The return value of cancel was overlooked when I modified everything to use CompletionStage. Thanks for pointing it out. It?s fixed now.

I agree about the name ?getCompletionStage?; I?m not sure about it. ?getResult? is better in some ways but it seems to me to be over promising. The method doesn?t get the result, it gets a CompletionStage which will be completed with the result. I?d be happy to change the name, but I?m not convinced that ?getResult? is the right name.

Douglas



> On Nov 23, 2017, at 7:00 AM, Sebastien Deleuze <sdeleuze at pivotal.io> wrote:
> 
> While reviewing ADBA API, I noticed that Submission#cancel returns Future
> [1], which in practice is mostly usable in a blocking way since get() is
> blocking, checking regularly isDone() is not convenient at all, and
> unlike CompletionStage it does not provide a way to register a complete
> callback.
> 
> Is there any specific reason why CompletionStage<Boolean> can't be used
> there as return type? In addition to providing a non-blocking API, that
> would be also more consistent.
> 
> I am also not sure about Submission#getCompletionStage method name. Maybe
> something like Submission#getResult would be more clear than just naming it
> against the return type, especially if there are 2 methods returning
> CompletionStage
> like in my proposal.
> 
> Any thoughts?
> 
> [1]
> http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/Submission.html#cancel--