From sdeleuze at pivotal.io Sun Dec 3 20:27:54 2017 From: sdeleuze at pivotal.io (Sebastien Deleuze) Date: Sun, 3 Dec 2017 12:27:54 -0800 Subject: CompletionStage instead of Future for Submission#cancel return type? In-Reply-To: <506657DC-C1D0-44C3-A154-B8CE18D013A7@oracle.com> References: <506657DC-C1D0-44C3-A154-B8CE18D013A7@oracle.com> Message-ID: I tend to think the CompletionStage return type clearly express the fact the the result could complete or not. Another option is to use some kind of convention for specifying the CompletionStage nature, like getAsyncResult() or getResultAsync(), but I don't find this very elegant and it duplicates an information we already have with the return type. Could you please indicate me where I can find the latest Javadoc and source code, since the URLs I have [1] [2] are outdated? [1] http://hg.openjdk.java.net/jdk10/sandbox/jdk/file/a31057bda7c5/src/java.sql/share/classes/java/sql2 [2] http://cr.openjdk.java.net/~lancea/apidoc/ On Mon, Nov 27, 2017 at 7:56 AM, Douglas Surber wrote: > The return value of cancel was overlooked when I modified everything to > use CompletionStage. Thanks for pointing it out. It?s fixed now. > > I agree about the name ?getCompletionStage?; I?m not sure about it. > ?getResult? is better in some ways but it seems to me to be over promising. > The method doesn?t get the result, it gets a CompletionStage which will be > completed with the result. I?d be happy to change the name, but I?m not > convinced that ?getResult? is the right name. > > Douglas > > > > > On Nov 23, 2017, at 7:00 AM, Sebastien Deleuze > wrote: > > > > While reviewing ADBA API, I noticed that Submission#cancel returns Future > > [1], which in practice is mostly usable in a blocking way since get() is > > blocking, checking regularly isDone() is not convenient at all, and > > unlike CompletionStage it does not provide a way to register a complete > > callback. > > > > Is there any specific reason why CompletionStage can't be used > > there as return type? In addition to providing a non-blocking API, that > > would be also more consistent. > > > > I am also not sure about Submission#getCompletionStage method name. Maybe > > something like Submission#getResult would be more clear than just naming > it > > against the return type, especially if there are 2 methods returning > > CompletionStage > > like in my proposal. > > > > Any thoughts? > > > > [1] > > http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/ > Submission.html#cancel-- > > From lance.andersen at oracle.com Sun Dec 3 20:41:06 2017 From: lance.andersen at oracle.com (Lance Andersen) Date: Sun, 3 Dec 2017 15:41:06 -0500 Subject: CompletionStage instead of Future for Submission#cancel return type? In-Reply-To: References: <506657DC-C1D0-44C3-A154-B8CE18D013A7@oracle.com> Message-ID: > On Dec 3, 2017, at 3:27 PM, Sebastien Deleuze wrote: > > I tend to think the CompletionStage return type clearly express the fact > the the result could complete or not. > Another option is to use some kind of convention for specifying the > CompletionStage nature, like getAsyncResult() or getResultAsync(), but I > don't find this very elegant and it duplicates an information we already > have with the return type. > > Could you please indicate me where I can find the latest Javadoc and source > code, since the URLs I have [1] [2] are outdated? > > [1] > http://hg.openjdk.java.net/jdk10/sandbox/jdk/file/a31057bda7c5/src/java.sql/share/classes/java/sql2 That is the old sandbox prior to the idk consolidation, the current URL is http://hg.openjdk.java.net/jdk/sandbox/file/b49fd5dbaab6/src/java.sql/share/classes/java/sql2 > [2] http://cr.openjdk.java.net/~lancea/apidoc/ This is the current javadoc based on the sandbox above. We have not updated this since prior to the US Thanksgiving Holiday. I expect we will updated it again within the next week or so > > On Mon, Nov 27, 2017 at 7:56 AM, Douglas Surber > wrote: > >> The return value of cancel was overlooked when I modified everything to >> use CompletionStage. Thanks for pointing it out. It?s fixed now. >> >> I agree about the name ?getCompletionStage?; I?m not sure about it. >> ?getResult? is better in some ways but it seems to me to be over promising. >> The method doesn?t get the result, it gets a CompletionStage which will be >> completed with the result. I?d be happy to change the name, but I?m not >> convinced that ?getResult? is the right name. >> >> Douglas >> >> >> >>> On Nov 23, 2017, at 7:00 AM, Sebastien Deleuze >> wrote: >>> >>> While reviewing ADBA API, I noticed that Submission#cancel returns Future >>> [1], which in practice is mostly usable in a blocking way since get() is >>> blocking, checking regularly isDone() is not convenient at all, and >>> unlike CompletionStage it does not provide a way to register a complete >>> callback. >>> >>> Is there any specific reason why CompletionStage can't be used >>> there as return type? In addition to providing a non-blocking API, that >>> would be also more consistent. >>> >>> I am also not sure about Submission#getCompletionStage method name. Maybe >>> something like Submission#getResult would be more clear than just naming >> it >>> against the return type, especially if there are 2 methods returning >>> CompletionStage >>> like in my proposal. >>> >>> Any thoughts? >>> >>> [1] >>> http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/ >> Submission.html#cancel-- >> >> Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From pm at netcetera.ch Mon Dec 4 10:14:46 2017 From: pm at netcetera.ch (Philippe Marschall) Date: Mon, 4 Dec 2017 11:14:46 +0100 Subject: [ADBA] Supporting BigInteger Message-ID: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> Hello One of the things that bothers me in JDBC is that there is no support for BigInteger. All arbitrary precision numbers, even if they have no decimal places, have to be expressed as BigDecimal. This is annoying because there is no way to communicate with the type that no decimal places are allowed or present. Currently ResultMap#get doesn't specify exactly which types have to be supported so I assume it's the same as with ResulSet#getObject. Cheers Philippe From jschauder at pivotal.io Mon Dec 4 10:39:43 2017 From: jschauder at pivotal.io (Jens Schauder) Date: Mon, 4 Dec 2017 11:39:43 +0100 Subject: Where is the input from the Expert Group? Message-ID: Hi again, if I got everything right, this is the mailing list of the JDBC Expert Group. So far almost all of the feedback to the proposed API seems to come from people involved with various projects relying on JDBC or related to Reactive (Reactive Streams and it's implementations, jOOQ). But I haven't seen any input from other Database Vendors besides Oracle although they seem to be part of the JDBC Expert Group. Do you get input from those via some other channel? If so: What is their feedback? If not: Would it be a good idea to reach out to them? Kind regards Jens Schauder From lance.andersen at oracle.com Mon Dec 4 11:58:53 2017 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 4 Dec 2017 06:58:53 -0500 Subject: Where is the input from the Expert Group? In-Reply-To: References: Message-ID: Hi Jens, > On Dec 4, 2017, at 5:39 AM, Jens Schauder wrote: > > Hi again, > > if I got everything right, this is the mailing list of the JDBC Expert > Group. This is the primary one though jsr-221-eg at jcp.org is still used > > So far almost all of the feedback to the proposed API seems to come from > people involved with various projects relying on JDBC or related to > Reactive (Reactive Streams and it's implementations, jOOQ). > But I haven't seen any input from other Database Vendors besides Oracle > although they seem to be part of the JDBC Expert Group. > > Do you get input from those via some other channel? We have had discussions with some of the JDBC EG members but have not gotten much feedback overall. Most of this was earlier this year > If so: What is their feedback? > If not: Would it be a good idea to reach out to them? Yes, and I have started reaching out directly to some of them again just recently When we hear back, the feedback will be shared on this alias. Thank you again for the suggestion, great minds think alike :-) Best Lance > > Kind regards > > Jens Schauder Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From lance.andersen at oracle.com Mon Dec 4 12:35:08 2017 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 4 Dec 2017 07:35:08 -0500 Subject: [ADBA] Supporting BigInteger In-Reply-To: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> References: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> Message-ID: Hi Philippe, We have never had a request for BigInteger support, We can consider this for a future release if others also see a common use case. > On Dec 4, 2017, at 5:14 AM, Philippe Marschall wrote: > > Hello > > One of the things that bothers me in JDBC is that there is no support for BigInteger. All arbitrary precision numbers, even if they have no decimal places, have to be expressed as BigDecimal. This is annoying because there is no way to communicate with the type that no decimal places are allowed or present. > > Currently ResultMap#get doesn't specify exactly which types have to be supported so I assume it's the same as with ResulSet#getObject. Yes that is correct. Keep in mind the javadocs are still a work in process Best Lance > > Cheers > Philippe Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From lukas.eder at gmail.com Mon Dec 4 12:53:44 2017 From: lukas.eder at gmail.com (Lukas Eder) Date: Mon, 4 Dec 2017 13:53:44 +0100 Subject: [ADBA] Supporting BigInteger In-Reply-To: References: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> Message-ID: I definitely see the use-case as well and have missed this functionality from JDBC in various jobs in the past. Thanks, Lukas 2017-12-04 13:35 GMT+01:00 Lance Andersen : > Hi Philippe, > > We have never had a request for BigInteger support, We can consider this > for a future release if others also see a common use case. > > > On Dec 4, 2017, at 5:14 AM, Philippe Marschall wrote: > > > > Hello > > > > One of the things that bothers me in JDBC is that there is no support > for BigInteger. All arbitrary precision numbers, even if they have no > decimal places, have to be expressed as BigDecimal. This is annoying > because there is no way to communicate with the type that no decimal places > are allowed or present. > > > > Currently ResultMap#get doesn't specify exactly which types have to be > supported so I assume it's the same as with ResulSet#getObject. > Yes that is correct. Keep in mind the javadocs are still a work in process > > Best > Lance > > > > Cheers > > Philippe > > > < > http://oracle.com/us/design/oracle-email-sig-198324.gif> > Lance Andersen| > Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > > From douglas.surber at oracle.com Mon Dec 4 16:20:05 2017 From: douglas.surber at oracle.com (Douglas Surber) Date: Mon, 4 Dec 2017 08:20:05 -0800 Subject: CompletionStage instead of Future for Submission#cancel return type? In-Reply-To: References: <506657DC-C1D0-44C3-A154-B8CE18D013A7@oracle.com> Message-ID: <563866C0-465E-4E94-912F-C6CBEEA8A501@oracle.com> There haven?t been many changes since the last uploaded version. I?ve been working on other things. Douglas > On Dec 3, 2017, at 12:41 PM, Lance Andersen wrote: > > > We have not updated this since prior to the US Thanksgiving Holiday. I expect we will updated it again within the next week or so > > > >> >> On Mon, Nov 27, 2017 at 7:56 AM, Douglas Surber > >> wrote: >> >>> The return value of cancel was overlooked when I modified everything to >>> use CompletionStage. Thanks for pointing it out. It?s fixed now. >>> >>> I agree about the name ?getCompletionStage?; I?m not sure about it. >>> ?getResult? is better in some ways but it seems to me to be over promising. >>> The method doesn?t get the result, it gets a CompletionStage which will be >>> completed with the result. I?d be happy to change the name, but I?m not >>> convinced that ?getResult? is the right name. >>> >>> Douglas >>> >>> >>> >>>> On Nov 23, 2017, at 7:00 AM, Sebastien Deleuze > >>> wrote: >>>> >>>> While reviewing ADBA API, I noticed that Submission#cancel returns Future >>>> [1], which in practice is mostly usable in a blocking way since get() is >>>> blocking, checking regularly isDone() is not convenient at all, and >>>> unlike CompletionStage it does not provide a way to register a complete >>>> callback. >>>> >>>> Is there any specific reason why CompletionStage can't be used >>>> there as return type? In addition to providing a non-blocking API, that >>>> would be also more consistent. >>>> >>>> I am also not sure about Submission#getCompletionStage method name. Maybe >>>> something like Submission#getResult would be more clear than just naming >>> it >>>> against the return type, especially if there are 2 methods returning >>>> CompletionStage >>>> like in my proposal. >>>> >>>> Any thoughts? >>>> >>>> [1] >>>> http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/ >>> Submission.html#cancel-- >>> >>> > > From douglas.surber at oracle.com Mon Dec 4 16:24:36 2017 From: douglas.surber at oracle.com (Douglas Surber) Date: Mon, 4 Dec 2017 08:24:36 -0800 Subject: [ADBA] Supporting BigInteger In-Reply-To: References: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> Message-ID: <5BC72510-3943-4D00-AF62-F0C14D30EC8A@oracle.com> +1 Douglas > On Dec 4, 2017, at 4:53 AM, Lukas Eder wrote: > > I definitely see the use-case as well and have missed this functionality > from JDBC in various jobs in the past. > > Thanks, > Lukas > > 2017-12-04 13:35 GMT+01:00 Lance Andersen : > >> Hi Philippe, >> >> We have never had a request for BigInteger support, We can consider this >> for a future release if others also see a common use case. >> >>> On Dec 4, 2017, at 5:14 AM, Philippe Marschall wrote: >>> >>> Hello >>> >>> One of the things that bothers me in JDBC is that there is no support >> for BigInteger. All arbitrary precision numbers, even if they have no >> decimal places, have to be expressed as BigDecimal. This is annoying >> because there is no way to communicate with the type that no decimal places >> are allowed or present. >>> >>> Currently ResultMap#get doesn't specify exactly which types have to be >> supported so I assume it's the same as with ResulSet#getObject. >> Yes that is correct. Keep in mind the javadocs are still a work in process >> >> Best >> Lance >>> >>> Cheers >>> Philippe >> >> >> < >> http://oracle.com/us/design/oracle-email-sig-198324.gif> >> Lance Andersen| >> Principal Member of Technical Staff | +1.781.442.2037 >> Oracle Java Engineering >> 1 Network Drive >> Burlington, MA 01803 >> Lance.Andersen at oracle.com >> >> >> >> From lance.andersen at oracle.com Mon Dec 4 18:21:32 2017 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 4 Dec 2017 13:21:32 -0500 Subject: [ADBA] Supporting BigInteger In-Reply-To: <5BC72510-3943-4D00-AF62-F0C14D30EC8A@oracle.com> References: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> <5BC72510-3943-4D00-AF62-F0C14D30EC8A@oracle.com> Message-ID: <98B90DA0-21C4-46E3-A5EE-F31D11889B72@oracle.com> I went back and looked at some of the older JDBC EG archives. There was a brief discussion on BigInteger at that time but there was not a consensus on the mapping. Input welcome to seed the discussion :-) Best Lance > On Dec 4, 2017, at 11:24 AM, Douglas Surber wrote: > > +1 > > Douglas > >> On Dec 4, 2017, at 4:53 AM, Lukas Eder wrote: >> >> I definitely see the use-case as well and have missed this functionality >> from JDBC in various jobs in the past. >> >> Thanks, >> Lukas >> >> 2017-12-04 13:35 GMT+01:00 Lance Andersen : >> >>> Hi Philippe, >>> >>> We have never had a request for BigInteger support, We can consider this >>> for a future release if others also see a common use case. >>> >>>> On Dec 4, 2017, at 5:14 AM, Philippe Marschall wrote: >>>> >>>> Hello >>>> >>>> One of the things that bothers me in JDBC is that there is no support >>> for BigInteger. All arbitrary precision numbers, even if they have no >>> decimal places, have to be expressed as BigDecimal. This is annoying >>> because there is no way to communicate with the type that no decimal places >>> are allowed or present. >>>> >>>> Currently ResultMap#get doesn't specify exactly which types have to be >>> supported so I assume it's the same as with ResulSet#getObject. >>> Yes that is correct. Keep in mind the javadocs are still a work in process >>> >>> Best >>> Lance >>>> >>>> Cheers >>>> Philippe >>> >>> >>> < >>> http://oracle.com/us/design/oracle-email-sig-198324.gif> >>> Lance Andersen| >>> Principal Member of Technical Staff | +1.781.442.2037 >>> Oracle Java Engineering >>> 1 Network Drive >>> Burlington, MA 01803 >>> Lance.Andersen at oracle.com >>> >>> >>> >>> > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com From pm at netcetera.ch Mon Dec 4 18:57:56 2017 From: pm at netcetera.ch (Philippe Marschall) Date: Mon, 4 Dec 2017 19:57:56 +0100 Subject: [ADBA] Supporting BigInteger In-Reply-To: <98B90DA0-21C4-46E3-A5EE-F31D11889B72@oracle.com> References: <43f43a6a-a569-24fa-b3d9-83b2dcd88473@netcetera.com> <5BC72510-3943-4D00-AF62-F0C14D30EC8A@oracle.com> <98B90DA0-21C4-46E3-A5EE-F31D11889B72@oracle.com> Message-ID: <6c4e0d68-bb9e-cc99-cd18-645dcfbfb6a6@netcetera.com> On 04.12.17 19:21, Lance Andersen wrote: > I went back and looked at some of the older JDBC EG archives. There was a brief discussion on BigInteger at that time but there was not a consensus on the mapping. > > > Input welcome to seed the discussion :-) Then I propose to allow mapping NUMBER(x, 0) to BigInteger in addition to all the existing supported mappings of NUMBER(x, 0). Two arguments can be made regarding the minimal supported value of x. First, the minimal supported value should be 19 as smaller values can be expressed with long. Second the minimal supported value should be 1 as #getLong is also supported for NUMBER(1, 0). I'm siding with the second argument. The existing integer types support various conversations. If these are going to be ported from JDBC to ADBA then for consistency the following should be considered: BigInteger reading could also be allowed from columns who's data type allows decimal places. This would be akin to supporting #getDate on a TIMESTAMP column. The JDBC specification currently allows this. BigInteger writing could also be allowed to columns who's data type allows decimal places. This would be akin to supporting #setDate on a TIMESTAMP column. The JDBC specification currently allows this. As #getLong is supported for TINYINT, SMALLINT, INTEGER, BIGINT, REAL, FLOAT, DOUBLE, DECIMAL, NUMERIC, BIT, BOOLEAN, CHAR, VARCHAR, LONGVARCHAR supporting these conversions for BigInteger would be consistent. I'm not found of many of those but if they're not revisited for ADBA then the consistency argument has some weight. If they're not revisited then I would welcome it if the conversion is defined stricter (what values can be converted, what pattern is to be used for parsing, what happens in case of a conversion error). Cheers Philippe From douglas.surber at oracle.com Fri Dec 15 22:21:22 2017 From: douglas.surber at oracle.com (Douglas Surber) Date: Fri, 15 Dec 2017 14:21:22 -0800 Subject: RowProcessorOperation In-Reply-To: References: <6aa21fc4-e4ed-f221-60cd-6f1904a5ed56@oracle.com> <01386292-0cd9-4004-a996-b9fc095169df@oracle.com> <128F2B26-A1A1-47DB-95AB-6C9B96C54301@oracle.com> <1c11ffc6-9a1d-4cb2-bf71-91baaf4d2267@Spark> <3E2D3086-C897-47CD-9725-3DFF8E0771C1@oracle.com> <1ccf064b-ec21-43d1-b6bd-33f10e722e15@Spark> Message-ID: Note: This discussion started as a private exchange but Lance noted, correctly, that it should be on this list. With everyone?s permission I?m continuing it here. This is the first post on this list so you?ll have to read the quoted messages to get the context. Mark, Every Operation has a result value. That is the value that completes the CompletionStage that is returned by Submission.getCompletionStage. This value is the result of the database action. The result of a RowOperation is the value returned by Collector.finisher. RowProcessorOperation also has a result value which is the last value published by the Processor. The ADBA driver publishes rows to the Processor and subscribes to the Processor for results. I know the JavaDoc that describes this is inadequate. I?m having a hard time finding the right language. Perhaps an example will help. This method takes a Connection and an invoice id and returns a CompletionStage producing the average price of the line items in the invoice. public CompletionStage flow(Connection conn, int invoiceId) { String sql = "select cost from line_item where invoice = :invoiceId"; Flow.Processor p = new Flow.Processor<>() { protected long sum = 0; protected long count = 0; protected long resultDemand = 0; protected Flow.Subscription rowSubscription; protected Flow.Subscription resultSubscription = new Flow.Subscription() { @Override public void request(long n) { // the driver will call this method to request a result value resultDemand += n; rowSubscription.request(n); } @Override public void cancel() { // Submission.cancel will call this method rowSubscription.cancel(); } }; protected Flow.Subscriber resultSubscriber; @Override public void onSubscribe(Flow.Subscription subscription) { rowSubscription = subscription; } @Override public void onNext(Result.Row item) { // the driver will call this repeatedly, once for each row sum += item.get("cost", Long.class); count++; rowSubscription.request(1); } @Override public void onError(Throwable throwable) { } @Override public void onComplete() { // the driver will call this when there are no more rows resultSubscriber.onNext(sum / count); // publish the result resultSubscriber.onComplete(); // no more results } @Override public void subscribe(Flow.Subscriber subscriber) { resultSubscriber = subscriber; resultSubscriber.onSubscribe(resultSubscription); } }; return conn.rowProcessorOperation(sql) .set("invoiceId", invoiceId) .rowProcessor(p) .submit() .getCompletionStage(); } My objection to PublisherOperation is that there is no result value. The driver publishes the rows to the Subscriber but never gets anything back. RowProcessorOperation does exactly the same thing but the Processor not only subscribes to the Rows but it also publishes results the last of which is the result value of the Operation. Douglas > On Dec 15, 2017, at 12:13 PM, Mark Paluch wrote: > > The result of the PublisherOperation is that the actual publisher receives a subscribe signal (Publisher#subscribe), the publisher notifies the subscriber with a subscription handle (Subscriber#onSubscribe) and starts emitting rows (Subscriber#onNext) as soon as the subscriber registers demand (Subscription#request). From a conceptional perspective, subscribing to the publisher is the last action that happens with the publisher, similar to what you?d achieve with a submission. PublisherOperation (in implementations most likely a delegate or communication component) is the one which should write requests to I/O and emit rows once received and decoded from I/O. > > A Processor is a Subscriber and a Publisher at the same time. It exposes onNext, onError and onComplete methods as well as a subscribe method. It requires to receive signals and pipes/broadcasts these to the subscribers that subscribe to the publisher. Typical use for a processor is to have a single upstream with multiple subscribers. Or what Lightbend did with their HTTP client as sort of a bridge: The processor receives HttpRequests and emits HttpResponses. For a database scenario that could be something like Processor where String represents a SQL string. I attached a few links [0] to [6] that point to implementations of processors in Project Reactor and RxJava 2. MongoDB follows a similar notion with their Reactive Streams driver to expose a fluent API (although their intermediate objects aren?t immutable) in which Publisher#subscribe is the action to initiate execution. > > Another aspect is that anyone passing in a processor can call methods on the processor. One could call onError/onComplete, change the state of the processor which makes it unusable for RowProcessorOperation as the processor is in terminated state and subsequent calls by RowProcessorOperation would fail as per ReactiveStreams (Flow) spec. > > [0] https://github.com/reactor/reactor-core/blob/master/reactor-core/src/main/java/reactor/core/publisher/DirectProcessor.java > [1] https://github.com/reactor/reactor-core/blob/master/reactor-core/src/main/java/reactor/core/publisher/EmitterProcessor.java > [2] https://github.com/reactor/reactor-core/blob/master/reactor-core/src/main/java/reactor/core/publisher/EventLoopProcessor.java > [3] https://github.com/reactor/reactor-core/blob/master/reactor-core/src/main/java/reactor/core/publisher/TopicProcessor.java > [4] https://github.com/ReactiveX/RxJava/blob/2.x/src/main/java/io/reactivex/processors/BehaviorProcessor.java > [5] https://github.com/ReactiveX/RxJava/blob/2.x/src/main/java/io/reactivex/processors/FlowableProcessor.java > [6] https://github.com/ReactiveX/RxJava/blob/2.x/src/main/java/io/reactivex/processors/ReplayProcessor.java > [7] https://github.com/mongodb/mongo-java-driver-reactivestreams/blob/master/driver/src/main/com/mongodb/reactivestreams/client/FindPublisher.java > > On 15. Dez. 2017, 18:23 +0100, Douglas Surber , wrote: >> But what is the result of a PublisherOperation? How does this tie into the rest of the API? That?s why I replaced PublisherOperation with ProcessorOperation. An Operation must have a result that is a function of the database action. That?s how Operations interact with the rest of the app and with each other. RowProcessorOperation is nearly identical to PublisherOperation except that it defines the result of the Operation in a natural way. >> >> I don?t understand your comment that way Processor is used in RowProcessorOperation is not typical. The Processor reads an input stream, the rows and transforms it into an output stream, the results. That?s exactly what a Processor does. I can?t see any way this is atypical. >> >> Douglas >> >>> On Dec 15, 2017, at 8:37 AM, Mark Paluch > wrote: >>> >>> Hi Douglas, >>> >>> Sorry it took me a while to get back to this topic. >>> >>> I think your (Lances) approach with PublisherOperation was quite promising. I created a proposal based on that interface. It does not reflect Operation and the relationship to collect(Collector) yet, let?s keep that for a later step. >>> >>> /** >>> * A publisher operation exposes a reactive execution model using >>> * Flow API. It provides an alternative model to {@link Operation#submit} >>> * allowing a streaming consumption of results. >>> * >>> * This {@link Operation} starts materializing this {@link Flow.Publisher} >>> * on {@link #subscribe subscription}. The {@link Operation} >>> * starts emitting {@link Result.Row rows} by calling {@link Flow.Subscriber#onNext} >>> * as soon as the {@link Flow.Subscription} registers demand. >>> * >>> * If the {@link Operation} causes an error it will call >>> * {@link Flow.Subscriber#onError} with the {@code Exception} then complete >>> * exceptionally with that {@code Exception}. If the {@link Operation} >>> * yields no result, it will call {@link Flow.Subscriber#onComplete} >>> * without emitting {@link Result#Row}. >>> * Calling {@link Flow.Subscription#cancel} requests cancellation similar to >>> * {@link Submission#cancel}. >>> * Each time a {@link Flow.Subscriber} subscribes to {@link PublisherOperation} >>> * it initiates a new execution of the {@link Operation}. >>> * >>> * Implementations are free to decide whether the actual >>> * execution should be initiated upon subscription or as soon as a >>> * {@link Flow.Subscription} registers demand through {@link Flow.Subscription#request}. >>> * >>> * @param the type of the result of the {@link Operation} >>> */ >>> public interface PublisherOperation extends Operation, Flow.Publisher { >>> >>> @Override >>> public void subscribe(Flow.Subscriber subscriber); >>> } >>> >>> Let me know what you think about it. >>> >>> Cheers, >>> Mark >>> >>> On 12. Dez. 2017, 17:55 +0100, Douglas Surber >, wrote: >>>> RowProcessorOperation does exactly what you asked for in the previous email. So you want something that does what RowProcessorOperation does but you want to get rid of RowProcessorOperation because the use of Processor isn?t typical. That?s fine but RowProcessorOperation is the best solution I have found to the problem. If you think it is not a good solution then by all means suggest an alternative. >>>> >>>> Douglas >>>> >>>>> On Dec 12, 2017, at 8:27 AM, Mark Paluch > wrote: >>>>> >>>>> Hi Douglas, >>>>> >>>>> I?ve seen RowProcessorOperation and wasn?t entirely happy with methods accepting a Flow.Processor. Typical use for processors is broadcasting/multiplexing of events. IMHO RowProcessorOperation is a good candidate to get rid of. >>>>> >>>>> Cheers, >>>>> Mark >>>>> >>>>> On 12. Dez. 2017, 17:15 +0100, Douglas Surber >, wrote: >>>>>> Have you looked at RowProcessorOperation? >>>>>> >>>>>> Douglas >>>>>> >>>>>>> On Dec 11, 2017, at 11:25 PM, Mark Paluch > wrote: >>>>>>> >>>>>>> Hi Douglas ? >>>>>>> >>>>>>> Thanks a lot for your answer. I didn?t want to stress Reactive Streams in the first place ? There are a couple of perspectives on this topic and I see your point in not requiring additional composition libraries to consume ADBA. >>>>>>> >>>>>>> I see it right now more from the perspective of enabling choice in places where it makes sense, similar to convenience methods you provide already in, e.g. Connection with the difference that Connection.connect() suggests synchronous execution on top of a non-blocking API. >>>>>>> >>>>>>> ADBA is limiting in the way of consuming results as it requires a collector and collects results until an operation is complete. Comparing to JDBC, a ResultSet is a pull-style iterator allowing a streaming consumption of results. That said, large results in ADBA would require paging and multiple queries to consume the data set. >>>>>>> >>>>>>> Reactive Streams comes with a dozen of opportunities that allow rethinking today?s data access of which back pressure is just one of these. I wouldn?t reduce back pressure/demand-awareness to only a measure to not overwhelm the consumer. >>>>>>> >>>>>>> Communicating demand to the data source (driver or in the future even to the database) is a sophisticated mechanism that allows drivers to react: Drivers can control fetch size (if not already set by the consumer), decide the amount of data to read and when to do the actual read. Demand can also suspend reading from the database if the client has no demand leading to a more efficient, demand-driven resource usage. >>>>>>> >>>>>>> Allowing e.g. RowOperation to be a Flow.Publisher (with keeping submissions) removes the collector limitation for Reactive Streams consumers and thus the limitation of issuing multiple queries to a database. Having callbacks (e.g. onNext(Consumer), similar to onError(?)) would as well help to remove these limitations. Without these limitations, consumers are able to reduce latency to first result and overall improve GC pressure. >>>>>>> >>>>>>> I?m happy to continue discussion on that topic on the mailing list. We integrated recently various drivers for Spring?s reactive data access and are happy to share more about effects and implications that come when used in applications. That?s primarily why I asked about EG and the way of collaboration. >>>>>>> >>>>>>> Cheers, >>>>>>> Mark >>>>>>> >>>>>>> >>>>>>> On 11. Dez. 2017, 19:33 +0100, Douglas Surber >, wrote: >>>>>>>>>>> which seem limiting to us in terms of a streaming consumption of results (Flow.Publisher). >>>>>>>> >>>>>>>> ADBA is not intended to be a pure reactive stream API. That is not and never has been a goal. This is not to say such an API would be a bad thing, but that is not what ADBA set out to be. What is a reasonable requirement however is that it be possible to write a real pure reactive stream API on top of ADBA. We believe that the current version of ADBA supports this. If not, please let us know what is missing. >>>>>>>> >>>>>>>> Keep in mind that if back pressure is of no use to the implementation/database then there is no reason for ADBA to support back pressure. The goal of back pressure is to eliminate the need for arbitrarily large queues. If the queue size is intrinsically limited (to something reasonable) then there is no need for back pressure. In such a case ADBA does not support reactive streams (or any other mechanism to provide back pressure) as it would be superfluous. A pure reactive stream API might have to expose hooks that would in theory provide back pressure in such a case. There would be no loss if those hooks did little or nothing as the goal of the back pressure, limited queue size, is already met. >>>>>>>> >>>>>>>> Douglas