OperationGroup extends AutoCloseable

[Douglas Surber]

An OperationGroup provides conditional execution, control of error response, and control of execution order.

Execution of one or more Operations may depend on the result of a previous Operation. Depending on the result of that previous Operation some other operations perhaps should not be executed. For example consider an account withdrawal. If the amount to withdraw exceeds the account balance the withdrawal operations should not be executed and an overdraft operation should be executed instead. It would be possible for the user thread to wait for the balance check Operation but that would block. Better would be to use the CompletionStage of the balance check Operation and submit the appropriate operation in a subsequent stage. But this is a common pattern and it would be better to encapsulate that pattern, which conditional OperationGroup does.

Not all Operations need to be executed in the order submitted. The most common example is a mass insert. The order in which the records are inserted doesn’t matter. A parallel OperationGroup gives the implementation the freedom to execute the Operations in any order. If some of the Operations have CompletionStage parameters this can be especially valuable.

OperationGroup also allows control of error response. By default if one Operation fails all subsequent Operations are skipped. That’s not always right. Consider the mass insert case. Just because one insert fails doesn’t mean they should all fail. An independent OperationGroup does this; the failure of one Operation has no impact on the execution of the rest.

As an OperationGroup is an Operation it must be submitted before its member Operations are executed. But this is not always the right thing. Again consider a mass insert. There is no point in waiting for the last insert Operation to be submitted before starting to execute the first. It needs to be possible to start execution of the member Operations before all the member Operations are submitted. So there must be a way to continue to submit member Operations after the OperationGroup is submitted.

The result of an Operation group depends on the results of its member Operations. Therefore an OperationGroup must know when all member Operations have been submitted because it cannot generate a result until all are completed. Since member Operations can be submitted after the OperationGroup has been submitted (see previous paragraph) there must be some other way to mark that all member Operations have been submitted. That’s the subject of this thread.

In the current version of the API there are two ways to submit an OperationGroup. OperationGroup.submit submits the OperationGroup and indicates that all member Operations have been submitted. OperationGroup.submitHoldingForMoreMembers does what it says; it submits the OperationGroup but indicates that more member Operations may be submitted. OperationGroup.releaseProhibitingMoreMembers indicates that all member Operations have been submitted. The problem with this is that failing to call releaseProhibitingMoreMembers after calling submitHoldingForMoreMembers will hang the execution queue. Since the OperationGroup is waiting for more member Operations it cannot produce a result. This does not fail fast and that’s a problem.

Rather than two ways to submit the change proposed in this thread is to disconnect submit from no-more-members. This change proposes that OperationGroup.submit enables execution of the member Operations but does not indicate all members have been submitted. That is always done separately by OperationGroup.close. I was concerned that programmers would too frequently forget to call close. After writing a lot more ADBA code I have concluded that this concern was misplaced. 

In many use cases try with resources is appropriate which insures close is called. Since programmers frequently use try with resources with OperationGroups, they will likely remember to call close in those use cases where try with resources is inappropriate. This also makes OperationGroup and Connection even more alike. Basically a Connection is just a special OperationGroup that has to predecessor Operations. Its first member Operation is executed immediately upon submission.

Does this help?

Douglas

> On Jul 20, 2018, at 12:54 AM, Mark Rotteveel <mark at lawinegevaar.nl> wrote:
> 
> I have to say I find these parts of ADBA a bit confusing, and I wonder if these kind of rules don't make it too complex and prone to incorrect use. Unfortunately, I still haven't had the time or energy to do a real deep dive into all parts of ADBA, so I can't really offer an insightful alternative yet.
> 
> For one, I don't really understand what an `OperationGroup` is supposed to represent, and how it is different from just a `Connection` (ie: why is it modeled separately), and why it should be 'submittable' at all. I also don't understand why you can do more things to it after a `submit()`, as submit sounds like a final operation to me. On the other hand, as the code below demonstrates, not accepting new operations would lead to usage errors. It sounds a bit like a transaction, but then `Transaction` is modeled separately.
> 
> Unfortunately, the javadoc on `OperationGroup` doesn't help much in understanding it.
> 
> Mark
> 
> On 2018-07-19 21:01, Douglas Surber wrote:
>> Yes, all the calls are non-blocking. The close doesn’t “terminate the
>> group”. It notifies the implementation that no more member Operations
>> will be added. It doesn’t stop anything from executing.
>> What I didn’t discuss but should have is the case where members are
>> added as a result of the execution of other Operations.
>>  OperationGroup grp = session.operationGroup();
>>  grp.submit();
>>  grp.rowOperation(selectSql)
>>    .collect(Collector.of( () -> null,
>>                                    (n, c) -> grp.rowCountOperation(insertSql)
>>                                                     .set(“param”,
>> c.at <http://c.at/>(“index”).get(Integer.class), AdbaType.INTEGER)
>>                                                     .submit(),
>>                                   (l, r) -> l,
>>                                   n -> { grp.close(); return n;}))
>>     .submit();
>> The above does an insert for each row selected. The OperationGroup
>> cannot be closed by try with resources because it would be closed
>> before all (any) of the inserts are added. The problem with this
>> pattern is failing to call grp.close() in the finalize method. If that
>> happens the OperationGroup will never be completed and execution will
>> hang.
>> This was the code pattern that motivated the ability to add members
>> after submit, but I don’t believe it will be the most common use case.
>> It’s unfortunate that it doesn’t fail overtly rather than hanging, but
>> the current design is no better in this regard. It just has more
>> explicit method names. I don’t think those method names carry their
>> weight compared to using try with resources.
>> You can write exactly the code above with a RowPublisherOperation. It
>> takes a few more lines of code to define the Subscriber, but it’s not
>> hard.
>> Douglas
>>> On Jul 19, 2018, at 11:11 AM, Dávid Karnok <akarnokd at gmail.com> wrote:
>>> Aren't those submit calls (supposed) asynchronous? In that case, won't the try-with-resources terminate the group before, most likely, anything could even happen?
>>> The scenario sounds like typical multi-valued asynchronous flow, for which a Flow.Publisher<Submission> source would be able to generate more submissions as well as indicate there won't be any further submissions.
>>> Douglas Surber <douglas.surber at oracle.com <mailto:douglas.surber at oracle.com>> ezt írta (időpont: 2018. júl. 19., Cs, 19:46):
>>> OperationGroup has a problem. It needs to be possible to add more member Operations to an OperationGroup after it is submitted. It is also necessary to know when all member Operations have been added as the OperationGroup cannot be completed until it knows no more member Operations will be added. Finally we want to minimize the likelyhood of Session execution hanging because an OperationGroup is waiting for more members when there are none.
>>> I approached this problem by specifying that by default no more member Operations can be added after the OperationGroup is submitted. For the case where members need to be added after submission there are two special methods.
>>> public Submission<T> submitHoldingForMoreMembers();
>>> public Submission<T> releaseProhibitingMoreMembers();
>>> The long, descriptive names are intentional. The goal is to clearly remind users that they must release the OperationGroup if they hold it. I’ve never liked this approach.
>>> I’d like to offer an alternative.
>>> - Remove submitHoldingForMoreMembers and releaseProhibitingMoreMembers.
>>> - Define OperationGroup to implement AutoCloseable.
>>> - Member Operations can be added to an open OperationGroup.
>>> - An OperationGroup will not be completed until after it is closed.
>>> - An OperationGroup must be submitted before it is closed.
>>> try (OperationGroup grp = session.operationGroup()) {
>>>  grp.rowCountOperation(insertSql)
>>>     .submit();
>>>  CompletionStage result = grp.submit().getCompletionStage();
>>>  grp.rowCountOperation(updateSql)
>>>     .submit();
>>> }
>>> A single usage pattern using try with resources seems much more reliable that the multiple usage patters with unfamiliar methods. Yes, the default case
>>>  OperationGroup grp = session.operationGroup();
>>>  grp.rowCountOperation(insertSql)
>>>     .submit();
>>>  grp.rowCountOperation(updateSql)
>>>     .submit();
>>>  return grp.submit().getCompletionStage();
>>> is perhaps simpler, but not by much.
>>> try (OperationGroup grp = session.operationGroup()) {
>>>  grp.rowCountOperation(insertSql)
>>>     .submit();
>>>  grp.rowCountOperation(updateSql)
>>>     .submit();
>>>  return grp.submit().getCompletionStage();
>>> }
>>> I’m starting to believe that the default case is not going to be the most common case. I think held OperationGroup are going to be much more common than I initially believed. A “held” group would look very similar to the “default” case.
>>> try (OperationGroup grp = session.operationGroup()) {
>>>  CompletionStage result = grp.submit().getCompletionStage();
>>>  grp.rowCountOperation(insertSql)
>>>     .submit();
>>>  grp.rowCountOperation(updateSql)
>>>     .submit();
>>>  return result;
>>> }
>>> The only difference is when the OperationGroup is submitted. If OperationGroups are always written using try with resources the forgetting to close one will be rare and easily noticed in code reviews. The tiny advantage of the default case, not having to call close, doesn’t seem to buy much compared to the complexity the current model adds to the held case. Uniform use of try with resources seems a much better approach.
>>> Thoughts?
>>> Douglas
>>> --
>>> Best regards,
>>> David Karnok