Re: RFR[8238286]: 'Add new flatMap stream operation that is more amenable to pushing’

[forax at univ-mlv.fr]

----- Mail original -----
> De: "Paul Sandoz" <paul.sandoz at oracle.com>
> À: "Remi Forax" <forax at univ-mlv.fr>
> Cc: "Peter Levart" <peter.levart at gmail.com>, "Patrick Concannon" <patrick.concannon at oracle.com>, "Julia Boes"
> <julia.boes at oracle.com>, "core-libs-dev" <core-libs-dev at openjdk.java.net>
> Envoyé: Mardi 7 Juillet 2020 21:04:09
> Objet: Re: RFR[8238286]: 'Add new flatMap stream operation that is more amenable to pushing’

> Hi Remi,
> 
> I stress again that the short-circuiting optimizations are an implementation
> detail that are partially applied. The behavior is not specified. It's already
> inconsistent.
> 
> We might be able to improve on certain cases, as in Peter’s example, and apply
> an optimization consistently for flatMap and mapMulti. But I would prefer we
> focus on what we have now and consider that later if need be.

I don't think an incremental evolution works here,
because apart from people trying to understand streams, you have also a whole ecosystem on top of the stream API like by example IDEs that can refactor, provide visual debugging, etc of streams.

> 
> Paul.

Rémi

> 
>> On Jul 7, 2020, at 11:20 AM, forax at univ-mlv.fr wrote:
>> 
>> At the same time,
>> mapMulti and flatMap are two sides of the same coins,
>> flatMap uses a pull API while mapMulti uses a push API.
>> 
>> so having mapMulti and flatMap doing cancellation differently is inconsistent.
>> 
>> Consistency is a good way to avoid to expect that all operations do
>> short-circuiting optimizations.
>> 
>> Rémi
>> 
>> ----- Mail original -----
>>> De: "Paul Sandoz" <paul.sandoz at oracle.com>
>>> À: "Peter Levart" <peter.levart at gmail.com>
>>> Cc: "Patrick Concannon" <patrick.concannon at oracle.com>, "Remi Forax"
>>> <forax at univ-mlv.fr>, "Julia Boes"
>>> <julia.boes at oracle.com>, "core-libs-dev" <core-libs-dev at openjdk.java.net>
>>> Envoyé: Mardi 7 Juillet 2020 20:09:01
>>> Objet: Re: RFR[8238286]: 'Add new flatMap stream operation that is more amenable
>>> to pushing’
>> 
>>> Hi Peter,
>>> 
>>> You guessed correctly about the motivation from the API perspective.
>>> 
>>> I would like to stress that short-circuiting optimizations are an implementation
>>> detail that are only partially applied. Surfacing the notion of cooperative
>>> cancellation for one operation has quite a cost, and will likely increase the
>>> expectation that it applies in all scenarios.
>>> 
>>> My sense is your example is likely more of on the edge than common. I believe
>>> it's simple to implement but would prefer to hold off and focus the current
>>> implementation and integrate, then we could reconsider afterwards if need be.
>>> 
>>> Paul.
>>> 
>>>> On Jul 5, 2020, at 1:44 AM, Peter Levart <peter.levart at gmail.com> wrote:
>>>> 
>>>> Hi Patric, Julia,
>>>> 
>>>> On 7/2/20 6:45 PM, Patrick Concannon wrote:
>>>>> http://cr.openjdk.java.net/~pconcannon/8238286/webrevs/webrev.03/
>>>>> <http://cr.openjdk.java.net/~pconcannon/8238286/webrevs/webrev.03/>
>>>> 
>>>> 
>>>> Then I'd like to discuss cancelation (shortcut Streams). Currently mapMulti does
>>>> not do anything to optimize processing if the downstream decides it does not
>>>> need more elements but just keeps pushing. This is a decision I guess to keep
>>>> the API surface clean and simple. All short-cutting ops or even intermediate
>>>> (like limit) must therefore ignore the surplus elements that are emitted by
>>>> mapMulti and they do so. If the intermediate operations up-stream mapMulti are
>>>> respecting the cancelation, this results in at most one invocation of mapMulti
>>>> function producing surplus elements, but if they don't, then multiple
>>>> invocations of mapMulti function is producing surplus elements.
>>>> 
>>>> For example:
>>>> 
>>>> 
>>>>    someStream
>>>> 
>>>>        .mapMulti((e, sink) -> { .... sink.accept(x); ... })
>>>> 
>>>>        .filter(x -> ...)
>>>> 
>>>>        .mapMulti((x, sink) -> { ... sink.accept(y); ...})
>>>> 
>>>>        .limit(10)
>>>> 
>>>>        ...
>>>> 
>>>> 
>>>> Here the 1st mapMulti emits elements and each of them is filtered and then maybe
>>>> passed to 2nd mapMulti. the 2nd mapMulti could skip calling its function if the
>>>> downstream limit(10) has already passed 10 elements.
>>>> 
>>>> 
>>>> WDYT?
>>>> 
>>>> 
>>>> Regards, Peter