stream.parallel().limit() on large streams

[Paul Sandoz]

On Oct 6, 2013, at 4:26 PM, Arne Siegel <v.a.ammodytes at googlemail.com> wrote:

> Hi Brian,
> 
> tried out both of your propositions, and indeed they do a good job on the
> order-independent scenario I was looking into:
> 
>            IntStream.range(0, maxToGenerate)
>                     .parallel()
>                     .unordered()
>                     .mapToObj(i -> generatorFunction.get())
>                     .filter(condition)
>                     .limit(needed)
>                     .forEach(action);
> 
> ==> runs quite good, but is on average a few percent less efficient than
> the implementation you cited in your mail;
> 
>            Stream.generate(generatorFunction)
>                  .parallel()
>                  .limit(maxToGenerate)
>                  .filter(condition)
>                  .limit(needed)
>                  .forEach(action);
> 
> ==> this is a really concise expression of the program's intents, and it
> performs equally well compared to the cited implementation. Nice!
> 

FWIW the former should be slightly more efficient than the latter since there is only one limit operation.


> One note: it's a complete different picture for scenarios where order is
> important. These need other implementations, and for these I found
> - parallel streams running much slower than serial streams and serial loop;
> - the ExecutorService-based approach running much faster than serial
> implementations, most of the time.
> 

How do you preserve order with the ExecutorService-based approach?

Paul