Stream.concat with varagrs

Éamonn McManus emcmanus at google.com
Wed Sep 17 20:21:18 UTC 2025


Guava has a Streams.concat
<https://github.com/google/guava/blob/e9ea5a982cad06ebd223ec6fdb5294eeb18654f6/guava/src/com/google/common/collect/Streams.java#L187>
method
that may suggest implementation possibilities.

On Wed, 17 Sept 2025 at 13:16, Olexandr Rotan <rotanolexandr842 at gmail.com>
wrote:

> So i have played around a bit and managed to come up with some
> implementation based on array of streams, you can find it here:
> https://github.com/Evemose/nconcat/blob/master/src/main/java/nconcat/NConcatSpliterator.java
>
> I have also added a small benchmark to the project, and the numbers are:
>
> Benchmark                                              (streamCount)  Mode
>  Cnt       Score       Error  Units
> NConcatBenchmark.nConcatFindFirst                                  4  avgt
>   10     131.616 �    15.474  ns/op
> NConcatBenchmark.nConcatFindFirst                                  8  avgt
>   10     187.929 �     6.544  ns/op
> NConcatBenchmark.nConcatFindFirst                                 16  avgt
>   10     322.342 �     6.940  ns/op
> NConcatBenchmark.nConcatFindFirst                                 32  avgt
>   10     659.856 �    85.509  ns/op
> NConcatBenchmark.nConcatFindFirst                                 64  avgt
>   10    1214.133 �    22.156  ns/op
> NConcatBenchmark.nConcatMethod                                     4  avgt
>   10    1910.150 �    25.269  ns/op
> NConcatBenchmark.nConcatMethod                                     8  avgt
>   10    3865.364 �   112.536  ns/op
> NConcatBenchmark.nConcatMethod                                    16  avgt
>   10    7743.097 �    74.655  ns/op
> NConcatBenchmark.nConcatMethod                                    32  avgt
>   10   15840.551 �   440.659  ns/op
> NConcatBenchmark.nConcatMethod                                    64  avgt
>   10   32891.336 �  1122.630  ns/op
> NConcatBenchmark.nConcatToListWithFilter                           4  avgt
>   10    9527.120 �   376.325  ns/op
> NConcatBenchmark.nConcatToListWithFilter                           8  avgt
>   10   20260.027 �   552.444  ns/op
> NConcatBenchmark.nConcatToListWithFilter                          16  avgt
>   10   44724.856 �  5040.069  ns/op
> NConcatBenchmark.nConcatToListWithFilter                          32  avgt
>   10   82577.518 �  2050.955  ns/op
> NConcatBenchmark.nConcatToListWithFilter                          64  avgt
>   10  181460.219 � 20809.669  ns/op
> NConcatBenchmark.nconcatToList                                     4  avgt
>   10    9268.814 �   712.883  ns/op
> NConcatBenchmark.nconcatToList                                     8  avgt
>   10   18164.147 �   786.803  ns/op
> NConcatBenchmark.nconcatToList                                    16  avgt
>   10   35146.891 �   966.871  ns/op
> NConcatBenchmark.nconcatToList                                    32  avgt
>   10   68944.262 �  5321.730  ns/op
> NConcatBenchmark.nconcatToList                                    64  avgt
>   10  136845.984 �  3491.562  ns/op
> NConcatBenchmark.standardStreamConcat                              4  avgt
>   10    1951.522 �    85.130  ns/op
> NConcatBenchmark.standardStreamConcat                              8  avgt
>   10    3990.410 �   190.517  ns/op
> NConcatBenchmark.standardStreamConcat                             16  avgt
>   10    8599.869 �   685.878  ns/op
> NConcatBenchmark.standardStreamConcat                             32  avgt
>   10   17923.603 �   361.874  ns/op
> NConcatBenchmark.standardStreamConcat                             64  avgt
>   10   46797.408 �  4458.069  ns/op
> NConcatBenchmark.standardStreamConcatFindFirst                     4  avgt
>   10     125.192 �     3.123  ns/op
> NConcatBenchmark.standardStreamConcatFindFirst                     8  avgt
>   10     303.791 �     8.670  ns/op
> NConcatBenchmark.standardStreamConcatFindFirst                    16  avgt
>   10     907.429 �    52.620  ns/op
> NConcatBenchmark.standardStreamConcatFindFirst                    32  avgt
>   10    2964.749 �   320.141  ns/op
> NConcatBenchmark.standardStreamConcatFindFirst                    64  avgt
>   10   11749.653 �   189.300  ns/op
> NConcatBenchmark.standardStreamConcatToList                        4  avgt
>   10    7059.642 �   740.735  ns/op
> NConcatBenchmark.standardStreamConcatToList                        8  avgt
>   10   13714.980 �   250.208  ns/op
> NConcatBenchmark.standardStreamConcatToList                       16  avgt
>   10   27028.052 �   565.047  ns/op
> NConcatBenchmark.standardStreamConcatToList                       32  avgt
>   10   53537.731 �   853.363  ns/op
> NConcatBenchmark.standardStreamConcatToList                       64  avgt
>   10  105847.755 �  3179.918  ns/op
> NConcatBenchmark.standardStreamConcatToListWithFilter              4  avgt
>   10    9736.527 �   154.817  ns/op
> NConcatBenchmark.standardStreamConcatToListWithFilter              8  avgt
>   10   20607.061 �   713.083  ns/op
> NConcatBenchmark.standardStreamConcatToListWithFilter             16  avgt
>   10   41241.199 �  1171.672  ns/op
> NConcatBenchmark.standardStreamConcatToListWithFilter             32  avgt
>   10   83029.244 �  1843.176  ns/op
> NConcatBenchmark.standardStreamConcatToListWithFilter             64  avgt
>   10  182349.009 � 11282.832  ns/op
>
> Basically, the conclusion is following (guilty of using AI for
> summarizing):
>
> The comprehensive benchmarks reveal that *NConcat significantly
>> outperforms the standard library for processing-intensive operations*
>> while trailing in simple collection scenarios. For short-circuit operations
>> like findFirst(), NConcat delivers 38-90% better performance as stream
>> count increases, reaching nearly 10x faster execution at 64 streams due to
>> superior scaling (19ns/stream vs 184ns/stream). Full traversal operations
>> like forEach consistently favor NConcat by 2-30%, with the advantage
>> growing at scale. However, simple collection operations (toList())
>> consistently run 22-24% faster with the standard library across all stream
>> counts.
>
>
>
> I have tried multiple approaches to optimize toList with know size of all
> sub-streams (which is clearly the reason why standard implementation wins
> here), and am sure that there is still plenty of room for improvement,
> especially in parallel, but the takeaway is, even a naive implementation
> like mine could bring a significant performance improvement to the table in
> early short-circuiting and full traversal cases that do not depend on size
> of the spliterator.
>
> Besides the performance part, of course, the most significant advantage of
> my proposal, as I think, is still developer experience, both reading and
> writing stream code.
>
> Please let me know your thoughts on the results of prototype and possible
> ways forward.
>
> Best regards
>
> On Wed, Sep 17, 2025 at 6:04 PM Olexandr Rotan <rotanolexandr842 at gmail.com>
> wrote:
>
>> Hello everyone! Thanks for your responses
>>
>> I will start of by answering to Viktor
>>
>> I guess a "simple" implementation of an N-ary concat could work, but it
>>> would have performance implications (think a recursive use of
>>> Stream.concat())
>>
>>
>> I too find just the addition of small reduction-performing sugar methods
>> rather unsatisfactory and most certainly not bringing enough value to be
>> considered a valuable addition. Moreover, I have not checked it myself, but
>> I would dare to guess that popular utility libraries such as Guava or
>> Apache Commons already provide this sort of functionality in their utility
>> classes. Though, if this method could bring some significant performance
>> benefits, I think it may be a valuable candidate to consider. Though, to me
>> as a user, the main value would be uniformity of the API and ease of use
>> and read. The main reason I am writing about this in the first place is the
>> unintuitive inconsistency with many other static methods-creators that
>> happily accept varargs
>>
>> I may play around with this spliterator code you have linked to to see if
>> I could make it generalized for arrays of streams
>>
>> Now, answering to Pavel
>>
>> Is it such a useful use case, though? I mean, it's no different from
>>> SequenceInputStream(...) or Math.min/max for that matter. I very rarely
>>> have to do Math.min(a, Math(min(b, c)) or some such.
>>
>>
>> I certainly see your point, but I would dare to say that most
>> applications rely on the streams much more than SequenceInputStream and
>> Math classes, and their lookalikes. Stream.concat is primarily a way to
>> merge a few datasource outputs into one, for later uniform processing,
>> which, in the nutshell, is one of the most common tasks in data-centric
>> applications. Of course, not every such use case has characteristics that
>> incline developers to use Stream.concat, such as combination of Stream.of
>> and Collection.stream() sources, and even if they do, not every case that
>> fits previous requirement requires to merge more than 2 sources. However,
>> for mid-to-large scale apps, for which java is known the most, I would say
>> it's fairly common. I went over our codebase and found that there were at
>> least 10+ usages of concat, and a few of them followed this kinda ugly
>> pattern of nested concates.
>>
>> Separately, it's not just one method. Consider that `concat` is also
>>> implemented in specialized streams such as IntStream, DoubleStream, and
>>> LongStream.
>>
>>
>> This is unfortunate, but I would dare to say that once Reference
>> spliterrator is implemented, others may also be derived by analogy fairly
>> quickly
>>
>> And last but not least, answering Daniel
>>
>> Not immediately obvious but you can create a Stream<Stream<T>> using
>>> Stream.of and reduce that using Stream::concat to obtain a Stream<T>.
>>
>> Something along those lines:
>>
>> ```
>>> var stream = Stream.of(Stream.of(1,2,3), Stream.of(4), Stream.of(5, 6,
>>> 7, 8)).reduce(Stream.empty(), Stream::concat, Stream::concat);
>>
>>
>> This is what I meant by "reduction-like" implementation, which is fairly
>> straightforward, but just from the looks of it, one could assume that this
>> solution will surely have performance consequences, even if using  flatmap
>> insead of reduce. Not sure though, how often people would want to use such
>> approach on the array of streams huge enough for the performance difference
>> to be noticable, though I would assume that there is a non-linear scale of
>> consumed time and resources from the length of streams array due to the
>> implementation of concat method.
>>
>> Nevertheless, this is an acceptable workaround for such cases, even
>> though not the most readable one. Even if this approach is accepted as
>> sufficient for such cases of n-sized array of streams merging, It would
>> probably make some sense to put note about it in the docs of the concat
>> method. Though, not having concat(Stream..) overload would still remain
>> unintuitive for many developers, including me
>>
>> Thanks everybody for the answers again
>>
>> Best regards
>>
>> On Wed, Sep 17, 2025 at 5:15 PM Pavel Rappo <pavel.rappo at gmail.com>
>> wrote:
>>
>>> >  this would be a great quality of life improvement
>>>
>>> Is it such a useful use case, though? I mean, it's no different from
>>> SequenceInputStream(...) or Math.min/max for that matter. I very
>>> rarely have to do Math.min(a, Math(min(b, c)) or some such. And those
>>> methods predate streams API by more than a decade.
>>>
>>> Separately, it's not just one method. Consider that `concat` is also
>>> implemented in specialized streams such as IntStream, DoubleStream,
>>> and LongStream.
>>>
>>> On Wed, Sep 17, 2025 at 2:58 PM Olexandr Rotan
>>> <rotanolexandr842 at gmail.com> wrote:
>>> >
>>> > Greetings to everyone on the list.
>>> >
>>> > When working on some routine tasks recently, I have encountered a,
>>> seemingly to me, strange decision in design of Stream.concat method,
>>> specifically the fact that it accepts exactly two streams. My concrete
>>> example was something along the lines of
>>> >
>>> > var studentIds = ...;
>>> > var teacherIds = ...;
>>> > var partnerIds = ...;
>>> >
>>> > return Stream.concat(
>>> >     studentIds.stream(),
>>> >     teacherIds.stream(),
>>> >     partnerIds.stream() // oops, this one doesn't work
>>> > )
>>> >
>>> > so I had to transform concat to a rather ugly
>>> > Stream.concat(
>>> >     studentIds.stream(),
>>> >     Stream.concat(
>>> >         teacherIds.stream(),
>>> >         partnerIds.stream()
>>> >     )
>>> > )
>>> >
>>> > Later on I had to add 4th stream of a single element (Stream.of), and
>>> this one became even more ugly
>>> >
>>> > When I first wrote third argument to concat and saw that IDE
>>> highlights it as error, I was very surprised. This design seems
>>> inconsistent not only with the whole java stdlib, but even with Stream.of
>>> static method of the same class. Is there any particular reason why concat
>>> takes exactly to arguments?
>>> >
>>> > I would say that, even if just in a form of sugar method that just
>>> does reduce on array (varagrs) of streams, this would be a great quality of
>>> life improvement, but I'm sure there also may be some room for performance
>>> improvement.
>>> >
>>> > Of course, there are workarounds like Stream.of + flatmap, but:
>>> >
>>> > 1. It gets messy when trying to concat streams of literal elements set
>>> (Stream.of) and streams of collections or arrays
>>> > 2. It certainly has significant performance overhead
>>> > 3. It still doesn't explain absence of varagrs overload of concat
>>> >
>>> > So, once again, is there any particular reason to restrict arguments
>>> list to exactly two streams? If not, I would be happy to contribute
>>> Stream.concat(Stream... streams) overload.
>>> >
>>> > Best regards
>>> >
>>> >
>>> >
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/core-libs-dev/attachments/20250917/68177d56/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5281 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.openjdk.org/pipermail/core-libs-dev/attachments/20250917/68177d56/smime-0001.p7s>


More information about the core-libs-dev mailing list