StreamOpFlag.* and OutOfMemoryError when going parall

[Paul Sandoz]

On Dec 6, 2012, at 4:23 AM, Brian Goetz <brian.goetz at Oracle.COM> wrote:

>> As Brian pointed out, because the limit() isn't fully lazy, this
>> results in an OutOfMemoryError because actually calculating _all_
>> prime numbers this way and _then_ limit the result set isn't really an
>> option. Now I wonder whether I could build a better limit() version
>> myself...
> 
> No, that's not exactly right.  The decomposition is left-spine (so it 
> should compute the leftmost ones earlier) and it cancels once its found 
> enough.  But it may well compute more than 1000.
> 
> It sounds like something else is wrong.   It shouldn't OOME on such a 
> small problem.
> 

It can just be limit(1) and it will still fail with an OOME.

The problem is the isLeftSpine() will always fail for the shape of tree generated by Iterators.spliterator() since nodes in the right spine will never complete.

I think we will need to put some logic into the leaf nodes such that on completion of a leaf it looks at all the siblings to the left of itself and short-cicuits and forces completion of the parent if a match of the slice/offset is found.

Also even though we don't know the complete size we may know sizes at certain depths, which could be leveraged to enable better slice calculations and avoid buffering on skipping.

Paul.