accumulate locally and mutatively but combine concurrently (a use case)

[John Rose]

On May 9, 2013, at 11:50 PM, Howard Lovatt <howard.lovatt at gmail.com> wrote:

> Ideally you would want to expand your example further, for a Stream<I> you would like a method:
> ...

Yes, your API frees up all the naturally independent type variables.  (The last step, toOutput, is pretty superficial, since a user can always tack on that last step after the reduction operation in the pipeline.  The toCombine step is the step I'm trying to discuss here.)

Here's what I hope will happen:

I'm hoping that someone will point out how the current APIs (including the simplified Collector design) accommodates my proposed use case.

Or else that someone will point out that that my use case is inferior to some other way of running a multi-core system that the current APIs accommodate.

My current "take" on mixing side effects and concurrency is that you want to keep your reads and your writes fenced and separated away from each other.  Add to that a reluctance to copy (or "marshal", for the GPU world), and you get a need for some sort of type or state transition.  (Hence the freezing step, which could also be compared with a file system "close, rewind, reopen for reading".  In fact, "rewind" is a nice metaphor, better than "freeze", for the change of data direction.)

So a third possible outcome would be, "yes, we need to support that rewind hook, and it should be an optional component of Collector".

— John

P.S.  If outcome #3 is the case, I understand from Brian that there are some nasty downstream problems associated with such a rewind hook, such as adjusting the end-type of a nested groupBy map, from Map<K, Map<L, AccumulatingAggregate>> to Map<K, Map<L, CollectibleChunk>> (given a rewind from AccumulatingAggregate to CollectibleChunk).  If I understand those concerns, I could speculate further on what they imply, from the point of view of (fortran) elemental functions, but will hold off for now.

http://fortranwiki.org/fortran/show/elemental