Effectively final

[Brian Goetz]

Yep, I think reflecting the parallelism in either the name or the type 
system is pretty much a requirement.  Trying to do "transparent 
parallelism" is going to be asking for trouble, just as "transparent 
remoting" was in a previous decade.  The name-based approaches that you 
suggest are a possibility, as is having "parallel views":

   collection.asParallel().forEach(block)

which is what PLinQ on .Net does.  But the user has to ask for it. 
(Scala made a choice similar to .Net.)

Cheers,
-Brian


On 8/19/2011 11:01 PM, Jim Mayer wrote:
> Hi Brian,
>
> Your post prompted a thought about the naming of library methods.  Since
> Java has its history and since it is easy to write non-functional code
> (call a method, the evil array hack, etc.), perhaps the methods that are
> intended to support fine grain parallelism should call that out in their
> names.
>
> I'm thinking of 'parallelReduce', ' pReduce', or something like that.  I
> suspect that lot of people will mentally model 'reduce' as a shorthand
> for a for loop and will assume sequential evaluation.
>
> Such a naming convention would also help avoid confusion with the
> 'Block' oriented methods that were discussed recently.  Finally, it
> would help avoid problems if mutation of locals was allowed in some
> future Java release.
>
> On the downside, the library would be more verbose.
>
> Thanks!
>
> -- Jim
>
> On Aug 18, 2011 12:46 PM, "Brian Goetz" <brian.goetz at oracle.com
> <mailto:brian.goetz at oracle.com>> wrote:
>  > This argument might work in another world, but ignores the reality when
>  > applied to Java.
>  >
>  > Yes, there *are* other programming models for concurrency that are less
>  > problematic than "everything is mutable and you have to use exclusion
>  > and/or partitioning to avoid shooting yourself in the foot."
>  >
>  > Yes, these models can be implemented in Java, provided that the
>  > programmer brings a sufficient degree of discipline to the situation and
>  > stays to the unenforced rules of the model. And in some of them,
>  > mutable local capture by lambdas would be convenient.
>  >
>  > Yes, people (myself included) have successfully written large, safe,
>  > concurrent systems in Java using this approach.
>  >
>  > Yes, it is a shame that Java chose the "mutation + locking" route to
>  > concurrency.
>  >
>  > But, if we allowed mutable capture, what is the most common idiom that
>  > would use it? It is virtually guaranteed it will be:
>  >
>  > int sum=0;
>  > x.forEach(#{ a -> sum += a.foo(); });
>  >
>  > And this is not one of those safer models that you hope will emerge into
>  > mainstream popularity in the future. This is making Java less safe, by
>  > creating a whole new category of serious hazards that didn't exist
>  > before -- data races on locals.
>  >
>  > Given that those safer models have not yet emerged into the mainstream,
>  > and this sort of unsafe willy-nilly mutation is definitely here, the
>  > tradeoff simply does not add up.
>  >
>  > I've already indicated we may choose to revisit this in the future,
>  > using an approach where the thread-confinement constraint can be
>  > enforced by the VM. But we will not be revisiting this now.
>  >
>  > I applaud your passion, but all of this is well-covered ground, and
>  > there's new ground to be covered before we're done. Let's move on.
>  >
>  >
>  > On 8/18/2011 12:00 PM, Tim Fox wrote:
>  >> On 18/08/11 16:30, Howard Lovatt wrote:
>  >>> On 16/08/2011, at 10:45 PM, Tim Fox<timvolpe at gmail.com
> <mailto:timvolpe at gmail.com>
>  >>> <mailto:timvolpe at gmail.com <mailto:timvolpe at gmail.com>>> wrote:
>  >>>
>  >>>> The point I was making was in response to your assertion that
>  >>>> inherently serial code will become less common as time goes on.
>  >>>> Actors frameworks (and other hybrid frameworks like node.x) are all
>  >>>> based around inherently serial code, and I don't see them dying
> out yet.
>  >>>
>  >>> I don't get what you are driving at, you have been posting that
>  >>> Lambdas should capture mutable local variables and have given Actors
>  >>> as an example of why that is necessary.
>  >> No, I didn't give actors as an example of why lambdas should mutate
>  >> local vars, I gave actors as a counterpoint to your assertion that
>  >> serial code will become less common as time came on (Actor code is all
>  >> serial and these frameworks are gaining in popularity)
>  >>
>  >> The point I was trying to make, in a nutshell, was this:
>  >>
>  >> Conventional wisdom Java concurrency involves letting many threads
>  >> execute your objects and using synchronization and what-not to make
>  >> those objects threadsafe. This is difficult. Concurrent programming
> is hard.
>  >>
>  >> There are other ways of making scalable concurrent systems that don't
>  >> involve having your code executed by multiple threads concurrently.
>  >> (Actor model is one example of this, there are others) This is very
>  >> desirable, since if you don't have to worry about making your code
>  >> threadsafe, you can open up to a legion of web developers who otherwise
>  >> would find concurrency too hard.
>  >>
>  >> My bet is the latter type of concurrent systems will eventually become
>  >> more popular. So... if in this future, developers will be writing
>  >> islands of single threaded coded, then it's safe to use lambdas that
>  >> mutate local vars in those islands since zero race conditions are
>  >> guaranteed.
>  >>
>  >> So far from being a throwback to the 50s, lambdas which mutate local
>  >> vars could be a very useful tool for the next generation of frameworks.
>  >> And if Java does not support such lambdas it will be sidelined as a good
>  >> programming language for these kinds of frameworks.
>  >>
>  >> Allowing such mutation would allow you to implement "node.js-like"
>  >> frameworks on the JVM. This is already possible in JS/Ruby/Groovy/etc
>  >> but is currently very clunky in Java because of lack of lambdas - but
>  >> even with Java 8 proposed lambdas it would still be clunky because of
>  >> the effectively-final rule.
>  >>
>  >>
>  >>> What I demonstrated, was that for Actors you wouldn't use Lambdas but
>  >>> classes and therefore your argument that Lambdas should capture local
>  >>> mutables because of Actors is non sequitur.
>  >>>
>  >>> Now you seem to be saying that you were talking about particular
>  >>> styles of code. Are you arguing about capturing mutables or not?
>  >>>
>  >>> PS Capturing a mutable variable in an Actor and passing it via message
>  >>> to another actor is problematic. Again indicating that Lambdas should
>  >>> not capture mutable state.
>  >>>
>  >>> PPS Also note that Scala has introduced map/reduce style collections
>  >>> (presumably because people prefer these to Actors - which it already
>  >>> has). Again map/reduce is best suited to not capturing mutable
> variables.
>  >>>
>  >>>> Quite the opposite, I see them getting more popular :)
>  >>>>
>  >>>> On 16/08/11 16:25, Howard Lovatt wrote:
>  >>>>> I don't see that the actor or FJ type frameworks are effected much,
>  >>>>> because the messages are immutable, e.g. a standard actor example:
>  >>>>>
>  >>>>> http://java.dzone.com/articles/scala-threadless-concurrent
>  >>>>>
>  >>>>> Recoded in Java would be something like this:
>  >>>>>
>  >>>>> public class Actor extends FJTask {
>  >>>>> private Queue messages ...
>  >>>>> public void accept( final Object message ) ...
>  >>>>> public Object acceptAndReply( final Object message ) ...
>  >>>>> public Future<Object> acceptAndReplyLater( final Object message ) ...
>  >>>>> protected void reply( final Object response ) ...
>  >>>>> protected Object getMessage() ...
>  >>>>> }
>  >>>>>
>  >>>>> |10.||public class Accumulator ||extends| |Actor {|
>  >>>>> |11.||public void run() {|
>  >>>>> |12.||int sum = ||0;|
>  >>>>> |13.||for(;;) {|
>  >>>>> |14.|final Object message = getMessage();
>  >>>>> |15.|if ( message instanceof |Accumulate ) { || sum +=
>  >>>>> ((Accumulate)message).n; ||}|
>  >>>>> |16.|else if ( message instanceof |Reset ) { sum = ||0; }|
>  >>>>> |17.|else if ( message instanceof Total ) { reply(sum); break;
>  >>>>> |18.||}|
>  >>>>> |19.||}|
>  >>>>> |20.||}|
>  >>>>> |21.||}|
>  >>>>>
>  >>>>>
>  >>>>> The lambdas might prove useful for the messages, that are immutable
>  >>>>> anyway, but the actors themselves would always be instances of a
>  >>>>> class because you need to be able to create more actors as the
>  >>>>> program runs and you can't do this with a lambda (or a closure many
>  >>>>> other language). As I said at the beginning I don't see that an
>  >>>>> Actor framework would change much one way or the other if Lambdas
>  >>>>> could mutate a variable or not.
>  >>>>>
>  >>>>> Cheers,
>  >>>>>
>  >>>>> -- Howard.
>  >>>>>
>  >>>>> Sent from my iPad
>  >>>>>
>  >>>>> On 16/08/2011, at 6:57 PM, Tim Fox<timvolpe at gmail.com
> <mailto:timvolpe at gmail.com>
>  >>>>> <mailto:timvolpe at gmail.com <mailto:timvolpe at gmail.com>>> wrote:
>  >>>>>
>  >>>>>> On 16/08/11 12:36, Howard Lovatt wrote:
>  >>>>>>> I think the correct decision has been made, inherently serial code
>  >>>>>>> will become less common as time goes on.
>  >>>>>> The growing popularity of actor model implementations (actors of
>  >>>>>> course
>  >>>>>> only allow serial execution of code in the actor), e.g. Erlang and
>  >>>>>> Akka,
>  >>>>>> and the growing popularity of languages which are inherently serial
>  >>>>>> (e.g. JavaScript) are a clear counterpoint to your argument.
> Also look
>  >>>>>> at webworkers, node.js. I could go on.
>  >>>>>>
>  >>>>>> I'd argue the current trend is quite opposite to what you see. Non
>  >>>>>> serial code will become unusual (because concurrency is hard), and
>  >>>>>> serial code will become the norm (because it's easier to code,
>  >>>>>> especially for the web developer masses). Systems will scale by
> having
>  >>>>>> multiple "islands" of serial code (e.g. multiple actors in the
>  >>>>>> actor model).
>  >>>>>>
>  >>>>>> Please note that just because all code is executed serially does not
>  >>>>>> mean you can't exploit parallelism. You can do fork/join type
> stuff by
>  >>>>>> sending messages to other "islands" and reassembling the results as
>  >>>>>> they
>  >>>>>> come back.
>  >>>>>>> Therefore optimizing new features for parallel execution is the
>  >>>>>>> correct path to take and also constant with Java's past of taking
>  >>>>>>> existing concepts into the mainstream. When Java was introduced
>  >>>>>>> the concentration on objects and garbage collection was not
>  >>>>>>> mainstream and was opposed by many. I think the same will happen
>  >>>>>>> with mutable data and serial execution and it is nice that Java is
>  >>>>>>> once again daring to be different and adhering to the mantra that
>  >>>>>>> less is more.
>  >>>>>>>
>  >>>>>>> Cheers,
>  >>>>>>>
>  >>>>>>> -- Howard.
>  >>>>>>>
>  >>>>>>> Sent from my iPad
>  >>>>>>>
>  >>>>>>> On 16/08/2011, at 4:56 PM, Steven Simpson<ss at comp.lancs.ac.uk
> <mailto:ss at comp.lancs.ac.uk>
>  >>>>>>> <mailto:ss at comp.lancs.ac.uk <mailto:ss at comp.lancs.ac.uk>>> wrote:
>  >>>>>>>
>  >>>>>>>> On 16/08/11 01:51, Stephen Colebourne wrote:
>  >>>>>>>>
>  >>>>>>>> [snip: lots of syntax options for permitting mutable locals]
>  >>>>>>>>> int #total = 0;
>  >>>>>>>>> list.apply(#{item -> total += item});
>  >>>>>>>>>
>  >>>>>>>>> ie. a way to introduce a local variable that can be managed
> safely.
>  >>>>>>>> For the simple example given, you could translate 'total' into an
>  >>>>>>>> AtomicInteger, but if there are other variables to be
> accessed, you'd
>  >>>>>>>> have to box them together, and make the whole lambda synchronize
>  >>>>>>>> on it,
>  >>>>>>>> or at least from the first use of the box to the last. Trying to
>  >>>>>>>> patch
>  >>>>>>>> the call site like this doesn't seem to be particularly
>  >>>>>>>> promising. The
>  >>>>>>>> alternative is to require List.apply to make additional
>  >>>>>>>> guarantees about
>  >>>>>>>> how it executes the lambda.
>  >>>>>>>>
>  >>>>>>>> Would it not be better to let List.apply get on with its potential
>  >>>>>>>> parallelism, and define other methods that do make extra
> guarantees,
>  >>>>>>>> e.g. that the lambda will be executed serially, or even on the
>  >>>>>>>> caller's
>  >>>>>>>> thread?
>  >>>>>>>>
>  >>>>>>>> Tim's cases include, for example, setTimeout(int, Runnable),
>  >>>>>>>> which must
>  >>>>>>>> make such a guarantee, if only informally in its documentation.
>  >>>>>>>> To be
>  >>>>>>>> more formal:
>  >>>>>>>>
>  >>>>>>>> * Declare setTimeout(int, @Callback Runnable).
>  >>>>>>>> * When a lambda is assigned to a @Callback Runnable, allow the
>  >>>>>>>> lambda body to mutate locals (without error or warning).
>  >>>>>>>> * Don't permit a @Callback Runnable (which is tainted) to be
>  >>>>>>>> assigned to a plain Runnable (without error or warning).
>  >>>>>>>>
>  >>>>>>>> This way, the likes of List.apply don't have to make any
> guarantees,
>  >>>>>>>> requiring the caller to make corresponding ones (automatically
>  >>>>>>>> achieved
>  >>>>>>>> by not being generally allowed to mutate locals). Meanwhile,
>  >>>>>>>> setTimout
>  >>>>>>>> makes additional guarantees, to the convenience of the caller,
> who is
>  >>>>>>>> specially permitted to mutate locals.
>  >>>>>>>>
>  >>>>>>>> Cheers,
>  >>>>>>>>
>  >>>>>>>> Steven
>  >>>>>>>>
>  >>>>>>
>  >>>>>>
>  >>>>
>  >>
>  >>
>  >