Parallel-safe lambdas

[John Rose]

On Feb 11, 2010, at 3:44 AM, Rémi Forax wrote:

> Perhaps the best option is to maintain the current status quo,
> local variable should be final, fields can be mutated.

Yes, that could work, especially with an "as if final" rule to make coding easier.

The trade-off is (and always has been) between transparency hazards and MP hazards.  A high-risk construct (opinions differ here!) should require an explicit programmer declaration before it becomes transparent into a nested scope (inner class and/or closure).

Unrestricted, implicit sharing of mutable local variables has *always been* (IMO) a bad idea, and it certainly gets worse every year as core counts grow.  So there's even less reason to make these transparent today, than there was in Java 1.1.

But, an explicit @Shared annotation, for when the user is willing to promise that shared state is OK, is (IMO) a good compromise, adding a transparency hazard to defend against an MP hazard.  (Maybe Doug will disagree about how to manage this trade-off?)

> lambda is already a restricted form of anonymous class.
> A lambda doesn't define a class, unlike anonymous class,
> so mutating something is less easy.

This thought made be realize that there's another design point here which needs discussion.  (Skip ahead to points 0..4 below for the quick proposal.)

Simple, transparent access to immutable names (fields, locals) in uplevel scopes should be a no-brainer, even given the current emphasis on MP safety.

Access to uplevel mutable names could be treated as another matter.

(In theory, methods needs special treatment too, since they can present MP hazards.  I wish we had pure methods!  But that would entail defining a pure sublanguage with its own subset type system and loopholes for interoperability.  So the reality is we need to rely on "cultural" conventions to guide programmers to use methods safely when doing MP programming.  The language as it stands can have only limited interaction with such conventions.)

Inner classes share access to the mutable fields of their enclosing classes.  It makes sense (IMO) that an inner object, since it is lexically a part of the outer object, has the same easy, unqualified access to outer fields as the outer object itself.  Programmers who use objects know that mutable fields are a risk to parallelism.  The programming culture deals adequately with mutable fields, for now.  (Still, we should invent pure objects and add them to Java someday, since hardware is changing.  Can't do that today.)

Now, the unqualified sharing of fields, enjoyed by inner classes, does not necessarily need to be true for closures.  Just as with @Shared, we could add an intentional transparency hazard whose resolution requires extra explicitness, for access to mutable object fields.  We could (and this is just another tweak to the "syntax dial", really) require that references to mutable fields of enclosing objects must be qualified with a suitable 'this'.  (That name, as many have pointed out, is not mutable.)

A safe and still useful tradeoff regarding uplevel members (fields, methods) could be this:
 0. no restrictions on qualified names
 1. allow unqualified references to immutable variables of all sorts
   ('unqualified' = 'simple name, no dots')
   ('immutable' = 'final, or a local implicitly markable as final')
 2. allow unqualified references to uplevel variables (all sorts) if they are marked @Shared
 3. allow unqualified references to uplevel methods if they are marked @Shared
 4. allow unqualified references to uplevel methods if they are "pure" (marked @Pure)

Point 0 lets the user of an API get to it always, by using the dot '.' syntax.  No change to selection expression semantics.

Point 1 lets demonstrably "safe" names be fully transparent.

Points 2,3,4 let the definer of an API (who can make general guarantees) contract with the user about MP safety, and grant easy access.  When coding, put @Shared on things (locals, fields, methods) you really mean to share (potentially) across execution contexts.

Point 4 may not be motivated by present-day use cases; I include it as a "taste" of what the pure sublanguage might feel like.  Pure things are a subset of things which are meant to be shared.  ...A most interesting subset.

The theory could be as follows:  Inner objects "inherit" complete access to the members of their enclosing objects.  They can handle this because they are objects, and can easily incorporate synchronization patterns.  (Although there is the well-known, and recently emphasized, problem of synchronizing on the "wrong this", a direct result of the decision to give every object, no matter how small, a monitor.)  But closures are not (not necessarily, and should not be IMO) first-class objects, which assume the whole setup of private methods, fields, 'this', etc.  So a closure does not necessarily have "full rights" to the fields of enclosing objects:  We can grant such rights based on the hazard trade-off mentioned above.

A lot of this is trying to guess Doug Lea's mind on what he perceives (based on deep experience) leads to racy code.  Doug, does any of this help with those concerns?

-- John