Yield as contextual keyword (was: Call for bikeshed -- break replacement in expression switch)

[Tagir Valeev]

Hello! Answering myself

> > The first token in a YieldStatement production is always preceded by one of these separator tokens: ;, {, }, ), or ->.
>
> Seems I'm missing something. Could you please illustrate in which case
> YieldStatement could be preceded by ')'?

Nevermind. if(foo) yield bar; is a good example. Other my points still apply.

> Also what about '->'? In
> lambda '->' is followed by an expression or block, but not a
> statement. In switch '->' is followed by block, throw or expression
> plus semicolon. Also could YieldStatement be preceded by ':' in old
> switch format? E.g.
>
> System.out.println(switch(0) { default: yield 1; }); // seems legit
>
> Also sections 16.1.7 and 16.1.8 are named identically. Probably
> there's some mistake.
>
> With best regards,
> Tagir Valeev.
>
> On Fri, May 24, 2019 at 7:25 PM Gavin Bierman <gavin.bierman at oracle.com> wrote:
> >
> > A draft spec including the compromise strategy below is available at:
> >
> > http://cr.openjdk.java.net/~gbierman/jep354-jls-20190524.html
> >
> > Comments welcomed!
> > Gavin
> >
> > On 22 May 2019, at 17:45, Brian Goetz <brian.goetz at oracle.com> wrote:
> >
> > We’ve been drilling into the spec and implementation of yield as a contextual keyword.  We have three possible strategies, all of which are specifiable and implementable, but with tradeoffs.
> >
> > The “dumb strategy” would be to say that `yield` is a keyword when it appears in the first position of a statement production (e.g., after an open brace or a semicolon.). This is simple to spec, and simple to implement, but it doesn’t so do well with variables named `yield`:
> >
> >    yield++;
> >    yield = 3;
> >    if (foo)
> >        yield += 3;
> >    yield[3] = 4;
> >
> > The “smart strategy” says that `yield` is a keyword only within the context of the YieldStatement production; the rest of the time it is an identifier.  This is also simple to spec, and does the right thing in all unambiguous cases, but requires unbounded lookahead, which compiler implementations may not like.  The one ambiguous case is
> >
> >    yield(e)
> >
> > which would match both YieldStatement and ExpressionStatement, and here we bias towards YieldStatement.  Naked yield() invocations can qualify the invocation:
> >
> >    this.yield(3)
> >    Thread.yield(4)
> >
> > The “compromise” strategy is like the smart strategy, except that it trades fixed lookahead for missing a few more method invocation cases.  Here, we look at the tokens that follow the identifier yield, and use those to determine whether to classify yield as a keyword or identifier.  (We’d choose identifier if it is an assignment op (=, +=, etc), left-bracket, dot, and a few others, plus a few two-token sequences (e.g., ++ and then semicolon), which is lookahead(2).
> >
> > The main difference between the compromise strategy and the smart strategy is the handling of method invocations that are not unary:
> >
> >    yield(3, 4)
> >
> > In the smart strategy, we’d figure out that this is a method call; in the compromise strategy, we’d require qualification just as we do with the unary method.
> >
> > The compromise strategy misses some cases we could parse unambiguously, but also offers a simpler user model: always qualify invocations of methods called yield when used as expression statements.  And it offers the better lookup behavior, which will make life easier for IDEs.
> >
> > So my recommendation here is the compromise strategy.
> >
> > On May 21, 2019, at 10:50 AM, Tagir Valeev <amaembo at gmail.com> wrote:
> >
> > I discussed this with colleagues and can confirm that for IntelliJ
> > IDEA parser it will be no problem to always consider yield as a
> > statement. At least it's much easier than to consider it as a
> > statement inside switchy blocks only.
> >
> > With best regards,
> > Tagir Valeev.
> >
> > On Tue, May 21, 2019 at 12:38 PM Tagir Valeev <amaembo at gmail.com> wrote:
> >
> >
> > So does this (option B plus your No) mean that IDEs would tend to color "yield" as a keyword (at the beginning of a statement) even if followed by "("?
> >
> >
> > My "No" was mostly against options C and D where symbol resolution
> > affects the parse tree. Sorry if it wasn't clear from my message. When
> > the context for the parsing is available inside the same Java file,
> > it's usually ok. See the 'var' restricted keyword:
> >
> > var var = 10; // first is highlighted as type, second as local variable
> > var = 20; // var is highlighted as local variable, despite it's at the
> > beginning of a statement.
> > var(1); // var is highlighted as a method call, despite it's at the
> > beginning of a statement.
> >
> > We have no very big problems parsing this.
> >
> > With best regards,
> > Tagir Valeev.
> >
> > On Tue, May 21, 2019 at 2:52 AM John Rose <john.r.rose at oracle.com> wrote:
> >
> >
> > On May 20, 2019, at 8:24 AM, Tagir Valeev <amaembo at gmail.com> wrote:
> >
> >
> > Assuming that we agreed on 'yield' the option B seems the most attractive. A big No to context-specific parse tree. It's a complete pain to IDEs. Don't forget that IDE often deals with incomplete code, missing dependencies, etc., and still needs to provide reasonable highlighting and completion. Imagine that 'yield' method is available via import static Foo.* or superclass. In this case we don't want to look into other files to build a correct parse tree.
> >
> >
> > So does this (option B plus your No) mean that IDEs would
> > tend to color "yield" as a keyword (at the beginning of a
> > statement) even if followed by "("?
> >
> > I suppose that would work.  It's hard to predict what that
> > would feel like, but it's logical.
> >
> > — John
> >
> >
> >