RFR: 8367531: Template Framework: use scopes and tokens instead of misbehaving immediate-return-queries

[Emanuel Peter]

I got some feedback from users of the Template Framework, especially @galderz . And personally, I already was slightly unsatisfied by some of the issues described below, but did not expect it to be as bad as it is.

So I'm sorry, but I think we need to do a significant re-design. It is now still early enough, and only trivial changes are required for the "real" uses of the framework. Only the framework internal tests require significant changes.

Many thanks to @galderz for trying out the framework, and reporting the issues. And thanks to @chhagedorn for spending a few hours in an offline meeting discussing the issue.

**Major issue with Template Framework: lambda vs token order**

The template rendering involves some state, such as keeping track of hashtag replacements, names and fuel cost.
Some methods have side-effects (`addDataName`, `let`, ...) and others are simple queries (`sample`, ...).
Sadly, the first version of the template framework was not very consistent, and created tokens (deferred evaluation, during token evaluation) for some, and for others it queried the state and returned the result immediately (during lambda execution). One nasty consequence is that an immediately returning query can "float" above a state affecting token. For example, `addDataName` generated a token (so that we know if it is to be added for the template frame or a hook anchoring frame), but answered sampling queries immediately (because that means we can use the returned value immediately and make decisions based on it immediately, which is nice). Looking at the example below, this had the confusing result that `addDataName` only generates a token at first, then `sample` does not have that name available yet, and only later during token evaluation is the name actually added.

var testTemplate = Template.make(() -> body(
    ...
    addDataName("name", someType, MUTABLE),
    let("name", dataNames(MUTABLE).exactOf(someType).sample().name()),
    ...
));


**Two possible solutions: all-in on lambda execution or all-in on tokens**

First, I thought I want to go all-in on lambda execution, and have everything have immediate effect and return results immediately. This would have the nice effect that the user feels like they are directly in control of the execution order. But I did not find a good way without exposing too many internals to the user, or getting rid of the nice "token lists" we currently have inside Templates (the list is directly concatenated). Look at the following example:

var testTemplate = Template.make(() -> body(
    ...
    template1.call(),
    "some code right here",
    template2.call(),
    ...
));

One way would have been that calling `template1` and `template2` directly inserts code. But in `testTemplate` we would execute `template2` before adding the `some code right here`, so that does not work. So maybe calling `template2` should only return some state that captures what happened inside `template2`, and that captured state is only applied once we collect all the tokens of the `testTemplate` `body`. But what if the user somehow calls `template2` but never adds the captured state to tokens? As long as the captured state is pure, i.e. has truly no side-effect, that would be fine. But what if it needs to insert code into some outer scope? That would become very messy quickly.

An alternative would have been to abandon the token list completely, and do something similar to `StringBuilder`, where the state is updated explicitly, and code would be added explicitly. But that does not look very nice either.

Christian had asked me from the beginning if I should not make everything into tokens. I was hesitant because I thought we could not do something like sampling without returning immediately. But eventually I realized we can just create a token that samples and then calls a lambda with the result:


var testTemplate = Template.make(() -> body(
    ...
    addDataName("name", someType, MUTABLE),
    dataNames(MUTABLE).exactOf(someType).sample((DataName dn) -> scope(
        ... code that can use the DataName dn ...
    ))),
    ...
));


**Minor issue: Hook.insert did not work without nested Template, and was implicitly transparent for Names**

Having to use a separate template for the code to be inserted is sometimes a bit cumbersome, and separates the code too far. And that insertion means the inserted template is implicitly transparent for names is also not great: if the template is used in insertion, its scope is transparent, but if it is used in regular template nesting it is non-transparent. That is not a great design: the template would have different semantics based on the context.

Now we can directly do `hook.insert(scope(...))`. And we have to explicitly allow transparency by either doing:
- `hook.insert(scope(...))`: non-transparent.
- `hook.insert(transparentScope(..))`: transparent for names.

When inserting templates, the scope of the template has to be specified to be transparent or non-transparent. This allows us to be very precise about when names escape into the anchor scope and when they stay local. And if a template with a transparent scope is used in regular template nesting, its scope is transparent as well. Hence, the behavior of a template is now more consistent, and does not depend on the context (insertion vs regular nesting).

**Summary of Changes**
- `Token`s instead of "immediate return functions":
  - Names: `sample`, `count`, `hasAny`, `toList`: this means we don't have these queries "float" above a `addDataName` or `addStructuralName`, which was very very confusing, and lead to misleading results and confusing bugs (e.g. no names found when sampling).
  - Hook: `isAnchored`. Prevents the `isAnchored` query from floating above the `hook.anchor`, which could lead to misleading results.
  - `let`: allows us to keep hashtag replacements local to nested scopes in a template. This is especially helpful when streaming over lists, where we want to have a `let` for each item.
- Generalize `TemplateToken` to `ScopeToken`, and `body` to `scope` (and its friends). This allows us to use scopes systematically in templates, limiting `Name`s, hashtags and `setFuelCost`. This required quite a bit of refactoring in the `Renderer`.
- Adjusted and improved the `TestTutorial.java`, as well as the `TestTemplate.java` (2.8k of the changes, i.e. the majority).

**Notes for Reviewers**

Make sure to look at these first:
- Changes in `TestTutorial.java`.
  - `generateWithHashtagAndDollarReplacements3` shows generalization of scopes.
  - `generateWithCustomHooks` shows that we can `Hook.insert` scopes and templates.
  - Replacing `generateWithDataNamesAndScopes1/2` with `generateWithScopes1`: bad "old" way (floating issues) replaced with new token-based and scope-based queries.
- Changes in `TestTemplate.java` (2k+ lines!)
  - Many tests are simply adapted (renaming only). But some are extended, modified or completely new additions.
  - Make sure to look first at `testLet2`, `testDataNames0a...d`, `testDataNames6`, `testStructuralNames3...6`, `testNestedScopes1/2`, `testHookAndScopes1...3`. You probably don't need to look at everything in absolute detail, just make sure you roughly know what's going on at first.
- Once you understand the new semantics of the scopes and queries, look at the `template_framework` changes.
  - Look at the new cases in `Token.java`
  - Look at the changes in `CodeFrame` and `TemplateFrame`: they implement the `Name`, hashtag and `setFuelCost` (non)transparency, which is fundamental to the scopes.
  - Look at the `ScopeTokenImpl`, and how it is used in `Renderer.java`.

I'm very sorry that this is a huge change, and that I did not get this right the first time. I'm realizing how difficult it is to develop API's 😅 
I left some comments in the code changes, so hopefully that helps in the review. Feel free to ask me for a code-walk-through, or if you have any questions ☺

-------------

Commit messages:
 - fix test
 - NestingToken -> ScopeToken
 - flat -> transparentScope
 - update other test
 - clean up tutorial
 - tutorial scope and DataNames
 - wip tutorial
 - extend tutorial
 - more tutorial improvements
 - tutorial scope with insert
 - ... and 86 more: https://git.openjdk.org/jdk/compare/2826d170...aceced65

Changes: https://git.openjdk.org/jdk/pull/27255/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=27255&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8367531
  Stats: 3973 lines in 35 files changed: 2918 ins; 333 del; 722 mod
  Patch: https://git.openjdk.org/jdk/pull/27255.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/27255/head:pull/27255

PR: https://git.openjdk.org/jdk/pull/27255