RFR: 8367531: Template Framework: use scopes and tokens instead of misbehaving immediate-return-queries [v4]

Tue Oct 14 15:54:13 UTC 2025

On Tue, 14 Oct 2025 05:13:36 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> I got some feedback from users of the Template Framework, especially @galderz . And personally, I already was slightly unsatisfied by some of the issues described below, but did not expect it to be as bad as it is.
>> 
>> So I'm sorry, but I think we need to do a significant re-design. It is now still early enough, and only trivial changes are required for the "real" uses of the framework. Only the framework internal tests require significant changes.
>> 
>> Many thanks to @galderz for trying out the framework, and reporting the issues. And thanks to @chhagedorn for spending a few hours in an offline meeting discussing the issue.
>> 
>> **Major issue with Template Framework: lambda vs token order**
>> 
>> The template rendering involves some state, such as keeping track of hashtag replacements, names and fuel cost.
>> Some methods have side-effects (`addDataName`, `let`, ...) and others are simple queries (`sample`, ...).
>> Sadly, the first version of the template framework was not very consistent, and created tokens (deferred evaluation, during token evaluation) for some, and for others it queried the state and returned the result immediately (during lambda execution). One nasty consequence is that an immediately returning query can "float" above a state affecting token. For example, `addDataName` generated a token (so that we know if it is to be added for the template frame or a hook anchoring frame), but answered sampling queries immediately (because that means we can use the returned value immediately and make decisions based on it immediately, which is nice). Looking at the example below, this had the confusing result that `addDataName` only generates a token at first, then `sample` does not have that name available yet, and only later during token evaluation is the name actually added.
>> 
>> var testTemplate = Template.make(() -> body(
>>     ...
>>     addDataName("name", someType, MUTABLE),
>>     let("name", dataNames(MUTABLE).exactOf(someType).sample().name()),
>>     ...
>> ));
>> 
>> 
>> **Two possible solutions: all-in on lambda execution or all-in on tokens**
>> 
>> First, I thought I want to go all-in on lambda execution, and have everything have immediate effect and return results immediately. This would have the nice effect that the user feels like they are directly in control of the execution order. But I did not find a good way without exposing too many internals to the user, or getting rid of the nice "token lists" we currently have inside Templates (the...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix TestMethodArguments.java after merge with master

Thank you for your continued effort on this, @eme64! Better to do this now than later.

I have so far only looked at the tutorial, but came up with some questions. I will continue looking at the rest, later.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 423:

> 421:             """
> 422:             static int v3d_#{x} = #a + #b;
> 423:             """

I do not understand how `a` escapes the let without a `transparentScope()` like for `b`. Also the paragraph above does not really explain what the trick is that lets us simulate a lambda-less `let`.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 469:

> 467:     // the top of the class, and insert a field.
> 468:     //
> 469:     // The choice of transparency of an insertion scope is quite important. A common use case

What is an "insertion scope"?

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 498:

> 496:             //
> 497:             // Which method is used is up to the user. General guidance is if the same code may be
> 498:             // inserted elsewhere, one should lean towards inserting templates. But in many cases

Suggestion:

            // Which method is used is up to the user. General guidance is if the same code may also
            // be inserted elsewhere, one should lean towards inserting templates. But in many cases

Just a small tweak to emphasize multiple usages.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 521:

> 519:             // Anchoring a Hook creates a scope, spanning the braces of the
> 520:             // "anchor" call. Any Hook.insert that happens inside this scope
> 521:             // goes to the top of that scope.

This first sentence is a bit strange when we have to write an explicit `scope` since we have to write it. But I do not have a better wording.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 577:

> 575:                 static { System.out.println("Defining static field $field"); }
> 576:                 public static int $field = #value;
> 577:                 """

Why do we not just write `public static int $field = 5;`? Is this just for demonstration purposes, or am I missing something more fundamental?

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 831:

> 829:         // Define a static field.
> 830:         // Note: it is very important that we use a "transparentScope" for the template here,
> 831:         //       so that the DataName can escape to outer scopes.

We could also use `hashtagScope` and achieve the same thing, could we not?

-------------

PR Review: https://git.openjdk.org/jdk/pull/27255#pullrequestreview-3336158396
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429525666
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429538149
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429547199
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429575893
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429591266
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2429605801