RFR: 8367531: Template Framework: use scopes and tokens instead of misbehaving immediate-return-queries [v11]

Wed Nov 5 09:58:16 UTC 2025

On Tue, 4 Nov 2025 16:04:22 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> I got some feedback from users of the Template Framework, especially @galderz . And personally, I already was slightly unsatisfied by some of the issues described below, but did not expect it to be as bad as it is.
>> 
>> So I'm sorry, but I think we need to do a significant re-design. It is now still early enough, and only trivial changes are required for the "real" uses of the framework. Only the framework internal tests require significant changes.
>> 
>> Many thanks to @galderz for trying out the framework, and reporting the issues. And thanks to @chhagedorn for spending a few hours in an offline meeting discussing the issue.
>> 
>> **Major issue with Template Framework: lambda vs token order**
>> 
>> The template rendering involves some state, such as keeping track of hashtag replacements, names and fuel cost.
>> Some methods have side-effects (`addDataName`, `let`, ...) and others are simple queries (`sample`, ...).
>> Sadly, the first version of the template framework was not very consistent, and created tokens (deferred evaluation, during token evaluation) for some, and for others it queried the state and returned the result immediately (during lambda execution). One nasty consequence is that an immediately returning query can "float" above a state affecting token. For example, `addDataName` generated a token (so that we know if it is to be added for the template frame or a hook anchoring frame), but answered sampling queries immediately (because that means we can use the returned value immediately and make decisions based on it immediately, which is nice). Looking at the example below, this had the confusing result that `addDataName` only generates a token at first, then `sample` does not have that name available yet, and only later during token evaluation is the name actually added.
>> 
>> var testTemplate = Template.make(() -> body(
>>     ...
>>     addDataName("name", someType, MUTABLE),
>>     let("name", dataNames(MUTABLE).exactOf(someType).sample().name()),
>>     ...
>> ));
>> 
>> 
>> **Two possible solutions: all-in on lambda execution or all-in on tokens**
>> 
>> First, I thought I want to go all-in on lambda execution, and have everything have immediate effect and return results immediately. This would have the nice effect that the user feels like they are directly in control of the execution order. But I did not find a good way without exposing too many internals to the user, or getting rid of the nice "token lists" we currently have inside Templates (the...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
> 
>   small adjustments after call with Roberto and Christian

Nice improvement! It makes it much more intuitive to use and avoids user mistakes where it is not immediately evident how to fix missing data names because you need to know more framework internals.

I'm already submitting some comments for `TestTutorial` which I walked through now. Very well written and it includes everything a normal user should know to get started without the need to further dig into the framework.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 371:

> 369:             // replacements escape the scope.
> 370:             transparentScope(
> 371:                 let("x", 11), // escape escopes the "transparentScope".

Did you mean the following?
Suggestion:

                let("x", 11), // escapes the "transparentScope".

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 499:

> 497:         var template2 = Template.make("x", (Integer x) -> scope(
> 498:             """
> 499:             // Let us go back to where we anchored the hook with anchor() and define a field named $field1 there.

Maybe add hint:

Suggestion:

            // Let us go back to where we anchored the hook with anchor() (see 'templateClass' below) and define a field
            // named $field1 there.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 501:

> 499:             // Let us go back to where we anchored the hook with anchor() and define a field named $field1 there.
> 500:             """,
> 501:             myHook.insert(scope( // <- insertion scope

I'm not yet clear on that one. What happens when we insert a transparentScope here and for example add a `let("y", 42)`. Where could we then use `#y` (might not be a thing one might want to do, though)? Anywhere in the anchor scope and/or in the caller scope?

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 720:

> 718:     //   variable from an outer Template. Luckily, the outer Templates have added their
> 719:     //   fields and variables, and you can now access them with "dataNames()". You can
> 720:     //   count them, get a list of them, or sample a random one.

(Sorry for also suggesting things for existing code, I just read through some of that to better understand the changes and found some things to improve)

Maybe for completeness we could add that any let() definition is only available in the current scope and nested scopes but do not cross a "template boundary"?

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 736:

> 734:     //
> 735:     // To get started, we show an example where all DataNames have the same type, and where
> 736:     // all Names are mutable. For simplicity, our type represents the primitive int type.

Side note: Now that we also have the library available, could we also use that one instead of defining our own `MySimpleInt`? If so, you might want to show an example and hint here that you could also just define your own type as an expert user.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 764:

> 762:             addDataName($("f1"), mySimpleInt, MUTABLE, 1),
> 763:             addDataName($("f2"), mySimpleInt, MUTABLE, 1),
> 764:             addDataName($("f3"), mySimpleInt, MUTABLE), // omit weight, default is 1.

It seems implicitly obvious but maybe we can add here that data names will only be available after adding them:

Suggestion:

// Also note that DataNames are only available once they are defined:

// Nothing defined, yet: dataNames() = {}
addDataName($("f1"), mySimpleInt, MUTABLE, 1),
// Only now dataNames() contains f1: dataNames() = {f1}
addDataName($("f2"), mySimpleInt, MUTABLE, 1),
// dataNames() = {f1, f2}
addDataName($("f3"), mySimpleInt, MUTABLE), // omit weight, default is 1.
// dataNames() = {f1, f2, f3}

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 778:

> 776:                     // the hashtag replacement "a".
> 777:                     """,
> 778:                     dataNames(MUTABLE).exactOf(mySimpleInt).sampleAndLetAs("a"),

What do you think about just naming it `sampleTo("a")`? Or do you think the mention of `let` is crucial here?

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 784:

> 782:                     // If we are also interested in the type of the field, we can do:
> 783:                     """,
> 784:                     dataNames(MUTABLE).exactOf(mySimpleInt).sampleAndLetAs("b", "bType"),

Just a suggestion: We could be more explicit to mention that the second one is the type like `sampleToWithType()` or something like that.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 860:

> 858:         // Define a local variable.
> 859:         // Note: it is very important that we use a "transparentScope" for the template here,
> 860:         //       so that the DataName can escape to outer scopes.

Might be obvious but you could add that it does not mean it's available for the entire outer scope but only for everything that follows this DataName definition/insertion point of this template.

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 898:

> 896:             // action based on the value. For that we have to capture the count
> 897:             // with a lambda and inner scope as above. If we only need to have
> 898:             // the count as a hashtag replacement, we can also use the follwing

Suggestion:

            // the count as a hashtag replacement, we can also use the following

-------------

PR Review: https://git.openjdk.org/jdk/pull/27255#pullrequestreview-3420222142
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493257567
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493447391
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493479782
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493566912
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493581451
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493716808
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493594823
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493599920
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493642120
PR Review Comment: https://git.openjdk.org/jdk/pull/27255#discussion_r2493682462