RFR: 8344942: Template-Based Testing Framework [v57]
Manuel Hässig
mhaessig at openjdk.org
Fri May 23 09:30:14 UTC 2025
On Thu, 22 May 2025 09:33:21 GMT, Emanuel Peter <epeter at openjdk.org> wrote:
>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>>
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>>
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>>
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>>
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>>
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>>
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>>
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>>
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
>
> Apply suggestions from code review
>
> Co-authored-by: Roberto Castañeda Lozano <robcasloz at users.noreply.github.com>
Thank you, @eme64, for bearing with us. The API has come a long way since v1.
I found some more typos, but otherwise it looks mighty fine.
test/hotspot/jtreg/compiler/lib/template_framework/DataName.java line 32:
> 30: * scope with {@link Template#addDataName}, and accessed with {@link Template#dataNames}, from where
> 31: * count, list or even sample random {@link DataName}s. Every {@link DataName} has a {@link DataName.Type},
> 32: * so that sampling can be restricted to these types.
Suggestion:
* {@link DataName}s represent things like fields and local variables, and can be added to the local
* scope with {@link Template#addDataName}, and accessed with {@link Template#dataNames}, to
* count, list or even sample random {@link DataName}s. Every {@link DataName} has a {@link DataName.Type},
* so that sampling can be restricted to these types.
The text flow was a bit weird. I hope this keeps the original meaning.
test/hotspot/jtreg/compiler/lib/template_framework/Name.java line 35:
> 33: * The name of the type, that can be used in code.
> 34: *
> 35: * @return The {@String} name of the name, that can be used in code.
Suggestion:
* The name of the name, that can be used in code.
*
* @return The {@String} name of the name, that can be used in code.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 150:
> 148: * {@link Template#make(String, Function)}. For each number of arguments there is an implementation
> 149: * (e.g. {@link Template.TwoArgs} for two arguments). This allows the use of Generics for the
> 150: * Template argument types which enables type checking of the Template arguments.
Suggestion:
* (e.g. {@link Template.TwoArgs} for two arguments). This allows the use of generics for the
* {@link Template} argument types which enables type checking of the {@link Template} arguments.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 214:
> 212: * <p>
> 213: * When working with {@link DataName}s and {@link StructuralName}s, it is important to be aware of the
> 214: * relevant scopes, as well as the execution order of the {@link Template} lambdas, as well as the evaluation
Suggestion:
* relevant scopes, as well as the execution order of the {@link Template} lambdas and the evaluation
Text flow nit
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 217:
> 215: * of the {@link Template#body} tokens. When a {@link Template} is rendered, its lambda is invoked. In the
> 216: * lambda, we generate the tokens, and create the {@link Template#body}. Once the lambda returns, the
> 217: * tokens are evaluated one by one. While evaluating the tokens, the Renderer might encounter a nested
Suggestion:
* tokens are evaluated one by one. While evaluating the tokens, the {@link Renderer} might encounter a nested
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 696:
> 694: * rendering a template with {@code render(fuel)} (e.g. {@link ZeroArgs#render(float)}).
> 695: */
> 696: float DEFAULT_FUEL = 100.0f;
Suggestion:
static final float DEFAULT_FUEL = 100.0f;
Is not mutated as far as i can tell.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 703:
> 701: * with {@link #setFuelCost(float)} inside {@link #body(Object...)}.
> 702: */
> 703: float DEFAULT_FUEL_COST = 10.0f;
Suggestion:
static final float DEFAULT_FUEL_COST = 10.0f;
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 761:
> 759: * or if we also allow it to be mutated.
> 760: * @param weight The weight of the {@link DataName}, which correlates to the probability
> 761: * of this {@link DataName} being chosen when we sample.
Should document the weight limit.
Suggestion:
* @param weight The weight of the {@link DataName}, which correlates to the probability
* of this {@link DataName} being chosen when we sample. Must be smaller than 1000.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 770:
> 768: }
> 769: boolean mutable = mutability == DataName.Mutability.MUTABLE;
> 770: return new AddNameToken(new DataName(name, type, mutable, weight));
"Input vaildation" for mutablity happens here for mutability, but inside the constructor for weight. Should both happen in the same place?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 332:
> 330: // The Hook is set for the Tokens inside the set braces.
> 331: // As long as the hook is anchored, we can insert code into the hook,
> 332: // here we can define static fields for example.
Perhaps this comment should mention that the code is inserted at the point where `myHook.anchor` is located.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 627:
> 625: // the current scope.
> 626: var templateSample = Template.make("type", (DataName.Type type) -> body(
> 627: let("name", dataNames(MUTABLE).exactOf(type).sample().name()),
Suggestion:
let("name", dataNames(MUTABLE_OR_IMMUTABLE).exactOf(type).sample().name()),
To get all available names you also need immutable names. There is no difference in this case, but this being a tutorial, the semantics should match the explanations.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 636:
> 634: var templateStatus = Template.make(() -> body(
> 635: let("ints", dataNames(MUTABLE).exactOf(myInt).count()),
> 636: let("longs", dataNames(MUTABLE).exactOf(myLong).count()),
Suggestion:
let("ints", dataNames(MUTABLE_OR_IMMUTABLE).exactOf(myInt).count()),
let("longs", dataNames(MUTABLE_OR_IMMUTABLE).exactOf(myLong).count()),
-------------
Changes requested by mhaessig (Author).
PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2863614753
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104076338
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104100212
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104137360
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104147066
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104147982
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104164360
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104166142
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104126982
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104129597
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104181130
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104189394
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2104190251
More information about the hotspot-compiler-dev
mailing list