RFR: 8344942: Template-Based Testing Framework [v61]
Christian Hagedorn
chagedorn at openjdk.org
Fri May 30 10:43:16 UTC 2025
On Fri, 23 May 2025 10:52:20 GMT, Emanuel Peter <epeter at openjdk.org> wrote:
>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>>
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>>
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>>
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>>
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>>
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>>
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>>
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>>
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with two additional commits since the last revision:
>
> - Merge branch 'JDK-8344942-TemplateFramework-v3' of https://github.com/eme64/jdk into JDK-8344942-TemplateFramework-v3
> - move verification
Thanks for all the updates and discussions! I've worked my way through the documentation in `Template` and the examples again in some more detail. It's much better and the new explanations are well done, excellent work!
I left some comments here and there but mostly minor things. I will have another look at the implementation - probably only finished by Monday. The design now looks great. I'm glad we could find a good solution now after some more iterations :-)
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 150:
> 148: * {@link Template#make(String, Function)}. For each number of arguments there is an implementation
> 149: * (e.g. {@link Template.TwoArgs} for two arguments). This allows the use of generics for the
> 150: * {@link Template} argument types which enables type checking of the {@link Template} arguments.
Maybe add:
Suggestion:
* {@link Template} argument types which enables type checking of the {@link Template} arguments.
* It is currently only allowed to use up to three arguments.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 160:
> 158: * Ideally, we would have used <a href="https://openjdk.org/jeps/430">string templates</a> to inject these Template arguments into the strings.
> 159: * But since string templates are not (yet) available, the Templates provide <strong>hashtag replacements</strong>
> 160:
These paragraphs should probably belong together? And maybe you want to wrap the long line above.
Suggestion:
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 174:
> 172: * <p>
> 173: * The dollar and hashtag names must have at least one character. The first character must be a letter
> 174: * or underscore (i.e. {@code a-zA-Z_}), the other characters can also be digits (i.e. {@code a-zA-Z0-9_}).
This does not seem to be enforced:
var testTemplate = Template.make(() -> body(
"""
public class Foo {
public static void main() {
int $1var = 34;
}
}
"""
));
System.out.println(testTemplate.render());
Results in:
public class Foo {
public static void main() {
int $1var = 34;
}
}
which compiles fine. I can also change it to `$$var` which renders to `$var_1` which also compiles fine.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 190:
> 188: * <p>
> 189: * The writer of recursive {@link Template}s must ensure that this recursion terminates. To unify the
> 190: * approach across {@link Template}s, we introduce the concept of {@link fuel}. Templates are rendered starting
Suggestion:
* approach across {@link Template}s, we introduce the concept of {@link #fuel}. Templates are rendered starting
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 201:
> 199: * are available, and if they are mutable or immutable. We model fields and variables with {@link DataName}s,
> 200: * which we can add to the current scope with {@link addDataName}. We can access the {@link DataName}s with
> 201: * {@link dataNames}. We can filter for {@link DataName}s of specific {@link DataName.Type}s, and then
Suggestion:
* which we can add to the current scope with {@link #addDataName}. We can access the {@link DataName}s with
* {@link #dataNames}. We can filter for {@link DataName}s of specific {@link DataName.Type}s, and then
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 222:
> 220: * that the lambda is always executed first, and the tokens are evaluated afterwards. A method like
> 221: * {@code dataNames(MUTABLE).exactOf(type).count()} is a method that is executed during the evaluation
> 222: * of the lambda. But a method like {@link addDataName} returns a token, and does not immediately add
Suggestion:
* of the lambda. But a method like {@link #addDataName} returns a token, and does not immediately add
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 252:
> 250: * // "otherTemplate" were to count the DataNames, the count would be increased
> 251: * // by 2 compared to "c1".
> 252: * otherTemplate.asToken(),
Last arg:
Suggestion:
* otherTemplate.asToken()
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 260:
> 258: * // evaluated later. By that time, the token for "v1" is evaluated, and so the
> 259: * // nested Template would observe an increment in the count.
> 260: * anotherTemplate.asToken(),
Last arg:
Suggestion:
* anotherTemplate.asToken()
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 324:
> 322: * A {@link Template} with one argument.
> 323: *
> 324: * @param arg0Name The name of the (first) argument, used for hashtag replacements in the {@link Template}.
Nit and I'm okay with both: Should we name the first argument arg1 instead of arg0? Starting from zero might not be expected.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 341:
> 339: * {@link Template}.
> 340: */
> 341: public TemplateToken asToken(A a) {
Could also be named `valueArg0` which is more expressive and it's easier when working in an IDE:
vs.
You could apply that change for the `render()` methods as well. Same for the two and three arg versions.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 539:
> 537: * @param arg0Name The name of the (first) argument for hashtag replacement.
> 538: * @return A {@link Template} with one argument.
> 539: */
Just a general thought and not really something we can enforce by the framework, but we might want to mention here as well that the `arg0Name` string should match the lambda parameter for easier application and consistency? Theoretically (and not very clever), you can do that:
var testTemplate = Template.make("a", "b", (Integer b, Integer a) -> body(
"""
public class Foo {
public static void main() {
int a1 = #a;
int b1 = #b;
""",
"int a2 = " + a + ";\n", // != a1, oops
"int b2 = " + b + ";\n", // != b1, oops
"""
}
}
"""
));
We could make the same remark in the two and three arg `make()` versions as well.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 668:
> 666: /**
> 667: * Define a hashtag replacement for {@code "#key"}, with a specific value, which is also captured
> 668: * by the provided {@code 'function'} with type {@code <T>}.
Suggestion:
* by the provided {@code function} with type {@code <T>}.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 683:
> 681: * @param value The value that the hashtag is replaced with.
> 682: * @param <T> The type of the value.
> 683: * @param function The function that is applied with the provided {@code 'value'}.
Suggestion:
* @param function The function that is applied with the provided {@code value}.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 696:
> 694: * rendering a template with {@code render(fuel)} (e.g. {@link ZeroArgs#render(float)}).
> 695: */
> 696: static final float DEFAULT_FUEL = 100.0f;
Fields defined in an interface are implicitly static and final
Suggestion:
float DEFAULT_FUEL = 100.0f;
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 703:
> 701: * with {@link #setFuelCost(float)} inside {@link #body(Object...)}.
> 702: */
> 703: static final float DEFAULT_FUEL_COST = 10.0f;
Suggestion:
float DEFAULT_FUEL_COST = 10.0f;
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 762:
> 760: * @param weight The weight of the {@link DataName}, which correlates to the probability
> 761: * of this {@link DataName} being chosen when we sample.
> 762: * Must be a value from 0 to 1000.
But you disallow 0 below. Should this be 1 to 1000?
Suggestion:
* Must be a value from 1 to 1000.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 771:
> 769: }
> 770: boolean mutable = mutability == DataName.Mutability.MUTABLE;
> 771: if (0 >= weight || weight > 1000) {
Could be more readable but up to you
Suggestion:
if (weight <= 0 || weight > 1000) {
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 810:
> 808: * @param weight The weight of the {@link StructuralName}, which correlates to the probability
> 809: * of this {@link StructuralName} being chosen when we sample.
> 810: * Must be a value from 0 to 1000.
Same here, 1 to 1000?
Suggestion:
* Must be a value from 1 to 1000.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 814:
> 812: */
> 813: static Token addStructuralName(String name, StructuralName.Type type, int weight) {
> 814: if (0 >= weight || weight > 1000) {
Could be more readable but up to you
Suggestion:
if (weight <= 0 || weight > 1000) {
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java line 61:
> 59: public class TestAdvanced {
> 60: private static final Random RANDOM = Utils.getRandomInstance();
> 61:
Unused
Suggestion:
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 66:
> 64: CompileFramework comp = new CompileFramework();
> 65:
> 66: // Add java source files.
Suggestion:
// Add Java source files.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 84:
> 82: // Hint: if you want to see the generated source code, you can enable
> 83: // printing of the source code that the CompileFramework receives,
> 84: // with -DCompileFrameworkVerbose=true
Maybe also add here that the printed output is not formatted and one might consider dumping it to an IDE or other tool to auto-format.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 151:
> 149: "System.out.println(", arg, ");\n", // capture arg via lambda argument
> 150: "System.out.println(#arg);\n", // capture arg via hashtag replacement
> 151: "System.out.println(#{arg});\n", // capture arg via hashtag replacement with brackets
It's not clear here why one should use brackets. If there is an argument for those further down, then you can cross reference. Otherwise, it might need some explanation here.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 154:
> 152: // The Template Framework allows two ways of formatting Strings, either
> 153: // by appending to the comma-separated list of Tokens, or by hashtag
> 154: // replacements. Appending as a Token works whenever one has a reference
Might be more clear:
Suggestion:
// by appending to the comma-separated list of Tokens passed to body(), or by hashtag
// replacements inside a single string. Appending as a Token works whenever one has a reference
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 199:
> 197: // In the code below, we use "var" as a local variable. But if we were
> 198: // to instantiate this template twice, the names could conflict. Hence,
> 199: // we automatically rename the names that have a $ prepended.
Suggestion:
// we automatically rename the names that have a $ prepended with
// var_1, var_2, etc.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 208:
> 206: var template2 = Template.make("x", (Integer x) ->
> 207: // Sometimes it can be helpful to not just create a hashtag replacement
> 208: // with let, but also to capture the variable.
Suggestion:
// with let, but also to capture the variable to use it as lambda parameter.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 258:
> 256:
> 257: // Render templateClass to String.
> 258: return templateClass.render();
When printing this, it starts at `var_2` and not `var_1`. Why is that?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 298:
> 296:
> 297: // In this example, we look at the use of Hooks. They allow us to reach back, to outer
> 298: // scopes. For example, we can reach out from inside a method body to a hook set at
Should we say "anchored" with the recent set -> anchor renaming?
Suggestion:
// scopes. For example, we can reach out from inside a method body to a hook anchored at
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 306:
> 304: var myHook = new Hook("MyHook");
> 305:
> 306: var template1 = Template.make("name", "value", (String name, Integer value) -> body(
One could generally think about using `_` for unused lambda parameters which I think is the common convention. But then I guess we would need to update the documentation about saying "name" and "String name" should be the same and make an exception for unused ones. I don't know.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 314:
> 312: var template2 = Template.make("x", (Integer x) -> body(
> 313: """
> 314: // Let us go back to the hook, and define a field named $field...
I guess one need to get used to the fact that "go back" means back in the lambda evaluation order which is reversed to the Java code order. Maybe we can clarify that with something like:
Suggestion:
// Let us go back to where we anchored the hook with anchor() and define a field named $field there.
// Note that in the Java code we have not defined anchor() on the hook, yet. But since it's a lambda
// expression, it is not evaluated, yet! Eventually, anchor() will be evaluated before insert() in
// this example.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 358:
> 356:
> 357: // We saw the use of custom hooks above, but now we look at the use of CLASS_HOOK and METHOD_HOOK
> 358: // from the Template Library.
Can you expand here on why it's better to use them instead of creating your own? Is it just readability/convenience?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 454:
> 452: // For every recursion depth, some fuel is automatically subtracted
> 453: // so that the fuel slowly depletes with the depth.
> 454: // We keep the recursion going until the fuel is depleted.
You can also note here that if we forget to check the `fuel()`, the renderer causes a stack overflow because the recursion never ends.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 487:
> 485: // in this scope, and in any nested scope, including nested Templates. This allows us to
> 486: // add some fields and registers in one Template, and later on, in another Template, we
> 487: // can access these fields and registers again with "dataNames()".
What do you mean by "registers"?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 525:
> 523: private static final MySimpleInt mySimpleInt = new MySimpleInt();
> 524:
> 525: // In this Example, we generate 3 fields, and add their names to the
Suggestion:
// In this example, we generate 3 fields, and add their names to the
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 596:
> 594: @Override
> 595: public boolean isSubtypeOf(DataName.Type other) {
> 596: return other instanceof MyPrimitive(String n) && n == name();
Is `==` intended? Should it be `equals()`?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 779:
> 777: // Having defined these helper methods, let us start with the first example.
> 778: // You should start reading this example bottum-up, starting at
> 779: // templateClass, then going to templateMain and last to templateInnner.
Suggestion:
// templateClass, then going to templateMain and last to templateInner.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 857:
> 855: public static int f1 = 42;
> 856: """,
> 857: // But why is this DataName now availabe inside the scope of
Suggestion:
// But why is this DataName now available inside the scope of
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 868:
> 866: // inserted into, i.e. the CLASS_HOOK. This is very important,
> 867: // if we did not make that scope transparent, we could not
> 868: // add any DataNames to the class scope any more, and we could
Suggestion:
// add any DataNames to the class scope anymore, and we could
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 1063:
> 1061: // And now some fields are different than before.
> 1062: """,
> 1063: myClassList.stream().map(c -> templateLoad.asToken(c)).toList(),
Suggestion:
myClassList.stream().map(c -> templateLoad.asToken(c)).toList(),
"""
// Now let us mutate some fields.
""",
myClassList.stream().map(c -> templateStore.asToken(c)).toList(),
"""
// And now some fields are different than before.
""",
myClassList.stream().map(c -> templateLoad.asToken(c)).toList(),
Suggestion:
myClassList.stream().map(templateLoad::asToken).toList(),
"""
// Now let us mutate some fields.
""",
myClassList.stream().map(templateStore::asToken).toList(),
"""
// And now some fields are different than before.
""",
myClassList.stream().map(templateLoad::asToken).toList(),
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 1081:
> 1079: // and we can read and write to them, they may be mutable or immutable.
> 1080: // We now introduce another set of "Names", the "StructuralNames". They are
> 1081: // useful for modeling method names an class names, and possibly more. Anything
Suggestion:
// useful for modeling method names and class names, and possibly more. Anything
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 1094:
> 1092: // caught.
> 1093: //
> 1094: // Let us show an example with Method names. But for simplicity, we assume they
Suggestion:
// Let us look at an example with Method names. But for simplicity, we assume they
test/hotspot/jtreg/testlibrary_tests/template_framework/tests/TestFormat.java line 49:
> 47:
> 48: public static void main(String[] args) {
> 49: List<FormatInfo> list = new ArrayList();
Suggestion:
List<FormatInfo> list = new ArrayList<>();
test/hotspot/jtreg/testlibrary_tests/template_framework/tests/TestTemplate.java line 79:
> 77: @Override
> 78: public boolean isSubtypeOf(DataName.Type other) {
> 79: return other instanceof MyPrimitive(String n) && n == name();
Is `==` wanted and not `equals()`?
-------------
PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2880263792
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115200599
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115203547
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115232385
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115240716
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115243151
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115253133
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115253959
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115254217
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115282245
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115277473
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115288457
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115294922
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115295164
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115306383
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115306586
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115317138
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115317698
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115318545
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115318856
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115207213
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115331173
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115338444
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115348344
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115354054
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115368253
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115374255
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115368728
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115382813
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115388737
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115397938
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115406391
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115448432
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115433085
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115438108
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115479497
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115497793
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115512637
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115515106
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115601955
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115604658
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115606795
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115320833
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2115327212
More information about the hotspot-compiler-dev
mailing list