RFR: 8344942: Template-Based Testing Framework [v9]
Christian Hagedorn
chagedorn at openjdk.org
Wed May 7 12:03:10 UTC 2025
On Wed, 7 May 2025 10:17:39 GMT, Emanuel Peter <epeter at openjdk.org> wrote:
>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>>
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>>
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>>
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>>
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>>
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>>
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>>
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>>
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 16 commits:
>
> - Merge branch 'master' into JDK-8344942-TemplateFramework-v3
> - Whitespace
> - Suggestions by Christian
>
> Co-authored-by: Christian Hagedorn <christian.hagedorn at oracle.com>
> - typo
> - For Christian: example and more intro
> - fix hashtag
> - manual merge
> - Apply suggestions from code review
>
> Co-authored-by: Christian Hagedorn <christian.hagedorn at oracle.com>
> - move library
> - Merge branch 'master' into JDK-8344942-TemplateFramework-v3
> - ... and 6 more: https://git.openjdk.org/jdk/compare/0844745e...fae7ced6
Next batch of comments. Will probably resume tomorrow :-)
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 6:
> 4: We want to make it easy to generate variants of tests. Often, we would like to have a set of tests, corresponding to a set of types, a set of operators, a set of constants, etc. Writing all the tests by hand is cumbersome or even impossible. When generating such tests with scripts, it would be preferable if the code generation happens automatically, and the generator script was checked into the code base. Code generation can go beyond simple regression tests, and one might want to generate random code from a list of possible templates, to fuzz individual Java features and compiler optimizations.
> 5:
> 6: The Template Framework provides a facility to generate code with Templates. Templates are essencially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of the Templates.
Suggestion:
The Template Framework provides a facility to generate code with Templates. Templates are essencially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of Templates.
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 8:
> 6: The Template Framework provides a facility to generate code with Templates. Templates are essencially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of the Templates.
> 7:
> 8: The Template Framework only generates code in the form of a String. This code can then be compiled and executed, for example with help of the [Compile Framework](../compile_framework/README.md).
Suggestion:
The Template Framework only generates code in the form of a String. This code can then be compiled and executed, for example with the help of the [Compile Framework](../compile_framework/README.md).
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 50:
> 48: * filled (replaced) by different values at each Template instantiation. For example, these "holes" can
> 49: * be filled with different types, operators or constants. Templates can also be nested, allowing a modular
> 50: * use of the Templates.
Suggestion:
* use of Templates.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 54:
> 52: * <p>
> 53: * <strong>Example:</strong>
> 54: * The following are snippets from the example test {@code TestAdvanced.java}.
Suggestion:
* The following snippets are from the example test {@code TestAdvanced.java}.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 57:
> 55: * First, we define a template that generates a {@code @Test} method for a given type, operator and
> 56: * constant generator. We define two constants {@code con1} and {@code con2}, and then use a multiline
> 57: * string with hashtag {@code #} "holes" that are then replaced by the template arguments and the
Suggestion:
* string with hashtags {@code #} (i.e. "holes") that are then replaced by the template arguments and the
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 81:
> 79: * }
> 80: *
> 81: * To get an executable test, we define a class Template, which takes a list of types,
Not entirely clear what you mean with "a class Template". Do you mean "we define a Template"?
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 82:
> 80: *
> 81: * To get an executable test, we define a class Template, which takes a list of types,
> 82: * and calls the test template for each type and operator. We use the {@code TestFramework}
Suggestion:
* and calls the {@code testTemplate} defined above for each type and operator. We use the {@code TestFramework}
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 105:
> 103: * """,
> 104: * // Call the testTemplate for each type and operator, generating a
> 105: * // list of list of TemplateWithArgs:
Suggestion:
* // list of lists of TemplateWithArgs:
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 126:
> 124: * new Type("long", () -> GEN_LONG.next(), List.of("+", "-", "*", "&", "|", "^")),
> 125: * new Type("float", () -> GEN_FLOAT.next(), List.of("+", "-", "*", "/")),
> 126: * new Type("double", () -> GEN_DOUBLE.next(), List.of("+", "-", "*", "/"))
Same here as commented earlier: You can directly use `GEN_X::next()`.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 166:
> 164: *
> 165: * <p>
> 166: * A {@link TemplateBinding} allows the recurisve use of {@link Template}s. With the indirection of such a binding,
Suggestion:
* A {@link TemplateBinding} allows the recursive use of {@link Template}s. With the indirection of such a binding,
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 170:
> 168: * with a certain amount of {@link #fuel}, which is decreased at each {@link Template} nesting by a certain amount
> 169: * (can be changed with {@link #setFuelCost}). Recursive templates are supposed to terminate once the {@link #fuel}
> 170: * is depleated (i.e. reaches zero).
Suggestion:
* is depleted (i.e. reaches zero).
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 175:
> 173: * Code generation often involves defining fields and variables, which are then available inside a defined
> 174: * scope, and can be sampled in any nested scope. To allow the use of names for multiple applications (e.g.
> 175: * fields, variables, methods, etc), we define a {@link Name}, which captures the {@link String} representation
Suggestion:
* fields, variables, methods, etc.), we define a {@link Name}, which captures the {@link String} representation
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 178:
> 176: * to be used in code, as well as its type and if it is mutable. One can add such a {@link Name} to the
> 177: * current code scope with {@link #addName}, and sample from the current or outer scopes with {@link #sampleName}.
> 178: * When generating code, one might want to create {@link Name}s (variables, fields, etc) in local scope, or
Suggestion:
* When generating code, one might want to create {@link Name}s (variables, fields, etc.) in local scope, or
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 179:
> 177: * current code scope with {@link #addName}, and sample from the current or outer scopes with {@link #sampleName}.
> 178: * When generating code, one might want to create {@link Name}s (variables, fields, etc) in local scope, or
> 179: * in some outer scope with the use of {@link Hook}s.
Maybe mention here again that all of the explained above can be found in tutorial like examples (I guess in `TestTutorial`)?. Because it was not that easy to grasp how these different options to create Templates now work in practice.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 61:
> 59: CompileFramework comp = new CompileFramework();
> 60:
> 61: // Add java source files.
Maybe it would also be nice to see the actually generated strings for the templates. Should we add an easy way to do this just for the tutorials in this file? Maybe we can do it by asking the user to pass an environment property like `-DPrintTemplates=true` or something like that. Or is there already a way provided by the framework to print the resulting templates on demand?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 85:
> 83: // This example shows the use of various Tokens.
> 84: public static String generateWithListOfTokens() {
> 85: // A Template is essencially a function / lambda that produces a
Suggestion:
// A Template is essentially a function / lambda that produces a
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 89:
> 87: var templateClass = Template.make(() -> body(
> 88: // The "body" method is filled by a sequence of "Tokens".
> 89: // This can be Strings and multi-line Strings, but also
Suggestion:
// These can be Strings and multi-line Strings, but also
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 98:
> 96: System.out.println("Hello World!");
> 97: """,
> 98: "int a = ", Integer.valueOf(1), ";\n",
Might be better to use `System.lineSeparator()` instead of `\n` to be platform independent.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 102:
> 100: // Special Float values are "smartly" formatted!
> 101: "float nan = ", Float.valueOf(Float.POSITIVE_INFINITY), ";\n",
> 102: "boolean c = ", Boolean.valueOf(true), ";\n",
Are these explicit calls to `valueOf()` necessary? Aren't these auto-boxed?
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 105:
> 103: // Lists of Tokens are also allowed:
> 104: List.of("int ", "d = 5", ";\n"),
> 105: // That can be great for streaming / mapping over an existing list:
By "that" you just mean the following line? Maybe rephrase to: "We can also stream / map over an existing list or one created on the fly:
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 106:
> 104: List.of("int ", "d = 5", ";\n"),
> 105: // That can be great for streaming / mapping over an existing list:
> 106: List.of(3, 5, 7, 11).stream().map(i -> "System.out.println(" + i + ");\n").toList(),
You can use `Stream.of()`:
Suggestion:
Stream.of(3, 5, 7, 11).map(i -> "System.out.println(" + i + ");\n").toList(),
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 131:
> 129: "System.out.println(", arg, ");\n", // capture arg via lambda argument
> 130: "System.out.println(#arg);\n", // capture arg via hashtag replacement
> 131: "if (#arg != ", arg, ") { throw new RuntimeException(\"mismatch\"); }\n"
When should I use the lambda argument and when the hashtag replacement? Maybe add a comment here for some guidance or link to later tutorials where it becomes obvious.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 143:
> 141: public static void main() {
> 142: """,
> 143: templateHello.withArgs(),
`withArgs()` looks strange when there are no args. Could we find a better name for it? But maybe I'm missing a pattern here.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 156:
> 154: }
> 155:
> 156: // Example with hashtag replacements (arguments and let), and $-name renamings.
Tacking a break now from reviewing. Bookmark for myself :-)
-------------
PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2821262179
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077333259
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077333828
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077338055
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077338763
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077341028
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077349886
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077350455
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077350740
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077355385
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077416870
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077418958
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077420881
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077421229
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077422864
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077453865
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077433478
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077434239
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077441698
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077435250
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077444105
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077436099
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077457338
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077460724
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2077465045
More information about the hotspot-compiler-dev
mailing list