RFR: 8344942: Template-Based Testing Framework [v16]
Manuel Hässig
mhaessig at openjdk.org
Wed May 14 13:16:11 UTC 2025
On Mon, 12 May 2025 14:26:37 GMT, Emanuel Peter <epeter at openjdk.org> wrote:
>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>>
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>>
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>>
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>>
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>>
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>>
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>>
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>>
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
>
> more documentation fixes
test/hotspot/jtreg/compiler/lib/template_framework/CodeFrame.java line 84:
> 82: * Creates a normal frame, which has a {@link parent} and which defines an inner
> 83: * {@link NameSet}, for the names that are generated inside this frame. Once this
> 84: * frame is exited, the name from inside this frame are not available any more.
Suggestion:
* frame is exited, the name from inside this frame are not available anymore.
Another typo nit.
test/hotspot/jtreg/compiler/lib/template_framework/CodeFrame.java line 91:
> 89:
> 90: /**
> 91: * Creates a special frame, which has a {@link parent} and but uses the {@link NameSet}
Suggestion:
* Creates a special frame, which has a {@link parent} but uses the {@link NameSet}
Not 100% sure, but I think you meant to use the "but".
test/hotspot/jtreg/compiler/lib/template_framework/Name.java line 40:
> 38: * @param type The type with which we restrict {@link Template#weighNames} and {@link Template#sampleName}.
> 39: * @param mutable Defines if the name is considered mutable or immutable.
> 40: * @param weight The weight measured by {@link Template#weighNames} and according to which we sample with {@link Template#sampleName}.
Suggestion:
* @param weight The weight measured by {@link Template#weightNames} and according to which we sample with {@link Template#sampleName}.
test/hotspot/jtreg/compiler/lib/template_framework/NameSet.java line 37:
> 35: */
> 36: class NameSet {
> 37: static final Random RANDOM = Utils.getRandomInstance();
IIUC, we can generate slightly different programs by sampling variables from name sets. For the purposes of reproducing tests it might be useful to seed the randomness and print the seed.
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 6:
> 4: We want to make it easy to generate variants of tests. Often, we would like to have a set of tests, corresponding to a set of types, a set of operators, a set of constants, etc. Writing all the tests by hand is cumbersome or even impossible. When generating such tests with scripts, it would be preferable if the code generation happens automatically, and the generator script was checked into the code base. Code generation can go beyond simple regression tests, and one might want to generate random code from a list of possible templates, to fuzz individual Java features and compiler optimizations.
> 5:
> 6: The Template Framework provides a facility to generate code with Templates. Templates are essencially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of Templates.
Suggestion:
The Template Framework provides a facility to generate code with Templates. Templates are essentially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of Templates.
test/hotspot/jtreg/compiler/lib/template_framework/Renderer.java line 215:
> 213:
> 214: // We need a CodeFrame to which the hook can insert code. That way, name
> 215: // definitions at the hook cannot excape the hookCodeFrame.
Suggestion:
// definitions at the hook cannot escape the hookCodeFrame.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 328:
> 326: /**
> 327: * Retrieves the dollar replacement of the {@code 'name'} for the
> 328: * current Template that is being instanciated. It returns the same
Suggestion:
* current Template that is being instantiated. It returns the same
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 357:
> 355: * let("b", a * 5),
> 356: * """
> 357: * System.out.prinln("Use a and b with hashtag replacement: #a and #b");
Suggestion:
* System.out.println("Use a and b with hashtag replacement: #a and #b");
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 405:
> 403:
> 404: /**
> 405: * The default amount of fuel spent per Template. It is suptracted from the current {@link #fuel} at every
Suggestion:
* The default amount of fuel spent per Template. It is subtracted from the current {@link #fuel} at every
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 411:
> 409:
> 410: /**
> 411: * The current remaining fuel for nested Templates. Every level of Template nestig
Suggestion:
* The current remaining fuel for nested Templates. Every level of Template nesting
test/hotspot/jtreg/compiler/lib/template_framework/TemplateBinding.java line 28:
> 26: /**
> 27: * To facilitate recursive uses of Templates, e.g. where a template uses
> 28: * itself, where a template needs to be referenced before it is fully defined,
Suggestion:
* itself and needs to be referenced before it is fully defined,
Nit for slightly better text flow.
test/hotspot/jtreg/compiler/lib/template_framework/TemplateFrame.java line 36:
> 34: * The {@link parent} relationship provides a trace for the use chain of templates.
> 35: * The {@link fuel} is reduced over this chain, to give a heuristic on how much time
> 36: * is spend on the code from the template corrsponding to the frame, and to give a
Suggestion:
* is spent on the code from the template corresponding to the frame, and to give a
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java line 80:
> 78: comp.compile();
> 79:
> 80: // Object ret = p.xyz.InnterTest.main();
Suggestion:
// Object ret = p.xyz.InnerTest.main();
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java line 96:
> 94: // - For a chosen type, operator, and generator.
> 95: // - The variable name "GOLD" and the test name "test" would get conflicts
> 96: // if we instanciate the template multiple times. Thus, we use the $ prefix
Suggestion:
// if we instantiate the template multiple times. Thus, we use the $ prefix
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java line 51:
> 49: comp.compile();
> 50:
> 51: // Object ret = p.xyz.InnterTest.test();
Suggestion:
// Object ret = p.xyz.InnerTest.test();
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 77:
> 75: comp.compile();
> 76:
> 77: // Object ret = p.xyz.InnterTest1.main();
Suggestion:
// Object ret = p.xyz.InnerTest1.main();
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 140:
> 138: // replacements. Appending as a Token works whenever one has a reference
> 139: // to the Object in Java code. But often, this is rather cumbersome and
> 140: // looks awkward, given al the additional quotes and commans required.
Suggestion:
// looks awkward, given all the additional quotes and commands required.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 249:
> 247: // We can define a custom hook.
> 248: // Note: generally we prefer using the pre-defined CLASS_HOOK and METHOD_HOOK from the library,
> 249: // when ever possible. See also the example after this one.
Perhaps this tutorial should quickly explain the concept of hooks before using them. That might be a duplication of the documentation in `Hook.java`, but it would help the reading flow of the tutorial to grok the concept a bit easier without having to jump between files.
I guess it is more of a problem when reading the tutorial on Github, as in the IDE you would get a preview for the documentation.
Perhaps like this:
Suggestion:
// In this example, we look at the use of Hooks. They allow us to refer back in the Template and
// to outer scopes, e.g. to define a field at the top of the class from inside a method.
public static String generateWithCustomHooks() {
// We can define a custom hook.
// Note: generally we prefer using the pre-defined CLASS_HOOK and METHOD_HOOK from the library,
// when ever possible. See also the example after this one.
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java line 299:
> 297:
> 298: // We saw the use of custom hooks above, but now we look at the use of CLASS_HOOK and METHOD_HOOK
> 299: // from the Temlate Library.
Suggestion:
// from the Template Library.
-------------
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088847371
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088849342
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088860008
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088871462
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088872395
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088876295
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088798775
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088798370
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088800322
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088809659
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088895839
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088899984
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088828038
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088828476
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088825587
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088824843
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088823501
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088690378
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088695925
More information about the hotspot-compiler-dev
mailing list