RFR: 8344942: Template-Based Testing Framework [v16]

Roberto Castañeda Lozano rcastanedalo at openjdk.org
Wed May 14 13:43:11 UTC 2025 

On Mon, 12 May 2025 14:26:37 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>> 
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>> 
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>> 
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>> 
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>> 
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>> 
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>> 
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>> 
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
> 
>   more documentation fixes

A few more documentation suggestions, will continue reviewing this changeset over the next days.

test/hotspot/jtreg/compiler/lib/template_framework/README.md line 6:

> 4: We want to make it easy to generate variants of tests. Often, we would like to have a set of tests, corresponding to a set of types, a set of operators, a set of constants, etc. Writing all the tests by hand is cumbersome or even impossible. When generating such tests with scripts, it would be preferable if the code generation happens automatically, and the generator script was checked into the code base. Code generation can go beyond simple regression tests, and one might want to generate random code from a list of possible templates, to fuzz individual Java features and compiler optimizations.
> 5: 
> 6: The Template Framework provides a facility to generate code with Templates. Templates are essencially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of Templates.

Suggestion:

The Template Framework provides a facility to generate code with Templates. A Template is essentially a list of tokens that are concatenated (i.e. rendered) to a String. The Templates can have "holes", which are filled (replaced) by different values at each Template instantiation. For example, these "holes" can be filled with different types, operators or constants. Templates can also be nested, allowing a modular use of Templates.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 46:

> 44:  *
> 45:  * <p>
> 46:  * The Template Framework provides a facility to generate code with Templates. Templates are essencially a list

Same grammar/spelling glitch as in the README file.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 50:

> 48:  * filled (replaced) by different values at each Template instantiation. For example, these "holes" can
> 49:  * be filled with different types, operators or constants. Templates can also be nested, allowing a modular
> 50:  * use of Templates.

I suggest replacing this text with a high-level summary of what is already written in the README file (or the other way round, write a short summary in the README file and point to this file for more background).

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 150:

> 148:  *
> 149:  * <p>
> 150:  * Given a {@link UnfilledTemplate}, one must apply the required number of arguments, i.e. fill

Suggestion:

 * Given an {@link UnfilledTemplate}, one must apply the required number of arguments, i.e. fill

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 152:

> 150:  * Given a {@link UnfilledTemplate}, one must apply the required number of arguments, i.e. fill
> 151:  * the Template, to arrive at a {@link FilledTemplate}. Note: {@link Template#make(Supplier)},
> 152:  * i.e. the making a Template with zero arguments directly returns a {@link FilledTemplate},

Suggestion:

 * i.e. making a Template with zero arguments directly returns a {@link FilledTemplate},

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 168:

> 166:  * <p>
> 167:  * When using nested Templates, there can be collisions with identifiers (e.g. variable names and method names).
> 168:  * For this, Templates provide <strong>dollar replacements</strong>, which automaticall rename any

Suggestion:

 * For this, Templates provide <strong>dollar replacements</strong>, which automatically rename any

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 228:

> 226: 
> 227:     /**
> 228:      * Creates a {@link UnfilledTemplate} with one argument.

Suggestion:

     * Creates an {@link UnfilledTemplate} with one argument.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 232:

> 230:      *
> 231:      * <p>
> 232:      * Here an example with template argument {@code 'a'}, captured once as string name

Suggestion:

     * Here is an example with template argument {@code 'a'}, captured once as string name

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 248:

> 246:      * @param <A> Type of the (first) argument.
> 247:      * @param arg0Name The name of the (first) argument for hashtag replacement.
> 248:      * @return A {@link UnfilledTemplate} with one argument.

Suggestion:

     * @return An {@link UnfilledTemplate} with one argument.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 255:

> 253: 
> 254:     /**
> 255:      * Creates a {@link UnfilledTemplate} with two arguments.

Suggestion:

     * Creates an {@link UnfilledTemplate} with two arguments.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 277:

> 275:      * @param <B> Type of the second argument.
> 276:      * @param arg1Name The name of the second argument for hashtag replacement.
> 277:      * @return A {@link UnfilledTemplate} with two arguments.

Suggestion:

     * @return An {@link UnfilledTemplate} with two arguments.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 284:

> 282: 
> 283:     /**
> 284:      * Creates a {@link UnfilledTemplate} with three arguments.

Suggestion:

     * Creates an {@link UnfilledTemplate} with three arguments.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 294:

> 292:      * @param <C> Type of the third argument.
> 293:      * @param arg2Name The name of the third argument for hashtag replacement.
> 294:      * @return A {@link UnfilledTemplate} with three arguments.

Suggestion:

     * @return An {@link UnfilledTemplate} with three arguments.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 328:

> 326:     /**
> 327:      * Retrieves the dollar replacement of the {@code 'name'} for the
> 328:      * current Template that is being instanciated. It returns the same

Suggestion:

     * current Template that is being instantiated. It returns the same

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 364:

> 362:      * @param key Name for the hashtag replacement.
> 363:      * @param value The value that the hashtag is replaced with.
> 364:      * @return A token that does nothing, so that the {@link #let} cal can easily be put in a list of tokens

Suggestion:

     * @return A token that does nothing, so that the {@link #let} can easily be put in a list of tokens

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 390:

> 388:      * @param <T> The type of the value.
> 389:      * @param function The function that is applied with the provided {@code 'value'}.
> 390:      * @return A token that does nothing, so that the {@link #let} cal can easily be put in a list of tokens

Suggestion:

     * @return A token that does nothing, so that the {@link #let} can easily be put in a list of tokens

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 390:

> 388:      * @param <T> The type of the value.
> 389:      * @param function The function that is applied with the provided {@code 'value'}.
> 390:      * @return A token that does nothing, so that the {@link #let} cal can easily be put in a list of tokens

The return type is something else in this case, no?

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 394:

> 392:      * @throws RendererException if there is a duplicate hashtag {@code key}.
> 393:      */
> 394:     static <T> TemplateBody let(String key, T value, Function<T, TemplateBody> function) {

I found it a bit confusing to find two methods called `let` that are pretty different in nature. Maybe you could rename this one to e.g. `letIn`?

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 405:

> 403: 
> 404:     /**
> 405:      * The default amount of fuel spent per Template. It is suptracted from the current {@link #fuel} at every

Suggestion:

     * The default amount of fuel spent per Template. It is subtracted from the current {@link #fuel} at every

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 411:

> 409: 
> 410:     /**
> 411:      * The current remaining fuel for nested Templates. Every level of Template nestig

Suggestion:

     * The current remaining fuel for nested Templates. Every level of Template nesting

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 419:

> 417:      * Example of a recursive Template, which checks the remaining {@link #fuel} at every level,
> 418:      * and terminates if it reaches zero. It also demonstrates the use of {@link TemplateBinding} for
> 419:      * the recursive use of Templates. We {@link FilledTemplate#render} with {@code 30} total fuel, and spending {@code 5} fuel at each recursion level.

Suggestion:

     * the recursive use of Templates. We {@link FilledTemplate#render} with {@code 30} total fuel, and spend {@code 5} fuel at each recursion level.

test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 461:

> 459:      * @param name The {@link Name} to be added to the current code frame.
> 460:      * @return The token that performs the defining action.
> 461:      */

The concept of "code frame" is not clear here, maybe you can introduce it or replace it by a concept that is already defined?

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 37:

> 35: public interface UnfilledTemplate {
> 36:     /**
> 37:      * A {@link UnfilledTemplate} with no arguments.

Suggestion:

     * An {@link UnfilledTemplate} with no arguments.

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 48:

> 46:         /**
> 47:          * Creates a {@link FilledTemplate} which can be used as a {@link Token} inside
> 48:          * a {@link UnfilledTemplate} for nested code generation, and it can also be used with

Suggestion:

         * an {@link UnfilledTemplate} for nested code generation, and it can also be used with

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 60:

> 58: 
> 59:     /**
> 60:      * A {@link UnfilledTemplate} with one argument.

Suggestion:

     * An {@link UnfilledTemplate} with one argument.

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 74:

> 72:         /**
> 73:          * Creates a {@link FilledTemplate} which can be used as a {@link Token} inside
> 74:          * a {@link UnfilledTemplate} for nested code generation, and it can also be used with

Suggestion:

         * an {@link UnfilledTemplate} for nested code generation, and it can also be used with

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 87:

> 85: 
> 86:     /**
> 87:      * A {@link UnfilledTemplate} with two arguments.

Suggestion:

     * An {@link UnfilledTemplate} with two arguments.

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 103:

> 101:         /**
> 102:          * Creates a {@link FilledTemplate} which can be used as a {@link Token} inside
> 103:          * a {@link UnfilledTemplate} for nested code generation, and it can also be used with

Suggestion:

         * an {@link UnfilledTemplate} for nested code generation, and it can also be used with

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 139:

> 137: 
> 138:     /**
> 139:      * A {@link UnfilledTemplate} with three arguments.

Suggestion:

     * An {@link UnfilledTemplate} with three arguments.

test/hotspot/jtreg/compiler/lib/template_framework/UnfilledTemplate.java line 157:

> 155:         /**
> 156:          * Creates a {@link FilledTemplate} which can be used as a {@link Token} inside
> 157:          * a {@link UnfilledTemplate} for nested code generation, and it can also be used with

Suggestion:

         * an {@link UnfilledTemplate} for nested code generation, and it can also be used with

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java line 96:

> 94:         // - For a chosen type, operator, and generator.
> 95:         // - The variable name "GOLD" and the test name "test" would get conflicts
> 96:         //   if we instanciate the template multiple times. Thus, we use the $ prefix

Suggestion:

        //   if we instantiate the template multiple times. Thus, we use the $ prefix

-------------

PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2839877707
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088754564
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088764847
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088762304
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088854248
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088857353
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088892230
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088947906
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088911264
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088956199
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088949253
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088956811
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088950157
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088957259
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088964462
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088918402
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088922386
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088970477
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088927510
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088944076
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088929241
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088931472
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088939632
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088957977
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088958376
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088958853
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088959282
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088959723
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088960151
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088960450
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088960797
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2088965249

More information about the hotspot-compiler-dev mailing list