RFR: 8344942: Template-Based Testing Framework [v2]
Christian Hagedorn
chagedorn at openjdk.org
Mon Mar 31 08:53:14 UTC 2025
On Thu, 27 Mar 2025 08:03:12 GMT, Emanuel Peter <epeter at openjdk.org> wrote:
>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>>
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>>
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>>
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>>
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>>
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>>
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>>
>> For a better experience, you may want to generate the `javadocs`:
>> `javadoc -sourcepath test/hotspot/jtreg:./test/lib compiler.lib.template_framework`
>>
>> **History**
>> @TobiHartmann and I have played with code generators for a while, and have had ...
>
> Emanuel Peter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision:
>
> - Merge branch 'master' into JDK-8344942-TemplateFramework-v3
> - fix tests
> - whitespace
> - whitespace
> - fix whitespace
> - JDK-8344942
Impressive work Emanuel! This is a very valuable framework to improve our testing coverage.
Here are some first comments after skimming some of the code and comments. I will deep dive more into the code and documentation later.
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 2:
> 1: # Template Framework
> 2: The Template Framework allows the generation of code with Templates. The goal is that these Templates are easy to write, and allow regression tests to cover a larger scope, and to make temlate based fuzzing easy to extend.
Suggestion:
The Template Framework allows the generation of code with Templates. The goal is that these Templates are easy to write, and allow regression tests to cover a larger scope, and to make template based fuzzing easy to extend.
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 6:
> 4: The Template Framework only generates code in the form of a String. This code can then be compiled and executed, for example with help of the [Compile Framework](../compile_framework/README.md).
> 5:
> 6: The basic functionalities of the Template Framework are described in the [Template Class](./Template.java), together with some examples. More examples can be found in [TestSimple.java](../../../testlibrary_tests/template_framework/examples/TestSimple.java) and [TestTutorial.java](../../../testlibrary_tests/template_framework/examples/TestTutorial.java).
Suggestion:
The basic functionalities of the Template Framework are described in the [Template Interface](./Template.java), together with some examples. More examples can be found in [TestSimple.java](../../../testlibrary_tests/template_framework/examples/TestSimple.java) and [TestTutorial.java](../../../testlibrary_tests/template_framework/examples/TestTutorial.java).
test/hotspot/jtreg/compiler/lib/template_framework/README.md line 8:
> 6: The basic functionalities of the Template Framework are described in the [Template Class](./Template.java), together with some examples. More examples can be found in [TestSimple.java](../../../testlibrary_tests/template_framework/examples/TestSimple.java) and [TestTutorial.java](../../../testlibrary_tests/template_framework/examples/TestTutorial.java).
> 7:
> 8: The [Template Library](../template_library/README.md) provides a large number of Templates which can be used to create anything from simple regression tests to complex fuzzers.
`template_library/README.md` does not exist.
Another thought: Should `template_library` be a subfolder in `template_framework`? Otherwise, it could suggest to be something separate from the Template Framework.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 32:
> 30:
> 31: /**
> 32: * {@link Template}s are used to generate code, based on {@link Token} which are rendered to {@link String}.
Tokens and Strings?
Suggestion:
* {@link Template}s are used to generate code, based on {@link Token}s which are rendered to {@link String}s.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 35:
> 33: *
> 34: * <p>
> 35: * A {@link Template} can have zero or more arguments, and for each number of arguments there is an implementation
Since the `README` refers to this file for more information (which is perfectly fine to avoid repetition), I naturally started to read here at the top. But in this paragraph, we are already explaining details of the implementation without giving a more general introduction and motivation for the Template Framework which can leave readers without background confused. I think it could be worth to spend some more time in the README (which you can then also just refer to when suggesting to use the framework in PRs). You could cover: Why is the framework useful/why should I care about it, some leading example/testing scenario and why it is really hard to cover that without the framework (i.e. what we've done so far until today), how does the framework roughly work, how easy is it to write tests (you can then reference example tests from there) etc.
You can still do many references from the `README` to classes and examples.
Side note: I admit that I could have extended the IR framework `README` with a more motivational introduction as well - maybe I will add that later.
test/hotspot/jtreg/compiler/lib/template_framework/Template.java line 41:
> 39: * the {@link Template}s provide <strong>hashtag replacements</strong> in the Strings: the {@link Template} argument
> 40: * names are captured, and the argument values automatically replace any {@code "#name"} in the Strings. See the
> 41: * different overloads of {@link make} for examples. Additional hashtag replacements can be defined with {@link let}.
General Javadocs comment: You should preceed method names with `#` in order to create a proper link.
Without:
With (i.e. `{@link make}` and `{@link #let})`:
-------------
PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2728506638
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020575952
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020584958
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020579026
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020593912
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020620543
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2020591523
More information about the hotspot-compiler-dev
mailing list