RFR: 8344942: Template-Based Testing Framework

Emanuel Peter epeter at openjdk.org
Tue Mar 25 10:09:27 UTC 2025


**Goal**
We want to generate Java source code:
- Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
- Enable the generation of domain specific fuzzers (e.g. random expressions and statements).

**How to get started**
When reviewing, please start by looking at:
https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76

We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.

And then for a "tutorial", look at:
`test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`

It shows these features:
- The `body` of a Template is essentially a list of `Token`s that are concatenated.
- Templates can be nested: a `TemplateWithArgs` is also a `Token`.
- We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
- We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
- The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
- The use of recursive templates, and `fuel` to limit the recursion.
- `Name`s: useful to register field and variable names in code scopes.

Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76

For a better experience, you may want to generate the `javadocs`:
`javadoc -sourcepath test/hotspot/jtreg:./test/lib compiler.lib.template_framework`

**History**
@TobiHartmann and I have played with code generators for a while, and have had the dream of doing that in a more principled way. And to hopefully make it more accessible for other VM developers, with the goal of improving test coverage.

@TobiHartmann started with an initial prototype. @tobiasholenstein took over, and worked with our intern @maasaid. Their approach was to take old tests and templetize them ([draft PR](https://github.com/openjdk/jdk/pull/22358)). Their templates had holes for replacements, and a `$` prefix for variables name replacement.

Once @tobiasholenstein left, I took over the project, and focused on nested template use, where templates could be passed arguments. I kept it string based, which worked, but the resulting syntax was a little cryptic. Debugging was difficult, as I had to produce custom stack traces, print available variables, etc. [Here is the string syntax based version](https://github.com/openjdk/jdk/pull/22483).

@theoweidmannoracle reviewed my draft, and had some very good feedback. He was frustrated with the complexity, and also the string syntax. Over several iterations, we kept most of the complexity (because code generation is a little complex), but changed the approach to use Java Generics. I took one of his prototypes and fleshed it out with all the other necessary features.

**Related Work**
There is lots of related work, for test generation:
- `Verify.java`: verify results.
- `Generators.java`: generate random inputs from "interesting distributions".
- `Compile Framework`: take string source code, compile and class-load it for execution.
- [JDK-8352861](https://bugs.openjdk.org/browse/JDK-8352861) Template-Framework Library: future work. Provide lots of useful templates, generate Expressions and nested Statements, etc. [Here a previous PR where I am experimenting with different Library features.](https://github.com/openjdk/jdk/pull/23418). I decided to already include the Hooks from the library, as they are useful to have in the tests and examples already.

-------------

Commit messages:
 - fix tests
 - whitespace
 - whitespace
 - fix whitespace
 - JDK-8344942

Changes: https://git.openjdk.org/jdk/pull/24217/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=24217&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8344942
  Stats: 3920 lines in 24 files changed: 3920 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/24217.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/24217/head:pull/24217

PR: https://git.openjdk.org/jdk/pull/24217


More information about the hotspot-compiler-dev mailing list