RFR: 8344942: Template-Based Testing Framework

[Emanuel Peter]

On Thu, 27 Mar 2025 05:19:43 GMT, Galder Zamarreño <galder at openjdk.org> wrote:

>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>> 
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>> 
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>> 
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>> 
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>> 
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>> 
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>> 
>> For a better experience, you may want to generate the `javadocs`:
>> `javadoc -sourcepath test/hotspot/jtreg:./test/lib compiler.lib.template_framework`
>> 
>> **History**
>> @TobiHartmann and I have played with code generators for a while, and have had ...
>
> Looks great though I'm not too familiar with the code to be able to do a reasonable review, but I had a question:
> 
> Have you got any practical use case that can show where you've used this and show what it takes to build such a use case? `VectorReduction2` or similar type of microbenchmarks would be great to see auto generated using this? 
> 
> The reason I ask this is because I feel that something that is missing in this PR is a small practical use case where this framework is put into action to actually generate some jtreg/IR/microbenchmark test and see how it runs as part of the CI in the PR. WDYT?

@galderz Thanks for your questions!

> Looks great though I'm not too familiar with the code to be able to do a reasonable review

Well the code is all brand new, so really anybody could review ;)

> Have you got any practical use case that can show where you've used this and show what it takes to build such a use case?

I actually have a list of experiments in this branch (it is linked in the PR description): https://github.com/openjdk/jdk/pull/23418
Some of them use the IR framework, though for now just as a testing harness, not for IR rules. Generating IR rules automatically requires quite a bit of logic...
I hope that is satisfactory for now?

Ah, but there was this test:
`test/hotspot/jtreg/compiler/loopopts/superword/TestDependencyOffsets.java`
I did now not refactor it, but it would not be too hard to see how to use the Templates for it. And I do generate IR rules in that one. I don't super like just refactoring old tests... there is always a risk of breaking it and then coverage is worse than before...

> The reason I ask this is because I feel that something that is missing in this PR is a small practical use case where this framework is put into action to actually generate some jtreg/IR/microbenchmark test and see how it runs as part of the CI in the PR. WDYT?

I also have a few tests in this PR that just generate regular JTREG tests, without the IR framework, did you see those?

> VectorReduction2 or similar type of microbenchmarks would be great to see auto generated using this?

I don't yet have a solution for microbenchmarks. It's mostly an issue of including the `test/hotspot/jtreg/compiler/lib` path... And I fear that JMH requires all benchmark code to be compiled beforehand, and not dynamically as I do with the class loader. But maybe there is a solution for that.

The patch is already quite large, and so I wanted to just publish the basic framework. Do you think that is ok?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/24217#issuecomment-2756920241