RFR: 8344942: Template-Based Testing Framework [v56]

[Roberto Castañeda Lozano]

On Thu, 22 May 2025 07:58:51 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>> 
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>> 
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>> 
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>> 
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>> 
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>> 
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>> 
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>> 
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - remove comment line for Roberto
>  - Apply suggestions from code review
>    
>    Co-authored-by: Roberto Castañeda Lozano <robcasloz at users.noreply.github.com>

Looks good, thank you for your patience through multiple design and review iterations!

A couple of observations:

- I measured the time to generate the code using this framework for [my case study](https://github.com/robcasloz/jdk/blob/JDK-8344942-template-testing/test/hotspot/jtreg/compiler/loopopts/TestArrayFillAntiDependenceTemplatedWithDelimiters.java) (using as fastdebug build) and it seems acceptable (in the milliseconds - same order of magnitude as executing it). The time to compile it using the compile framework is noticeable though (a couple of seconds). If we start adopting this methodology in the large, we might have to find ways to mitigate the increased test execution time.

- I struggled understanding the interplay between names, scopes, hooks, etc. (e.g. in the `generateWithDataNamesAndScopes2` example). I think what makes it difficult is that it requires understanding the internals of the framework (order of expansion/evaluation etc.). But I guess this is unavoidable complexity for "advanced" fuzzing use cases only, and I trust that with some effort, the available documentation and examples would be enough to understand the details if necessary.

- The `DataName` and `StructuralName` concepts seem to belong to a lower level of abstraction (Java language) compared to the rest of the framework's API (abstract language). Just an observation, since I imagine it is difficult (or feels unnecessary) to abstract these concepts when we need to express very Java-like things like mutability or inheritance.

test/hotspot/jtreg/compiler/lib/template_framework/DataName.java line 42:

> 40:  *               {@link DataName} when sampling later on.
> 41:  */
> 42: public record DataName(String name, DataName.Type type, boolean mutable, int weight) implements Name {

Since `DataName` represents "fields and variables", you might consider renaming it to `VariableName` or `VarName` for clarity - fields can be considered variables too (as in [instance variables](https://en.wikipedia.org/wiki/Instance_variable)). This is just a minor, optional suggestion, in case you still have some fuel left for this PR ;)

-------------

Marked as reviewed by rcastanedalo (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/24217#pullrequestreview-2860094632
PR Review Comment: https://git.openjdk.org/jdk/pull/24217#discussion_r2102054527