RFR: 8344942: Template-Based Testing Framework [v39]

[Emanuel Peter]

On Fri, 16 May 2025 15:31:42 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> **Goal**
>> We want to generate Java source code:
>> - Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
>> - Enable the generation of domain specific fuzzers (e.g. random expressions and statements).
>> 
>> Note: with the Template Library draft I was already able to find a [list of bugs](https://bugs.openjdk.org/issues/?jql=labels%20%3D%20template-framework%20ORDER%20BY%20created%20DESC%2C%20summary%20DESC).
>> 
>> **How to get started**
>> When reviewing, please start by looking at:
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java#L60-L76
>> 
>> We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments `template.withArgs(42, "7")`, producing a `TemplateWithArgs`. This can then be `render`ed to a String. And then that can be compiled and executed with the CompileFramework.
>> 
>> Second, look at this advanced test:
>> https://github.com/openjdk/jdk/blob/77079807042fc5a3af04e0ccccad4ecd89e21cdb/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java#L102-L119
>> 
>> And then for a "tutorial", look at:
>> `test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java`
>> 
>> It shows these features:
>> - The `body` of a Template is essentially a list of `Token`s that are concatenated.
>> - Templates can be nested: a `TemplateWithArgs` is also a `Token`.
>> - We can use `#name` replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
>> - We can use `$var` to make variable names unique: if we applied the same template twice, we would get variable collisions. `$var` is then replaced with e.g. `var_7` in one template use and `var_42` in the other template use.
>> - The use of `Hook`s to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
>> - The use of recursive templates, and `fuel` to limit the recursion.
>> - `Name`s: useful to register field and variable names in code scopes.
>> 
>> Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.
>> https://github.com/openjdk/jdk/blob/d21a8aabaf3b191e851b6997c11bb30fcd0f942f/test/hotspot/jtreg/compiler/lib/template_framework/Template.java#L31-L76
>> 
>> For a better experience, you may want...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
> 
>   tutorial with mutable and subtyping

I'm realizing that we have to do a bit more work on the `Name`s. I'm not yet happy with the API, and I'll need your help here :)

As a reminder: the `Name`s are there to model variables and fields, but possibly also other "names" like methods, classes, etc. Necessary features:
- Add a `Name` for the current code scope.
- Sample a random `Name` from the current code scope.
- The `Name`s each have a `Type`. For example, this is the Java type of a variable/field. These `Type`s could be in subtype/supertype relations with each other.
  - This allows us to sample from only the `int` variables/fields, or from a fields/variables that are of a specific `Type`, or subtype thereof.
- Each `Name` has a weight, so that sampling can be biased, i.e. some variables/fields are used more often than others.
- We want to check if there are any variables of a type or subtype, so we can decide if we can even sample, or have to do something else.

The current API:
- `addName(new Name(String name, Type type, boolean onlyMutable, int weight))`
- `sampleName(Type type, boolean onlyMutable)` -> samples from names of `type` and subtype thereof, possibly constraining to only mutable names if requested.
- `weighNames(Type type, boolean onlyMutable)` -> weighs all names of `type` and subtype thereof, possibly constraining to only mutable names if requested.

So far so good.

One thing I missed: in some cases, you also want to sample from a `Type` or any supertype, and not just subtypes. I had so far only considered loading values from random variables: in that scenario, you must receive an Object/value that fits in your requested `Type`, which can be of that `Type` or any subtype. But there is the inverse problem: you want to store a given value, and that value can only be stored to a `Type` or a supertype thereof. And currently, the API does not allow you to only sample from a `Type` or supertypes.

The API already has methods with multiple arguments, adding a `subtype/supertype` flag will make it even more clunky.
Plus, it may be helpful to the user to also have a way to retrieve all `Name`s so that they can understand better what is happening in their code generation.

I've been thinking about a new interface like this:
- `addName(String name, Type type, boolean onlyMutable, int weight)`, i.e. removing the need for the user to instantiate the `new Name`.
  - We could consider reducing the number of arguments here.
  - `addMutableName(String name, Type type)` could give a mutable name with weight 1.
  - `addImmutableName(String name, Type type)` could give a immutable name with weight 1.
- Access to the `Name`s could be given in more of a stream-like interface:
  - `names()` returns a `NameSetView`.
  - We can sample from it with `names().sample()`, picking a random element biased by the weights.
  - We can further filter the `NameSetView`, with:
    - `names().onlyMutable()` to only get mutable `Names`, and return a filtered `NameSetView`.
    - `names().subtypeOf(Type type)` and `names().supertypeOf(Type type)`
    - We could even add a custom `NameSetView.map(filter)`, with a binary predicate `filter`, so the user has even more control.
  - Next to `names().sample()`, we could also have `names().weight()` and even a `names().toList()` so the user can access all names directly.
Another benefit of this `NameSetView` approach is that it is much easier to extend.

@chhagedorn We can discuss the options this afternoon in an offline meeting, I just thought I wanted to sketch my ideas already now :)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/24217#issuecomment-2889913886