RFR: 8344942: Template-Based Testing Framework [v16]

[Emanuel Peter]

On Tue, 20 May 2025 17:06:34 GMT, Roberto Castañeda Lozano <rcastanedalo at openjdk.org> wrote:

>> A few more documentation suggestions, will continue reviewing this changeset over the next days.
>
>> @robcasloz I addressed all your comments :)
> 
> Thanks @eme64!

@robcasloz Thanks for all the suggestions and comments!

> Looks good, thank you for your patience through multiple design and review iterations!
> 
> A couple of observations:
> 
> * I measured the time to generate the code using this framework for [my case study](https://github.com/robcasloz/jdk/blob/JDK-8344942-template-testing/test/hotspot/jtreg/compiler/loopopts/TestArrayFillAntiDependenceTemplatedWithDelimiters.java) (using as fastdebug build) and it seems acceptable (in the milliseconds - same order of magnitude as executing it). The time to compile it using the compile framework is noticeable though (a couple of seconds). If we start adopting this methodology in the large, we might have to find ways to mitigate the increased test execution time.

In my experience, compilation has a significant overhead. But the biggest overhead is **starting** the compilation, and gets only slightly slower with a larger file. So if you have a test that is big enough, eventually running the compiled code is the biggest factor.

> * I struggled understanding the interplay between names, scopes, hooks, etc. (e.g. in the `generateWithDataNamesAndScopes2` example). I think what makes it difficult is that it requires understanding the internals of the framework (order of expansion/evaluation etc.). But I guess this is unavoidable complexity for "advanced" fuzzing use cases only, and I trust that with some effort, the available documentation and examples would be enough to understand the details if necessary.

Yes, the interplay between `DataName`s and scopes is non-trivial. It all comes from the fact that Templates are evaluated in two passes: lambda evaluation and token evaluation. My string based approach only had a single pass, that was an advantage there, then the order is very clear. An alternative is a StringBuilder based approach, where you have to serialize everything with `append` style calls. But that clutters the interface.

The current approach is to have a comma separated list. But that means that those comma separated tokens are generated during the lambda evaluation, and then in a second pass, those tokens are evaluated. It's a nice interface, but leads to some possible confusion about the order.

> * The `DataName` and `StructuralName` concepts seem to belong to a lower level of abstraction (Java language) compared to the rest of the framework's API (abstract language). Just an observation, since I imagine it is difficult (or feels unnecessary) to abstract these concepts when we need to express very Java-like things like mutability or inheritance.

Right, Templates are just about string generation, and with the scopes/hook insertions it becomes sort of an abstract language. I suppose one could have a language without variables, class and function names... but that would be hard to imagine. I think that `DataName` and `StructuralName` are still at a very high level of abstraction, they could work for the languages I know at least. Plus: all we are currently trying to target is Java, and maybe Jasm. Maybe possibly one day other JVM languages.

My goal was to keep the basic framework as simple as possible, and build everything Java in follow-up RFEs, to build up a library of Templates.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/24217#issuecomment-2900729675