RFR: 8254129: IR Test Framework to support regex-based matching on the IR in JTreg compiler tests [v3]

[John Tortugo]

On Mon, 19 Apr 2021 12:49:39 GMT, Christian Hagedorn <chagedorn at openjdk.org> wrote:

>> This RFE provides an IR test framework to perform regex-based checks on the C2 IR shape of test methods emitted by the VM flags `-XX:+PrintIdeal` and `-XX:+PrintOptoAssembly`. The framework can also be used for other non-IR matching (and non-compiler) tests by providing easy to use annotations for commonly used testing patterns and compiler control flags.
>> 
>> The framework is based on the ideas of the currently present IR test framework in [Valhalla](https://github.com/openjdk/valhalla/blob/e9c78ce4fcfd01361c35883e0d68f9ae5a80d079/test/hotspot/jtreg/compiler/valhalla/inlinetypes/InlineTypeTest.java) (mainly implemented by @TobiHartmann) which is being used with great success. This new framework aims to replace the old one in Valhalla at some point.
>> 
>> A detailed description about how this new IR test framework works and how it is used is provided in the [README.md](https://github.com/chhagedorn/jdk/blob/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/README.md) file and in the [Javadocs](https://github.com/chhagedorn/jdk/blob/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/doc/jdk/test/lib/hotspot/ir_framework/package-summary.html) written for the framework classes.
>> 
>> To finish a first version of this framework for JDK 17, I decided to leave some improvement possibilities and ideas to be followed up on in additional RFEs. Some ideas are mentioned in "Future Work" in [README.md](https://github.com/chhagedorn/jdk/blob/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/README.md) and were also created as subtasks of this RFE.
>> 
>> Testing (also described in "Internal Framework Tests in [README.md](https://github.com/chhagedorn/jdk/blob/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/README.md)):
>> There are various tests to verify the correctness of the test framework which can be found as JTreg tests in the [tests](https://github.com/chhagedorn/jdk/tree/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/tests) folder. Additional testing was performed by converting all compiler Inline Types test of project Valhalla (done by @katyapav in [JDK-8263024](https://bugs.openjdk.java.net/browse/JDK-8263024)) that used the old framework to the new framework. This provided additional testing for the framework itself. We ran the converted tests with all the flag settings used in hs-tier1-9 and hs-precheckin-comp. For sanity checking, this was also done with a sample IR test in mainline.
>> 
>> Some stats about the framework code added to [ir_framework](https://github.com/chhagedorn/jdk/tree/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework):
>> 
>> -  without the [Javadocs files](https://github.com/chhagedorn/jdk/tree/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/doc) : 60 changed files, 13212 insertions, 0 deletions.
>>    - without the [tests](https://github.com/chhagedorn/jdk/tree/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/tests)  and [examples](https://github.com/chhagedorn/jdk/tree/aa005f384a4567c6c0b5f08f7c5df57f705dc540/test/lib/jdk/test/lib/hotspot/ir_framework/examples) folder: 40 files changed, 6781 insertions
>>       - comments: 2399 insertions (calculated with `git diff --cached !(tests|examples) | grep -c -E "(^[+-]\s*(/)?*)|(^[+-]\s*//)"`)
>>       - which leaves 4382 lines of code inserted 
>> 
>> Big thanks to:
>> - @TobiHartmann for all his help by discussing the new framework and for providing insights from his IR test framework in Valhalla.
>> - @katyapav for converting the Valhalla tests to use the new framework which found some harder to catch bugs in the framework and also some actual C2 bugs.
>> - @iignatev for helping to simplify the framework usage with JTreg and with the framework internal VM calling structure.
>> - and others who provided valuable feedback.
>> 
>> Thanks,
>> Christian
>
> Christian Hagedorn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Apply review comments in code from Igor I. and Vladimir K.

test/lib/jdk/test/lib/hotspot/ir_framework/IRMethod.java line 32:

> 30:  */
> 31: class IRMethod {
> 32:     final private Method method;

The order of these keywords seems to be inverted.

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 43:

> 41: The simplest form of testing provides a single `@Test` annotated method which the framework will invoke as part of the testing. The test method has no or well-defined arguments that the framework can automatically provide. 
> 42: 
> 43: More information on base tasts with a precise definition can be found in the [Javadocs](./doc/jdk/test/lib/hotspot/ir_framework/Test.html). Concrete examples on how to specify a base test can be found in [BaseTestsExample](./examples/BaseTestExample.java).

"tasts"

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 48:

> 46: The base tests do not provide any way of verification by user code. A checked test enabled that by allowing the user to define an additional `@Check` annotated method which is invoked directly after the `@Test` annotated method. This allows the user to perform various checks about the test method including return value verification.
> 47: 
> 48: More information on checked tasts with a precise definition can be found in the [Javadocs](./doc/jdk/test/lib/hotspot/ir_framework/Check.html). Concrete examples on how to specify a checked test can be found in [CheckedTestsExample](./examples/CheckedTestExample.java).

Typo "tasts"

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 53:

> 51: Neither the base nor the checked tests provide any control over how a `@Test` annotated method is invoked in terms of customized argument values and/or conditions for the invocation itself. A custom run test gives full control over the invocation of the `@Test` annotated method to the user. The framework calls a dedicated `@Run` annotated method from which the user can invoke the `@Test` method according to his/her needs.
> 52: 
> 53: More information on checked tasts with a precise definition can be found in the [Javadocs](./doc/jdk/test/lib/hotspot/ir_framework/Run.html). Concrete examples on how to specify a custom run test can be found in [CustomRunTestsExample](./examples/CustomRunTestExample.java).

Typo: "tasts" -> "tests".

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 74:

> 72: 
> 73: ### 2.3 Test VM Flags and Scenarios
> 74: The recommended way to use the framework is by using defining single `@run driver` statement in the JTreg header which, however, does not allow the specification of additional test VM flags. Instead, the user has the possibility to provide VM flags by calling `TestFramework.runWithFlags()` or by creating a `TestFramework` builder object on which `addFlags()` can be called.

Typo: missing article or duplicated words? "is by using defining single"

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 110:

> 108: 
> 109: The framework will spawn a new "test VM" to execute the user defined tests. The test VM collects all tests of the test class specified by the user code in `main()` and ensures that there is no violation of the required format by the framework. In a next step, the framework does the following for each test in general:
> 110: 1. Warm the test up for a predefined number of times (default 2000). This can also be adapted for all tests by using `testFrameworkobject.setDefaultWarmup(100)` or for individual tests with an additional [@Warmup](./doc/jdk/test/lib/hotspot/ir_framework/Warmup.html) annotation. 

The previous section seems to indicate that the default number of warm-up iterations is 200. "-Dwarmup=200".

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 112:

> 110: 1. Warm the test up for a predefined number of times (default 2000). This can also be adapted for all tests by using `testFrameworkobject.setDefaultWarmup(100)` or for individual tests with an additional [@Warmup](./doc/jdk/test/lib/hotspot/ir_framework/Warmup.html) annotation. 
> 111: 2. After the warm-up is finished, the framework compiles the associated `@Test` annotated method at the specified compilation level (default: C2).
> 112: 3. After the compilation, the test is invokes one more time.

Typoe: "invokes" -> "invoked".

test/lib/jdk/test/lib/hotspot/ir_framework/README.md line 116:

> 114: Once the test VM terminates, IR verification (if possible) is performed on the output of the test VM. If any test throws an exception during its execution or if IR matching fails, the failures are collected and reported in a pretty format. Check the standard error and output for more information and how to reproduce these failures.
> 115: 
> 116: Some of the steps above can be different due to the kind of the test or due to using non-default annotation properties. These details and differences are discribed in the Javadocs for the three tests (see section 2.1 Different Tests).

Typo: "discribed" -> "described".

test/lib/jdk/test/lib/hotspot/ir_framework/examples/BaseTestExample.java line 65:

> 63: 
> 64:     public static void main(String[] args) {
> 65:         TestFramework.run(); // equivalent to TestFramework.run(TestSimpleTest.class)

Did you mean same as `TestFramework.run(BaseTestExample.class)` ?

test/lib/jdk/test/lib/hotspot/ir_framework/examples/CheckedTestExample.java line 72:

> 70: 
> 71:     public static void main(String[] args) {
> 72:         TestFramework.run(); // equivalent to TestFramework.run(TestSimpleTest.class)

Did you mean same as `TestFramework.run(CheckedTestExample.class)` ?

test/lib/jdk/test/lib/hotspot/ir_framework/examples/CustomRunTestExample.java line 80:

> 78: 
> 79:     public static void main(String[] args) {
> 80:         TestFramework.run(); // equivalent to TestFramework.run(TestSimpleTest.class)

Did you mean same as `TestFramework.run(CustomRunTestExample.class)` ?

test/lib/jdk/test/lib/hotspot/ir_framework/examples/IRExample.java line 145:

> 143:     @IR(failOn = {IRNode.STORE, IRNode.LOOP}) // LOOP regex not found but STORE regex, letting the rule fail
> 144:     @IR(failOn = {IRNode.LOOP, IRNode.STORE}) // Order does not matter
> 145:     @IR(failOn = {IRNode.STORE, IRNode.LOAD}) // LOOP and STORE regex, letting the rule fail

LOOP regex?

-------------

PR: https://git.openjdk.java.net/jdk/pull/3508