RFR: 8337221: CompileFramework: test library to conveniently compile java and jasm sources for fuzzing
Emanuel Peter
epeter at openjdk.org
Thu Sep 12 15:57:26 UTC 2024
**Motivation**
I want to write small dedicated fuzzers:
- Generate `java` and `jasm` source code: just some `String`.
- Quickly compile it (with this framework).
- Execute the compiled code.
The primary users of the CompileFramework are Compiler-Engineers. Imagine you are working on some optimization. You already have a list of **hand-written tests**, but you are worried that this does not give you good coverage. You also do not trust that an existing Fuzzer will catch your bugs (at least not fast enough). Hence, you want to **script-generate** a large list of tests. But where do you put this script? It would be nice if it was also checked in on git, so that others can modify and maintain the test easily. But with such a script, you can only generate a **static test**. In some cases that is good enough, but sometimes the list of all possible tests your script would generate is very very large. Too large. So you need to randomly sample some of the tests. At this point, it would be nice to generate different tests with every run: a "mini-fuzzer" or a **fuzzer dedicated to a compiler feature**.
**The CompileFramework**
Java sources are compiled with `javac`, jasm sources with `asmtools` that are delivered with `jtreg`.
An important factor: Integration with the IR-Framwrork (`TestFramework`): we want to be able to generate IR-rules for our tests.
I implemented a first, simple version of the framework. I added some tests and examples.
**Example**
CompileFramework comp = new CompileFramework();
comp.add(SourceCode.newJavaSourceCode("XYZ", "<source-code-here>"));
comp.compile();
comp.invoke("XYZ", "test", new Object[] {5}); // XYZ.test(5);
https://github.com/openjdk/jdk/blob/e869cce8092ee995cf2f3ad1ab2bca69c5e256ab/test/hotspot/jtreg/testlibrary_tests/compile_framework/examples/SimpleJavaExample.java#L42-L74
**Below some use cases: tests that would have been better with the CompileFramework**
**Use case: test/hotspot/jtreg/compiler/loopopts/superword/TestAlignVectorFuzzer.java**
I needed to test loops with various `init / stride / limit / scale / unrolling-factor / ...`.
For this I used `MethodHandle constant = MethodHandles.constant(int.class, value);`,
to be able to chose different values before the C2 compilation, and then the C2 compilation would see them as constants and optimize assuming those constants. This works, but is difficult to extract reproducers once something inevitably breaks in the VM code (i.e. most likely in loop-opts or SuperWord).
**Use case : test/hotspot/jtreg/compiler/loopopts/superword/TestDependencyOffsets.java**
Currently, I generate it with a `generator.py` that I update and keep in the latest related JBS issue. Not great, because the generator code is not properly version-controlled. And it is python which is not really used elsewhere in our stack.
Basically, I generate different loops with different parameters. I need to iterate over a list of distances and generate one loop for each of them. With this generator.py I must chose a fixed list of values, but with a fuzzer I could both pick a list of fixed values but also sprinkle in some random values for higher coverage.
**Use case: test/hotspot/jtreg/compiler/loopopts/superword/TestMemorySegment.java**
Test memory segments with different kinds of backing data types (array, native, etc). Then use various different address shapes. Verify that they vectorize as expected. It is very cumbersome to generate all the examples by hand.
**More use cases**
I'm currently looking into extending both `MergeStores` and `SuperWord`. This will require more tests. The test coverage could be improved even for current features. Often when I add a big test I seem to catch some already existing bug.
-------------
Commit messages:
- fix paths for windows, had compile issue
- increase compile timeut
- rm unnecessary test
- Merge branch 'master' into fuzzer-test
- Merge branch 'master' into fuzzer-test
- name timeout better
- stub of TestMergeStoresFuzzer
- private source and classes directory per CompileFramework
- give javac the classesDir from jasm compilation, so java files can reference jasm classes
- make it multi-threading safe
- ... and 41 more: https://git.openjdk.org/jdk/compare/8fce5275...881c76bf
Changes: https://git.openjdk.org/jdk/pull/20184/files
Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=20184&range=00
Issue: https://bugs.openjdk.org/browse/JDK-8337221
Stats: 1266 lines in 13 files changed: 1266 ins; 0 del; 0 mod
Patch: https://git.openjdk.org/jdk/pull/20184.diff
Fetch: git fetch https://git.openjdk.org/jdk.git pull/20184/head:pull/20184
PR: https://git.openjdk.org/jdk/pull/20184
More information about the hotspot-compiler-dev
mailing list