RFR: 8370315: [IR-Framework] Allow scenarios to be run in parallel

[Christian Hagedorn]

On Thu, 30 Oct 2025 16:27:11 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

> ## Issue
> Today, the only practical ways to run IR Framework scenarios in parallel seems to be:
> * spawning threads manually in a single test, or
> * letting jtreg treat each scenario as a separate test (the only way to potentially distribute across hosts).
> 
> This makes it a bit cumbersome to use host CPU cores efficiently when running multiple scenarios within the same test.
> 
> ## Change
> This change introduces a method `TestFramework::startParallel` to execute multiple scenarios concurrently. The implementation:
> * launches one task per scenario and runs them in parallel (by default, the maximum concurrency should match the host’s available cores)
> * captures each task’s `System.out` into a dedicated buffer and flushes it when the task completes to avoid interleaved output between scenarios (Note: only call paths within the `compile.lib.ir_framework` package are modified to per-task output streams. `ProcessTools` methods still write directly to `stdout`, so their output may interleave).
> * adds an option `-DForceSequentialScenarios=true` to force all scenarios to be run sequentially.
> 
> ## Testing
> * Tier 1-3+
> * explicit `ir_framework.tests` runs
>   * added IR-Framework test `TestDForceSequentialScenarios.java` to test forcing sequential testing (checkin the output order) and added a parallel run to `TestScenatios.java` (as well as adding `ForceSequentialScenarios` flag to `TestDFlags.java`)
> 
> As reference: a comparison of the execution time between sequential and parallel of all IR-Framework tests using scenarios on our machines (linux x64/aarch64, macosx x64/aarch64, windows x64 with different number of cores, so the results for a single test might not be relevant) gave me an average speedup of 1.9.

Overall looks good, thanks for improving this! I left a few suggestions.

Now the only question remaining is which tests would already benefit from using the parallel version. I guess we can investigate that separately.

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 456:

> 454:             }
> 455:         } else {
> 456:             startWithScenarios(!FORCE_SEQUENTIAL_SCENARIOS && parallel);

Maybe we can already handle `FORCE_SEQUENTIAL_SCENARIOS` directly in `startParallel()`. Then `parallel` really means parallel. You could also add an additional API comment for `startParallel()` that we can force disable it with the corresponding property flag.

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 762:

> 760:                 outcome = new Outcome(scenario, null, null);
> 761:             } catch (TestFormatException e) {
> 762:                 outcome = new Outcome(scenario, e, null);

Why do you collect the format exceptions here and only throw them later? Is a fail-fast not possible?

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 772:

> 770:                         System.out.println(output);
> 771:                     }
> 772:                 }

This will probably also not be sorted by scenario index? Could we also just gather it here and then dump it after the stream? Maybe we can put `output` into `Outcome` as well as the exceptions by using a `ConcurrentSkipListMap<Scenario, Outcome>` map in the parallel case or a normal `TreeMap` in the non-parallel case.

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 787:

> 785:         outcomes.stream()
> 786:                 .filter(o -> o.other() != null)
> 787:                 .forEach(o -> exceptionMap.put(o.scenario(), o.other()));

You could use a `ConcurrentSkipListMap` in the parallel case instead of a tree map. This would allow us to modify the map in parallel in the stream processing and simplify the code. Moreover, it will be sorted by scenario index which I'm not sure is currently the case?

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 847:

> 845:      * test VM flags, which also determine if IR matching should be done, and then starts the test VM to execute all tests.
> 846:      */
> 847:     private void start(Scenario scenario, PrintStream printStream) {

It might be time to refactor this method and create a scenario version and a non-scenario version. But that's for another day...

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 927:

> 925:         if (testVMProcess == null) {
> 926:             throw new TestFrameworkException("TestVMProcess is null");
> 927:         }

You can use this utility method instead:
Suggestion:

        TestFramework.check(testVMProcess != null, "TestVMProcess must not be null");

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestScenarios.java line 2:

> 1: /*
> 2:  * Copyright (c) 2021, 2023, Oracle and/or its affiliates. All rights reserved.

We can finally comment on hidden non-modified code 🙏 You should also update the copyright here:

Suggestion:

 * Copyright (c) 2021, 2025, Oracle and/or its affiliates. All rights reserved.

-------------

PR Review: https://git.openjdk.org/jdk/pull/28065#pullrequestreview-3409620122
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485608824
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485465527
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485582995
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485476754
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485599238
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485585464
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2485612986