RFR: 8370315: [IR-Framework] Allow scenarios to be run in parallel [v4]

[Christian Hagedorn]

On Wed, 5 Nov 2025 16:36:26 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

>> ## Issue
>> Today, the only practical ways to run IR Framework scenarios in parallel seems to be:
>> * spawning threads manually in a single test, or
>> * letting jtreg treat each scenario as a separate test (the only way to potentially distribute across hosts).
>> 
>> This makes it a bit cumbersome to use host CPU cores efficiently when running multiple scenarios within the same test.
>> 
>> ## Change
>> This change introduces a method `TestFramework::startParallel` to execute multiple scenarios concurrently. The implementation:
>> * launches one task per scenario and runs them in parallel (by default, the maximum concurrency should match the host’s available cores)
>> * captures each task’s `System.out` into a dedicated buffer and flushes it when the task completes to avoid interleaved output between scenarios (Note: only call paths within the `compile.lib.ir_framework` package are modified to per-task output streams. `ProcessTools` methods still write directly to `stdout`, so their output may interleave).
>> * adds an option `-DForceSequentialScenarios=true` to force all scenarios to be run sequentially.
>> 
>> ## Testing
>> * Tier 1-3+
>> * explicit `ir_framework.tests` runs
>>   * added IR-Framework test `TestDForceSequentialScenarios.java` to test forcing sequential testing (checkin the output order) and added a parallel run to `TestScenatios.java` (as well as adding `ForceSequentialScenarios` flag to `TestDFlags.java`)
>> 
>> As reference: a comparison of the execution time between sequential and parallel of all IR-Framework tests using scenarios on our machines (linux x64/aarch64, macosx x64/aarch64, windows x64 with different number of cores, so the results for a single test might not be relevant) gave me an average speedup of 1.9.
>
> Damon Fenacci has updated the pull request incrementally with one additional commit since the last revision:
> 
>   JDK-8370315: fix typo

Sorry, I dropped the ball on this - thanks for the updates! Some more comments but then I think it looks good!

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 433:

> 431:      * Start the testing of the implicitly (by {@link #TestFramework()}) or explicitly (by {@link #TestFramework(Class)})
> 432:      * set test class. Scenarios are run in parallel. Note: scenarios could still be run sequentially if flag
> 433:      *  {@code -DForceSequentialScenarios=true} is given.

Suggestion:

     *  {@code -DForceSequentialScenarios=true} is used.

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 749:

> 747:      * scenarios without prematurely throwing an exception. Format violations, however, are wrong for all scenarios
> 748:      * and thus is reported immediately on the first scenario execution.
> 749:      * @param parallel Run tests concurrently

Suggestion:

     * 
     * @param parallel Run tests concurrently

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 753:

> 751:     private void startWithScenarios(boolean parallel) {
> 752:         Map<Scenario, Exception> exceptionMap = new ConcurrentSkipListMap<>(Comparator.comparingInt(Scenario::getIndex));
> 753:         record Outcome(Scenario scenario, Exception other) {}

Can now be removed

Suggestion:

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java line 775:

> 773:                     if (!output.isEmpty()) {
> 774:                         System.out.println(output);
> 775:                     }

We probably also need to do a similar trick as for the exceptions in order to have ordered stdouts for the scenarios?

test/hotspot/jtreg/compiler/lib/ir_framework/shared/TestFormat.java line 33:

> 31:  */
> 32: public class TestFormat {
> 33:     private static final ThreadLocal<List<String>> threadLocalFailures = ThreadLocal.withInitial(ArrayList<String>::new);

Suggestion:

    private static final ThreadLocal<List<String>> threadLocalFailures = ThreadLocal.withInitial(ArrayList::new);

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestBadFormat.java line 128:

> 126:     private static void expectTestFormatException(Class<?> clazz, Class<?>... helpers) {
> 127:         // Single test
> 128:         boolean exceptionCatched = false;

Nit: catched -> caught

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestScenarios.java line 52:

> 50: public class TestScenarios {
> 51:     public static void main(String[] args) {
> 52:         TestFramework testFramework;

Seems unused

Suggestion:

-------------

PR Review: https://git.openjdk.org/jdk/pull/28065#pullrequestreview-3567845856
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2610902845
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2610908883
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2610906921
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2611008476
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2611036168
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2611048022
PR Review Comment: https://git.openjdk.org/jdk/pull/28065#discussion_r2610967451