RFR: 8369232: testlibrary_tests/ir_framework/tests/TestScenariosCrossProduct.java timed out [v2]

[Emanuel Peter]

On Tue, 14 Oct 2025 08:41:49 GMT, Christian Hagedorn <chagedorn at openjdk.org> wrote:

>> Generally looks ok. But I'm still a little sad that we don't even get to test a single case end-to-end now. Can we not do at least a 2x2 case?
>
>> Generally looks ok. But I'm still a little sad that we don't even get to test a single case end-to-end now. Can we not do at least a 2x2 case?
> 
> Thanks for your review! That's true, we don't have any end-to-end tests anymore now. I thought about keeping one but the problem is even with a 2x2, we will execute 4 scenarios, i.e. spawning 4 test VMs. But if we assume that, whenever we have the right `Scenario` objects on the `TestFramework.scenarios` list, we can trust the IR framework to do the right things, then it's not required. 
> 
> We currently have scenario end-to-end tests here which should us give this confidence:
> https://github.com/openjdk/jdk/blob/master/test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestScenarios.java
> 
> But I see a missing gap here: `TestScenarios.java` only does end-to-end testing by using `TestFramework.addScenarios()`. But `TestScenariosCrossProduct.java` only checks the `TestFramework.scenarios` object. By only looking at this as a black box, `TestFramework.addScenarios()` could use something different from `TestFramework.scenarios`. Thus, I suggest to add another test that verifies that `TestFramework.addCrossProductScenarios()` creates the same `TestFramework.scenarios` state as if we used `TestFramework.addScenarios()`. I pushed an update with such a test. Let me know what you think.

@chhagedorn How much time would it really take for an end-to-end test with 2x2? The issue up to now was that we had multiple end-to-end tests, and that cumulatively was very much. But just a 2x2 might be some extra seconds, but not too many? There is just always a risk that we test the wrong "partial" things if we don't do end-to-end. That's just my intuition, I can also be ok with what you suggested.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27672#issuecomment-3400769600