RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently

[Jatin Bhateja]

On Tue, 23 Dec 2025 06:45:46 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

> The test fails intermittently with the following error:
> 
> 
> Caused by: java.lang.RuntimeException: assertEqualsWithTolerance: expected 0.0 but was 1.1754945E-38 (tolerance: 1.4E-44, diff: 1.1754945E-38)
> at compiler.vectorapi.TestVectorOperationsWithPartialSize.verifyAddReductionFloat(TestVectorOperationsWithPartialSize.java:231)
> at compiler.vectorapi.TestVectorOperationsWithPartialSize.testAddReductionFloat(TestVectorOperationsWithPartialSize.java:260)
> 
> 
> The root cause is that the Vector API `reduceLanes()` does not guarantee a specific calculation order for floating-point reduction operations [1]. When the array contains extreme values, this can produce results outside the tolerance range compared to sequential scalar addition.
> 
> For example, given array elements:
> 
> [0.0f, Float.MIN_NORMAL, Float.MAX_VALUE, -Float.MAX_VALUE]
> 
> 
> Sequential scalar addition produces:
> 
> 0.0f + Float.MIN_NORMAL + Float.MAX_VALUE - Float.MAX_VALUE = 0.0f
> 
> 
> However, `reduceLanes()` might compute:
> 
> (0.0f + Float.MIN_NORMAL) + (Float.MAX_VALUE - Float.MAX_VALUE) = Float.MIN_NORMAL
> 
> 
> The difference of the two times of calculation is `Float.MIN_NORMAL` (1.1754945E-38), which exceeds the tolerance of `Math.ulp(0.0f) * 10.0f = 1.4E-44`. Even with a 10x rounding error factor, the tolerance is insufficient for such edge cases.
> 
> Since `reduceLanes()` does not require a specific calculation order, differences from scalar results can be significantly larger when special or extreme maximum/minimum values are present. Using a fixed tolerance is inappropriate for such corner cases.
> 
> This patch fixes the issue by initializing the float array in test with random normal values within a specified range, ensuring the result gap stays within the defined tolerance. 
> 
> Tested locally on my AArch64 and X86_64 machines 500 times, and I didn't observe the failure again.
> 
> [1] https://docs.oracle.com/en/java/javase/25/docs/api/jdk.incubator.vector/jdk/incubator/vector/FloatVector.html#reduceLanes(jdk.incubator.vector.VectorOperators.Associative)

test/hotspot/jtreg/compiler/vectorapi/TestVectorOperationsWithPartialSize.java line 80:

> 78:         random.fill(random.longs(), la);
> 79:         random.fill(random.uniformFloats(1.0f, 5.0f), fa);
> 80:         random.fill(random.uniformDoubles(1.0, 5.0), da);

Ideally our tolerance window should be narrow, and increasing the tolerance range to accomodate outliers as you mentioned in your issue description may defeat the purpose.

Unlike auto-vectorization which adhears strict ordering JLS semantics, vectorAPI relaxes the reduction order to give backends leeway to use parallel reduction not strictly following the sequential order.

There are multiple considerations involed, fallback implimentation performs reduction sequentially, inline expander always relaxes the strict ordering, intrinsification of Add/Mul reductions are only supported by Aarch64, X86 and riscv. 

Computing expected value using parallel reduction can be other alternative but then we may face similar problems on targets which does not intrinsify unordered reductions.

Tolerance modeling is a complex topic and involves relative and absolute error, current 10ULP absolute limit is not generic enough to handle entier spectrum of values, what you have enforced now is a range based tolerance did you try widening the input value range and confirm if 10ULP tolerance limit is sufficient ?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28960#discussion_r2645002228