RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently [v2]

[Jie Fu]

On Thu, 25 Dec 2025 07:06:39 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

>> The test fails intermittently with the following error:
>> 
>> 
>> Caused by: java.lang.RuntimeException: assertEqualsWithTolerance: expected 0.0 but was 1.1754945E-38 (tolerance: 1.4E-44, diff: 1.1754945E-38)
>> at compiler.vectorapi.TestVectorOperationsWithPartialSize.verifyAddReductionFloat(TestVectorOperationsWithPartialSize.java:231)
>> at compiler.vectorapi.TestVectorOperationsWithPartialSize.testAddReductionFloat(TestVectorOperationsWithPartialSize.java:260)
>> 
>> 
>> The root cause is that the Vector API `reduceLanes()` does not guarantee a specific calculation order for floating-point reduction operations [1]. When the array contains extreme values, this can produce results outside the tolerance range compared to sequential scalar addition.
>> 
>> For example, given array elements:
>> 
>> [0.0f, Float.MIN_NORMAL, Float.MAX_VALUE, -Float.MAX_VALUE]
>> 
>> 
>> Sequential scalar addition produces:
>> 
>> 0.0f + Float.MIN_NORMAL + Float.MAX_VALUE - Float.MAX_VALUE = 0.0f
>> 
>> 
>> However, `reduceLanes()` might compute:
>> 
>> (0.0f + Float.MIN_NORMAL) + (Float.MAX_VALUE - Float.MAX_VALUE) = Float.MIN_NORMAL
>> 
>> 
>> The difference of the two times of calculation is `Float.MIN_NORMAL` (1.1754945E-38), which exceeds the tolerance of `Math.ulp(0.0f) * 10.0f = 1.4E-44`. Even with a 10x rounding error factor, the tolerance is insufficient for such edge cases.
>> 
>> Since `reduceLanes()` does not require a specific calculation order, differences from scalar results can be significantly larger when special or extreme maximum/minimum values are present. Using a fixed tolerance is inappropriate for such corner cases.
>> 
>> This patch fixes the issue by initializing the float array in test with random normal values within a specified range, ensuring the result gap stays within the defined tolerance. 
>> 
>> Tested locally on my AArch64 and X86_64 machines 500 times, and I didn't observe the failure again.
>> 
>> [1] https://docs.oracle.com/en/java/javase/25/docs/api/jdk.incubator.vector/jdk/incubator/vector/FloatVector.html#reduceLanes(jdk.incubator.vector.VectorOperators.Associative)
>
> Xiaohong Gong has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Extend the float/double value range

The testing logic has been proved wrong with negative floats.
This is because the rounding errors are mainly related to the inputs, not the final result.

Not sure whether the test would always pass without fixing that error logic.

-------------

Changes requested by jiefu (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28960#pullrequestreview-3612075682