RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently

[Jie Fu]

On Tue, 23 Dec 2025 06:45:46 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

> The test fails intermittently with the following error:
> 
> 
> Caused by: java.lang.RuntimeException: assertEqualsWithTolerance: expected 0.0 but was 1.1754945E-38 (tolerance: 1.4E-44, diff: 1.1754945E-38)
> at compiler.vectorapi.TestVectorOperationsWithPartialSize.verifyAddReductionFloat(TestVectorOperationsWithPartialSize.java:231)
> at compiler.vectorapi.TestVectorOperationsWithPartialSize.testAddReductionFloat(TestVectorOperationsWithPartialSize.java:260)
> 
> 
> The root cause is that the Vector API `reduceLanes()` does not guarantee a specific calculation order for floating-point reduction operations [1]. When the array contains extreme values, this can produce results outside the tolerance range compared to sequential scalar addition.
> 
> For example, given array elements:
> 
> [0.0f, Float.MIN_NORMAL, Float.MAX_VALUE, -Float.MAX_VALUE]
> 
> 
> Sequential scalar addition produces:
> 
> 0.0f + Float.MIN_NORMAL + Float.MAX_VALUE - Float.MAX_VALUE = 0.0f
> 
> 
> However, `reduceLanes()` might compute:
> 
> (0.0f + Float.MIN_NORMAL) + (Float.MAX_VALUE - Float.MAX_VALUE) = Float.MIN_NORMAL
> 
> 
> The difference of the two times of calculation is `Float.MIN_NORMAL` (1.1754945E-38), which exceeds the tolerance of `Math.ulp(0.0f) * 10.0f = 1.4E-44`. Even with a 10x rounding error factor, the tolerance is insufficient for such edge cases.
> 
> Since `reduceLanes()` does not require a specific calculation order, differences from scalar results can be significantly larger when special or extreme maximum/minimum values are present. Using a fixed tolerance is inappropriate for such corner cases.
> 
> This patch fixes the issue by initializing the float array in test with random normal values within a specified range, ensuring the result gap stays within the defined tolerance. 
> 
> Tested locally on my AArch64 and X86_64 machines 500 times, and I didn't observe the failure again.
> 
> [1] https://docs.oracle.com/en/java/javase/25/docs/api/jdk.incubator.vector/jdk/incubator/vector/FloatVector.html#reduceLanes(jdk.incubator.vector.VectorOperators.Associative)

Here is an example which shows that the tolerance may be still not big enough  even with `ROUNDING_ERROR_FACTOR_ADD = 10000000.0f`.

Note: the test range of `a` and `b` is only `[0.0f, 1.0f]`.


class T {
  public static void main(String[] args) {

    float ROUNDING_ERROR_FACTOR_ADD = 10000000.0f;

    Float a = 0.0f + (ROUNDING_ERROR_FACTOR_ADD + 1) * Math.ulp(0.0f);
    Float b = 1.0f;

    Float expected = a + b - b;
    Float actual   = a + (b - b);

    float tolerance = Math.ulp(expected) * ROUNDING_ERROR_FACTOR_ADD;
    if (Math.abs(expected - actual) > tolerance) {
      System.out.println("Error: Out of tolerance!");
    }
  }
}


So I'm afraid the sum-based tolerance should be improved.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28960#issuecomment-3689857047