RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently

[Xiaohong Gong]

On Tue, 23 Dec 2025 08:37:48 GMT, Jie Fu <jiefu at openjdk.org> wrote:

> Shall we change the calculation of tolerance?
> 
> e.g.
> 
> ```
> max_abs = max(abs(arr[0]), abs(arr[1]), ...)
> tolerance = Math.ulp(max_abs) * vlen
> ```

Thanks for looking at this issue. It's really a good question about the definition of a more reasonable `tolerance`, which is fundamentally a numerical analysis problem per my understanding.

For a floating‑point reduction test, the goal is to check that the API is implemented correctly. And what we really care about is how close the reduction result is to the mathematically expected value. In other words, the tolerance should be derived from the expected sum itself. For example, if the float array is `[1.0f, -1.0f, 2.0f, -2.0f]`, we genuinely expect a result very close to `0.0f`, not something near `2.0f`.

Using `max_abs` to set the tolerance risks inflating the admissible error range (since `max_abs` here would be 2.0f), which I'm afraid might make the test much less effective. WDYT?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28960#issuecomment-3685912527