RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently

[Jie Fu]

On Thu, 25 Dec 2025 01:35:16 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

> > > So that's just the issue that I want to avoid. We didn't have a more reasonable golden value as we do not know the calculation order in Vector API, right?
> > 
> > 
> > Agreed.
> > I would suggest the testing range also covers negative floats, not only positives.
> 
> Extending the range to include both negative and positive floats would produce discrepancies like the ones you observed, and would force us to adopt a much larger tolerance, which is undesirable. As mentioned above, it is not realistic to use a single tolerance that works well for all possible input values in the Vector API. Since reduceLanes is already thoroughly tested under jdk/jdk/incubator/vector with a wide variety of float inputs, this particular test is intended to focus on verifying the generated IR rather than exhaustively validating numerical behavior.

What I want to show here is that the current sum-based tolerance seems unreasonable.
This is because the rounding errors are related to all the inputs, not the final result.

Also, if the test range are all positive floats, `Math.ulp(sum)` is always larger than `Math.ulp(max-abs-of-inputs)`.
So even for positive floats, I would suggest a max-abs-based tolerance like this

    float baseUlp = Math.ulp(maxAbs_of_the_inputs);    //  which is smaller than Math.ulp(sum)
    float tolerance = baseUlp * ROUNDING_ERROR_FACTOR_ADD;
    tolerance = Math.max(tolerance, Float.MIN_NORMAL);

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28960#issuecomment-3690733113