RFR: 8373722: [TESTBUG] compiler/vectorapi/TestVectorOperationsWithPartialSize.java fails intermittently

[Xiaohong Gong]

On Thu, 25 Dec 2025 02:03:36 GMT, Jie Fu <jiefu at openjdk.org> wrote:

> > > > So that's just the issue that I want to avoid. We didn't have a more reasonable golden value as we do not know the calculation order in Vector API, right?
> > > 
> > > 
> > > Agreed.
> > > I would suggest the testing range also covers negative floats, not only positives.
> > 
> > 
> > Extending the range to include both negative and positive floats would produce discrepancies like the ones you observed, and would force us to adopt a much larger tolerance, which is undesirable. As mentioned above, it is not realistic to use a single tolerance that works well for all possible input values in the Vector API. Since reduceLanes is already thoroughly tested under jdk/jdk/incubator/vector with a wide variety of float inputs, this particular test is intended to focus on verifying the generated IR rather than exhaustively validating numerical behavior.
> 
> What I want to show here is that the current sum-based tolerance seems unreasonable. This is because the rounding errors are related to all the inputs, not the final result.
> 
> Also, if the test range are all positive floats, `Math.ulp(sum)` is always larger than `Math.ulp(max-abs-of-inputs)`. So even for positive floats, I would suggest a max-abs-based tolerance like this
> 
> ```
>     float baseUlp = Math.ulp(maxAbs_of_the_inputs);    //  which is smaller than Math.ulp(sum)
>     float tolerance = baseUlp * ROUNDING_ERROR_FACTOR_ADD;
>     tolerance = Math.max(tolerance, Float.MIN_NORMAL);
> ```

Thanks for the suggestion. I understand your point, and it sounds reasonable to me. In this test, the tolerance calculation is based on the existing Vector API jtreg reduction tests. For example, here is the approach used in `Float128VectorTests`:
https://github.com/openjdk/jdk/blob/73a8629c5b52b678febcc9d339e01ebcc5277909/test/jdk/jdk/incubator/vector/Float128VectorTests.java#L143-L156

Where `rc` and `r[i]` are the expected values calculated by scalar addition result and `relativeErrorFactor` for the add reduction is `10.0`.

Since the tests under `jdk/incubator/vector` have been exercised for a long time, I think they provide a high level of confidence in the way results are verified. Given that choosing an appropriate tolerance for floating‑point computations is a non‑trivial problem, I would rather follow the established approach used in those tests than introduce additional complexity in this fix.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28960#issuecomment-3691061476