RFR: 8273139: C2: assert(f <= 1 && f >= 0) failed: Incorrect frequency [v2]

[Emanuel Peter]

On Wed, 19 Jan 2022 10:23:11 GMT, Roland Westrelin <roland at openjdk.org> wrote:

> I think Dean's suggestion is to only to use double locally that is in PathFrequency::to(). I suppose that would be feasible. What do you think?

That would be feasible. But it will not fix the issue, since bad rounding can already happen outside (computing the probabilities and counts). This is most likely why this bug was reported: inside the method we paid attention to round correctly, but the outside numbers most likely had rounding errors, such that if they were added in some adversarial order, they add up to `f>1`.

Assume a `float` comes in with a rounding error of `2^-23`. If it is converted to a `double` now in this method, it still still has a rounding error of `2^-23`, right? Most likely the mantissa of 23bit is just extended with zeros to 52bit mantissa.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7113