RFR: 8294816: C2: Math.min/max vectorization miscompilation

[Tobias Hartmann]

On Wed, 2 Nov 2022 12:06:14 GMT, Bhavana Kilambi <bkilambi at openjdk.org> wrote:

> C2 miscompiles during auto-vectorization of MinI/MaxI nodes when "short" type operands are involved. When a short and an integer value is compared, C2 generates vector min/max nodes for "short" types which does not result in correct output as it disregards the higher order bits of the integer input. Java API for Math.min/max also only supports the int, long, float and double types but not the subword integer types namely - char, byte and short. Hence, char/short/byte min/max vector instructions should not be generated.
> This patch ensures that MaxV and MinV vector nodes are only generated for the "int" type for MaxI and MinI nodes during auto-vectorization.

Looks good to me. I'll run testing and report back once it passed.

-------------

Marked as reviewed by thartmann (Reviewer).

PR: https://git.openjdk.org/jdk/pull/10944