RFR: 8294816: C2: Math.min/max vectorization miscompilation

[Tobias Hartmann]

On Wed, 2 Nov 2022 12:06:14 GMT, Bhavana Kilambi <bkilambi at openjdk.org> wrote:

> C2 miscompiles during auto-vectorization of MinI/MaxI nodes when "short" type operands are involved. When a short and an integer value is compared, C2 generates vector min/max nodes for "short" types which does not result in correct output as it disregards the higher order bits of the integer input. Java API for Math.min/max also only supports the int, long, float and double types but not the subword integer types namely - char, byte and short. Hence, char/short/byte min/max vector instructions should not be generated.
> This patch ensures that MaxV and MinV vector nodes are only generated for the "int" type for MaxI and MinI nodes during auto-vectorization.

Changes requested by thartmann (Reviewer).

test/hotspot/jtreg/compiler/c2/TestMinMaxSubword.java line 64:

> 62:     // should not generate vectorized Min/Max nodes for them.
> 63:     @Test
> 64:     @IR(failOn = {IRNode.Min_V})

The test currently fails compilation because the IRNodes were renamed by https://github.com/openjdk/jdk/pull/10695 to ` MIN_V`.

-------------

PR: https://git.openjdk.org/jdk/pull/10944