[jdk17] RFR: JDK-8267652: c2 loop unrolling by 8 results in reading memory past array

[Nils Eliasson]

I moved this PR to be on top of JDK-17. (The old one can be found here: https://github.com/openjdk/jdk/pull/4527)

Currently there are a bunch of AVX instructions on x86 that operate on memory that read a full 16-bytes even though only 8 are used. This means we can read out of bounds. This can be reproduced by using -XX:MaxLoopUnrollFactor=8 or -XX:MaxVectorLength=8.

I've tried creating test cases where a complete unroll results in a 8 byte vector. Then we will choose none-AVX instructions.

I've tried to patch x86.ad, looking for all uses of LoadVector on instructions that require AVX. I add a predicate that the vector length must be more than 8 bytes. This forces the use the reg-reg variants when the vector length is 8.

What I am missing is some kind of verification that the fix covers all cases.

Another additional complexity is that we are using the same instructions in assembler_x86.cpp. I've seen no obvious out-of-bounds reads, but they might be there.

Best regards,
Nils Eliasson

-------------

Commit messages:
 - JDK-8267652: c2 loop unrolling by 8 results in reading memory past array

Changes: https://git.openjdk.java.net/jdk17/pull/108/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk17&pr=108&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8267652
  Stats: 49 lines in 1 file changed: 27 ins; 0 del; 22 mod
  Patch: https://git.openjdk.java.net/jdk17/pull/108.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk17 pull/108/head:pull/108

PR: https://git.openjdk.java.net/jdk17/pull/108