RFR: JDK-8267652: c2 loop unrolling by 8 results in reading memory past array

[Nils Eliasson]

Hi,

Currently there are a bunch of AVX instructions on x86 that operate on memory that read a full 16-bytes even though only 8 are used. This means we can read out of bounds. This can be reproduced by using -XX:MaxLoopUnrollFactor=8 or -XX:MaxVectorLength=8.

I've tried creating test cases where a complete unroll results in a 8 byte vector. Then we will choose none-AVX instructions. 

I've tried to patch x86.ad, looking for all uses of LoadVector on instructions that require AVX. I add a predicate that the vector length must be more than 8 bytes. This forces the use the reg-reg variants when the vector length is 8. 

What I am missing is some kind of verification that the fix covers all cases.

Another additional complexity is that we are using the same instructions in assembler_x86.cpp. I've seen no obvious out-of-bounds reads, but they might be there. 

Best regards,
Nils Eliasson

-------------

Commit messages:
 - remove line
 - JDK-8267652: c2 loop unrolling by 8 results in reading memory past array

Changes: https://git.openjdk.java.net/jdk/pull/4527/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4527&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8267652
  Stats: 46 lines in 1 file changed: 24 ins; 0 del; 22 mod
  Patch: https://git.openjdk.java.net/jdk/pull/4527.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/4527/head:pull/4527

PR: https://git.openjdk.java.net/jdk/pull/4527