RFR: JDK-8267652: c2 loop unrolling by 8 results in reading memory past array

[Sandhya Viswanathan]

On Fri, 18 Jun 2021 09:01:11 GMT, Nils Eliasson <neliasso at openjdk.org> wrote:

> Hi,
> 
> Currently there are a bunch of AVX instructions on x86 that operate on memory that read a full 16-bytes even though only 8 are used. This means we can read out of bounds. This can be reproduced by using -XX:MaxLoopUnrollFactor=8 or -XX:MaxVectorLength=8.
> 
> I've tried creating test cases where a complete unroll results in a 8 byte vector. Then we will choose none-AVX instructions. 
> 
> I've tried to patch x86.ad, looking for all uses of LoadVector on instructions that require AVX. I add a predicate that the vector length must be more than 8 bytes. This forces the use the reg-reg variants when the vector length is 8. 
> 
> What I am missing is some kind of verification that the fix covers all cases.
> 
> Another additional complexity is that we are using the same instructions in assembler_x86.cpp. I've seen no obvious out-of-bounds reads, but they might be there. 
> 
> Best regards,
> Nils Eliasson

@neliasso Thanks a lot for identifying and fixing this issue. Your approach is correct. This is not limited to auto-vectorizer. Smaller vectors can come from Vector API as well.  
I went through the x86.ad file for all usages of $Address. You have covered most of the cases. Only the following are missing:
vfmaF_mem
vfmaD_mem
vpternlog_mem
vroundD_mem

-------------

PR: https://git.openjdk.java.net/jdk/pull/4527