RFR: 8319747: galoisCounterMode_AESCrypt stack walking broken

[Sandhya Viswanathan]

On Wed, 8 Nov 2023 18:11:58 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

> This PR fixes stack walking by removing incorrect RBP change in the middle of a stub.
> 
> I also changed the register assignment to avoid saving RSI on Windows.
> 
> Testing:
> - test/hotspot/jtreg/compiler/codegen/aes/TestAESMain.java still passes, both on Windows and Linux
> - mach5 tier1-2 (including security and TLS-heavy networking tests) still pass
> - the generated flamegraphs (attached in JBS) show the correct stack trace with this fix

src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp line 243:

> 241:   const Register state = r13;
> 242:   const Address subkeyH_mem(rbp, 8 * wordSize);
> 243:   const Register subkeyHtbl = r11;

Changing the register from r14 to r11 here doesn't look correct. As aesgcm_encrypt() uses r11 as temporary for windows at line 3046 and overwrites it at line 3074 etc in the original code.  
3046 const Register ghash_pos = NOT_WIN64( r14) WIN64_ONLY( r11 );
3074   __ movl(ghash_pos, 0); // Pointer for ghash read and store operations

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16567#discussion_r1388845844