Integrated: 8344802: Crash in StubRoutines::verify_mxcsr with -XX:+EnableX86ECoreOpts and -Xcheck:jni

[Volodymyr Paprotski]

On Tue, 10 Dec 2024 23:45:37 GMT, Volodymyr Paprotski <vpaprotski at openjdk.org> wrote:

> (Also see `8319429: Resetting MXCSR flags degrades ecore`)
> 
> This PR fixes two issues:
> - the original issue is a crash caused by `__ warn` corrupting the stack on Windows only
> - This issue also uncovered that -Xcheck:jni test cases were getting 65k lines of warning on HelloWorld (on both Linux _and_ windows):
> 
> OpenJDK 64-Bit Server VM warning: MXCSR changed by native JNI code, use -XX:+RestoreMXCSROnJNICall
> 
> 
> First, the crash. Caused when FXRSTOR is attempting to write reserved bits into MXCSR. If those bits happen to be set, crash. (Hence the crash isn't deterministic. But frequent enough if `__ warn` is used). It is caused by the binding not reserving stack space for register parameters ()
> ![image](https://github.com/user-attachments/assets/4ad63908-088b-4e9d-9e7d-a3509bee046a)
> Prolog of the warn function then proceeds to store the for arg registers onto the stack, overriding the fxstore save area. (See https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#calling-convention-defaults)
> 
> Fix uses `frame::arg_reg_save_area_bytes` to bump the stack pointer.
> 
> ---
> 
> I also kept the fix to `verify_mxcsr` since without it, `-Xcheck:jni` is practically unusable when `-XX:+EnableX86ECoreOpts` are set (65k+ lines of warnings)

This pull request has now been integrated.

Changeset: 55097dd4
Author:    Volodymyr Paprotski <vpaprotski at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/55097dd4cbb5d691c12cb0247d66dce593759d59
Stats:     127 lines in 9 files changed: 66 ins; 54 del; 7 mod

8344802: Crash in StubRoutines::verify_mxcsr with -XX:+EnableX86ECoreOpts and -Xcheck:jni

Reviewed-by: jwaters, kvn, sviswanathan

-------------

PR: https://git.openjdk.org/jdk/pull/22673