AMD64: JVM crashes during uncommon_trap_blob and deopt_blob

[Schoesser, Andreas]

Hi all,

we experienced crashes of the JVM during uncommon traps and deoptimization on x64.

Cause: The uncommon trap and deoptimization blobs call into the VM (Deoptimization::unpack_frames) with an 8-byte aligned stack pointer. The ABI requires 16-byte alignment.
In our case, the runtime eventually called the Windows API function "RtlCaptureContext". This issues an "fxsave" instruction which requires a 16-byte aligned buffer which in turn is only guaranteed if it's stack frame is 16-byte aligned. The 8-byte aligned stack frame lead to an EXCEPTION_ACCESS_VIOLATION and the JVM crashed.

You can easily reproduce this when turning on "create user stack database" using the Microsoft GFlags tool on Windows x64. This will trigger "RtlCaptureContext" quite often. Running the hotspot "compiler" test suite will immediately fail.

There's already a BugID for it:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6636110

and the bug seems also to be known "outside" for a while:
http://jpassing.com/2008/05/11/the-case-of-the-mysterious-jvm-x64-crashes/

I have a suggestion for a fix which I filed as a WebRev:
http://www.sapjvm.com/asc/webrevs/unaligned_stack/webrev/

The fix aligns the stack pointer as required by x64 ABI before calling into the VM. It fixes the uncommon_trap_blob as well as the deopt_blob. Few adjustments have to be made:
Since due to the aligned SP the runtime cannot easily retrieve the return PC any more we have to store it into the last java frame as well and adapt the call site's oop map accordingly. The deoptimization blob does SP-relative addressing after the runtime call, so we have to restore the unaligned SP before doing this using the stored value.

Best regards,
Andreas