AMD64: JVM crashes during uncommon_trap_blob and deopt_blob

[Tom Rodriguez]

On Oct 27, 2011, at 2:52 AM, Schoesser, Andreas wrote:

> Hi all,
> 
> we experienced crashes of the JVM during uncommon traps and deoptimization on x64.
> 
> Cause: The uncommon trap and deoptimization blobs call into the VM (Deoptimization::unpack_frames) with an 8-byte aligned stack pointer. The ABI requires 16-byte alignment.
> In our case, the runtime eventually called the Windows API function "RtlCaptureContext". This issues an "fxsave" instruction which requires a 16-byte aligned buffer which in turn is only guaranteed if it's stack frame is 16-byte aligned. The 8-byte aligned stack frame lead to an EXCEPTION_ACCESS_VIOLATION and the JVM crashed.
> 
> You can easily reproduce this when turning on "create user stack database" using the Microsoft GFlags tool on Windows x64. This will trigger "RtlCaptureContext" quite often. Running the hotspot "compiler" test suite will immediately fail.
> 
> There's already a BugID for it:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6636110
> 
> and the bug seems also to be known "outside" for a while:
> http://jpassing.com/2008/05/11/the-case-of-the-mysterious-jvm-x64-crashes/
> 
> I have a suggestion for a fix which I filed as a WebRev:
> http://www.sapjvm.com/asc/webrevs/unaligned_stack/webrev/
> 
> The fix aligns the stack pointer as required by x64 ABI before calling into the VM. It fixes the uncommon_trap_blob as well as the deopt_blob. Few adjustments have to be made:
> Since due to the aligned SP the runtime cannot easily retrieve the return PC any more we have to store it into the last java frame as well and adapt the call site's oop map accordingly. The deoptimization blob does SP-relative addressing after the runtime call, so we have to restore the unaligned SP before doing this using the stored value.

Thanks for sending this in.  I think your change looks good, though the imprecision of the last_pc slightly bugs me.  We could conceivably fix it up after emitting the actual call but that seems too clever for this problem.  I'll take your bits and run some of our deopt tests on it and see how it looks.

tom

> 
> Best regards,
> Andreas
> 
>