RFR(S): JDK-8146697 : VM crashes in test Test7005594

[David Holmes]

Hi Fred,

Thanks for explaining things to me offline. This change seems fine in 
making the code more robust.

Thanks,
David

On 9/08/2016 12:55 AM, Frederic Parain wrote:
> Greetings,
>
> Please review this small fix for JDK-8146697
> https://bugs.openjdk.java.net/browse/JDK-8146697
>
> Summary: The JVM sometimes tries to re-enable the Reserved Stack Area
> while it is currently not disabled, leading to the following assertion
> failure:
>
> share/vm/runtime/thread.cpp:2551 assert(_stack_guard_state !=
> stack_guard_enabled) failed: already enabled
>
> This problem occurred while running different tests including tests
> where stack overflows are unlikely. It is rare and very hard to
> reproduce. At the beginning of the investigation, I've been able to
> reproduce it three times out of 1,000+ runs of metaspace stress test
> (the fact that was is a metaspace test doesn't matter). But once I've
> instrumented the JVM, the bug didn't show up again, even after 30,000+
> runs.
>
> So, I've investigated it with the limited material I had. The failures
> always occurred on x86/32bits platforms.
> Regarding that some failures occurred on tests where stack overflows are
> unlikely (no recursive calls, small call stack), and that all failures
> occurred in interpreted Java code, my guess is that the issue is in the
> test performed on interpreted method exit to determine if the Reserved
> Stack Area should be enabled or not.
>
> The test on method exit compares the SP of the caller frame to an
> activation SP address stored in the JavaThread object when the Reserved
> Stack Area has been disabled. Without a reproducible test case, I've not
> been able to find what was the issue between the two values (de-opt,
> OSR, other?). So, I've slightly changed the test to make it more robust
> against the situation causing the assertion failure. Now the test checks
> the status of the guard pages, and if no guard pages have been disabled,
> the method exits normally. This means there's always only one test on
> interpreted method exit if Reserved Stack Area has not been used, so no
> difference on performances for most cases. If this first test detects
> that guard pages have been disabled, then the previous test (caller SP
> vs activation SP) is performed, to determine if this is the place where
> the Reserved Stack Area should be re-enabled or not.
>
> Even if the root cause of the bug is still unknown, the fix should make
> the code more robust and prevent unnecessary re-enabling of the Reserved
> Stack Area.
>
> Webrev:
> http://cr.openjdk.java.net/~fparain/8146697/webrev.00/
>
> Thank you,
>
> Fred