RFR 8009302: JVM crash on infinite recursion on Appkit Thread

[Gerard Ziemski]

hi guys,

Here is a trivial fix (workaround) for a particular XNU (Mac OS X 
kernel) behavior causing JavaVM problems.

Explanation:

There is a case where XNU will map SIGBUS into SIGSEGV, which only 
happens on main thread (secondary threads receive the original SIGBUS) 
due to the way XNU determines stack address and its size. To complicate 
the issue, XNU will only deliver that converted SIGSEGV signal if the 
BSD signal catcher uses an alternate stack.

Since JavaVM BSD signal catcher does not use alt stack, we end up 
missing the signal and the Java process results in a native crash which 
is then presented to the user via Mac CrashReporter app.

We do not want to use an actual alternate stack, which among other 
things would result in an increased RAM usage and possibly result in a 
regression, so we use a workaround, where we declare we will use an 
alternate stack, without actually creating one.

This workaround is only for Mac (ie. we use #ifdef __APPLE__) and it 
works because we install our own guard pages, and that's where the 
faulting address occurs, so we are assured that our stack does have 
enough room to handle the signal by the existing JavaVM code, which 
removes the guard pages before handling the signal.

I have filed an issue with Apple (id: 14047928) to fix this inconsistent 
(wrt threads) behavior in XNU, and wrote native tests that check for 
this particular behavior, which will catch any future change, should 
Apple ever elect to fix it.


Testing:

- the issue's own use case
- vm.quick.testlist
- vm.signal.testlist
- nsk.stack.testlist
- JPRT
- JCK lang
- JCK vm


References:

Webrev: http://cr.openjdk.java.net/~hseigel/bug_8009302/

OpenJDK bug  http://bugs.sun.com/view_bug.do?bug_id=8009302

JBS bug: https://jbs.oracle.com/bugs/browse/JDK-8009302