Can't get hs_err log on native stack overflow on Linux

[Yasumasa Suenaga]

Hi,

I encountered native stack overflow at JNI code on Linux (Fedora 15 and Ubuntu 11).
I got coredump image, however, I could not get hs_err log.

In the case of SIGSEGV, hs_err log is generated in signal handler. If native
stack overflow occurred, Linux can't use stack area. So, SIGSEGV handler
(JVM_handle_linux_signal) is never called.

manpage of sigaltstack(2):
/****************/
NOTES
       The most common usage of an alternate signal stack is to handle the SIGSEGV sig‐
       nal that is generated if the space available for the  normal  process  stack  is
       exhausted:  in  this case, a signal handler for SIGSEGV cannot be invoked on the
       process stack; if we wish to handle it, we must use an alternate signal stack.
/****************/


If this patch is applied, we can get hs_err log on native stack overflow as follows:

/****************/
#
#  SIGSEGV (0xb) at pc=0x00007fb23f1265f7, pid=25748, tid=140403650643712
#  java.lang.StackOverflowError: Native stack
#
# JRE version: 8.0
# Java VM: OpenJDK 64-Bit Server VM (22.0-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [liboverflow.so+0x5f7]  Java_Main_doStackOverflow+0x3b
/****************/


I've attached this patch and testcase in this email. Please check it.


I would like to contribute this patch, and I hope to apply this patch to
JDK 6 / 7 / 8.


Please cooperate.

Best regards,
Yasumasa
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testcase.zip
Type: application/x-zip-compressed
Size: 1057 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20110809/d58bd1ac/attachment.bin 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stackguard.patch
Url: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20110809/d58bd1ac/attachment.ksh