Can't get hs_err log on native stack overflow on Linux

[David Holmes]

Hi,

I could be mistaken here but I believe the intent/hope is that any
stackoverflow will be caught when the guard pages set up by the VM are
accessed. In that way we haven't run out of true native stack and so we
can still process the signal that indicates the stack overflow. This is
not a perfect mechanism of course and there may be situations where you
can jump over the guard pages and truly exhaust the stack.

I also believe there is a bit of bad history here, where we had problems
trying to use alternative signal stacks on Linux. It will take me a bit
of archaeology to dig up relevant info on that.

David Holmes

Yasumasa Suenaga said the following on 08/09/11 16:06:
> Hi,
> 
> I encountered native stack overflow at JNI code on Linux (Fedora 15 and Ubuntu 11).
> I got coredump image, however, I could not get hs_err log.
> 
> In the case of SIGSEGV, hs_err log is generated in signal handler. If native
> stack overflow occurred, Linux can't use stack area. So, SIGSEGV handler
> (JVM_handle_linux_signal) is never called.
> 
> manpage of sigaltstack(2):
> /****************/
> NOTES
>        The most common usage of an alternate signal stack is to handle the SIGSEGV sig‐
>        nal that is generated if the space available for the  normal  process  stack  is
>        exhausted:  in  this case, a signal handler for SIGSEGV cannot be invoked on the
>        process stack; if we wish to handle it, we must use an alternate signal stack.
> /****************/
> 
> 
> If this patch is applied, we can get hs_err log on native stack overflow as follows:
> 
> /****************/
> #
> #  SIGSEGV (0xb) at pc=0x00007fb23f1265f7, pid=25748, tid=140403650643712
> #  java.lang.StackOverflowError: Native stack
> #
> # JRE version: 8.0
> # Java VM: OpenJDK 64-Bit Server VM (22.0-b01 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C  [liboverflow.so+0x5f7]  Java_Main_doStackOverflow+0x3b
> /****************/
> 
> 
> I've attached this patch and testcase in this email. Please check it.
> 
> 
> I would like to contribute this patch, and I hope to apply this patch to
> JDK 6 / 7 / 8.
> 
> 
> Please cooperate.
> 
> Best regards,
> Yasumasa
>