Question about stack overflows in native code

[Frederic Parain]

When the yellow zone is hit and the thread state is not in
_thread_in_java (which means thread state is _thread_in_native or
_thread_in_vm), the yellow zone is silently disabled and the thread
is allowed to resume its execution.

When the red zone is hit, what ever the current thread state is,
the red zone is disabled and VMError::report_and_die() is called,
which should generate a hs_err file unless the generation of the
error file requires more memory than the red zone provides.

Fred

On 04/03/2017 02:08 PM, Thomas Stüfe wrote:
> Hi,
>
> Today we wondered what would happen when a stack overflow occurs in native
> code running in a java thread (an attached thread or one created by the VM).
>
> In that case yellow and red pages are in place, but this would not help
> much, would it not, because the native code would not do any stack banging?
>
> So, native code would hit the yellow page, and then there would probably
> not be enough space left on the stack to invoke the signal handler. The
> result would be immediate VM death - not even an hs-err file - is that
> correct?
>
> Also, we would hit the our own yellow page, not the guard page the OS may
> or may not have established, so - on UNIX - this would show up as
> "Segmentation Fault", not "Stack Overflow", or?
>
> Thank you,
>
> Thomas
>