RFR: 8197429: Increased stack guard causes segfaults on x86-32

[David Holmes]

Hi Andrew,

On 10/02/2018 2:51 AM, Andrew Haley wrote:
> 32-bit Linux x86 HotSpot allocates an executable memory region just
> below the end of the stack.  This is a workaround for JDK-8023956,
> which in turn relates to a bug in the RHEL 5 & 6 kernels on old
> (pre-NX) CPUs:
> 
>    To summarize: to emulate NX feature on X86_32 code segment is used
>    to limit execution to the highest executable VA. There is a tiny
>    race on SMP MM invalidation code which can cause the lazy CS update
>    code in trap handling to think a general protection fault wasn't
>    cause by itself. This results in sending the JVM a useless SIGSEGV
>    with si_code:SI_KERNEL, results in JVM signal handling forcing a
>    dump.
> 
>    The suggested work around (limited to 32 bit Linux): is to enable
>    execution (PROT_EXEC) on a high address and execute some code.
> 
> To be more precise: on 32-bit Linux kernels the top of the main stack
> is at about 3G (0xC0000000), and HotSpot creates an executable mapping
> of a single page just a little way below the main stack and executes
> an instruction in it.  (It then leaves the region mapped; I do not
> know why.  It could be that the region could be removed at this
> point.)
> 
> Some new Linux kernels by default have a stack guard of a megabyte
> between the main stack and any allocated memory region.  See
> CVE-2017-1000364.  (Note that this megabyte is a default: it can be
> changed at boot time.)
> 
> So, when the stack grows to within a megabyte of the executable region
> HotSpot installed, the process segfaults and is killed.  This only
> happens when we're running on the main stack, and that only happens
> when using the JNI invocation interface.
> 
> 
> I have looked at several ways to fix this.  One was to probe to find
> out what the stack guard size is, and to place the executable mapping
> an appropriate distance from the stack; this can be done, but it is
> complex.  I believe it's also unnecessary, because the workaround for
> JDK-8023956 isn't needed with the newer kernels that have the larger
> stack guard gap.
> 
> The fix I'm proposing here first bangs down the stack to the Java
> stack limit, then tries to map the executable memory region.  On
> systems with a large stack guard gap this mapping attempt will fail,
> and we return and continue.  On older systems which do not have a
> large stack gap it will continue and install the executable memory
> region.

How does this interact with the use of DisablePrimordialThreadGuardPages?

Thanks,
David

> There are other possible fixes.  Rather than failing, we could loop
> trying to install the executable mapping until we succeed.
> 
> http://cr.openjdk.java.net/~aph/8197429-1/
>