RFR: 8197429: Increased stack guard causes segfaults on x86-32

[Andrew Haley]

32-bit Linux x86 HotSpot allocates an executable memory region just
below the end of the stack.  This is a workaround for JDK-8023956,
which in turn relates to a bug in the RHEL 5 & 6 kernels on old
(pre-NX) CPUs:

  To summarize: to emulate NX feature on X86_32 code segment is used
  to limit execution to the highest executable VA. There is a tiny
  race on SMP MM invalidation code which can cause the lazy CS update
  code in trap handling to think a general protection fault wasn't
  cause by itself. This results in sending the JVM a useless SIGSEGV
  with si_code:SI_KERNEL, results in JVM signal handling forcing a
  dump.

  The suggested work around (limited to 32 bit Linux): is to enable
  execution (PROT_EXEC) on a high address and execute some code.

To be more precise: on 32-bit Linux kernels the top of the main stack
is at about 3G (0xC0000000), and HotSpot creates an executable mapping
of a single page just a little way below the main stack and executes
an instruction in it.  (It then leaves the region mapped; I do not
know why.  It could be that the region could be removed at this
point.)

Some new Linux kernels by default have a stack guard of a megabyte
between the main stack and any allocated memory region.  See
CVE-2017-1000364.  (Note that this megabyte is a default: it can be
changed at boot time.)

So, when the stack grows to within a megabyte of the executable region
HotSpot installed, the process segfaults and is killed.  This only
happens when we're running on the main stack, and that only happens
when using the JNI invocation interface.


I have looked at several ways to fix this.  One was to probe to find
out what the stack guard size is, and to place the executable mapping
an appropriate distance from the stack; this can be done, but it is
complex.  I believe it's also unnecessary, because the workaround for
JDK-8023956 isn't needed with the newer kernels that have the larger
stack guard gap.

The fix I'm proposing here first bangs down the stack to the Java
stack limit, then tries to map the executable memory region.  On
systems with a large stack guard gap this mapping attempt will fail,
and we return and continue.  On older systems which do not have a
large stack gap it will continue and install the executable memory
region.

There are other possible fixes.  Rather than failing, we could loop
trying to install the executable mapping until we succeed.

http://cr.openjdk.java.net/~aph/8197429-1/

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. <https://www.redhat.com>
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671