JVM inserting guard pages into stack of initial thread - will that stop with Java 10?

[David Holmes]

Please Note that Tomas's email was sent October 10 and only just 
appeared on the mailing list (presumably he was not subscribed at the time).

Since then this has all been discussed and an additional workaround in 
the pipeline through:

https://bugs.openjdk.java.net/browse/JDK-8189170

Thanks,
David

On 24/10/2017 4:31 AM, Thomas Stüfe wrote:
> Rereading I found my first answer a bit shortish and I wish to elaborate 
> a bit on my answer, even though David is probably in a better position 
> to do so.
> 
> We had a discussion about this on hotspot.runtime 
> (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/thread.html#24859), 
> AFAIK David plans to implement the switch to disable guard stack 
> creation on the primordial thread.
> 
> Both jdk9 and jdk10 already contain David's fix for 8170307, which 
> changes the clamp-down size for the primordial thread on Linux to 8M.
> 
> Latest versions are here:
> http://hg.openjdk.java.net/jdk9/hs/hotspot/file/c68024d52834/src/os/linux/vm/os_linux.cpp
> http://hg.openjdk.java.net/jdk10/hs/file/1aecd400f2fa/src/hotspot/os/linux/os_linux.cpp 
> 
> 
> Kind Regards, Thomas
> 
> On Tue, Oct 10, 2017 at 11:36 AM, Tomas Kalibera 
> <tomas.kalibera at gmail.com <mailto:tomas.kalibera at gmail.com>> wrote:
> 
>     Dear hotspot developers,
> 
>     I am one of the core R developers and what has been causing us a lot
>     of trouble is the 2M cap for stack size of the initial thread on
>     Linux. In short, R has its own stack overflow checking, the stack
>     size is taken from rlimit, and R needs a lot of stack (often more
>     than 2M). Via rJava package, R uses Java, so it initializes Java via
>     JNI. As a workaround for some old Linux problem, hotspot caps the
>     stack at 2M by inserting guard pages. Consequently, R crashes when
>     the recursion gets too deep, and the crash is ugly as R's stack
>     overflow checking does not now about the shrinkage of the stack.
>     Users have no chance of diagnosing what's wrong and it is causing us
>     trouble increasingly more lately, as we have more recursive calls
>     (JIT implemented in R) and as newer gcc versions produce bigger
>     frames from our interpreter loop. The problem is still present in
>     Java 9.
> 
>     We're so hopeless that I implemented a workaround for rJava where we
>     fill up the stack just enough before initializing the JVM so that
>     is_initial_thread returns false even for the initial thread, so the
>     guard page is not inserted. This is probably something we will use
>     for now - or is there a simpler workaround? (the code is here, but
>     probably you get the idea just from the description of it:
>     https://github.com/s-u/rJava/pull/102/files
>     <https://github.com/s-u/rJava/pull/102/files>)
> 
>     As you surely know well, the 2M cap comes from a bug report:
>     http://bugs.java.com/view_bug.do?bug_id=4466587
>     <http://bugs.java.com/view_bug.do?bug_id=4466587>
>     And there is a cleanup of this code discussed in
>     http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html
>     <http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html>
> 
>     If I understand correctly the discussion, this is the new version
>     for Java 10 - is that correct?
>     http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html
>     <http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html>
> 
>     Now, if I am reading this patch correctly, I think that the 2M cap
>     is gone. There is a new 8M cap, which however only applies when the
>     rlimit stack size is unlimited, which is something we could probably
>     live with. Is this a correct interpretation of the code? If so, we
>     should be fine for Java 10.
> 
>     Even though, having an option to initialize the JVM without stack
>     overflow checking for the initial thread might be even better... for
>     cases like ours, when we have our own stack overflow checking.
> 
>     Thanks,
>     Tomas
> 
>