JVM inserting guard pages into stack of initial thread - will that stop with Java 10?

[Tomas Kalibera]

Hi Thomas,

thanks for the summary, I've read the thread about the CSR and I am 
happy with the planned new option to disable the guard pages on the 
primordial thread. For older versions of Java that won't have that 
option, we can use the workaround in R/rJava we have now. In Java 9 we 
do get the guard pages inside the primordial thread stack as well, but 
it is because of -Xss and the default thread stack size, not because the 
2M cap which is no longer in Java 9. The workaround still works, though, 
as long as it fills up the stack above -Xss (which is by default <2M).

Best
Tomas


On 10/23/2017 08:31 PM, Thomas Stüfe wrote:
> Rereading I found my first answer a bit shortish and I wish to 
> elaborate a bit on my answer, even though David is probably in a 
> better position to do so.
>
> We had a discussion about this on hotspot.runtime 
> (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/thread.html#24859), 
> AFAIK David plans to implement the switch to disable guard stack 
> creation on the primordial thread.
>
> Both jdk9 and jdk10 already contain David's fix for 8170307, which 
> changes the clamp-down size for the primordial thread on Linux to 8M.
>
> Latest versions are here:
> http://hg.openjdk.java.net/jdk9/hs/hotspot/file/c68024d52834/src/os/linux/vm/os_linux.cpp
> http://hg.openjdk.java.net/jdk10/hs/file/1aecd400f2fa/src/hotspot/os/linux/os_linux.cpp 
>
>
> Kind Regards, Thomas
>
> On Tue, Oct 10, 2017 at 11:36 AM, Tomas Kalibera 
> <tomas.kalibera at gmail.com <mailto:tomas.kalibera at gmail.com>> wrote:
>
>     Dear hotspot developers,
>
>     I am one of the core R developers and what has been causing us a
>     lot of trouble is the 2M cap for stack size of the initial thread
>     on Linux. In short, R has its own stack overflow checking, the
>     stack size is taken from rlimit, and R needs a lot of stack (often
>     more than 2M). Via rJava package, R uses Java, so it initializes
>     Java via JNI. As a workaround for some old Linux problem, hotspot
>     caps the stack at 2M by inserting guard pages. Consequently, R
>     crashes when the recursion gets too deep, and the crash is ugly as
>     R's stack overflow checking does not now about the shrinkage of
>     the stack. Users have no chance of diagnosing what's wrong and it
>     is causing us trouble increasingly more lately, as we have more
>     recursive calls (JIT implemented in R) and as newer gcc versions
>     produce bigger frames from our interpreter loop. The problem is
>     still present in Java 9.
>
>     We're so hopeless that I implemented a workaround for rJava where
>     we fill up the stack just enough before initializing the JVM so
>     that is_initial_thread returns false even for the initial thread,
>     so the guard page is not inserted. This is probably something we
>     will use for now - or is there a simpler workaround? (the code is
>     here, but probably you get the idea just from the description of
>     it: https://github.com/s-u/rJava/pull/102/files
>     <https://github.com/s-u/rJava/pull/102/files>)
>
>     As you surely know well, the 2M cap comes from a bug report:
>     http://bugs.java.com/view_bug.do?bug_id=4466587
>     <http://bugs.java.com/view_bug.do?bug_id=4466587>
>     And there is a cleanup of this code discussed in
>     http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html
>     <http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html>
>
>     If I understand correctly the discussion, this is the new version
>     for Java 10 - is that correct?
>     http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html
>     <http://cr.openjdk.java.net/%7Edholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html>
>
>     Now, if I am reading this patch correctly, I think that the 2M cap
>     is gone. There is a new 8M cap, which however only applies when
>     the rlimit stack size is unlimited, which is something we could
>     probably live with. Is this a correct interpretation of the code?
>     If so, we should be fine for Java 10.
>
>     Even though, having an option to initialize the JVM without stack
>     overflow checking for the initial thread might be even better...
>     for cases like ours, when we have our own stack overflow checking.
>
>     Thanks,
>     Tomas
>
>