JVM inserting guard pages into stack of initial thread - will that stop with Java 10?

[Thomas Stüfe]

On Mon, Oct 23, 2017 at 6:16 PM, Severin Gehwolf <sgehwolf at redhat.com>
wrote:

> On Tue, 2017-10-10 at 11:36 +0200, Tomas Kalibera wrote:
> > Dear hotspot developers,
> >
> > I am one of the core R developers and what has been causing us a lot of
> > trouble is the 2M cap for stack size of the initial thread on Linux. In
> > short, R has its own stack overflow checking, the stack size is taken
> > from rlimit, and R needs a lot of stack (often more than 2M). Via rJava
> > package, R uses Java, so it initializes Java via JNI. As a workaround
> > for some old Linux problem, hotspot caps the stack at 2M by inserting
> > guard pages. Consequently, R crashes when the recursion gets too deep,
> > and the crash is ugly as R's stack overflow checking does not now about
> > the shrinkage of the stack. Users have no chance of diagnosing what's
> > wrong and it is causing us trouble increasingly more lately, as we have
> > more recursive calls (JIT implemented in R) and as newer gcc versions
> > produce bigger frames from our interpreter loop. The problem is still
> > present in Java 9.
> >
> > We're so hopeless that I implemented a workaround for rJava where we
> > fill up the stack just enough before initializing the JVM so that
> > is_initial_thread returns false even for the initial thread, so the
> > guard page is not inserted. This is probably something we will use for
> > now - or is there a simpler workaround? (the code is here, but probably
> > you get the idea just from the description of it:
> > https://github.com/s-u/rJava/pull/102/files)
> >
> > As you surely know well, the 2M cap comes from a bug report:
> > http://bugs.java.com/view_bug.do?bug_id=4466587
> > And there is a cleanup of this code discussed in
> > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-
> option-Xss-is-ignored-td292960.html
> >
> > If I understand correctly the discussion, this is the new version for
> > Java 10 - is that correct?
> > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/
> os/linux/vm/os_linux.cpp.udiff.html
> >
> > Now, if I am reading this patch correctly, I think that the 2M cap is
> > gone. There is a new 8M cap, which however only applies when the rlimit
> > stack size is unlimited, which is something we could probably live with.
> > Is this a correct interpretation of the code? If so, we should be fine
> > for Java 10.
> >
> > Even though, having an option to initialize the JVM without stack
> > overflow checking for the initial thread might be even better... for
> > cases like ours, when we have our own stack overflow checking.
>
> This maybe what you are looking for:
> https://bugs.openjdk.java.net/browse/JDK-8189170
>
> Cheers,
> Severin
>

We also had a lengthy discussion about this, see:

http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html

Best Regards, Thomas