JVM inserting guard pages into stack of initial thread - will that stop with Java 10?

[Tomas Kalibera]

Dear hotspot developers,

I am one of the core R developers and what has been causing us a lot of 
trouble is the 2M cap for stack size of the initial thread on Linux. In 
short, R has its own stack overflow checking, the stack size is taken 
from rlimit, and R needs a lot of stack (often more than 2M). Via rJava 
package, R uses Java, so it initializes Java via JNI. As a workaround 
for some old Linux problem, hotspot caps the stack at 2M by inserting 
guard pages. Consequently, R crashes when the recursion gets too deep, 
and the crash is ugly as R's stack overflow checking does not now about 
the shrinkage of the stack. Users have no chance of diagnosing what's 
wrong and it is causing us trouble increasingly more lately, as we have 
more recursive calls (JIT implemented in R) and as newer gcc versions 
produce bigger frames from our interpreter loop. The problem is still 
present in Java 9.

We're so hopeless that I implemented a workaround for rJava where we 
fill up the stack just enough before initializing the JVM so that 
is_initial_thread returns false even for the initial thread, so the 
guard page is not inserted. This is probably something we will use for 
now - or is there a simpler workaround? (the code is here, but probably 
you get the idea just from the description of it: 
https://github.com/s-u/rJava/pull/102/files)

As you surely know well, the 2M cap comes from a bug report: 
http://bugs.java.com/view_bug.do?bug_id=4466587
And there is a cleanup of this code discussed in
http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html

If I understand correctly the discussion, this is the new version for 
Java 10 - is that correct?
http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html

Now, if I am reading this patch correctly, I think that the 2M cap is 
gone. There is a new 8M cap, which however only applies when the rlimit 
stack size is unlimited, which is something we could probably live with. 
Is this a correct interpretation of the code? If so, we should be fine 
for Java 10.

Even though, having an option to initialize the JVM without stack 
overflow checking for the initial thread might be even better... for 
cases like ours, when we have our own stack overflow checking.

Thanks,
Tomas