JVM inserting guard pages into stack of initial thread - will that stop with Java 10?

[Severin Gehwolf]

On Tue, 2017-10-10 at 11:36 +0200, Tomas Kalibera wrote:
> Dear hotspot developers,
> 
> I am one of the core R developers and what has been causing us a lot of 
> trouble is the 2M cap for stack size of the initial thread on Linux. In 
> short, R has its own stack overflow checking, the stack size is taken 
> from rlimit, and R needs a lot of stack (often more than 2M). Via rJava 
> package, R uses Java, so it initializes Java via JNI. As a workaround 
> for some old Linux problem, hotspot caps the stack at 2M by inserting 
> guard pages. Consequently, R crashes when the recursion gets too deep, 
> and the crash is ugly as R's stack overflow checking does not now about 
> the shrinkage of the stack. Users have no chance of diagnosing what's 
> wrong and it is causing us trouble increasingly more lately, as we have 
> more recursive calls (JIT implemented in R) and as newer gcc versions 
> produce bigger frames from our interpreter loop. The problem is still 
> present in Java 9.
> 
> We're so hopeless that I implemented a workaround for rJava where we 
> fill up the stack just enough before initializing the JVM so that 
> is_initial_thread returns false even for the initial thread, so the 
> guard page is not inserted. This is probably something we will use for 
> now - or is there a simpler workaround? (the code is here, but probably 
> you get the idea just from the description of it: 
> https://github.com/s-u/rJava/pull/102/files)
> 
> As you surely know well, the 2M cap comes from a bug report: 
> http://bugs.java.com/view_bug.do?bug_id=4466587
> And there is a cleanup of this code discussed in
> http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html
> 
> If I understand correctly the discussion, this is the new version for 
> Java 10 - is that correct?
> http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html
> 
> Now, if I am reading this patch correctly, I think that the 2M cap is 
> gone. There is a new 8M cap, which however only applies when the rlimit 
> stack size is unlimited, which is something we could probably live with. 
> Is this a correct interpretation of the code? If so, we should be fine 
> for Java 10.
> 
> Even though, having an option to initialize the JVM without stack 
> overflow checking for the initial thread might be even better... for 
> cases like ours, when we have our own stack overflow checking.

This maybe what you are looking for:
https://bugs.openjdk.java.net/browse/JDK-8189170

Cheers,
Severin