RFR: 8170307: Stack size option -Xss is ignored

[David Holmes]

Bug: https://bugs.openjdk.java.net/browse/JDK-8170307

The bug is not public unfortunately for non-technical reasons - but see 
my eval below.

Background: if you load the JVM from the primordial thread of a process 
(not done by the java launcher since JDK 6), there is an artificial 
stack limit imposed on the initial thread (by sticking the guard page at 
the limit position of the actual stack) of the minimum of the -Xss 
setting and 2M. So if you set -Xss to > 2M it is ignored for the main 
thread even if the true stack is, say, 8M. This limitation dates back 
10-15 years and is no longer relevant today and should be removed (see 
below). I've also added additional explanatory notes.

webrev: http://cr.openjdk.java.net/~dholmes/8170307/webrev/

Testing was manually done by modifying the launcher to not run the VM in 
a new thread, and checking the resulting stack size used.

This change will only affect hosted JVMs launched with a -Xss value > 2M.

Thanks,
David
-----

Bug eval:

JDK-4441425 limits the stack to 8M as a safeguard against an unlimited 
value from getrlimit in 1.3.1, but further constrained that to 2M in 
1.4.0 due to JDK-4466587.

By 1.4.2 we have the basic form of the current problematic code:

#ifndef IA64
   if (rlim.rlim_cur > 2 * K * K) rlim.rlim_cur = 2 * K * K;
#else
   // Problem still exists RH7.2 (IA64 anyway) but 2MB is a little small
   if (rlim.rlim_cur > 4 * K * K) rlim.rlim_cur = 4 * K * K;
#endif

   _initial_thread_stack_size = rlim.rlim_cur & ~(page_size() - 1);

   if (max_size && _initial_thread_stack_size > max_size) {
      _initial_thread_stack_size = max_size;
   }

This was added by JDK-4678676 to allow the stack of the main thread to 
be _reduced_ below the default 2M/4M if the -Xss value was smaller than 
that.** There was no intent to allow the stack size to follow -Xss 
arbitrarily due to the operational constraints imposed by the OS/glibc 
at the time when dealing with the primordial process thread.

** It could not actually change the actual stack size of course, but set 
the guard pages to limit use to the expected stack size.

In JDK 6, under JDK-6316197, the launcher was changed to create the JVM 
in a new thread, so that it was not limited by the idiosyncracies of the 
OS or thread library primordial thread handling. However, the stack size 
limitations remained in place in case the VM was launched from the 
primordial thread of a user application via the JNI invocation API.

I believe it should be safe to remove the 2M limitation now.