[9] RFR (S) 6762191: Setting stack size to 16K causes segmentation fault

[Chris Plummer]

This is an initial review for 6762191. I'm guessing there will be 
recommendations to fix in a different way, but thought this would be a 
good time to start the discussion.

https://bugs.openjdk.java.net/browse/JDK-6762191
http://cr.openjdk.java.net/~cjplummer/6762191/webrev.00.jdk/
http://cr.openjdk.java.net/~cjplummer/6762191/webrev.00.hotspot/

The bug is that if the -Xss size is set to something very small (like 
16k), on linux there will be a crash due to overwriting the end of the 
stack. This happens before hotspot can compute its stack needs and 
verify that the stack is big enough.

It didn't seem viable to move the hotspot stack size check earlier. It 
depends on too much other work done before that point, and the changes 
would have been disruptive. The stack size check is currently done in 
os::init_2().

What is needed is a check before the thread is created. That way we can 
create a thread with a big enough stack to handle all needs up to the 
point of the check in os::init_2(). This initial check does not need to 
be the final check. It just needs to confirm that we have enough stack 
to get us to the check in os::init_2().

I decided to check in java.c if the -Xss size is too small, and set it 
to a larger size if it is. I hard coded this size to 32k (I'll explain 
why 32k later). I suspect this is the part that will result in some 
debate. If you have better suggestions let me know. If it does stay 
here, then probably the 32k needs to be a #define, and maybe even an OS 
porting interface, but I'm not sure where to put it.

The reason I chose 32k is because this is big enough for all platforms 
to get to the stack size check in os::init_2(). It is also smaller than 
the actual minimum stack size allowed on any platform. 32-bit windows 
has the smallest requirement at 64k. I add some printfs to print the 
minimum stack requirement, and then ran a simple JTReg test with every 
JPRT supported platform to get the results.

The TooSmallStackSize.sh will run "java -version" with -Xss16k, -Xss32k, 
and -XXss<minsize>, where <minsize> is the size from the error message 
produced by the JVM, such as in the following:

$ java -Xss32k -version
The stack size specified is too small, Specify at least 100k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I ran this test through JPRT on all platforms, and they all pass.

One thing to point out is that Windows behaves a bit different than the 
other platforms. It always rounds the stack size up to a multiple of 64k 
, so even if you specify -Xss16k, you get a 64k stack. On 32-bit Windows 
with C1, 64k is also the minimum requirement, so there is no error 
produced in this case. However, on 32-bit Windows with C2, 68k is the 
minimum, so an error is produced since the stack will only be 64k. There 
is no bug here. It's just a bit confusing.

thanks,

Chris