RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM

[David Holmes]

Looking for a runtime Reviewer please.

Also Tomas: are you able to test this patch with R?

Thanks,
David

On 14/11/2017 8:39 PM, David Holmes wrote:
> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170
> 
> webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/
> 
> 
> There are three parts to this which made it somewhat more complicated 
> than I had envisaged:
> 
> 1. Add the flag and disable the guards.
> 
> This is relatively straight-forward:
> 
>   void JavaThread::create_stack_guard_pages() {
>     if (!os::uses_stack_guard_pages() ||
>         _stack_guard_state != stack_guard_unused ||
>         (DisablePrimordialThreadGuardPages && 
> os::is_primordial_thread())) {
>         log_info(os, thread)("Stack guard page creation for thread "
>                              UINTX_FORMAT " disabled", 
> os::current_thread_id());
>       return;
> 
> with a tweak in JavaThread::stack_guards_enabled as well.
> 
> 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread()
> 
> We have three platforms that already implement this functionality: 
> Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do 
> anything different for the primordial thread, and we don't have a way to 
> detect the primordial thread - so is_primordial_thread() just returns 
> false.
> 
> I also tidied up some comments regarding terminology.
> 
> 3. Thomas noticed that we unconditionally subtract 2*page_size() from 
> the rlimit stack size, without checking it was bigger than 2*page_size() 
> to start with. I was unsure how to tackle this. It's no good leaving 
> stack_size at zero so I opted to ensure we would have at least one page 
> left. Of course in such cases we would hit the bug in libc (if it still 
> exists, which seems unlikely but time consuming to establish).
> 
> Testing:
>    - Tier 1 (JPRT) testing with a modified launcher that uses the 
> primordial thread
>      - with guard pages enabled
>      - with guard pages disabled
>    - Tier 1 (JPRT) normal JDK build (which should be unaffected by this 
> change)
> 
> The testing was primarily to ensure that disabling the stack guards on 
> the primordial thread wasn't totally broken. A number of tests fail:
> - setting -Xss32m fails for the primordial thread when guards are 
> enabled and the rlimit is 8m
> - tests that would hit StackOverflowError crash the VM when guards are 
> disabled (as you would expect).
> - runtime/execstack/Testexecstack.java crashes when the guards are disabled
> 
> Thanks,
> David