RFR (S) JDK-7107135 - Stack guard pages becomes writable

[Ioi Lam]

Thanks to everyone for the reviews. I just came back from two days of 
vacation and two days of flu ....

Please see my comments in-line:

On 02/20/2013 10:07 PM, David Holmes wrote:
> On 21/02/2013 4:40 AM, Ioi Lam wrote:
>> [Moving the discussion to hotspot-runtime-dev at openjdk.java.net]
>> Please review:
>> http://javaweb.us.oracle.com/~iklam/webrev/7107135/stack_guard_001/
>>
>> Bug: Stack guard pages are no more protected after loading a shared
>>       library with executable stack
>> https://jbs.oracle.com/bugs/browse/JDK-7107135
>
> This seems somewhat more complex than necessary.
>
> First dll_open_inner should only ever be called via the VMOperation 
> and that will only happen if:
>  - LoadExecStackDllInVMThread == true
>  - os::uses_stack_guard_pages() == true
>  - is_init_completed() == true
>
> so there is no need to check any of these inside dll_open_inner.
>
That's right. I will fix it.

> Related to that back in dll_load I don't understand why we don't 
> simply have:
>
>      if (!LoadExecStackDllInVMThread) {
>         result = ::dl_open(..)
>      else {
>         // use Vm op
>      }
>
The VM op is used for both loading the DLL (in case of 
LoadExecStackDllInVMThread==true) and for fixing the stack after loading 
a "bad" DLL.

> Also I think you will find that Java threads (reference-processor and 
> finalizer threads) have been created before init_completed is set.
>
I found that when (is_init_completed() == false), all the DLLs are 
loaded before any Java threads are created. I will try to fix the 
comments and put an assert() to that effect.

- Ioi


> David
> -----
>
>
>> Background:
>>
>>      Recent versions of Linux support Non-Executable Stack
>>      protection -- by default, the stack is made non-executable to
>>      prevent code injection via overflowing on-stack buffers.
>>
>>      However, some old Linux DLLs require the stack to be
>>      executable. For backwards compatibility, after loading such
>>      DLLs, the Linux dynamic loader makes the stack executable.
>>
>>      Due to a limitation of the Linux system call API, the Linux
>>      dynamic loader makes the stack readable/writable as well. This
>>      disables Java's stack guard.
>>
>> Summary of fix:
>>
>>      1. Check if DLL requires executable stack by inspecting ELF header.
>>      2. Enter a Safepoint and load such DLLs in the VM thread.
>>         - immediately after loading, change all Java stack guards
>>           back to PROT_NONE.
>>      3. Leave Safepoint to resume Java execution.
>>
>>      I also added a global flag LoadExecStackDllInVMThread to load
>>      such "bad" DLLs outside of the safepoint (in case the DLL
>>      invokes JNI functions inside static constructors, which are
>>      executed before dlopen() returns).
>>
>> Note:
>>
>>      I got this code from an outside contributor and I don't really
>>      understand what this block does. Please comment if it's
>>      correct:
>>
>>      1877:    ThreadInVMfromNative tiv(JavaThread::current());
>>      1878:    debug_only(VMNativeEntryWrapper vew;)
>>
>> Tests executed:
>>
>>      * JPRT
>>      * UTE (vm.quick.testlist)
>>      * JTREG (hotspot/tests/runtime, hotspot/tests/closed/runtime)
>>
>> Thanks,
>> Ioi
>>