RFR (S) JDK-7107135 - Stack guard pages becomes writable

[Ioi Lam]

> On 02/20/2013 08:25 AM, Dean Long wrote:
>> Which part seems buggy, changing the permissions?  It seems necessary 
>> as long as executable stacks is
>> a supported feature.  It would be nice if dlopen() would just fail to 
>> load these libraries based on some flag
>> in the executable's ELF file.  I think the existing flag means "no 
>> executable stacks used by this module",
>> but "no executable stacks used by this process" would make more sense 
>> for Java.
>>

I found that it's possible to forbid dlopen() from loading shared 
libraries that requires executable stacks. This can be done by calling 
munmap() on the Java stack guard. Later, when dlopen calls mprotect to 
make the stack RWX, the mprotect() system call will fail with an "memory 
unavailable" error. As a result, dlopen would abort the loading.

I plan to make this as a separate commit with a new bug report. This 
feature would be controlled by a run-time switch like 
-XX:+ForbidLinuxExecStackDll. I really want to make this TRUE by 
default, for "security out of the box", but there is worry about 
backwards-compatibility for some customers that may depend on such old Dlls.

- Ioi