Request for review [new bug](S): Stack guard pages are no more protected after loading a shared library with executable stack.

[Lindenmaier, Goetz]

Hi,

I am Goetz Lindenmaier from SAP AG, working in the SAP JVM Jit Team.

We found an issue loading jni libraries where the stack is set to 'execstack', or
where this flag is not set at all.  After loading such a library, the stack guard
pages are lost and though stack overflows can no more be detected.
We had a lot of libraries in our test systems where the execstack
flag was missing.

This webrev contains a small test and a possible fix for the problem:
http://www.sapjvm.com/gl/webrevs/noexecstack/
As I don't have a bug ID yet, I just used an arbitrary number to make
the test executable with jtreg .  Please open a bug for this issue.
I'll fix the test if I have a proper number.

This problem exists since 7019808, which adds -z noexecstack to the
linker command on linux.

The fix I propose does the following:
It reads the elf file to detect whether loading the library can change
stack properties.  If so, it requests a safepoint and loads the library
during the safepoint.  After loading the library, it tests the guard
pages with SafeFetch.  If they are no more protected, it reprotects
the guard pages of all Java stacks.

There might be cases where reading the elf file does not suffice
to know that the stack access rights will change.
As I understand, if a properly compiled jni library loads another
library which requires execstack, the stack changes access rights, too.
This is detected by the SafeFetch, and the stacks are fixed.  But in
this case the stacks are unprotected for a short time.
To improve on this, one would have to request a SafePoint for each
library loaded.  But anyways, there is no bulletproof solution,
as the loaded code could do an mprotect() at some point.

Best regards,
  Goetz.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20111026/307c59af/attachment.html