RFR: Disable PaX mprotect on executables

[Mikael Vidstedt]

Alpine by default has PaX [1] enabled. Among other things, PaX tries to prevent various attack vectors by controlling the ways in which page protection can be updated/changed. For example, it does not allow marking previously writable pages as executable. Guess what hotspot likes to do?

This change is incomplete, and I could use some help figuring out how to complete it.

http://cr.openjdk.java.net/~mikael/webrevs/portola/paxctl/webrev.00/webrev/

Basically, it all comes down to needing to use a tool - paxctl - to mark the executables in a way so that the kernel knows to disable/relax some of the PaX features for the process. The patch adds support for finding the paxctl tool, and for using the tool (if found) to mark the excutable. However, here are two of the main reasons why this patch is incomplete:

* There are two different JDKs which may or may not need the special paxctl treatment: buildjdk and target jdk. When building the JDK on Alpine (really: on a PaX enabled system) the buildjdk needs to have its executables marked as well, but when cross compiling that’s not necessarily needed.
* The patch currently bases the decision on whether to mark the executables on whether the target CLIB is “musl”, where in fact that doesn’t really have anything to do with musl
* Not all executables necessarily load and run the JVM, so not all of them necessarily need to be marked
* It’s not really the user space that decides whether the PaX relaxation is needed - it’s the kernel. That means that when running in, say, Docker, even if the base image would be Alpine (or some other distribution which has PaX enabled by default), it’s really the configuration of the host system kernel which comes into play.

I’m not sure what the right solution is here. Taking suggestions.

Cheers,
Mikael

[1] http://pax.grsecurity.net