Check for PaX during initialization

[Poonam Parhar]

Hello Mikael,

A couple of comments:


+     vm_exit_out_of_memory(size, OOM_MMAP_ERROR, "failed to allocate memory for PaX check.");

'failed' should be Failed.


+                                   "    kernel.pax.softmode = 0\n"
+                                   "\n"
+                                   "In particular, if the value is 0 (zero), then PaX is enabled.\n"

I think, it would be good to mention how to disable Pax. For example, adding the following in the error message would be handy for people after they get this failure at launch time.  

sysctl -w kernel.pax.softmode=1

Thanks,
Poonam

> -----Original Message-----
> From: Mikael Vidstedt
> Sent: Monday, May 01, 2017 2:03 PM
> To: portola-dev at openjdk.java.net
> Subject: Check for PaX during initialization
> 
> 
> Please review the following change, which adds code to check for the
> presence of PaX/MPROTECT (https://pax.grsecurity.net/docs/mprotect.txt
> <https://pax.grsecurity.net/docs/mprotect.txt>) during VM
> initialization, and prints out a helpful message if PaX is enabled and
> interfering with the VM. Specifically, the code checks if a writable
> page can be made executable (mimicking what the JIT would typically
> do). Since ZERO doesn’t generate code the check is not performed there.
> 
> Copy+paste from the actual change:
> // Some linux distributions (notably: Alpine Linux) include the //
> grsecurity in the kernel by default. Of particular interest from a //
> JVM perspective is PaX (https://pax.grsecurity.net/), which adds //
> some security features related to page attributes. Specifically, // the
> MPROTECT PaX functionality //
> (https://pax.grsecurity.net/docs/mprotect.txt) prevents dynamic // code
> generation by disallowing a (previously) writable page to be // marked
> as executable. This is, of course, exactly what HotSpot does // for
> both JIT compiled method, as well as for stubs, adapters, etc.
> //
> // Instead of crashing "lazily" when trying to make a page executable,
> // this code probes for the presence of PaX and reports the failure //
> eagerly.
> 
> http://cr.openjdk.java.net/~mikael/webrevs/portola/paxcheck/webrev.02/h
> otspot/webrev/
> 
> Cheers,
> Mikael
>