RFR: 8223307: enable the Stack Execution Disable flag for JDK binaries on AIX - was : AIX : -bnorwexec linker flag

[Baesken, Matthias]

Hello,  I created a webrev for the previously discussed issue (AIX : -bnorwexec linker flag).

AIX supports a binary hardening option called SED, see :

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.security/stack_exec_disable.htm

System wide configuration can be done with  the sedmgr tool :

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds5/sedmgr.htm


The system-wide configuration supports various modes, very often  "select" mode is configured where 
binaries can opt-in  to  use  SED :

"select:
Only a select set of files are enabled and monitored for SED protection. The select set of files are chosen by reviewing the SED related flags in the executable program binary headers.
The executable program header enables SED related flags to request to be included in the select mode."


We can set a linker option on AIX to configure "select" mode for the JDK binaries; our tests show that this does not "kill" the JIT (jitted code).

changed binary shows then the opt-in flag ("request" ) :

bash-4.4$ sedmgr -d  /rs6000_64/nightly/output-jdk-test/images/jdk/bin/java
/rs6000_64/nightly/output-jdk-test/images/jdk/bin/java : request

while the unchanged binary does not have the flag set ("system") :

bash-4.4$ sedmgr -d /rs6000_64/nightly/output-jdk11-test/images/jdk/bin/java
/rs6000_64/nightly/output-jdk11-test/images/jdk/bin/java : system



Bug/webrev :

https://bugs.openjdk.java.net/browse/JDK-8223307

http://cr.openjdk.java.net/~mbaesken/webrevs/8223307.0/


Thanks, Matthias



> -----Original Message-----
> From: Erik Joelsson <erik.joelsson at oracle.com>
> Sent: Freitag, 12. April 2019 15:30
> To: Baesken, Matthias <matthias.baesken at sap.com>; ppc-aix-port-
> dev at openjdk.java.net; 'build-dev at openjdk.java.net' <build-
> dev at openjdk.java.net>
> Subject: Re: AIX : -bnorwexec linker flag
> 
>  From a build point of view, the patch looks good. I cannot comment on
> the validity of adding the flag though.
> 
> /Erik
> 
> On 2019-04-12 02:15, Baesken, Matthias wrote:
> > Hello,  I have a question regarding the  AIX  -bnorwexec linker flag .
> > I think it is related to an AIX  security feature  SED , see also  :
> >
> >
> https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.
> aix.security/stack_exec_disable_flags.htm
> >
> > When building with the additional  -bnorwexec linker flag    we signal  the
> OS  that we "request"  the SED  feature .
> > Please compare  a patched  and an unpatched  java   ( patched is flagged
> "request"  while   unpatched uses the  "system" setting ).
> >
> > bash-4.3$ sedmgr -d /patched_jdk/images/jdk/bin/java
> > /patched_jdk/images/images/jdk/bin/java : request
> >
> >
> > bash-4.3$ sedmgr -d /normal_jdk/images/jdk/bin/java
> > /normal_jdk/images/jdk/bin/java : system
> >
> >
> > System config on the example machine is "normal" (default) select :
> > bash-4.3$ sedmgr
> > Stack Execution Disable (SED) mode: select
> > SED configured in kernel: select
> >
> >
> > In our  internal tests  I noticed so far no issues when setting the  -
> bnorwexec linker flag  in OpenJDK  on AIX  .
> > Do you have any experience  with it, do you see issues when setting the
> flag ?
> >
> >
> > The documentation of the  flag is a bit short .
> >
> >
> >
> https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.
> aix.cmds3/ld.htm
> >
> >
> > norwexec
> >
> > Specifies that if the system's sed_config setting is not off, the process'
> private data areas will have non-execute permission.
> >
> >
> >
> >
> > Patch would be :
> >
> > diff -r 0d7fb7f07134 make/autoconf/flags-ldflags.m4
> > --- a/make/autoconf/flags-ldflags.m4    Mon Apr 08 06:56:37 2019 +0100
> > +++ b/make/autoconf/flags-ldflags.m4 Mon Apr 08 10:50:07 2019 +0200
> > @@ -1,5 +1,5 @@
> > #
> > -# Copyright (c) 2011, 2018, Oracle and/or its affiliates. All rights reserved.
> > +# Copyright (c) 2011, 2019, Oracle and/or its affiliates. All rights reserved.
> > # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> > #
> > # This code is free software; you can redistribute it and/or modify it
> > @@ -88,7 +88,7 @@
> >       BASIC_LDFLAGS_JVM_ONLY="-library=%none -mt -z noversion"
> >     elif test "x$TOOLCHAIN_TYPE" = xxlc; then
> > -    BASIC_LDFLAGS="-b64 -brtl -bnolibpath -bexpall -bernotok -
> btextpsize:64K \
> > +    BASIC_LDFLAGS="-b64 -brtl -bnorwexec -bnolibpath -bexpall -bernotok
> -btextpsize:64K \
> >           -bdatapsize:64K -bstackpsize:64K"
> >       # libjvm.so has gotten too large for normal TOC size; compile with
> qpic=large and link with bigtoc
> >       BASIC_LDFLAGS_JVM_ONLY="-Wl,-lC_r -bbigtoc"
> >
> >
> > Best regards, Matthias
> >