RFR: 8270199: Most SA tests are skipped on macosx-aarch64 because all executables are signed

[Chris Plummer]

For any SA test that attaches to an OSX process (this would be all SA tests except for those that test core file support), there is a check to make sure that the target jvm process is not a signed binary. If it is, SkippedException is thrown, and the test passes without doing anything. This is all we can do since being signed implied being notarized, and debuggers cannot attach to a notarized binary.

I noticed that on macosx-aarch64, all our SA tests that attach to a process were being skipped because the binary was signed, even for debug builds. It turns out that for macosx-aarch64, the linker always ads what is known as ad-hoc signing. You can find some info on ad-hoc signing here:

https://eclecticlight.co/2020/08/22/apple-silicon-macs-will-require-signed-code/

The tests use the codesign tool to determine if the binary is signed. Normally the check just relies on getting an error code of 1 when not signed, but since all binaries are now signed on macosx-aarch64, we need to modify the check to ignore ad-hoc signing.

Using "codesign --display" on an an ad-hoc signed binary shows some output that is of interest:

bash-3.2$ codesign --display --verbose=4 a.out
CodeDirectory v=20400 size=254 flags=0x20002(adhoc,linker-signed) hashes=5+0 location=embedded 

Looking for flags=0x20002(adhoc,linker-signed) will tell us if the binary is adhoc signed, which means we can assume that the SA tests can still be run against it. 

Looking for flags=0x10000(runtime) will tell us that the binary is hardened. These are binaries that we cannot use with SA.

If the binary is not signed at all, which should only happen on macosx-x64, codesign will print the message "code object is not signed at all". We can also run SA against these binaries.

Due to fixing the issue, we now have to problem list a few more core file tests on macosx-aarch64 due to 8269982. This issue was not noticed in the past because the tests were not being run. This was due to most of our macosx-aarch64 machines not being capable of producing a core file. Without this fix the test assumes it couldn't produce the core file because the binary was signed. With this fix it, for binaries that are not hardened the test will fail if it does not produce a core file.

-------------

Commit messages:
 - Merge branch 'master' into 8270199-osx_hardened
 - Fix OSX test issues with determining if binary is hardened or not.

Changes: https://git.openjdk.java.net/jdk/pull/6906/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=6906&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8270199
  Stats: 64 lines in 5 files changed: 27 ins; 18 del; 19 mod
  Patch: https://git.openjdk.java.net/jdk/pull/6906.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/6906/head:pull/6906

PR: https://git.openjdk.java.net/jdk/pull/6906