RFR: 8308503: AArch64: SIGILL when running with -XX:UseBranchProtection=pac-ret on hardware without PAC feature [v3]

[Hao Sun]

> When revisiting the behavior of UseBranchProtection [1], we get one SIGILL error when running with -XX:UseBranchProtection=pac-ret on hardware without PAC.
> 
> Problem:
> 
> We build and run `java --version` with the following configuration matrix `Config X VMoption X Machine`.
> 
> 
>   Config = {--enable-branch-protection, null}
>   VMoption = {-XX:UseBranchProtection=pac-ret, -XX:UseBranchProtection=standard}
>   Machine = {w/ PAC, w/o PAC}
> 
> 
> VM crashes with SIGILL error for configure `Config=null, VMoption=pac-ret, Machine=w/o PAC`. The unrecognized instruction is `pacia x30, x29`, i.e. `pacia(lr, rfp)` generated by function `MacroAssembler::protect_return_address()`. [2]
> 
> Root cause:
> 
> 1. Instruction `pacia` is not in the NOP space. That's why `Config=null, VMoption=pac-ret` passes on `hardware w/ PAC`, but fails on `hardware w/o PAC`.
> 
> 2. -XX:UseBranchProtection=pac-ret behaves differently from the document [3], i.e.
> 
> 
>   In order to use Branch Protection features in the VM,
>   --enable-branch-protection must be used
> 
> 
> `_rop_protection` is not turned off for `Config=null`. That's why `VMoption=pac-ret, Machine=w/o PAC` passes with
> `Config=--enable-branch-protection` but fails with `Config=null`.
> 
> Fix:
> 
> This patch refines the parsing of -XX:UseBranchProtection=pac-ret:
> 
> 1. We handle "pac-ret" and "standard" in the same way, since only one type of branch protection is implemented for now, i.e. "pac-ret". We may update "standard" in the future if "bti" protection is added.
> 
> 2. `_rop_protection` is not turned on unless all the three conditions are satisfied [4]. Otherwise, it's kept off and one warning message is emitted.
> 
> 
> // Enable PAC if this code has been built with branch-protection, the
> // CPU/OS supports it, and incompatible preview features aren't enabled.
> 
> 
> [1] https://bugs.openjdk.org/browse/JDK-8287325?focusedCommentId=14581099&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14581099
> [2] https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp#L5976
> [3] https://github.com/openjdk/jdk/blob/master/doc/building.md#branch-protection
> [4] https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp#L457

Hao Sun has updated the pull request incrementally with one additional commit since the last revision:

  Update the warning message

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14095/files
  - new: https://git.openjdk.org/jdk/pull/14095/files/fbbf86a8..b22fa23d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14095&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14095&range=01-02

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/14095.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14095/head:pull/14095

PR: https://git.openjdk.org/jdk/pull/14095