RFR: 8337536: AArch64: Enable BTI branch protection for runtime part [v3]

[Fei Gao]

On Tue, 3 Sep 2024 09:25:55 GMT, Andrew Haley <aph at openjdk.org> wrote:

> What is the effect on JNI? Is there full interworking with non-branch-protected libraries?

@theRealAph, thanks for your review!

It should be no problem to have libjvm.so built with BTI and a JNI library built without BTI.

BTI marks code pages as "Guarded". For executable pages that have been guarded, all indirect branches must have a destination that is a BTI instruction of the appropriate type. But for unguarded pages, we don’t do this check. This allows BTI to be incrementally turned on for a specific codebase. BTI would then protect the branches within the libraries with BTI  but not those without BTI.

When we're jumping from JNI to libjvm, it's OK because BTI is enabled for libjvm.so and all the entry points have landing pads. When we're jumping from libjvm to JNI, it's also OK because the code cache pages have BTI disabled so it doesn't need landing pads.

To verify it, after patching this PR, I disabled the `-mbranch-protection=standard` flag for all other libraries and enabled it only for jvm, we can have libjvm.so built with BTI and all other libraries built without BTI. Jtreg tests passed. Before the patch, on mainline, we have all other libraries built with BTI but libjvm.so built without BTI, we also have no BTI failures.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/20491#issuecomment-2328972872