RFR: 8328306: AArch64: MacOS lazy JIT "write xor execute" switching

[Andrew Dinn]

On Wed, 30 Jul 2025 17:34:48 GMT, Andrew Haley <aph at openjdk.org> wrote:

> In MacOS/AArch64 HotSpot, we have to deal with the fact that a thread must be in one of two modes: it either may write to code cache memory or it may execute (and read) code or data in it. A system call `pthread_jit_write_protect_np(int enabled)` changes from one to the other.
> 
> Today, we change mode whenever making a transition from interpreter to VM. This means that we change mode a lot: experiments have shown that during `jshell` startup we change mode 4 million times. Other experiments have shown that we only needed to change mode 45 thousand times.
> 
> This "eager" mode switching is perhaps too eager, and we'd be better off switching lazily. While the system call that changes mode is very fast, mode switching still amounts to about 100ms of startup time. Switching eagerly also means that some native calls (e.g. to do arithmetic) are disproportionately expensive, given that they have no need of mode switching at all.
> 
> The approach in this PR is to defer transitioning from exec-but-don't-write mode (`WXExec`) to write-but-don't-exec mode (`WXWrite`) until we need to write. Instead of enabling `WXWrite` immediately, we switch to a mode called `WXArmedForWrite`. When in this mode, when we need to write into code memory we call `os_bsd_jit_exec_enabled(false)` to enable writing and then set the current mode to `WXWrite`.
> 
> We mark all sites that we know will write to code memory with
> `MACOS_AARCH64_ONLY(os::thread_wx_enable_write());` Judicious placement of these markers, such as when entering patching code, means that we have a fairly small number of these.
> 
> We also keep track (in thread-local storage) of the current state of `pthread_jit_write_protect_np` in order to avoid making the system call unnecessarily.
> 
> It is possible that we have missed some sites where we do need to make a transition from write-protected to -enabled. While we haven't seen any in testing, we have a fallback path. An attempt to write into code memory triggers a `SIGILL` signal. A signal handler detects this, and if the current mode `WXArmedForWrite` it changes mode to write-enabled and returns. In addition, the handler "heals" the VM entry point so that next time the same point is entered (and for the rest of the lifetime of the VM) it will immediately transition to `WXWrite`.
> 
> One other possibility remains: we could omit all of the `wx_enable_write` markers and use healing instead. We've experimented with this. It works well enough, but is rather crude, and it's better to be able to ...

src/hotspot/os/bsd/os_bsd.cpp line 821:

> 819:   // Avoiding excessive CAS operations to hot RW locations is critical.
> 820:   // See https://blogs.oracle.com/dave/entry/cas_and_cache_trivia_invalidate
> 821:   // https://web.archive.org/web/20131214182431/https://blogs.oracle.com/dave/entry/cas_and_cache_trivia_invalidate

Something is wrong here. Do we need both the original URL and the wayback machine link?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26562#discussion_r2268996730