RFR: 8221261: Deadlock on macOS in JFXPanel app when handling IME calls

[Kevin Rushforth]

On Wed, 10 Jan 2024 14:54:47 GMT, Kevin Rushforth <kcr at openjdk.org> wrote:

> As described in the JBS bug, there is a long-standing deadlock that happens on macOS between the AWT EDT and the JavaFX Application thread (which on macOS is the AppKit thread) when processing Input Method Events (IME) in a WebView node in a JFXPanel.
> 
> This PR fixes the deadlock by adding `"AWTRunLoopMode"` to the list of modes that will accept the calls to `performSelectorOnMainThread` used by `_submitForLaterInvocation` and `_invokeAndWait`. These two native methods are the macOS implemention `runLater` and `runAndWait`, respectively, and are used to run Java Runnables on the JavaFX Application Thread.
> 
> This deadlock is happening much more often on recent macOS versions, and in macOS 14, pressing CAPS LOCK is sufficient to trigger IME calls.
> 
> The OS calls the AWT IME methods on the AppKit thread, which is also the JavaFX Application Thread. The AWT IME callback methods, for example `characterIndexForPoint`, call invokeAndWait to run the IME handler on the AWT Event thread. In the case of JFXPanel, the registered IME handler is in JavaFX code and we often need to run something on the JavaFX Application Thread to avoid threading problems or crashes. See [JDK-8196011](https://bugs.openjdk.org/browse/JDK-8196011) and [JDK-8322703](https://bugs.openjdk.org/browse/JDK-8322703).
> 
> A similar deadlock was observed while trying to fix a threading problem in JFXPanel's handling of  the IME calls as described in [JDK-8322784](https://bugs.openjdk.org/browse/JDK-8322784), so it isn't necessarily limited to using WebView.
> 
> Anton Tarasov @forantar proposed a fix (in a comment in the JBS bug) in AWT that would change the invokeAndWait calls used by the IME methods in CInputMethod to call `doAWTRunLoop` with the `processEvents` flag set to true, which will allow all events to be processed while in `doAWTRunLoop`.
> 
> NOTE: I had created Draft PR openjdk/jdk#17290 with Anton's proposed fix, expanded to cover all of the cases, but during the discussion it was pointed out that `doAWTRunLoop` should already be able to run selectors via `performSelectorOnMainThread` -- see [this comment](https://github.com/openjdk/jdk/pull/17290#issuecomment-1880001813). The reason `doAWTRunLoop` doesn't process our messages is that it runs the loop in a mode that only handles messages that include a custom `"AWTRunLoopMode"` mode.    
>           
> Adding `"AWTRunLoopMode"` to the `performSelectorOnMainThread` calls used by `_submitForLaterInvocation` and `_invokeAndWait` allows them to run when in...

Reviewers: @prrace @jaybhaskar

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1327#issuecomment-1885299231