RFR: 8339178: [macos] Swing InterOp Platform.exit() crash

[Kevin Rushforth]

On Mon, 26 Aug 2024 21:11:03 GMT, Kevin Rushforth <kcr at openjdk.org> wrote:

> A Swing / FX interop app will crash if an application creates a new AWT / Swing window after calling Platform.exit. The root cause of the crash is that AWT caches the JNI env pointer for the AppKit thread, and assumes that it is valid for the life of the application. In the case where JavaFX is the owner of the NSApplication, we detach the AppKit thread from the JVM, after which AWT's env pointer is no longer valid. AWT will therefore crash the next time it does a JNI upcall.
> 
> This PR fixes the crash by leaving the macOS AppKit thread attached to the JVM after the JavaFX main event loop terminates. This requires attaching the AppKit thread to the JVM as a daemon thread when JavaFX is the NSApplication owner, matching what AWT does when it is the owner. In order to prevent a JavaFX application from exiting prematurely, create a non-daemon "KeepAlive" thread that can be terminated when the FX toolkit exits. This also solves a somewhat-related problem where the JavaFX toolkit will exit prematurely if the AWT toolkit is started first, and all AWT windows are disposed.
> 
> This fix is in addition to the AWT fix: [JDK-8190329](https://bugs.openjdk.org/browse/JDK-8190329) / openjdk/jdk#20688. Either the AWT fix or the JavaFX fix is sufficient to avoid this specific problem, but there is value in fixing it in both places, so I cloned the AWT bug to create a JavaFX bug that we can use for this PR.
> 
> Summary of the changes:
> 
> * Attach the AppKit thread to the JVM as a daemon
> * Do not detach the thread when the FX main event loop terminates
> * Create and start a KeepAlive thread in MacApplication, when the FX toolkit starts
> * Terminate the KeepAlive thread when the FX toolkit finishes
> 
> Testing:
> 
> I created automated systems tests from the manual test programs in the bugs as well as a test to ensure that we don't regress and exit prematurely in the pure JavaFX case (which would happen without the KeepAlive thread). Two of the new tests fail on macOS without the fix and pass with the fix. All three pass on all platforms with the fix.

tests/system/src/test/java/test/launchertest/TestAppPlatformExitAWT.java line 50:

> 48:             Platform.exit();
> 49:         });
> 50:         d.setVisible(true);

By code inspection, I realized that the `JDialog` might be shown before `Platform::exit` has been called, meaning that the test could pass even if the bug is present. I'll push a fix to move the setVisible inside the runLater lambda, using a `SwingUtilities::invokeLater`. This will ensure that it provokes the bug.

I ran the updated test and confirm that the test still fails without the fix and passes with the fix.

-------------

PR Review Comment: https://git.openjdk.org/jfx/pull/1545#discussion_r1736501384