RFR: 8190329: [macos] Swing InterOp Platform.exit() crash

[Kevin Rushforth]

On Fri, 23 Aug 2024 17:56:57 GMT, Phil Race <prr at openjdk.org> wrote:

>> The testcase calls Platform.exit before the dialog is made visible but
>> on macOS, JavaFX takes over the AppKit thread and uses that as the FX application thread. As part of the FX platform shutdown process, it detaches that thread from the JVM. This means that the AppKit thread is no longer available to Java processes
>> so AWT crashes when it ties to access appkit thread.
>> Fix is made to check if the thread has been detached for move-resize notification event before proceeding..
>
> src/java.desktop/macosx/native/libawt_lwawt/awt/AWTWindow.m line 640:
> 
>> 638:     if (ret == JNI_EDETACHED) {
>> 639:         return;
>> 640:     }
> 
> This seems like a completely arbitrary location in the AWT code for a test of something that would seem completely unexpected.
> Can you PROVE that this is the only place that this will ever be needed ?
> And if you can't, this isn't the right fix.
> If you can it needs some comments.
> 
> Also : does AWT detach the AppKit thread from the VM on shutdown ?
> IF not why does FX have to do it ? If it does, then could FX also crash if FX is embedded in an AWT app.

I'm also not sure that this is the right fix. It seems like it will fix one specific place where AWT could do an upcall on the AppKit thread after FX has detached without addressing the problem as a whole.

Another possible fix would be to have AWT not use the cached JNI Env when it is not the NSApplication owner. I don't know how viable this fix would be, or whether it is the right fix, but it would prevent the crash in all possible places.

> Also : does AWT detach the AppKit thread from the VM on shutdown ?

I doubt it, since there isn't AFAIK an explicit toolkit shutdown in AWT like there is in FX.

> IF not why does FX have to do it ? If it does, then could FX also crash if FX is embedded in an AWT app.

FX only detaches the AppKit thread when it is the NSApplication owner, and thus attached the thread in the first place. In the case of FX embedded in an AWT app, it won't detach the thread.

We could explore what it would take for FX to not detach the AppKit thread from Java. At a minimum, it would require attaching the AppKit thread as a daemon (which AWT already does), and then creating a non-daemon Java thread that we would then terminate at FX toolkit shutdown. This would need to be tested carefully, but might be the cleanest solution.

One thing to note is that anything we do here will be about avoiding a crash in AWT, not about having the UI remain functional. In this case FX is the "owner" of the native Window system event loop (and NSApplication on macOS). Applications should not shutdown the FX toolkit (meaning, not call `Platform.exit`) and expect AWT to continue to work.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/20688#discussion_r1729612142