RFR: 8242544: CMD+ENTER key event crashes the application when invoked on dialog

[Martin Fox]

On Wed, 29 Dec 2021 00:44:44 GMT, Andreas Heger <duke at openjdk.java.net> wrote:

> This also solves [JDK-8205915 [macOS] Accelerator assigned to button in dialog fires menuItem in owning stage](https://bugs.openjdk.java.net/browse/JDK-8205915l).
> 
> I must admit that I don't have 100% insight about what actually caused the problems and how the event flow completely works.
> 
> In MacOS, key events with a command or control modifier are handled via the performKeyEquivalent method, first. If no view or window handles such a shortcut key event (indicated by a return value NO in performKeyEquivalent), the key event is sent a second time via the sendEvent method. Before this fix, the method  **sendJavaKeyEvent** in **GlassViewDelegate.m** seemed to be called twice: first by performKeyEquivalent and then by sendEvent, since no responder returned YES in performKeyEquivalent. For not handling the same event twice in JFX, there seemed to be a workaround in  **sendJavaKeyEvent** in **GlassViewDelegate.m**. It only handled the first call. The second call checked if the given event was exactly the same like in the last call. In this case it simply returned and did not forward the event. This means that all key events with a CMD or CTRL modifier were handled in JFX in the performKeyEquivalent event loop, even though they actually weren't used by the MacOS keyEquivale
 nt functionality and all responders returned NO in performKeyEquivalent. So MacOS sent the event again to all responders in the sendEvent loop. I assume that the window has been destroyed already, when MacOS tried to send the second event to it, because the closing was handled in the first call. Sending an event to a no longer existing window probably caused the crash by an unhandled exception.
> 
> It seems that this problem existed in the past and there was a workaround in **GlassWindow.m**. In the method **performKeyEquivalent**, it was checked if the window was closed and if so, it returned YES. However, nowadays the window closing check did not work... self->gWindow->isClosed returned NO when I debugged it, although the window should be closed by the key event. Returning YES in case of a closed window is not a clean solution, anyway, because YES should mean that the shortcut key equivalent is handled. Another point is that Apple writes that overwriting performKeyEquivalent in an NSWindow subclass is discouraged. So, I completely deleted the method from **GlassWindow.m**, since it only returned the value of the super function, except for the non working closing check.
> 
> Since the default version of performKeyEquivalent in NSWindow always returns NO, only deleting the method from **GlassWindow.m** did not fix the crash. So I tried to solve the problem that a shortcut key event is handled in the performKeyEquivalent loop. The call of **sendJavaKeyEvent** in **GlassViewDelegate.m** which is caused by a performKeyEquivalent should not be handled, actually. So, I simply removed this call which is done in **GlassView3D.m** **performKeyEquivalent** method. By removing the call, there is also no longer any need to check if the same key event was passed to **sendJavaKeyEvent** in **GlassViewDelegate.m**, so this check is also removed.
> 
> I'm curious about your comments and reviews and I wonder if this is a clean solution. All my tests so far seemed to be fine.

I reviewed the Mac glass code and it seems the smallest change we can make to prevent the crash is to set the NSWindow's contentView to a dummy NSView instead of nil. I don't see any place in the code where the contentView is inspected or is assumed to be a glass NSView subclass. I can submit the PR since I already have a branch set up to test it.

I suspect performKeyEquivalent was originally implemented because before macOS 10.5 it was the only way to receive key events with the Cmd modifier. These days we can receive those events via keyDown. For now it's main purpose is to ensure we get key events before the system menu bar does.

@jperedadnr The test case for JDK-8205915 tests a particular aspect of performKeyEquivalent that I just learned about (I've been watching some exciting WWDC videos lately). If the main and key windows are not the same they can both end up processing the same key event via performKeyEquivalent. That's a great test case.

I don't think the javafxports test case reveals the underlying ordering issue. Cmd+V should be handled by the textField directly and the MenuItem should not fire. I suspect with performKeyEquivalent removed the opposite is happening and only the MenuItem is firing.

-------------

PR: https://git.openjdk.java.net/jfx/pull/704