RFR: JDK-8284445: macOS 12 prints a warning when a function key shortcut is assigned to a menu

[Martin Fox]

On Mon, 2 Oct 2023 12:51:52 GMT, vlaaad <duke at openjdk.org> wrote:

> This PR addresses [JDK-8284445](https://bugs.openjdk.org/browse/JDK-8284445).
> 
> Context: the warning might be more important than it seems. Our JavaFX-based desktop app has [an issue](https://github.com/defold/defold/issues/7845) — when the user opens 2 instances of the app, one of the instances hangs after a while. While debugging the issue, I found that it's somehow related to the system menu bar — not refreshing it fixes the issue. Further investigations revealed that the app hangs only if we show menus that use F1-F24 shortcuts. This reminded me of the warnings we get about `NSEventModifierFlagFunction specified to -setKeyEquivalentModifierMask` for such shortcuts. I experimented with removing the modifier for these shortcuts, and it helped with our issue!

With this change the Quit menu item shows up as Cmd+F2.

To fix the bug we just need to stop setting the NSFunctionKeyMask in the equivalent's modifier mask. The rest of the code removed in this PR needs to be restored.

When the core code is converting from Java internals to the data structure passed to Glass it only has one field to specify the shortcut. Normally this is a character but for keys like F2 it passes down a Java KeyCode. To signal the difference it sets the MODIFIER_FUNCTION flag. Otherwise it can't distinguish KeyCode F2 from character Q.

The naming of this flag is confusing since it's not a modifier but a bit used solely for this purpose. And I don't know why this flag is set only for the function keys, it should be set whenever the core code is passing down a KeyCode rather than a character. This is probably part of what's going wrong in [JDK-8088445](https://bugs.openjdk.org/browse/JDK-8088445).

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1252#issuecomment-1774203116