RFR: 8359108: Mac - When Swing starts First, native application menu doesn't work for JavaFX [v3]

[Martin Fox]

On Tue, 30 Sep 2025 23:31:20 GMT, Pabulaner IV <duke at openjdk.org> wrote:

>> This pull request fixes the system menu bar on MacOS when combining windows of Swing and JavaFX.
>> 
>> # Behavior before
>> 
>> If for some reason You needed to initialize AWT before JavaFX and You wanted to install the system menu bar from the JavaFX side, this wasn't possible. This issue is persistent even when You didn't open a Swing window. 
>> 
>> One scenario where this kind of issue happens is when You use install4j (see https://www.ej-technologies.com/install4j). In this case AWT is initialized by install4j and therefore You can't use the JavaFX system menu bar anymore.
>> 
>> 
>> # Behavior after
>> 
>> The fix allows JavaFX to install a system menu bar even if it is initialized after AWT. This is achieved by only changing code inside JavaFX. Each JavaFX window stores the previously installed menu bar when gaining focus and will restore this menu bar if the focus was lost. This only happens if the system menu bar installed by the JavaFX window is still unchanged.
>> 
>> 
>> # Tests
>> 
>> This PR introduces tests for the system menu bar in addition to verifying its own behavior / changes. The tests include single- and multi-window tests while interacting with Swing. The tests ensure that the menu bar stays the same for each window, no matter how You switch focus between them.
>> 
>> 
>> # Additional benifits 
>> 
>> This fix is not specifically for AWT, but allows JavaFX to interact much more compatibly with other frameworks that make use of the system menu bar.
>> 
>> 
>> # Review from AWT
>> 
>> In the previous PR related to this one, the comment was made that the folks from AWT should take a look at this fix. It would be great and much appreciated if someone could initiate it.
>> 
>> 
>> # Add disable flag?
>> 
>> We could also add a flag to prevent JavaFX from installing a system menu bar for users who have found other fixes for their projects / setups. This could be used to restore the previous behavior when AWT is initialized first.
>> 
>> 
>> Co-Author: @FlorianKirmaier
>
> Pabulaner IV has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - 8359108: Mac - When Swing starts First, native application menu doesn't work for JavaFX
>  - 8359108: Mac - When Swing starts First, native application menu doesn't work for JavaFX

I’ve been reviewing how the platform code interacts with the rest of the system. For others who want to take a look at this PR here’s a quick run-down of how the current master branch works.

When JavaFX is not embedded the toolkit builds a single NSMenu to use as the system menu. It installs that NSMenu as NSApp.mainMenu when the first top-level window is shown but before it becomes the key window. This happens inside the _setMenubar call in GlassWindow.m. That menu stays in place as NSApp.mainMenu for the lifetime of the JavaFX application. Subsequent calls to _setMenubar and the logic that sets NSApp.mainMenu inside windowDidBecomeKey: are basically no-ops since it’s always the same NSMenu and it has already been installed.

It might look like the system is setting a separate menubar for each window but it’s not. There’s only one, it stays in place, and the toolkit rebuilds its content as windows gain and lose focus. And it always contains at least one menu, namely the default application menu.

The system also calls _setMenubar to hand the system NSMenu to each PopupWindow when it’s first shown. There doesn’t seem to be any point to this; the NSMenu was already installed by the popup’s owner and MenuBarSkin ignores popups when updating the system menu bar.

When JavaFX is embedded none of the above happens.

This PR doesn’t seem to change any behavior in the non-embedded case. In the embedded case everything changes. JavaFX now builds a system-wide NSMenu and hands it off to each top-level window but it only gets installed when that window becomes key. The menu is un-installed when the window resigns key (and the previous mainMenu is restored). That’s all new behavior but it seems to be working reliably.

Also new with this PR: if a popup is displayed in an FXPanel the system will call _setMenubar to set the system menu. The platform code is handling this correctly (basically ignoring it) but the checks it’s making shouldn’t be necessary. It would be far better if createTKPopupStage stopped calling _setMenubar.

This PR is turning on some code further up in the system that can lead to some minor bugs. For example, if you add a MenuBar to an FXPanel’s scene and then turn on useSystemMenuBar the entire menubar will become inaccessible. This is similar to another bug that’s already present in the master branch; if you add a MenuBar to a PopupWindow and then turn on useSystemMenuBar the entire menubar will disappear. These issues are all contrived and it’s easy for developers to avoid them but it would be good to tighten up the code a bit. The bugs I’ve found so far are minor enough not to block this PR and could be addressed in a separate issue. This should be easy to clean up though I can’t personally provide further assistance until November.

(The primary problem is in MenuBarSkin.setSystemMenu. It crawls up the stage ownership chain to decide which MenuBar to use to populate the system menu bar. It should ignore windows that aren’t eligible for the system menu bar like embedded and popup stages. One possible solution is to call getMenubar on each stage to see if the toolkit initialized it with the system menu.)

Still no show-stoppers, just some unnecessary complications from the way PopupWindows keep trying to force the system menu onto the platform code and a few very minor bugs that no developer is likely to notice.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1904#issuecomment-3377746473