RFR: 8359601: Fix window button states of an extended stage

[Cormac Redmond]

On Sun, 15 Jun 2025 20:01:11 GMT, Michael Strauß <mstrauss at openjdk.org> wrote:

> The window button states (disabled/hidden) of extended stages with a `HeaderButtonOverlay` or custom header buttons are inconsistent with what we would expect from the OS (Windows and Linux). To figure out what we would expect, I started with gathering some data. The following table shows the button states of system-decorated windows on various platforms:
> 
> #### Windows 11
> 
> | Window attributes | Iconify | Maximize | Close |
> |---|---|---|---|
> | resizable, not modal | visible | visible | visible |
> | not resizable, not modal | visible | visible, disabled | visible |
> | resizable, modal | visible, disabled | visible | visible |
> | not resizable, modal | hidden | hidden | visible, utility-style |
> 
> #### Ubuntu 24 / Fedora 41 (GNOME)
> 
> | Window attributes | Iconify | Maximize | Close |
> |---|---|---|---|
> | resizable, not modal | visible | visible | visible |
> | not resizable, not modal | visible | hidden | visible |
> | resizable, modal | visible, _not working_ | visible, _not working_ | visible |
> | not resizable, modal | visible, _not working_ | hidden | visible |
> 
> #### Kubuntu 24 (KDE)
> 
> | Window attributes | Iconify | Maximize | Close |
> |---|---|---|---|
> | resizable, not modal | visible | visible | visible |
> | not resizable, not modal | visible | hidden | visible |
> | resizable, modal | visible, _not working_ | visible | visible |
> | not resizable, modal | visible, _not working_ | hidden | visible |
> 
> ## Observations
> 
> 1. On Windows, buttons are generally disabled when their operation is not possible with the given window attributes.
>    * Exception: modal/non-resizable windows look like utility windows (iconify and maximize are hidden)
> 2. On GNOME and KDE, buttons are generally hidden when their operation is not possible.
>    * Exception: iconify and maximize on modal windows is not hidden, but seems to simply not do anything (bug?)
> 
> ## Permitted window button operations
> 
> Given the gathered observations and some simple logic, this is the table of operations that are permitted for all combinations of modal and resizable window attributes:
> 
> | Window attributes | Iconify | Maximize | Close |
> |---|---|---|---|
> | resizable, not modal | yes | yes | yes |
> | not resizable, not modal | yes | no | yes |
> | resizable, modal | no | yes | yes |
> | not resizable, modal | no | no | yes |
> 
> ## Fixes
> 
> This PR includes the following changes:
> 1. Unused code relating to window modality is removed.
> 2. The disabled states of `HeaderButtonOverlay` as well as `HeaderButtonBehavior` are changed to mat...

Most of this seems to be working well!

But I have found a likely bug. Hopefully fairly self-explanatory from the code.

Minimise icon is normally disabled, but still enabled when EXTENDED is used & when dialog is resizable. 

Just flip comment on `dialog.initStyle(StageStyle.EXTENDED)` to see difference.



import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.control.ButtonType;
import javafx.scene.control.Dialog;
import javafx.scene.layout.StackPane;
import javafx.stage.Modality;
import javafx.stage.Stage;
import javafx.stage.StageStyle;

public class MinimiseIconEnabledBug extends Application {
    @Override
    public void start(Stage primaryStage) {
        Button button = new Button("Click");
        button.setOnAction(e -> {
            final Dialog<Object> dialog = new Dialog<>();
            dialog.initOwner(primaryStage);
            // Existing behaviour: minimise icon disabled when commented
            // Possible "bug": Minimise icon enabled when uncommented
            // dialog.initStyle(StageStyle.EXTENDED);
            dialog.initModality(Modality.NONE);
            dialog.setResizable(true); // This is important
            dialog.setTitle("My Dialog");
            dialog.setContentText("Lorem ipsum dolor sit amet, consectetur adipiscing...");
            dialog.getDialogPane().getButtonTypes().addAll(ButtonType.OK);
            dialog.show();
        });

        StackPane root = new StackPane(button);
        Scene scene = new Scene(root, 300, 200);

        primaryStage.setTitle("My First JavaFX App");
        primaryStage.setScene(scene);
        primaryStage.show();
    }

    public static void main(String[] args) {
        launch(args);
    }
}

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1831#issuecomment-2978439674