RFR: 8273485: Deadlock when also using Swing and exiting Fullscreen on Mac [v9]

[Martin Fox]

On Mon, 18 Jul 2022 12:18:49 GMT, Florian Kirmaier <fkirmaier at openjdk.org> wrote:

>> When using Swing it's possible to generate a Deadlock.
>>  It's related to the nested eventloop started in enterFullScreenExitingLoop - and the RenderLock aquired when using setView in Scene.
>>  Sample Programm and Threaddump are added to the ticket.
>> 
>> Removing the nested loop fixes the Problem. 
>> I hope this doesn't have any side effect - so far i don't know of any.
>
> Florian Kirmaier has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 10 commits:
> 
>  - JDK-8273485
>    Fixing issue caused by wrong merge.
>  - Merge remote-tracking branch 'origin/master' into JDK-8273485_swing-deadlock
>    
>    # Conflicts:
>    #	modules/javafx.graphics/src/main/native-glass/mac/GlassViewDelegate.m
>  - JDK-8273485
>    Added check for null when calling initScreens
>  - JDK-8273485
>    Fixing toggle fullscreen!
>  - JDK-8273485
>    removed the toggle fullscreen before closing - to avoid the beep and improve the user experience
>  - JDK-8273485
>    Fixed Beep sound when closing a fullscreen window
>  - JDK-8273485
>    small cleanup of the changes.
>  - JDK-8273485
>    Removed the enter/leave nested event loop logic, for mac fullscreen
>  - JDK-8273485
>    Added unit-test
>  - JDK-8273485
>    Fixing deadlock when switching to fullscreen, when also swing is used.

I took a look at this since I'm trying to better understand the Mac fullscreen code (there's a bunch of cruft I would like to remove).

The stack trace in the original bug report confused me. In `WindowStage.setScene` the window be should taken out of fullscreen mode before setScene acquires the renderLock and sets the view to null. Exiting fullscreen is failing because at this point in the close process the stage is hidden and you can't change the fullscreen state of a stage while it's hidden (see `WindowStage.applyFullScreen`).

I think it's a bug in the core Java code that you can hide a fullscreen window without having it automatically exit fullscreen. Hiding a fullscreen window doesn't even work on the Mac since the OS just ignores the `orderOut:` call.

As an experiment I tweaked the Mac code to exit fullscreen when the window is hidden using exactly the same code as in _setView and it avoided the deadlock state reported in the original bug. I haven't done a full test run with the changes.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/622#issuecomment-1625983080