RFR: JDK-8285893: Hiding dialog and showing new one causes dialog to be frozen [v6]

[Martin Fox]

On Tue, 5 Mar 2024 23:37:18 GMT, Marius Hanl <mhanl at openjdk.org> wrote:

>> This PR fixes the dialog freeze problem once and for all. 
>> 
>> This one is a bit tricky to understand, here is how it works:
>> This bug happens on every platform, although the implementation of nested event loops differs on every platform.
>> E.g. on Linux we use `gtk_main` and `gtk_main_quit`, on Windows and Mac we have an own implementation of a nested event loop (while loop), controlled by a boolean flag.
>> 
>> Funny enough, the reason why this bug happens is always the same: Timing.
>> 
>> 1. When we hide a dialog, `_leaveNestedEventLoop` is called. 
>> 2. This will call native code to get out of the nested event loop, e.g. on Windows we try to break out of the while loop with a boolean flag, on Linux we call `gtk_main_quit`.
>> 3. Now, if we immediately open a new dialog, we enter a new nested event loop via `_enterNestedEventLoop`, as a consequence we do not break out of the while loop on Windows (the flag is set back again, the while loop is still running), and we do not return from `gtk_main` on Linux.
>> 4. And this will result in the Java code never returning and calling `notifyLeftNestedEventLoop`, which we need to recover the UI.
>> 
>> So it is actually not trivial to fix this problem, and we can not really do anything on the Java side. We may can try to wait until one more frame has run so that things will hopefully be right, but that sounds rather hacky.
>> 
>> I therefore analyzed, if we even need to return from `_enterNestedEventLoop`. Turns out, we don't need to. 
>> There is a return value which we actually do not use (it does not have any meaning to us, other that it is used inside an assert statement).
>> Without the need of a return value, we also do not need to care when `_enterNestedEventLoop` is returning - instead we cleanup and call `notifyLeftNestedEventLoop` in `_leaveNestedEventLoop`, after the native code was called.
>> 
>> Lets see if this is the right approach (for all platforms).
>> Testing appreciated.
>> #
>> - [x] Tested on Windows
>> - [x] Tested on Linux
>> - [x] Tested on Mac
>> - [ ] Tested on iOS (although the nested event loop code is the same as for Mac) (I would appreciate if someone can do this as I have no access to an iOS device)
>> - [x] Adjust copyright
>> - [x] Write Systemtest
>> - [ ] Document the new behaviour - in general, there shoul be more information what to expect
>
> Marius Hanl has updated the pull request incrementally with four additional commits since the last revision:
> 
>  - test for multiple nested event loops
>  - try leave outer loop after finishing inner loop
>  - update copyright
>  - trigger an outer nested event loop leave attempt

The Mac is still failing the NestedEventLoop test.

At the top of InvokeLaterDispatcher.java there's a long explanation on how it's trying to coordinate with the event loop. To do this it needs to have an accurate picture of what's happening at the platform level. In this PR you're telling the dispatcher that an event loop has exited before the platform has actually exited the loop (before enterNestedEventLoop has returned). This is causing the dispatcher to send runLater runnables to the platform earlier than expected. On Windows the runnables will get deferred to the next event loop cycle but on Mac the runnables might get executed in the current event loop cycle. I think this is one of the "native system limitations" mentioned in the comment in InvokeLaterDispatcher.java.

I've created a branch with my proposed fix for this problem (https://github.com/beldenfox/jfx/tree/eventloopjam). The fix prevents the dispatcher from jamming when the event loop it thought was leaving enters a new loop instead. Over in the Mac Glass code I also added a comment with a few more details on what's going on.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1324#issuecomment-1981400654