RFR: 8285893: Hiding dialog and showing new one causes dialog to be frozen

[Martin Fox]

On Sat, 4 May 2024 22:55:13 GMT, Martin Fox <mfox at openjdk.org> wrote:

> This PR is based on a discussion that happened over in PR #1324. Some of this explanation is copied from that thread.
> 
> When `exitNestedEventLoop` is called on the innermost loop the invokeLaterDispatcher suspends operation until the loop finishes. But if you immediately start a new event loop the previous one won't finish and the dispatcher will jam up and stop dispatching indefinitely.
> 
> When the invokeLaterDispatcher is told that the innermost loop is exiting it sets `leavingNestedEventLoop` to true expecting it to be set to false when the loop actually exits. When the dispatcher is told that a new event loop has started it is not clearing `leavingNestedEventLoop` which is causing the jam. Basically it should follow the same logic used in glass; leaving the innermost loop updates a boolean indicating that the loop should exit but if a new loop is started the boolean is set back to a running state since it now applies to the new loop, not the previous one.
> 
> I suspect the invokeLaterDispatcher exists in part to deal with the specifics of how deferred runnables are handled on the Mac. I investigated this a bit and wrote up some comments in the Mac glass code.

I now understand what’s going wrong on Linux but I don’t know how to fix it.

When the test case tries to leave the innermost loop (let’s call it A) the cross-platform Java code calls Application.leaveNestedEventLoop to tell glass that the loop should exit. But before the loop can exit the test case starts a new loop leading to a call to Application.enterNestedEventLoop.

On Mac the leaveNestedEventLoop call sets a few global state variables in glass and the enterNestedEventLoop call wipes out all those changes. So as far as glass is concerned the leaveNestedEventLoop call was never made. This works because the cross-platform Java code is holding onto an object that represents loop A; it’s down one level in the event loop stack and marked as LEAVING. When that object reaches the top of the event loop stack the core will once again call Application.leaveNestedEventLoop to exit loop A.

On Linux leaveNestedEventLoop makes changes in glass that cannot be erased or undone (it calls gtk_main_quit which updates internal GTK state that we don’t have access to). In the end GTK will exit the loops in the correct order but it will exit loop A before the core code has a chance to call Application.leaveNestedEventLoop again. This is throwing off the bookkeeping including some debug checks.

(JavaFX struggles with this scenario because the original test case behaves in such an unexpected way. While processing a click event on a modal stage the event handler hides that modal and then immediately starts a new modal. The original modal can’t really go away; the handler for the click event originated there and is still in progress. That first modal might not be visible but it can’t truly close until the second modal is dismissed.)

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1449#issuecomment-2110892622