RFR: 8285893: Hiding dialog and showing new one causes dialog to be frozen

[Martin Fox]

On Sat, 4 May 2024 22:55:13 GMT, Martin Fox <mfox at openjdk.org> wrote:

> This PR is based on a discussion that happened over in PR #1324. Some of this explanation is copied from that thread.
> 
> When `exitNestedEventLoop` is called on the innermost loop the invokeLaterDispatcher suspends operation until the loop finishes. But if you immediately start a new event loop the previous one won't finish and the dispatcher will jam up and stop dispatching indefinitely.
> 
> When the invokeLaterDispatcher is told that the innermost loop is exiting it sets `leavingNestedEventLoop` to true expecting it to be set to false when the loop actually exits. When the dispatcher is told that a new event loop has started it is not clearing `leavingNestedEventLoop` which is causing the jam. Basically it should follow the same logic used in glass; leaving the innermost loop updates a boolean indicating that the loop should exit but if a new loop is started the boolean is set back to a running state since it now applies to the new loop, not the previous one.
> 
> I suspect the invokeLaterDispatcher exists in part to deal with the specifics of how deferred runnables are handled on the Mac. I investigated this a bit and wrote up some comments in the Mac glass code.

I'll look into the Linux failure. The core EventLoop code passes an object into the application's leaveNestedEventLoop and expects to see that object returned from the applications's matching enterNestedEventLoop call. On Mac and Windows this object is passed through to glass as an argument on the stack. On Linux this value is handled as an application global. I suspect the Linux bookkeeping isn't robust enough to handle this case but it will take a bit to nail down the details.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1449#issuecomment-2096926552