RFR: 8335468: [XWayland] JavaFX hangs when calling java.awt.Robot.getPixelColor

[Harshitha Onkar]

On Fri, 15 Nov 2024 01:35:16 GMT, Alexander Zvegintsev <azvegint at openjdk.org> wrote:

> This is a linux only issue and this PR contains two parts(they are available as two separate commits):
> 
> a. [8335468](https://bugs.openjdk.org/browse/JDK-8335468): [XWayland] JavaFX hangs when calling java.awt.Robot.getPixelColor
> b. [8335469](https://bugs.openjdk.org/browse/JDK-8335469): [XWayland] crash when an AWT ScreenCast session overlaps with an FX ScreenCast session
> 
> `b.` is not reproducible without `a.` being fixed.
> 
> More about both cases:
> 
> ------
> <a href="#a-part" id="a-part">`a.`</a>
> 
> Previously we used a blocking [`g_main_context_iteration`](https://docs.gtk.org/glib/method.MainContext.iteration.html) call, this prevents normal execution if there is a GTK main loop running in the process.
> 
> Now we are handling 3 cases:
> 1. If there is no GTK main loop running
> _Example: just a JDK only application._
> In this case we call `g_main_context_iteration(NULL, TRUE)` as before (when [`gtk_main_level() == 0`](https://docs.gtk.org/gtk3/func.main_level.html)).
> 
> 2. If there is a GTK main loop running, but we are not requesting pixels on its thread
> _Example: application showing a `JFXPanel`, but are requesting pixels from a main thread._
> Now we are not trying to block thread: `g_main_context_iteration(NULL, FALSE)` (when `gtk_main_level() > 0`) .
> 
> 3. If there is a GTK main loop running, and we are requesting pixels on its thread
> _Example: a JavaFX application trying to get pixels on the FX application thread, e.g. from a button callback._
> Now we go nested with [`gtk_main()`](https://docs.gtk.org/gtk3/func.main.html)
> 
> [jdk commit](https://github.com/openjdk/jdk/pull/22131/commits/f439eb9739cf53ed6196a98062032b3045dd1cbe)
> 
> ------
> 
> <a href="#b-part" id="b-part">`b.`</a>
> 
> After fixing `a.` [the crash](https://bugs.openjdk.org/browse/JDK-8335469) appears:
> 
> Internally the ScreenCast session keeps open for 2s (both JDK and JFX, and their implementations are almost identical).
> This is to reduce overhead in case of frequent consecutive screen captures.
> 
> When we perform a [cleanup](https://github.com/openjdk/jdk/blob/db56266ad164b4ecae59451dc0a832097dbfbd8e/src/java.desktop/unix/native/libawt_xawt/awt/screencast_pipewire.c#L91) to close the session, we internally call [`pw_deinit`](https://docs.pipewire.org/group__pw__pipewire.html#gafa6045cd7391b467af4575c6752d6c4e).
> 
> It becomes a problem if these sessions overlap in time, so that the second session cleanup crashes when it tries to call pipewire functions without initializing the pipewire system by  [`pw_init`](https://do...

LGTM, tested without/with jdk fix and without/with jfx fix combinations.
Interop tests work as expected with jdk + jfx fix.

-------------

Marked as reviewed by honkar (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/22131#pullrequestreview-2456632078