[External] : On UI testing in Wayland

[Maxim Kartashev]

(Following up on our discussion at the meeting on Feb 26, 2026)

As you know, each window in Wayland has a coordinate system of its own with
the top-left corner placed at (0, 0) in that coordinate system. There is no
way to programmatically set or obtain the location in the global desktop's
coordinate system.
This complicates automatic testing to the extreme; many UI tests follow
this pattern:
   Point p = component.getLocationOnScreen();
   Color c = robot.getPixelColor(p.x + dx, p.y + dy);
   if (!c.equals(...))...

Since getLocationOnScreen() returns (0, 0) for top-level windows, the
coordinates supplied to the Robot call will almost certainly be incorrect
in the global coordinate space that Robot uses.

There are a few routes that can be taken to at least mitigate this problem.
1. Make Robot operate in window's coordinate space and look into pixels
obtained from window'
s SurfaceData without involving the desktop compositor.
This obviously doesn't work when there's more than one window and does not
account for popups. Still, this was sufficient to run more than half of the
existing tests.
Another problem with this approach is that client applications, which are
not tests, need to take proper screenshots of the composited desktop.
Therefore, some kind of switch between two Robot modes is required (perhaps
a system property?).

2. Enhance Robot API to specify which window's pixels need to be fetched.
This will require bulk updates to most tests under java/awt and javax/swing
in addition to changes on all currently supported platforms.

3. Run tests only in a more controlled Wayland environment that exposes the
global coordinate system to the client applications (like Weston or Gnome
KIOSK, perhaps with extra plugins).
This was initially attempted at JetBrains with Weston+plugin. A side note:
each test had to run in a new Weston instance because tests often crashed
Weston.

4. Wait for new Wayland protocols (like
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/230)
dedicated to UI testing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/wakefield-dev/attachments/20260227/ed2e514f/attachment-0001.htm>