From maxim.kartashev at jetbrains.com Fri Feb 27 06:37:42 2026 From: maxim.kartashev at jetbrains.com (Maxim Kartashev) Date: Fri, 27 Feb 2026 10:37:42 +0400 Subject: [External] : On UI testing in Wayland Message-ID: (Following up on our discussion at the meeting on Feb 26, 2026) As you know, each window in Wayland has a coordinate system of its own with the top-left corner placed at (0, 0) in that coordinate system. There is no way to programmatically set or obtain the location in the global desktop's coordinate system. This complicates automatic testing to the extreme; many UI tests follow this pattern: Point p = component.getLocationOnScreen(); Color c = robot.getPixelColor(p.x + dx, p.y + dy); if (!c.equals(...))... Since getLocationOnScreen() returns (0, 0) for top-level windows, the coordinates supplied to the Robot call will almost certainly be incorrect in the global coordinate space that Robot uses. There are a few routes that can be taken to at least mitigate this problem. 1. Make Robot operate in window's coordinate space and look into pixels obtained from window' s SurfaceData without involving the desktop compositor. This obviously doesn't work when there's more than one window and does not account for popups. Still, this was sufficient to run more than half of the existing tests. Another problem with this approach is that client applications, which are not tests, need to take proper screenshots of the composited desktop. Therefore, some kind of switch between two Robot modes is required (perhaps a system property?). 2. Enhance Robot API to specify which window's pixels need to be fetched. This will require bulk updates to most tests under java/awt and javax/swing in addition to changes on all currently supported platforms. 3. Run tests only in a more controlled Wayland environment that exposes the global coordinate system to the client applications (like Weston or Gnome KIOSK, perhaps with extra plugins). This was initially attempted at JetBrains with Weston+plugin. A side note: each test had to run in a new Weston instance because tests often crashed Weston. 4. Wait for new Wayland protocols (like https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/230) dedicated to UI testing. -------------- next part -------------- An HTML attachment was scrubbed... URL: