RFR: 8280982: [Wayland] [XWayland] java.awt.Robot taking screenshots [v7]

Fri May 26 15:36:00 UTC 2023

On Thu, 25 May 2023 16:29:39 GMT, Alexander Zvegintsev <azvegint at openjdk.org> wrote:

>> Modern Linux systems often come with [Wayland](https://wayland.freedesktop.org/) by default.
>> This comes with some difficulties, and one of them is the inability to get screenshots from the system.
>> This is because we now use the [X Window System API](https://en.wikipedia.org/wiki/X_Window_System) to capture screenshots and it cannot access data outside the [XWayland server](https://wayland.freedesktop.org/xserver.html) 
>> 
>> But this functionality is a very important part of automated testing.
>> 
>> 
>> At the moment there are two obvious solutions to this problem, and both use [xdg-desktop-portal](https://github.com/flatpak/xdg-desktop-portal):
>> 
>> 1. [org.freedesktop.portal.Screenshot DBUS API](https://flatpak.github.io/xdg-desktop-portal/#gdbus-org.freedesktop.portal.Screenshot)
>> It has several drawbacks though:
>> + It saves a screenshot to disk, which must be read and deleted(may add some delays depending on the type of a disk drive).
>> + There is no way to disable the visual "screen flash" after screenshot
>> + It asks a user confirmation to save a screenshot. This confirmation can be saved on Gnome 43+. 
>> Since we would like Ubuntu 22.04 LTS which comes with Gnome 42 this option is not acceptable for us because it would require user confirmation for each screenshot.
>> But we still can consider this option as a fallback.
>> 
>> 
>> 
>> 2. [org.freedesktop.portal.ScreenCast](https://flatpak.github.io/xdg-desktop-portal/#gdbus-org.freedesktop.portal.ScreenCast)
>> It typically used by applications that need to capture the contents of the user's screen or a specific window for the purpose of sharing, recording, or streaming.
>> This might be a bit of overkill, but it avoids several of the problems mentioned in the Screenshot API.
>> 
>> + implementation is more complicated comparing to Screenshot API
>> + no intermediate file, screenshot data can be obtained from memory
>> + Permission to make screenshots can be stored with [`restore_token`](https://flatpak.github.io/xdg-desktop-portal/#gdbus-method-org-freedesktop-portal-ScreenCast.SelectSources)
>> 
>> 
>> So this PR adds the ability to take screenshots using the ScreenCast API. This functionality is currently disabled by default.
>> 
>> This change also introduces some new behavior for the robot:
>> A system window now appears asking for confirmation from the user to capture the screen.
>> + The user can refuse the screen capture completely. In this case a security exception will be thrown.
>> + The user can allow...
>
> Alexander Zvegintsev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   rework token storage

Tried running this version on my machine (Ubuntu 22.04, two displays with 100% and 200% scaling). A few observations:
1. I couldn't get any of the screenshot tests working through `jtreg` (screenshots are all black, no permission is ever asked). Does anybody know how to do that properly? Can this be made to work out of the box?

2. Running manually is OK for the most part except for some tests in certain configurations:

* Test `test/jdk/java/awt/Robot/HiDPIScreenCapture/ScreenCaptureGtkTest.java` consistently fails with `sun.java2d.uiScale` other than `1`. For example,

$ java -Dsun.java2d.uiScale=3 ScreenCaptureGtkTest.java
Creating screen capture of java.awt.Rectangle[x=89,y=99,width=100,height=100]
Exception in thread "main" java.lang.SecurityException: Screen Capture in the selected area was not allowed
	at java.desktop/sun.awt.screencast.ScreencastHelper.getRGBPixels(ScreencastHelper.java:161)
	at java.desktop/sun.awt.X11.XRobotPeer.getRGBPixels(XRobotPeer.java:139)
	at java.desktop/java.awt.Robot.createCompatibleImage(Robot.java:606)
	at java.desktop/java.awt.Robot.createScreenCapture(Robot.java:477)
	at ScreenCaptureGtkTest.captureImageOf(ScreenCaptureGtkTest.java:130)
	at ScreenCaptureGtkTest.main(ScreenCaptureGtkTest.java:96)

and this is when I allow both screens to be captured in the system dialog that appears after the start of the test.
The same failure can be observed with `java -Dsun.java2d.uiScale=2 HiDPIRobotScreenCaptureTest.java`. Both tests hang after throwing those exceptions, by the way.

* The same test fails in a different fashion with `-Djdk.gtk.version=2`. For example:

$ java -Djdk.gtk.version=2 -Dsun.java2d.uiScale=1 ScreenCaptureGtkTest.java
WARNING: the GTK 2 library is deprecated and its support will be removed in a future release
Gtk-Message: 19:17:15.122: Failed to load module "canberra-gtk-module"
Creating screen capture of java.awt.Rectangle[x=89,y=99,width=100,height=100]
Could not load native libraries for ScreencastHelper
Checking color at 139, 149 to be equal to java.awt.Color[r=0,g=255,b=0]... Mismatch: found java.awt.Color[r=0,g=0,b=0] instead
Check image.png
Exception in thread "main" java.lang.RuntimeException: Wrong screen pixel color
	at ScreenCaptureGtkTest.checkPixelColors(ScreenCaptureGtkTest.java:115)
	at ScreenCaptureGtkTest.main(ScreenCaptureGtkTest.java:100)

* Test `test/jdk/java/awt/Robot/HiDPIScreenCapture/ScreenCaptureResolutionTest.java` fails and hangs like so:

$ java ScreenCaptureResolutionTest.java
Exception in thread "AWT-EventQueue-0" java.lang.RuntimeException: Exception while validating ScreenCapture
	at ScreenCaptureResolutionTest$2.run(ScreenCaptureResolutionTest.java:75)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13803#issuecomment-1564568586