RFR: 8280982: [Wayland] [XWayland] java.awt.Robot taking screenshots [v9]

Phil Race prr at openjdk.org
Wed May 31 22:35:15 UTC 2023 

On Wed, 31 May 2023 15:03:04 GMT, Alexander Zvegintsev <azvegint at openjdk.org> wrote:

>> Modern Linux systems often come with [Wayland](https://wayland.freedesktop.org/) by default.
>> This comes with some difficulties, and one of them is the inability to get screenshots from the system.
>> This is because we now use the [X Window System API](https://en.wikipedia.org/wiki/X_Window_System) to capture screenshots and it cannot access data outside the [XWayland server](https://wayland.freedesktop.org/xserver.html) 
>> 
>> But this functionality is a very important part of automated testing.
>> 
>> 
>> At the moment there are two obvious solutions to this problem, and both use [xdg-desktop-portal](https://github.com/flatpak/xdg-desktop-portal):
>> 
>> 1. [org.freedesktop.portal.Screenshot DBUS API](https://flatpak.github.io/xdg-desktop-portal/#gdbus-org.freedesktop.portal.Screenshot)
>> It has several drawbacks though:
>> + It saves a screenshot to disk, which must be read and deleted(may add some delays depending on the type of a disk drive).
>> + There is no way to disable the visual "screen flash" after screenshot
>> + It asks a user confirmation to save a screenshot. This confirmation can be saved on Gnome 43+. 
>> Since we would like Ubuntu 22.04 LTS which comes with Gnome 42 this option is not acceptable for us because it would require user confirmation for each screenshot.
>> But we still can consider this option as a fallback.
>> 
>> 
>> 
>> 2. [org.freedesktop.portal.ScreenCast](https://flatpak.github.io/xdg-desktop-portal/#gdbus-org.freedesktop.portal.ScreenCast)
>> It typically used by applications that need to capture the contents of the user's screen or a specific window for the purpose of sharing, recording, or streaming.
>> This might be a bit of overkill, but it avoids several of the problems mentioned in the Screenshot API.
>> 
>> + implementation is more complicated comparing to Screenshot API
>> + no intermediate file, screenshot data can be obtained from memory
>> + Permission to make screenshots can be stored with [`restore_token`](https://flatpak.github.io/xdg-desktop-portal/#gdbus-method-org-freedesktop-portal-ScreenCast.SelectSources)
>> 
>> 
>> So this PR adds the ability to take screenshots using the ScreenCast API. This functionality is currently disabled by default.
>> 
>> This change also introduces some new behavior for the robot:
>> A system window now appears asking for confirmation from the user to capture the screen.
>> + The user can refuse the screen capture completely. In this case a security exception will be thrown.
>> + The user can allow...
>
> Alexander Zvegintsev has updated the pull request incrementally with four additional commits since the last revision:
> 
>  - improve retVal processing
>  - address token storage comments
>  - removing non-ascii
>  - EXCEPTION_CHECK_DESCRIBE_CLEAR -> EXCEPTION_CHECK_DESCRIBE

I spoke too soon :-(
I remembered about an odd result from last week and it seems that whenever I have a full screen window capture fails - at least on my system and I think one lab system.
See test and output below from a fully patched Ubuntu 22.04.02 system

import java.awt.Frame;
import java.awt.GraphicsDevice;
import java.awt.Rectangle;
import java.awt.Robot;

public class FSW {

    public static void main(String args[]) throws Exception {
        Robot robot = new Robot();
        Frame frame = new Frame("FSW");
        GraphicsDevice gd = frame.getGraphicsConfiguration().getDevice();
        try {
            gd.setFullScreenWindow(frame);
            robot.delay(5000);
            robot.createScreenCapture(new Rectangle(50, 50, 10, 10));
        } finally {
            gd.setFullScreenWindow(null);
            frame.dispose();
        }
    }
}

% java FSW
cropTo:163 Unexpected stride / 4: 0 srcW: 1920

and the capture fails every time.


And again with debugging turned on 

ScreencastWatcher: started
// getRGBPixels affectedScreenBounds [java.awt.Rectangle[x=0,y=0,width=1920,height=1200]]
// getTokens exact matches 1. [Token: 41dc61fb-279b-4069-b408-30873451fb03
	java.awt.Rectangle[x=0,y=0,width=1920,height=1200]
]
// getTokens same sizes 2. [Token: 41dc61fb-279b-4069-b408-30873451fb03
	java.awt.Rectangle[x=0,y=0,width=1920,height=1200]
]
// storeToken old: |41dc61fb-279b-4069-b408-30873451fb03| new |41dc61fb-279b-4069-b408-30873451fb03| allowed bounds [0, 0, 1920, 1200]
// Storing TokenItem:
Token: 41dc61fb-279b-4069-b408-30873451fb03
	java.awt.Rectangle[x=0,y=0,width=1920,height=1200]

initXdgDesktopPortal:236 connection/sender name :1.525 / 1_525
checkVersion:190 ScreenCast protocol version 4
Java_sun_awt_screencast_ScreencastHelper_getRGBPixelsImpl:833 taking screenshot at 
	x:    50 y    50 w    10 h    10 with token |41dc61fb-279b-4069-b408-30873451fb03|
initXdgDesktopPortal:236 connection/sender name :1.525 / 1_525
checkVersion:190 ScreenCast protocol version 4
callbackScreenCastStart:611 available screen count 1
rebuildScreenData:87 
==== screenId#35
rebuildScreenData:132 -----------------------
rebuildScreenData:133 screenId#35
||	bounds         x     0 y     0 w  1920 h  1200
||	capture area   x     0 y     0 w     0 h     0 shouldCapture 0

rebuildScreenData:134 #---------------------#

callbackScreenCastStart:617 rebuildScreenData result |0|
callbackScreenCastStart:630 restore_token |41dc61fb-279b-4069-b408-30873451fb03|
storeRestoreToken:653 saving token, old: |41dc61fb-279b-4069-b408-30873451fb03| > new: |41dc61fb-279b-4069-b408-30873451fb03|
portalScreenCastStart:709 ScreenCastStartResult |0|
getPipewireFd:886 portalScreenCastStart result |0|
checkCanCaptureAllRequiredScreens:847 Found allowed screen bounds in affected screen bounds 0 0 1920 1200
getPipewireFd:900 --- portalScreenCastStart
getPipewireFd:907 pwFd 17
doLoop:557 screenId#35[loc(0,0) size(1920x1200)] @@@ adding screen 0
checkScreen:472 screenId#35
||	bounds         x     0 y     0 w  1920 h  1200
||	capture area   x    50 y    50 w    10 h    10 shouldCapture 1

connectStream:373 @@@ using screen 0
connectStream:411 screenId#35
||	bounds         x     0 y     0 w  1920 h  1200
||	capture area   x    50 y    50 w    10 h    10 shouldCapture 1

startStream:355 screenId#35: stream connecting 0x7f04548e3600
onStreamStateChanged:303 screenId#35[loc(0,0) size(1920x1200)] state 0 (unconnected) -> 1 (connecting) err (null)
onStreamStateChanged:303 screenId#35[loc(0,0) size(1920x1200)] state 1 (connecting) -> 2 (paused) err (null)
onStreamParamChanged:197 screenId#35[loc(0,0) size(1920x1200)] param event id 4
onStreamParamChanged:218 screenId#35[loc(0,0) size(1920x1200)] stream format: Spa:Enum:VideoFormat:BGRx (8)	1920x1200
connectStream:424 screenId#35[loc(0,0) size(1920x1200)] frame size: 1920x1200
doLoop:563 screenId#35[loc(0,0) size(1920x1200)] @@@ screen processed 0
onStreamStateChanged:303 screenId#35[loc(0,0) size(1920x1200)] state 2 (paused) -> 3 (streaming) err (null)
onStreamProcess:236 screenId#35[loc(0,0) size(1920x1200)] hasFormat 1 captureDataReady 0 shouldCapture 1
onStreamProcess:271 screenId#35
||	bounds         x     0 y     0 w  1920 h  1200
||	capture area   x    50 y    50 w    10 h    10 shouldCapture 1

onStreamProcess:272 screenId#35[loc(0,0) size(1920x1200)] got a frame of size 0 offset 0 stride 0 flags 0 FD 31 captureDataReady 0
onStreamProcess:292 screenId#35[loc(0,0) size(1920x1200)] data ready
Java_sun_awt_screencast_ScreencastHelper_getRGBPixelsImpl:856 
all data ready
Java_sun_awt_screencast_ScreencastHelper_getRGBPixelsImpl:864 screenId#35[loc(0,0) size(1920x1200)] @@@ copying screen data 0, captureData 0x7f03d40042a0
	||	x    50 y    50 w    10 h    10 requested area
	||	x     0 y     0 w  1920 h  1200 screen bound
	||	x    50 y    50 w    10 h    10 in-screen coords capture area

onStreamParamChanged:197 screenId#35[loc(0,0) size(1920x1200)] param event id 4
onStreamStateChanged:303 screenId#35[loc(0,0) size(1920x1200)] state 3 (streaming) -> 2 (paused) err (null)
onStreamStateChanged:303 screenId#35[loc(0,0) size(1920x1200)] state 2 (paused) -> 0 (unconnected) err (null)
doCleanup:109 STOPPING loop

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13803#issuecomment-1571048110

More information about the build-dev mailing list