Robot's API createScreenCapture() question

[Mario Torre]

On Wed, Sep 22, 2021 at 11:42 AM Olivier Fourdan <ofourdan at redhat.com> wrote:
>
> Hi all,
>
> In a Wayland desktop, Xwayland cannot capture the content of the Wayland
> native windows (as those are handled by the Wayland compositor and not
> Xwayland), but X11's GetImage works on X11 toplevel windows in Xwayland,
> except for the root window which in Xwayland has no storage (when Xwayland
> is run rootless).
>
> IOW:
>
> $ xterm &
> $ xwd -id $WINDOWID | xwud
>
> Works fine in Xwayland, but:
>
> $ xwd -root | xwud
>
> Fails with a BadMatch for the reason mentioned above.
>
> Now I was wondering if we could make that work in Xwayland, but then I had
> some doubts as to how it currently works with Robot's API
> createScreenCapture().
>
> Typically, in a composited environment on Xorg, capturing the content of
> the root window will not fetch the content of all the other windows, just
> the bare content of the root window itself.
>
> xwd works around that by fetching the image of each toplevel window visible
> on screen and rebuilding a global image by copying those, basically, trying
> to replicate what the X11 compositor does.
>
> How does Robot's API createScreenCapture() deal with such composited setups
> even on plain Xorg?

Hi Olivier,

If you download the OpenJDK wakefiled repository fork, you can find it under:

wakefield/src/java.desktop/unix/native/libawt_xawt/awt/awt_Robot.c

The Robot.createScreenCapture calls on peer.getRGBPixels which then
for the X11 implementation goes to
Java_sun_awt_X11_XRobotPeer_getRGBPixelsImpl.

>From there there are two possible paths, if the underlying toolkit
uses GTK we use the GTK api for screenshots by calling
get_drawable_data, otherwise we try to determine if it's a composited
desktop by checking for the XComposite extensions and then grabbing
the actual owner via the NET_WM_CM_Sn hint and a call to
XGetSelectionOwner.

The call to getWindowImage uses code derived from xwd and retrieves
the ReadAreaToImage function, so it works identically from this point
on.

> Corollary question, what actually fails when trying to use the
> createScreenCapture() API in Xwayland?

I need to check this, I think we get a black pixel or an empty pixel
array? I'll get back to you with this if no one else replies before.

> I believe if we could make that work in Xwayland it would probably help the
> Robot API in Xwayland (even though it would be limited to X11 windows in
> Xwayland, it would work at least for Java own windows).

Yes, that would be very useful!

Cheers,
Mario


-- 
Mario Torre
Manager, Software Engineering, core OpenJDK
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898