Experiments with screenshots, graphics using shm

[Jonas Ådahl]

On Thu, Sep 23, 2021 at 03:03:27PM +0200, Zdeněk Žamberský wrote:
> Hi,
> 
> I have done some experiments with dbus APIs:
> 
> portals dbus API:
> - libportal is not available on rhel-8, however dbus API/libs are there
>   (I think it is provided by xdg-desktop-portals* packages)
> - I have made simple program calling dbus API directly [1]
> - Screenshot call is asynchronous, this means you perform call to make
> screenshot (it returns immediately),
>   url to the screenshot file is delivered later by signal (dbus)
> - this means you need to create even loop (to wait for signal)
> - When screenshot is requested, user is presented with dialog asking for
> confirmation (see rhel-8 example [2]).
>   There seem no way to avoid it (even with "interactive" option to false)
> - API works on RHEL-8/gnome-shell(3), F34/gnome-shell(40), but confirmation
> dialog is shown, as I wrote earlier
> - I also tried it on f33/XFCE/X11 screenshot call was successful, but no
> dialog was shown
>   and dbus signal was not delivered (so test app basically hanged waiting
> for signal...)
> - All in all, I don't have really good impression of this API (user
> interaction needed, eventloop needed),
>   in current form it does not fit our needs very well, I would see it as
> last resort
> 
> gnome-shell dbus API:
> - obviously gnome only (there are other desktop specific backends)
> - In the past I have created experimental patch using it as a backend for
> Robot on wayland
> - I have quickly rebased it to current jdk [3]
> - This API is syncronous, screenshot call directly returns path to
> screenshot file
> - It does not require user interaction (does not show any dialogs)
> - API works on RHEL-8/gnome-shell(3), F34/gnome-shell(40)
> - Theoretically could be one of the options to try by openjdk
>   (I have seen this in multiple screenshoting apps as on of their backends)

I would like to point out that this API will not be usable starting with
F35 or anything other that ships GNOME 41, so advice against relying on
it.

What I suggest instead are these two options:

 * Reconsider the screen cast portal using PipeWire option for robot screenshots

I think this somewhat maps relatively correctly to what the API needs;
you want more or less direct on-demand access to the content of one or
more monitors, without user interaction after an initial dialog where
the user granted permission.

This should, in theory, be possible using PipeWire without being an
unnecessary strain on resources, with a few minor modifications to the
currently primary use case (active screen sharing / screen recording):

  1. Open a PipeWire screen share session on-demand whenever anything
  asks for a screenshot. Let it stay alive for the lifetime of the
  application, or deactivate it after a timeout, or some other
  heuristics. Whenever a screenshot is requested, use the existing
  screen cast session.

  2. To avoid making a screen cast consume resources when unused, it is
  possible to "reverse" the responsibility of what entity is responsible
  for triggering new screen cast frames. Thus, in theory, it should be
  possible for Java to create a screen cast stream with a specific
  number of buffers (e.g. 1), then only ask for an actual screen cast
  frame when anything from Java asks for it. The compositor would not
  record frames when not asked to, and Java wouldn't receive any when
  not asked to.

This might need adaptations in the screen cast producer side (e.g.
mutter / gnome-shell), as the screen sharing so far has been targeting
users such as WebRTC and OBS Studio which wants a constantly updating
stream, but I'd be happy to assist with that, including writing an
proof of concept how to do this.

This would allow for high performing screen content copying directly via
the GPU using DMA buffers, or via shared memory buffers. What method
would be used is something that is negotiated using PipeWire.


 * Reach out to the Flatpak portal community about the screenshot portal

The second, less attractive, alternative I see is continuing with the
portal screenshot path, while reaching out the Flatpak portal community
about potential changes or additions can be made to the screenshot
portal, to meet your needs.

Note that what is being asked for is, from a privacy and security point
of view, equivalent to sharing the screen via a screen cast, i.e. a
practically unhindered view of the screen content once the user
initially granted permission, so anything that moves the screenshot
portal in that direction would in practice conflict - feature
wise - with the intended use case of the screencast portal that uses
PipeWire.


Jonas