Layering JavaFX onto an external rendering context

[Mark Raynsford]

On 2021-02-15T13:40:43 +0000
Mark Raynsford <org.openjdk at io7m.com> wrote:

> Hello!
> 
> I'd like to use JavaFX for the UI of an application that will
> involve rendering using an existing Vulkan-based renderer. For the sake
> of example, assume that the application looks and behaves a bit like
> the Unreal Engine 4 editing tools....

Apologies for the delay in responding (I've read and digested every
reply).

So... Having read everything, I'm reasonably sure that the ability to
do this in the "traditional" way - that is, having a node in the
JavaFX scene graph that displays the contents of a GPU-side
image/texture - isn't going to be happening any time soon. There are
multiple intersecting issues that make this intractably difficult to
achieve:

  1. There are software boundaries; JavaFX doesn't expose the internal
     rendering context it uses and, even if it did, there is a
     combinatorial explosion of possibilities with regards to which
     rendering API the _user_ might be using, and which rendering API
     JavaFX might be using (if any!). Traditional stateful APIs like
     OpenGL make it far too easy for external code to screw up the
     rendering context anyway, so exposing JavaFX's own context would
     be a bad idea.

  2. There are hardware and concurrency issues; Let's assume that
     JavaFX is using OpenGL or DirectX internally. Let's also assume
     that the user is using some kind of API that explicitly provides
     for managing concurrency with primitives such as fences,
     semaphores, etc. The user would have to somehow magically arrange
     for an image transfer on the GPU into an image that JavaFX knew
     about, and would have to somehow set up all of the right memory
     barriers and notify JavaFX when the transfer was completed. This
     is likely to be intensely error-prone at the very least. The
     thought of implementing this directly brings me out in a cold
     sweat.

So how about another approach entirely?

In the past, I've rendered UI elements using entirely software rendering
and then uploaded the software-rendered UI as a texture to be overlaid
on top of the 3D rendered view. I could obviously continue to do this,
but maintaining entire software-rendered UI toolkits myself is a burden,
and it would be great if I could use an existing known-good toolkit
instead like JavaFX.

JavaFX _does_ have a software renderer. What if we could have JavaFX
work with entirely software rendering into an offscreen BufferedImage?

To make this work, we'd need the following:

  1. We'd need to be able to initialize JavaFX in a manner that
     _didn't_ require JavaFX to "own" the primary stage. That is,
     right now JavaFX will invoke a user's Application subclass
     with a Stage that JavaFX created by itself. This won't work
     for most rendering engines, as typically the programmer has
     to do a fair bit of work to open a graphics context with
     the correct settings. Consider the typical code needed to
     open a window over GLFW with Vulkan...
     
        https://github.com/io7m/jcoronado/blob/develop/com.io7m.jcoronado.examples/src/main/java/com/io7m/jcoronado/examples/HelloVulkan.java

  2. We'd need to be able to force JavaFX to operate entirely
     using Java2D software rendering. This shouldn't be a problem
     at all.

  3. We'd need to be able to make JavaFX render to an offscreen
     BufferedImage. It would be the responsibility of the programmer
     to submit keyboard and mouse events to JavaFX manually, because
     JavaFX would no longer "own" the primary application window.
     I believe the first half of this is already doable in the sense
     that the Node class allows for taking snapshots of a scene; these
     snapshots could be uploaded to the GPU as textures. The second
     half I believe is not exposed in any sense; JavaFX opens its own
     windows, and consumes input events from those windows directly.

I wonder if it would be possible to, for example, add some kind of
WindowFactory API to JavaFX so that, when JavaFX wants to create a
Window, there's the option to delegate window creation to the
user. The created windows might not even be operating system windows
(for window-in-window UI systems)[0]. As long as the returned Window
instance acted like a Window in terms of returning dimensions,
producing the right input events... Would JavaFX even know that it
wasn't dealing with an operating system window?

[0] https://softcamel.com/wp-content/uploads/2019/03/Transport-Tycoon-Deluxe-2.png 

-- 
Mark Raynsford | https://www.io7m.com