Layering JavaFX onto an external rendering context

[Tom Schindl]

Hi,

If I got you right you now changed your idea to render JavaFX into an 
PixelBuffer and integrate that into a scene renderer with Vulkan, ... .

This exactly how the integration into SWT and Swing is done, the 
internal API for that is found in "com.sun.javafx.embed". Before we 
implemented DriftFX we integrated a JavaFX-Scene into our application 
exactly this way.

For that purpose we hacked FXCanvas and published the changed code at 
our customers github account [1]. You see there at [2] how one fills the 
buffer from an offscreen scene.

My gut feeling is that getting a "com.sun.javafx.embed"-API is much more 
likely than an DriftFX-Like-API but that just my personal opinion.

Anyways our/my focus still is DriftFX and I/we are quite confident that 
what we have today in DriftFX is a good API where we today support 
JavaFX-8 to JavaFX-15 (I haven't tested 16 but I have not seen any 
changes going into it that would require us to change our code)

Tom

[1] 
https://github.com/eclipsesource/FXCanvas/blob/master/FXCanvas/src/javafx/embed/swt/FXCanvas.java
[2] 
https://github.com/eclipsesource/FXCanvas/blob/master/FXCanvas/src/javafx/embed/swt/FXCanvas.java#L631

Am 07.03.21 um 16:35 schrieb Mark Raynsford:
> On 2021-03-07T12:22:11 +0000
> Neil C Smith <neil at codelerity.com> wrote:
> 
>> Hi,
>>
>> A few comments from my perspective here, mainly coming from
>> frustration at the number of times I've had to tell clients or users
>> that JavaFX isn't a viable option for them ..
>>
>> On Sat, 6 Mar 2021 at 12:22, Mark Raynsford <org.openjdk at io7m.com> wrote:
>>> there is a
>>>       combinatorial explosion of possibilities with regards to which
>>>       rendering API the _user_ might be using, and which rendering API
>>>       JavaFX might be using (if any!).
>>
>> I agree that's a difficult one, but also surely not an insurmountable
>> one?!  There are precedents around Java APIs that provide querying and
>> access to optional support on a platform by platform basis...
> 
> I think it's insurmountable due to the complexity of the semantics of
> the thing (the GPU) that's being abstracted over. The basic primitive
> that would be required is "transfer this image to this other image".
> You'd need to expose that operation in way that would work for every
> possible pair of rendering APIs, including getting all of the
> synchronization right that lower level APIs like Vulkan and DX12
> require. The complexity of handling that would need to be pushed onto
> each person trying to use the API. JavaFX would effectively need to
> know when it was safe to use an image that was being transferred in,
> without actually being able to know where the image was coming from or
> who to talk to about it. The only methods it _could_ use would involve
> CPU-side fences, which would then have performance implications.
> 
> To clarify a bit: Let's say we lived in an ideal world and JavaFX had a
> Vulkan backend, and contained code that was willing to cooperate with
> the user's Vulkan-based rendering code. The user could tell JavaFX
> "here's the image to use for this node, but don't use it until this
> semaphore becomes available". This is generally highly efficient as a
> semaphore (in Vulkan terms) is a GPU-side primitive that just controls
> parallelism across different queue instances on the GPU, with no
> CPU-side interaction at all.
> 
> We have no way to get to that ideal world whilst JavaFX and the user's
> code have no common API over which to communicate. Everything would
> have to go via CPU-based abstractions.
> 
>>> Traditional stateful APIs like
>>>       OpenGL make it far too easy for external code to screw up the
>>>       rendering context anyway, so exposing JavaFX's own context would
>>>       be a bad idea.
>>
>> I remain of the view that this shouldn't really be that concerning?!
>> It's a bug in the library if it does so.
> 
> It's definitely concerning. Have you worked with OpenGL much? If you're
> *lucky* you'll get a crash, because at least then you'll see the
> exact location where the mistake occured. What you'll typically get
> instead is corrupted rendering, a black screen, or just bad performance
> as the driver desperately tries to make sense of what you asked it to
> do, usually in completely unrelated parts of the rendering process. It's
> the prime reason I jumped ship to Vulkan the second it came out. :)
> 
>>> JavaFX _does_ have a software renderer. What if we could have JavaFX
>>> work with entirely software rendering into an offscreen BufferedImage?
>>
>> Why BufferedImage?  What about a reuse / extension / parallel of
>> PixelBuffer?
> 
> Yes indeed. I said BufferedImage just because I assumed that would be
> more familiar. I'd not personally care where the output went as long as
> I could get the raw bytes out of it in order to upload to the GPU.
>