Integrating native Open GL code in JavaFX and alternatives

[Artem Ananiev]

A few comments from my side:

1. If a native Window/Scene handle is provided, it will be enough to 
embed whatever you need. It will even be possible to embed D3D content 
into JavaFX running with OpenGL pipeline :)

The question is what is the native handle? On Windows, HWND is the 
answer. On Mac: NSView or CALayer? In the latter case we'll also need 
mechanism to flush embedded layer content into the parent (window) 
layer, and the thing will get complicated.

2. Another challenge with heavyweight embedding is integration into FX 
scene. Focus traversal, transforms, effects, modality, etc.

3. Alternatively, we can try to implement semi-heavyweight embedding. 
What we need is to make Prism be able to render externally managed 
textures (GL, D3D). Input events will be handled as for any other FX 
nodes, transforms/effects will also be of no problem. What's 
questionable here is a) performance b) sharing GL context or D3D device 
between FX and application code. I'm not a graphics expert, but suspect 
there may be problems in this area.

Thanks,

Artem

On 1/25/2013 6:28 PM, Scott Palmer wrote:
> +1
>
> I see two basic needs:
>
> 1: A JAWT equivalent so we can at least get a handle to the Stage
> window.
>
> 2: Some means of embedding our own rendering surface.  Likely via a
> "texture" described in the native format for the graphics pipeline
> used (OpenGL or Direct3D)
>
> In our Swing product (on Windows) we grab the native window handle of
> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
> - hardware assisted with a colorspace conversion, if I recall
> correctly.  JavaFX offers no equivalent.
>
> I know that JavaFX will likely never allow the concept of embedding a
> heavyweight native window into a Scene.  I don't think that would be
> the right approach for it anyway.  But we can't even make a separate
> preview window natively and give it a proper parent window with the
> current APIs.
>
> Scott
>
> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
> wrote:
>
>> Hi,
>>
>> just to avoid, this gets buried in the other long thread where it
>> originated from (Migrating commercial applications from Swing to JFX),
>> I'm posting this again as a separate thread. I hope this is OK.
>>
>> We are evaluating migrating our Swing Application which includes a
>> video player built using native code for decoding, rendering and
>> effects to JavaFX. Performance requirements are very ambitious (like
>> being close to the playback performance of the best native player, in
>> our case Quicktime for most formats we care about, for high definition
>> video).
>>
>> Currently there is no way to do the same thing in JavaFX, which the
>> JOGL folks have already run into themselves (see
>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>
>> We would either need support for getting a native Open GL context to
>> draw to a JFX component (panel, canvas whatever), so we can use native
>> code to do all sorts of stuff that happens in our video player or we
>> would need support for the following operations in JFX:
>>
>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>> - shaders and frame buffer objects (render to texture) for applying
>> layered effects to video frames
>> - pixel buffer objects for transferring video images to textures (and back)
>> - interoperability with opencl and cuda (processing of video frames
>> with opencl/cuda and rendering with opengl)
>> - reuse of opengl code across platforms (including android and iOS)
>>
>> We achieve that currently by having cross-platform C code that is
>> connected to each platform via a rather thin layer for communication
>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>> controlled by a Java-UI). My guess is that it is more likely that you
>> will be able to expose the native context than providing all the
>> listed things. It is also a maybe a philosophical question whether you
>> want that but IMHO you close the door on an interesting range of
>> applications if you don't provide anything like that in some way
>> because you cannot implement a serious media application of a certain
>> kind without it and why reinvent the wheel? IMHO Java technologies in
>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>> from being good ideas but not making it to a production-quality state
>> because doing all that would have meant reinventing the wheel, which
>> in complex areas like the one described here is simply not feasible so
>> they IMHO remained being mostly of educational value for academic
>> environments. Acknowledging that in some areas it does make sense to
>> have some native code doesn't diminish the value of Java as a
>> platform. Having a lot of portable code in an application is still a
>> lot better than having one application per platform. Quite the
>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>> area if that was supported some more.
>>
>> Sorry for being a bit philosophical in the end ;-).
>>
>> Thanks a lot for your help. I'm really hoping we can come to the
>> conclusion that migrating to JavaFX is the right way to go for us.
>>
>> Robert
>