Integrating native Open GL code in JavaFX and alternatives

[Scott Palmer]

On 2013-01-25, at 2:16 PM, Artem Ananiev <artem.ananiev at oracle.com> wrote:

> 
> A few comments from my side:
> 
> 1. If a native Window/Scene handle is provided, it will be enough to embed whatever you need. It will even be possible to embed D3D content into JavaFX running with OpenGL pipeline :)

There will be issues in JavaFX that we can't work around as easily as we code in Swing.
For example clipping.  How can my video preview overlay participate in a Scene such that I can put it inside a ScrollPane or SplitPane?  With Swing I could manage this by placing a heavyweight component in the UI.  Rectangular clipping was easy to deal with by wrapping in another heavyweight component - the OS windowing system took care of it.  Later Swing got smarter about mixing heavy/light components.  JavaFX doesn't have something comparable.

> The question is what is the native handle? On Windows, HWND is the answer. On Mac: NSView or CALayer?

CALayer seems to be the way things are going. NSView was the Java 6 native layer, now in Java 7 it is CALayer.  But this is native code… let there be a way to get either from a platform specific structure.

> In the latter case we'll also need mechanism to flush embedded layer content into the parent (window) layer, and the thing will get complicated.
> 
> 2. Another challenge with heavyweight embedding is integration into FX scene. Focus traversal, transforms, effects, modality, etc.

Can be depending on the use case.  For a video preview panel, I wouldn't' need most of that.  #3 would be best.

> 3. Alternatively, we can try to implement semi-heavyweight embedding. What we need is to make Prism be able to render externally managed textures (GL, D3D). Input events will be handled as for any other FX nodes, transforms/effects will also be of no problem. What's questionable here is a) performance b) sharing GL context or D3D device between FX and application code. I'm not a graphics expert, but suspect there may be problems in this area.

I agree.  But I think it is worth getting a graphics expert (I'm not a OpenGL/D3D person either) to think about it over the weekend :-)

Cheers,


Scott

> 
> Thanks,
> 
> Artem
> 
> On 1/25/2013 6:28 PM, Scott Palmer wrote:
>> +1
>> 
>> I see two basic needs:
>> 
>> 1: A JAWT equivalent so we can at least get a handle to the Stage
>> window.
>> 
>> 2: Some means of embedding our own rendering surface.  Likely via a
>> "texture" described in the native format for the graphics pipeline
>> used (OpenGL or Direct3D)
>> 
>> In our Swing product (on Windows) we grab the native window handle of
>> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
>> - hardware assisted with a colorspace conversion, if I recall
>> correctly.  JavaFX offers no equivalent.
>> 
>> I know that JavaFX will likely never allow the concept of embedding a
>> heavyweight native window into a Scene.  I don't think that would be
>> the right approach for it anyway.  But we can't even make a separate
>> preview window natively and give it a proper parent window with the
>> current APIs.
>> 
>> Scott
>> 
>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
>> wrote:
>> 
>>> Hi,
>>> 
>>> just to avoid, this gets buried in the other long thread where it
>>> originated from (Migrating commercial applications from Swing to JFX),
>>> I'm posting this again as a separate thread. I hope this is OK.
>>> 
>>> We are evaluating migrating our Swing Application which includes a
>>> video player built using native code for decoding, rendering and
>>> effects to JavaFX. Performance requirements are very ambitious (like
>>> being close to the playback performance of the best native player, in
>>> our case Quicktime for most formats we care about, for high definition
>>> video).
>>> 
>>> Currently there is no way to do the same thing in JavaFX, which the
>>> JOGL folks have already run into themselves (see
>>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>> 
>>> We would either need support for getting a native Open GL context to
>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>> code to do all sorts of stuff that happens in our video player or we
>>> would need support for the following operations in JFX:
>>> 
>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>> - shaders and frame buffer objects (render to texture) for applying
>>> layered effects to video frames
>>> - pixel buffer objects for transferring video images to textures (and back)
>>> - interoperability with opencl and cuda (processing of video frames
>>> with opencl/cuda and rendering with opengl)
>>> - reuse of opengl code across platforms (including android and iOS)
>>> 
>>> We achieve that currently by having cross-platform C code that is
>>> connected to each platform via a rather thin layer for communication
>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>> controlled by a Java-UI). My guess is that it is more likely that you
>>> will be able to expose the native context than providing all the
>>> listed things. It is also a maybe a philosophical question whether you
>>> want that but IMHO you close the door on an interesting range of
>>> applications if you don't provide anything like that in some way
>>> because you cannot implement a serious media application of a certain
>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>> from being good ideas but not making it to a production-quality state
>>> because doing all that would have meant reinventing the wheel, which
>>> in complex areas like the one described here is simply not feasible so
>>> they IMHO remained being mostly of educational value for academic
>>> environments. Acknowledging that in some areas it does make sense to
>>> have some native code doesn't diminish the value of Java as a
>>> platform. Having a lot of portable code in an application is still a
>>> lot better than having one application per platform. Quite the
>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>> area if that was supported some more.
>>> 
>>> Sorry for being a bit philosophical in the end ;-).
>>> 
>>> Thanks a lot for your help. I'm really hoping we can come to the
>>> conclusion that migrating to JavaFX is the right way to go for us.
>>> 
>>> Robert
>>