Integrating native Open GL code in JavaFX and alternatives

[Artem Ananiev]

On 1/26/2013 12:04 AM, Scott Palmer wrote:
>
> On 2013-01-25, at 2:16 PM, Artem Ananiev <artem.ananiev at oracle.com>
> wrote:
>
>> A few comments from my side:
>>
>> 1. If a native Window/Scene handle is provided, it will be enough
>> to embed whatever you need. It will even be possible to embed D3D
>> content into JavaFX running with OpenGL pipeline :)
>
> There will be issues in JavaFX that we can't work around as easily as
> we code in Swing. For example clipping.  How can my video preview
> overlay participate in a Scene such that I can put it inside a
> ScrollPane or SplitPane?  With Swing I could manage this by placing a
> heavyweight component in the UI.  Rectangular clipping was easy to
> deal with by wrapping in another heavyweight component - the OS
> windowing system took care of it.  Later Swing got smarter about
> mixing heavy/light components.  JavaFX doesn't have something
> comparable.

To use heavyweight component in a scroll/split pane, we need heavyweight 
scroll/split pane :) In other words, such scenarios don't work well in 
Swing too.

Do you mean limited heavyweight/lightweight component mixing, introduced 
to 6u10 and the forward-ported to 7? Note the word "limited" here: the 
mixing is really limited, it doesn't cover non-rectangular shapes, 
transparent components, and many more. It actually works fine for 
simplest scenarios only. The code we put to AWT to support the limited 
mixing is not something I would like to see in JavaFX.

>> The question is what is the native handle? On Windows, HWND is the
>> answer. On Mac: NSView or CALayer?
>
> CALayer seems to be the way things are going. NSView was the Java 6
> native layer, now in Java 7 it is CALayer.  But this is native code…
> let there be a way to get either from a platform specific structure.

Believe it or not, we will support both CALayer-based and NSView-based 
embedded windows in JDK8 (and probably next JDK7 updates). Check the 
following JDK bug for details:

https://jbs.oracle.com/bugs/browse/JDK-7154778

Classes are not public (java.*, javax.*) APIs, though, but are supposed 
to be backwards compatible. If we implement this functionality in 
JavaFX, we'll likely support both CALayers and NSViews.

>> In the latter case we'll also need mechanism to flush embedded
>> layer content into the parent (window) layer, and the thing will
>> get complicated.
>>
>> 2. Another challenge with heavyweight embedding is integration into
>> FX scene. Focus traversal, transforms, effects, modality, etc.
>
> Can be depending on the use case.  For a video preview panel, I
> wouldn't' need most of that.  #3 would be best.
>
>> 3. Alternatively, we can try to implement semi-heavyweight
>> embedding. What we need is to make Prism be able to render
>> externally managed textures (GL, D3D). Input events will be handled
>> as for any other FX nodes, transforms/effects will also be of no
>> problem. What's questionable here is a) performance b) sharing GL
>> context or D3D device between FX and application code. I'm not a
>> graphics expert, but suspect there may be problems in this area.
>
> I agree.  But I think it is worth getting a graphics expert (I'm not
> a OpenGL/D3D person either) to think about it over the weekend :-)
>
> Cheers,
>
> Scott

Thanks,

Artem

>> Thanks,
>>
>> Artem
>>
>> On 1/25/2013 6:28 PM, Scott Palmer wrote:
>>> +1
>>>
>>> I see two basic needs:
>>>
>>> 1: A JAWT equivalent so we can at least get a handle to the Stage
>>> window.
>>>
>>> 2: Some means of embedding our own rendering surface.  Likely via a
>>> "texture" described in the native format for the graphics pipeline
>>> used (OpenGL or Direct3D)
>>>
>>> In our Swing product (on Windows) we grab the native window handle of
>>> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
>>> - hardware assisted with a colorspace conversion, if I recall
>>> correctly.  JavaFX offers no equivalent.
>>>
>>> I know that JavaFX will likely never allow the concept of embedding a
>>> heavyweight native window into a Scene.  I don't think that would be
>>> the right approach for it anyway.  But we can't even make a separate
>>> preview window natively and give it a proper parent window with the
>>> current APIs.
>>>
>>> Scott
>>>
>>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> just to avoid, this gets buried in the other long thread where it
>>>> originated from (Migrating commercial applications from Swing to JFX),
>>>> I'm posting this again as a separate thread. I hope this is OK.
>>>>
>>>> We are evaluating migrating our Swing Application which includes a
>>>> video player built using native code for decoding, rendering and
>>>> effects to JavaFX. Performance requirements are very ambitious (like
>>>> being close to the playback performance of the best native player, in
>>>> our case Quicktime for most formats we care about, for high definition
>>>> video).
>>>>
>>>> Currently there is no way to do the same thing in JavaFX, which the
>>>> JOGL folks have already run into themselves (see
>>>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>>>
>>>> We would either need support for getting a native Open GL context to
>>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>>> code to do all sorts of stuff that happens in our video player or we
>>>> would need support for the following operations in JFX:
>>>>
>>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>>> - shaders and frame buffer objects (render to texture) for applying
>>>> layered effects to video frames
>>>> - pixel buffer objects for transferring video images to textures (and back)
>>>> - interoperability with opencl and cuda (processing of video frames
>>>> with opencl/cuda and rendering with opengl)
>>>> - reuse of opengl code across platforms (including android and iOS)
>>>>
>>>> We achieve that currently by having cross-platform C code that is
>>>> connected to each platform via a rather thin layer for communication
>>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>>> controlled by a Java-UI). My guess is that it is more likely that you
>>>> will be able to expose the native context than providing all the
>>>> listed things. It is also a maybe a philosophical question whether you
>>>> want that but IMHO you close the door on an interesting range of
>>>> applications if you don't provide anything like that in some way
>>>> because you cannot implement a serious media application of a certain
>>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>>> from being good ideas but not making it to a production-quality state
>>>> because doing all that would have meant reinventing the wheel, which
>>>> in complex areas like the one described here is simply not feasible so
>>>> they IMHO remained being mostly of educational value for academic
>>>> environments. Acknowledging that in some areas it does make sense to
>>>> have some native code doesn't diminish the value of Java as a
>>>> platform. Having a lot of portable code in an application is still a
>>>> lot better than having one application per platform. Quite the
>>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>>> area if that was supported some more.
>>>>
>>>> Sorry for being a bit philosophical in the end ;-).
>>>>
>>>> Thanks a lot for your help. I'm really hoping we can come to the
>>>> conclusion that migrating to JavaFX is the right way to go for us.
>>>>
>>>> Robert
>>>
>