Integrating native Open GL code in JavaFX and alternatives

[Kevin Rushforth]

It would interfere if we tried to allow arbitrary shaders that could be 
set on arbitrary objects. If we took the approach of enabling (somehow) 
an OpenGL surface in a manner similar to the Canvas node that users 
could scribble into, it might be solvable by using a separate GL context 
(or equivalently, saving / restoring all state).

There are other barriers to doing this, though. In addition to the 
effort / security concerns / etc of exposing native handles, there is 
also the issue that we use D3D on Windows platforms so we would need to 
do something like what WebGL does -- a GL to D3D bridge.

In any case, this is not feasible in the short-medium term (i.e., not 
for FX 8). One possible angle would be to use JOGL's GLJPanel and the 
new-in-FX8 Swing node.

-- Kevin


Richard Bair wrote:
> One question I have on this is related to security. I'm more than a little worried about exposing native handles in any way as that means a reasonable likelihood that a bug in our code could lead to an exploit. What are the safety mechanisms for something like this?
>
> The other issue is one of shaders. I would love to be able to give you an OpenGL surface that you can scribble into and have it be a normal part of the scene graph. I think that's a requirement (I would hate to end up in a place where it has to be a separate window). However one thing I don't know about is whether any shaders you develop would interfere with the shaders we use for normal rasterization etc.
>
> Kevin, Jim?
>
> On Jan 25, 2013, at 6:28 AM, Scott Palmer <swpalmer at gmail.com> wrote:
>
>   
>> +1
>>
>> I see two basic needs:
>>
>> 1: A JAWT equivalent so we can at least get a handle to the Stage window.
>>
>> 2: Some means of embedding our own rendering surface.  Likely via a "texture" described in the native format for the graphics pipeline used (OpenGL or Direct3D)
>>
>> In our Swing product (on Windows) we grab the native window handle of a heavyweight 'placholder' via JAWT and blit a bitmap directly to it - hardware assisted with a colorspace conversion, if I recall correctly.  JavaFX offers no equivalent.
>>
>> I know that JavaFX will likely never allow the concept of embedding a heavyweight native window into a Scene.  I don't think that would be the right approach for it anyway.  But we can't even make a separate preview window natively and give it a proper parent window with the current APIs.
>>
>> Scott
>>
>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de> wrote:
>>
>>     
>>> Hi,
>>>
>>> just to avoid, this gets buried in the other long thread where it
>>> originated from (Migrating commercial applications from Swing to JFX),
>>> I'm posting this again as a separate thread. I hope this is OK.
>>>
>>> We are evaluating migrating our Swing Application which includes a
>>> video player built using native code for decoding, rendering and
>>> effects to JavaFX. Performance requirements are very ambitious (like
>>> being close to the playback performance of the best native player, in
>>> our case Quicktime for most formats we care about, for high definition
>>> video).
>>>
>>> Currently there is no way to do the same thing in JavaFX, which the
>>> JOGL folks have already run into themselves (see
>>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>>
>>> We would either need support for getting a native Open GL context to
>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>> code to do all sorts of stuff that happens in our video player or we
>>> would need support for the following operations in JFX:
>>>
>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>> - shaders and frame buffer objects (render to texture) for applying
>>> layered effects to video frames
>>> - pixel buffer objects for transferring video images to textures (and back)
>>> - interoperability with opencl and cuda (processing of video frames
>>> with opencl/cuda and rendering with opengl)
>>> - reuse of opengl code across platforms (including android and iOS)
>>>
>>> We achieve that currently by having cross-platform C code that is
>>> connected to each platform via a rather thin layer for communication
>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>> controlled by a Java-UI). My guess is that it is more likely that you
>>> will be able to expose the native context than providing all the
>>> listed things. It is also a maybe a philosophical question whether you
>>> want that but IMHO you close the door on an interesting range of
>>> applications if you don't provide anything like that in some way
>>> because you cannot implement a serious media application of a certain
>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>> from being good ideas but not making it to a production-quality state
>>> because doing all that would have meant reinventing the wheel, which
>>> in complex areas like the one described here is simply not feasible so
>>> they IMHO remained being mostly of educational value for academic
>>> environments. Acknowledging that in some areas it does make sense to
>>> have some native code doesn't diminish the value of Java as a
>>> platform. Having a lot of portable code in an application is still a
>>> lot better than having one application per platform. Quite the
>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>> area if that was supported some more.
>>>
>>> Sorry for being a bit philosophical in the end ;-).
>>>
>>> Thanks a lot for your help. I'm really hoping we can come to the
>>> conclusion that migrating to JavaFX is the right way to go for us.
>>>
>>> Robert
>>>       
>
>