Integrating native Open GL code in JavaFX and alternatives

[Scott Palmer]

On 2013-01-25, at 12:32 PM, Kevin Rushforth <kevin.rushforth at oracle.com> wrote:

> It would interfere if we tried to allow arbitrary shaders that could be set on arbitrary objects. If we took the approach of enabling (somehow) an OpenGL surface in a manner similar to the Canvas node that users could scribble into, it might be solvable by using a separate GL context (or equivalently, saving / restoring all state).
> 
> There are other barriers to doing this, though. In addition to the effort / security concerns / etc of exposing native handles, there is also the issue that we use D3D on Windows platforms so we would need to do something like what WebGL does -- a GL to D3D bridge.

Would it be unacceptable to defer the kind of surface to a platform-specific type?  On Windows you get a D3D surface and on Mac/Linux you get an OpenGL surface?
I'm thinking that JNI is involved to render to this surface anyway, so we are already in the land of platform-specific code.
My point being that avoiding the need for a bridge would get us where we want to go sooner.

Also in terms of security: The whole idea is that we are painting into this with native code.  If we are allowed to run native code, the door is open.  Perhaps it is solved if you don't let us get the handle from the Java side - just like JAWT.

Btw, is there such a thing as an OpenGL pipeline for JavaFX on Windows, or is it fixed and always D3D?


> In any case, this is not feasible in the short-medium term (i.e., not for FX 8). One possible angle would be to use JOGL's GLJPanel and the new-in-FX8 Swing node.

Is GLJPanel a lightweight component?

Regards,

Scott


> 
> -- Kevin
> 
> 
> Richard Bair wrote:
>> 
>> One question I have on this is related to security. I'm more than a little worried about exposing native handles in any way as that means a reasonable likelihood that a bug in our code could lead to an exploit. What are the safety mechanisms for something like this?
>> 
>> The other issue is one of shaders. I would love to be able to give you an OpenGL surface that you can scribble into and have it be a normal part of the scene graph. I think that's a requirement (I would hate to end up in a place where it has to be a separate window). However one thing I don't know about is whether any shaders you develop would interfere with the shaders we use for normal rasterization etc.
>> 
>> Kevin, Jim?
>> 
>> On Jan 25, 2013, at 6:28 AM, Scott Palmer <swpalmer at gmail.com> wrote:
>> 
>>   
>>> +1
>>> 
>>> I see two basic needs:
>>> 
>>> 1: A JAWT equivalent so we can at least get a handle to the Stage window.
>>> 
>>> 2: Some means of embedding our own rendering surface.  Likely via a "texture" described in the native format for the graphics pipeline used (OpenGL or Direct3D)
>>> 
>>> In our Swing product (on Windows) we grab the native window handle of a heavyweight 'placholder' via JAWT and blit a bitmap directly to it - hardware assisted with a colorspace conversion, if I recall correctly.  JavaFX offers no equivalent.
>>> 
>>> I know that JavaFX will likely never allow the concept of embedding a heavyweight native window into a Scene.  I don't think that would be the right approach for it anyway.  But we can't even make a separate preview window natively and give it a proper parent window with the current APIs.
>>> 
>>> Scott
>>> 
>>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de> wrote:
>>> 
>>>     
>>>> Hi,
>>>> 
>>>> just to avoid, this gets buried in the other long thread where it
>>>> originated from (Migrating commercial applications from Swing to JFX),
>>>> I'm posting this again as a separate thread. I hope this is OK.
>>>> 
>>>> We are evaluating migrating our Swing Application which includes a
>>>> video player built using native code for decoding, rendering and
>>>> effects to JavaFX. Performance requirements are very ambitious (like
>>>> being close to the playback performance of the best native player, in
>>>> our case Quicktime for most formats we care about, for high definition
>>>> video).
>>>> 
>>>> Currently there is no way to do the same thing in JavaFX, which the
>>>> JOGL folks have already run into themselves (see
>>>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>>> 
>>>> We would either need support for getting a native Open GL context to
>>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>>> code to do all sorts of stuff that happens in our video player or we
>>>> would need support for the following operations in JFX:
>>>> 
>>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>>> - shaders and frame buffer objects (render to texture) for applying
>>>> layered effects to video frames
>>>> - pixel buffer objects for transferring video images to textures (and back)
>>>> - interoperability with opencl and cuda (processing of video frames
>>>> with opencl/cuda and rendering with opengl)
>>>> - reuse of opengl code across platforms (including android and iOS)
>>>> 
>>>> We achieve that currently by having cross-platform C code that is
>>>> connected to each platform via a rather thin layer for communication
>>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>>> controlled by a Java-UI). My guess is that it is more likely that you
>>>> will be able to expose the native context than providing all the
>>>> listed things. It is also a maybe a philosophical question whether you
>>>> want that but IMHO you close the door on an interesting range of
>>>> applications if you don't provide anything like that in some way
>>>> because you cannot implement a serious media application of a certain
>>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>>> from being good ideas but not making it to a production-quality state
>>>> because doing all that would have meant reinventing the wheel, which
>>>> in complex areas like the one described here is simply not feasible so
>>>> they IMHO remained being mostly of educational value for academic
>>>> environments. Acknowledging that in some areas it does make sense to
>>>> have some native code doesn't diminish the value of Java as a
>>>> platform. Having a lot of portable code in an application is still a
>>>> lot better than having one application per platform. Quite the
>>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>>> area if that was supported some more.
>>>> 
>>>> Sorry for being a bit philosophical in the end ;-).
>>>> 
>>>> Thanks a lot for your help. I'm really hoping we can come to the
>>>> conclusion that migrating to JavaFX is the right way to go for us.
>>>> 
>>>> Robert
>>>>       
>> 
>>