Integrating native Open GL code in JavaFX and alternatives

[Richard Bair]

One question I have on this is related to security. I'm more than a little worried about exposing native handles in any way as that means a reasonable likelihood that a bug in our code could lead to an exploit. What are the safety mechanisms for something like this?

The other issue is one of shaders. I would love to be able to give you an OpenGL surface that you can scribble into and have it be a normal part of the scene graph. I think that's a requirement (I would hate to end up in a place where it has to be a separate window). However one thing I don't know about is whether any shaders you develop would interfere with the shaders we use for normal rasterization etc.

Kevin, Jim?

On Jan 25, 2013, at 6:28 AM, Scott Palmer <swpalmer at gmail.com> wrote:

> +1
> 
> I see two basic needs:
> 
> 1: A JAWT equivalent so we can at least get a handle to the Stage window.
> 
> 2: Some means of embedding our own rendering surface.  Likely via a "texture" described in the native format for the graphics pipeline used (OpenGL or Direct3D)
> 
> In our Swing product (on Windows) we grab the native window handle of a heavyweight 'placholder' via JAWT and blit a bitmap directly to it - hardware assisted with a colorspace conversion, if I recall correctly.  JavaFX offers no equivalent.
> 
> I know that JavaFX will likely never allow the concept of embedding a heavyweight native window into a Scene.  I don't think that would be the right approach for it anyway.  But we can't even make a separate preview window natively and give it a proper parent window with the current APIs.
> 
> Scott
> 
> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de> wrote:
> 
>> Hi,
>> 
>> just to avoid, this gets buried in the other long thread where it
>> originated from (Migrating commercial applications from Swing to JFX),
>> I'm posting this again as a separate thread. I hope this is OK.
>> 
>> We are evaluating migrating our Swing Application which includes a
>> video player built using native code for decoding, rendering and
>> effects to JavaFX. Performance requirements are very ambitious (like
>> being close to the playback performance of the best native player, in
>> our case Quicktime for most formats we care about, for high definition
>> video).
>> 
>> Currently there is no way to do the same thing in JavaFX, which the
>> JOGL folks have already run into themselves (see
>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>> 
>> We would either need support for getting a native Open GL context to
>> draw to a JFX component (panel, canvas whatever), so we can use native
>> code to do all sorts of stuff that happens in our video player or we
>> would need support for the following operations in JFX:
>> 
>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>> - shaders and frame buffer objects (render to texture) for applying
>> layered effects to video frames
>> - pixel buffer objects for transferring video images to textures (and back)
>> - interoperability with opencl and cuda (processing of video frames
>> with opencl/cuda and rendering with opengl)
>> - reuse of opengl code across platforms (including android and iOS)
>> 
>> We achieve that currently by having cross-platform C code that is
>> connected to each platform via a rather thin layer for communication
>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>> controlled by a Java-UI). My guess is that it is more likely that you
>> will be able to expose the native context than providing all the
>> listed things. It is also a maybe a philosophical question whether you
>> want that but IMHO you close the door on an interesting range of
>> applications if you don't provide anything like that in some way
>> because you cannot implement a serious media application of a certain
>> kind without it and why reinvent the wheel? IMHO Java technologies in
>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>> from being good ideas but not making it to a production-quality state
>> because doing all that would have meant reinventing the wheel, which
>> in complex areas like the one described here is simply not feasible so
>> they IMHO remained being mostly of educational value for academic
>> environments. Acknowledging that in some areas it does make sense to
>> have some native code doesn't diminish the value of Java as a
>> platform. Having a lot of portable code in an application is still a
>> lot better than having one application per platform. Quite the
>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>> area if that was supported some more.
>> 
>> Sorry for being a bit philosophical in the end ;-).
>> 
>> Thanks a lot for your help. I'm really hoping we can come to the
>> conclusion that migrating to JavaFX is the right way to go for us.
>> 
>> Robert
>