Integrating native Open GL code in JavaFX and alternatives

[Daniel Zwolenski]

Regarding the security issue is it worth considering the option that there are features in javafx that just aren't allowed in the web sandbox ever? 

i.e. trying to use this native gl stuff in an applet/jnlp app will always result in a security exception even if you have all permissions allowed. 

I'd personally be keen to see this general philosophy extend to anything where supporting web deployment would hinder the rest of the platform. As I understand it the web deployment options are on the end of the curve and we are mostly keeping them around for legacy reasons. In that case they are conceptually deprecated and for my money should be treated as such - ie they continue to work as is, but if you want newer features, move to the newer options. 



On 26/01/2013, at 7:04 AM, Scott Palmer <swpalmer at gmail.com> wrote:

> 
> On 2013-01-25, at 2:16 PM, Artem Ananiev <artem.ananiev at oracle.com> wrote:
> 
>> 
>> A few comments from my side:
>> 
>> 1. If a native Window/Scene handle is provided, it will be enough to embed whatever you need. It will even be possible to embed D3D content into JavaFX running with OpenGL pipeline :)
> 
> There will be issues in JavaFX that we can't work around as easily as we code in Swing.
> For example clipping.  How can my video preview overlay participate in a Scene such that I can put it inside a ScrollPane or SplitPane?  With Swing I could manage this by placing a heavyweight component in the UI.  Rectangular clipping was easy to deal with by wrapping in another heavyweight component - the OS windowing system took care of it.  Later Swing got smarter about mixing heavy/light components.  JavaFX doesn't have something comparable.
> 
>> The question is what is the native handle? On Windows, HWND is the answer. On Mac: NSView or CALayer?
> 
> CALayer seems to be the way things are going. NSView was the Java 6 native layer, now in Java 7 it is CALayer.  But this is native code… let there be a way to get either from a platform specific structure.
> 
>> In the latter case we'll also need mechanism to flush embedded layer content into the parent (window) layer, and the thing will get complicated.
>> 
>> 2. Another challenge with heavyweight embedding is integration into FX scene. Focus traversal, transforms, effects, modality, etc.
> 
> Can be depending on the use case.  For a video preview panel, I wouldn't' need most of that.  #3 would be best.
> 
>> 3. Alternatively, we can try to implement semi-heavyweight embedding. What we need is to make Prism be able to render externally managed textures (GL, D3D). Input events will be handled as for any other FX nodes, transforms/effects will also be of no problem. What's questionable here is a) performance b) sharing GL context or D3D device between FX and application code. I'm not a graphics expert, but suspect there may be problems in this area.
> 
> I agree.  But I think it is worth getting a graphics expert (I'm not a OpenGL/D3D person either) to think about it over the weekend :-)
> 
> Cheers,
> 
> 
> Scott
> 
>> 
>> Thanks,
>> 
>> Artem
>> 
>> On 1/25/2013 6:28 PM, Scott Palmer wrote:
>>> +1
>>> 
>>> I see two basic needs:
>>> 
>>> 1: A JAWT equivalent so we can at least get a handle to the Stage
>>> window.
>>> 
>>> 2: Some means of embedding our own rendering surface.  Likely via a
>>> "texture" described in the native format for the graphics pipeline
>>> used (OpenGL or Direct3D)
>>> 
>>> In our Swing product (on Windows) we grab the native window handle of
>>> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
>>> - hardware assisted with a colorspace conversion, if I recall
>>> correctly.  JavaFX offers no equivalent.
>>> 
>>> I know that JavaFX will likely never allow the concept of embedding a
>>> heavyweight native window into a Scene.  I don't think that would be
>>> the right approach for it anyway.  But we can't even make a separate
>>> preview window natively and give it a proper parent window with the
>>> current APIs.
>>> 
>>> Scott
>>> 
>>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> just to avoid, this gets buried in the other long thread where it
>>>> originated from (Migrating commercial applications from Swing to JFX),
>>>> I'm posting this again as a separate thread. I hope this is OK.
>>>> 
>>>> We are evaluating migrating our Swing Application which includes a
>>>> video player built using native code for decoding, rendering and
>>>> effects to JavaFX. Performance requirements are very ambitious (like
>>>> being close to the playback performance of the best native player, in
>>>> our case Quicktime for most formats we care about, for high definition
>>>> video).
>>>> 
>>>> Currently there is no way to do the same thing in JavaFX, which the
>>>> JOGL folks have already run into themselves (see
>>>> https://jogamp.org/bugzilla/show_bug.cgi?id=607).
>>>> 
>>>> We would either need support for getting a native Open GL context to
>>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>>> code to do all sorts of stuff that happens in our video player or we
>>>> would need support for the following operations in JFX:
>>>> 
>>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>>> - shaders and frame buffer objects (render to texture) for applying
>>>> layered effects to video frames
>>>> - pixel buffer objects for transferring video images to textures (and back)
>>>> - interoperability with opencl and cuda (processing of video frames
>>>> with opencl/cuda and rendering with opengl)
>>>> - reuse of opengl code across platforms (including android and iOS)
>>>> 
>>>> We achieve that currently by having cross-platform C code that is
>>>> connected to each platform via a rather thin layer for communication
>>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>>> controlled by a Java-UI). My guess is that it is more likely that you
>>>> will be able to expose the native context than providing all the
>>>> listed things. It is also a maybe a philosophical question whether you
>>>> want that but IMHO you close the door on an interesting range of
>>>> applications if you don't provide anything like that in some way
>>>> because you cannot implement a serious media application of a certain
>>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>>> from being good ideas but not making it to a production-quality state
>>>> because doing all that would have meant reinventing the wheel, which
>>>> in complex areas like the one described here is simply not feasible so
>>>> they IMHO remained being mostly of educational value for academic
>>>> environments. Acknowledging that in some areas it does make sense to
>>>> have some native code doesn't diminish the value of Java as a
>>>> platform. Having a lot of portable code in an application is still a
>>>> lot better than having one application per platform. Quite the
>>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>>> area if that was supported some more.
>>>> 
>>>> Sorry for being a bit philosophical in the end ;-).
>>>> 
>>>> Thanks a lot for your help. I'm really hoping we can come to the
>>>> conclusion that migrating to JavaFX is the right way to go for us.
>>>> 
>>>> Robert
>>> 
>