Integrating native Open GL code in JavaFX and alternatives

Mon Jan 28 07:48:32 PST 2013

The link

https://jbs.oracle.com/bugs/browse/JDK-7154778

does not work for "outsiders" - I think you are hinting at

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7154778

-Sven

On Mon, Jan 28, 2013 at 4:16 PM, Artem Ananiev <artem.ananiev at oracle.com>wrote:

>
> On 1/26/2013 12:04 AM, Scott Palmer wrote:
>
>>
>> On 2013-01-25, at 2:16 PM, Artem Ananiev <artem.ananiev at oracle.com>
>> wrote:
>>
>>  A few comments from my side:
>>>
>>> 1. If a native Window/Scene handle is provided, it will be enough
>>> to embed whatever you need. It will even be possible to embed D3D
>>> content into JavaFX running with OpenGL pipeline :)
>>>
>>
>> There will be issues in JavaFX that we can't work around as easily as
>> we code in Swing. For example clipping.  How can my video preview
>> overlay participate in a Scene such that I can put it inside a
>> ScrollPane or SplitPane?  With Swing I could manage this by placing a
>> heavyweight component in the UI.  Rectangular clipping was easy to
>> deal with by wrapping in another heavyweight component - the OS
>> windowing system took care of it.  Later Swing got smarter about
>> mixing heavy/light components.  JavaFX doesn't have something
>> comparable.
>>
>
> To use heavyweight component in a scroll/split pane, we need heavyweight
> scroll/split pane :) In other words, such scenarios don't work well in
> Swing too.
>
> Do you mean limited heavyweight/lightweight component mixing, introduced
> to 6u10 and the forward-ported to 7? Note the word "limited" here: the
> mixing is really limited, it doesn't cover non-rectangular shapes,
> transparent components, and many more. It actually works fine for simplest
> scenarios only. The code we put to AWT to support the limited mixing is not
> something I would like to see in JavaFX.
>
>
>  The question is what is the native handle? On Windows, HWND is the
>>> answer. On Mac: NSView or CALayer?
>>>
>>
>> CALayer seems to be the way things are going. NSView was the Java 6
>> native layer, now in Java 7 it is CALayer.  But this is native code…
>> let there be a way to get either from a platform specific structure.
>>
>
> Believe it or not, we will support both CALayer-based and NSView-based
> embedded windows in JDK8 (and probably next JDK7 updates). Check the
> following JDK bug for details:
>
> https://jbs.oracle.com/bugs/**browse/JDK-7154778<https://jbs.oracle.com/bugs/browse/JDK-7154778>
>
> Classes are not public (java.*, javax.*) APIs, though, but are supposed to
> be backwards compatible. If we implement this functionality in JavaFX,
> we'll likely support both CALayers and NSViews.
>
>
>  In the latter case we'll also need mechanism to flush embedded
>>> layer content into the parent (window) layer, and the thing will
>>> get complicated.
>>>
>>> 2. Another challenge with heavyweight embedding is integration into
>>> FX scene. Focus traversal, transforms, effects, modality, etc.
>>>
>>
>> Can be depending on the use case.  For a video preview panel, I
>> wouldn't' need most of that.  #3 would be best.
>>
>>  3. Alternatively, we can try to implement semi-heavyweight
>>> embedding. What we need is to make Prism be able to render
>>> externally managed textures (GL, D3D). Input events will be handled
>>> as for any other FX nodes, transforms/effects will also be of no
>>> problem. What's questionable here is a) performance b) sharing GL
>>> context or D3D device between FX and application code. I'm not a
>>> graphics expert, but suspect there may be problems in this area.
>>>
>>
>> I agree.  But I think it is worth getting a graphics expert (I'm not
>> a OpenGL/D3D person either) to think about it over the weekend :-)
>>
>> Cheers,
>>
>> Scott
>>
>
> Thanks,
>
> Artem
>
>  Thanks,
>>>
>>> Artem
>>>
>>> On 1/25/2013 6:28 PM, Scott Palmer wrote:
>>>
>>>> +1
>>>>
>>>> I see two basic needs:
>>>>
>>>> 1: A JAWT equivalent so we can at least get a handle to the Stage
>>>> window.
>>>>
>>>> 2: Some means of embedding our own rendering surface.  Likely via a
>>>> "texture" described in the native format for the graphics pipeline
>>>> used (OpenGL or Direct3D)
>>>>
>>>> In our Swing product (on Windows) we grab the native window handle of
>>>> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
>>>> - hardware assisted with a colorspace conversion, if I recall
>>>> correctly.  JavaFX offers no equivalent.
>>>>
>>>> I know that JavaFX will likely never allow the concept of embedding a
>>>> heavyweight native window into a Scene.  I don't think that would be
>>>> the right approach for it anyway.  But we can't even make a separate
>>>> preview window natively and give it a proper parent window with the
>>>> current APIs.
>>>>
>>>> Scott
>>>>
>>>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
>>>> wrote:
>>>>
>>>>  Hi,
>>>>>
>>>>> just to avoid, this gets buried in the other long thread where it
>>>>> originated from (Migrating commercial applications from Swing to JFX),
>>>>> I'm posting this again as a separate thread. I hope this is OK.
>>>>>
>>>>> We are evaluating migrating our Swing Application which includes a
>>>>> video player built using native code for decoding, rendering and
>>>>> effects to JavaFX. Performance requirements are very ambitious (like
>>>>> being close to the playback performance of the best native player, in
>>>>> our case Quicktime for most formats we care about, for high definition
>>>>> video).
>>>>>
>>>>> Currently there is no way to do the same thing in JavaFX, which the
>>>>> JOGL folks have already run into themselves (see
>>>>> https://jogamp.org/bugzilla/**show_bug.cgi?id=607<https://jogamp.org/bugzilla/show_bug.cgi?id=607>
>>>>> ).
>>>>>
>>>>> We would either need support for getting a native Open GL context to
>>>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>>>> code to do all sorts of stuff that happens in our video player or we
>>>>> would need support for the following operations in JFX:
>>>>>
>>>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB
>>>>> ....)
>>>>> - shaders and frame buffer objects (render to texture) for applying
>>>>> layered effects to video frames
>>>>> - pixel buffer objects for transferring video images to textures (and
>>>>> back)
>>>>> - interoperability with opencl and cuda (processing of video frames
>>>>> with opencl/cuda and rendering with opengl)
>>>>> - reuse of opengl code across platforms (including android and iOS)
>>>>>
>>>>> We achieve that currently by having cross-platform C code that is
>>>>> connected to each platform via a rather thin layer for communication
>>>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>>>> controlled by a Java-UI). My guess is that it is more likely that you
>>>>> will be able to expose the native context than providing all the
>>>>> listed things. It is also a maybe a philosophical question whether you
>>>>> want that but IMHO you close the door on an interesting range of
>>>>> applications if you don't provide anything like that in some way
>>>>> because you cannot implement a serious media application of a certain
>>>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>>>> from being good ideas but not making it to a production-quality state
>>>>> because doing all that would have meant reinventing the wheel, which
>>>>> in complex areas like the one described here is simply not feasible so
>>>>> they IMHO remained being mostly of educational value for academic
>>>>> environments. Acknowledging that in some areas it does make sense to
>>>>> have some native code doesn't diminish the value of Java as a
>>>>> platform. Having a lot of portable code in an application is still a
>>>>> lot better than having one application per platform. Quite the
>>>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>>>> area if that was supported some more.
>>>>>
>>>>> Sorry for being a bit philosophical in the end ;-).
>>>>>
>>>>> Thanks a lot for your help. I'm really hoping we can come to the
>>>>> conclusion that migrating to JavaFX is the right way to go for us.
>>>>>
>>>>> Robert
>>>>>
>>>>
>>>>
>>

-- 
Sven Reimers

* Senior Expert Software Architect
* NetBeans Dream Team Member: http://dreamteam.netbeans.org
* NetBeans Governance Board Member:
http://netbeans.org/about/os/governance.html
* Community Leader  NetBeans: http://community.java.net/netbeans
                              Desktop Java:
http://community.java.net/javadesktop
* Duke's Choice Award Winner 2009
* Blog: http://nbguru.blogspot.com

* XING: https://www.xing.com/profile/Sven_Reimers8
* LinkedIn: http://www.linkedin.com/in/svenreimers

Join the NetBeans Groups:
* XING: http://www.xing.com/group-20148.82db20
* NUGM: http://haug-server.dyndns.org/display/NUGM/Home
* LinkedIn: http://www.linkedin.com/groups?gid=1860468
                   http://www.linkedin.com/groups?gid=107402
                   http://www.linkedin.com/groups?gid=1684717
* Oracle: https://mix.oracle.com/groups/18497