Blur effect on live scene?

[Matthias Hänel]

Hi Jim,


> On 8/10/2015 11:44 PM, Matthias Hänel wrote:
>>> If we had a "snapshot to texture" mechanism then that might reduce the memory copying of the work around technique.
>> 
>> Well, did you say there is no snapshot to texture in JavaFX? In plain OpenGL you can use FBO's (Frame Buffer Objects) to render them. This is nothing more than a snapshot. They are even sliceable and strechable.
> 
> To be clear, there is no API to directly specify "snapshot this Node/Group to a texture and give me a handle to it to manipulate".  As I said later, there is a mechanism to get Node/trees rendered into a texture and that is the Node cache property, but we do that behind the scenes, the developer doesn't get a handle to the texture in that case.

Okay, that are different OSI layers. On the lowest OpenGL Layer it could be a FBO implementation. On higher layers there must be APIs that support this.



>>> I'd argue that we sort of do have something like that - it is the cache flag.  If a Node is cached then we do have a copy of it in a texture and that can help make the Blur Effect work more efficiently, but there may be some additional copies between textures if everything isn't set up right.  Still, that is an avenue for someone to check to see if there isn't a better way to achieve this effect in the short term...
>> 
>> I am not pretty sure what cache does. Probably some hasmap that holds objects and they are not instantly destroyed in the graphics RAM?
> 
> It is not a hashmap.
> 
> It is a hint to save the rendering of that node in a buffer:
> 
> https://docs.oracle.com/javase/8/javafx/api/javafx/scene/Node.html#cacheProperty
> 
> I don't like the way that this doc comment is worded as it implies that using it on a node that is blurred is unwise, but if the node is animated over even GPU acceleration of the rendering and blurring operations" are going to have some cost that it could save.

That documentation says nodes are cached as Bitmaps. In GPU or in CPU space? It is not clear here. I suspect it is in CPU RAM space. 
That will just lead to more copy-tasks from CPU to GPU or not? If optimized there is no benefit at all as stated in the docu "note that on some platforms such as GPU accelerated platforms there is little benefit". 

Actually, I would expect that renered Nodes are textures in GPU-VRAM to get the most performance out of it.


>> From my current point the major problem with JavaFX is still the same.
>> 
>> 1. Has a good API
>> 2. renders most of its stuff in software, hence does not run performant
>> 3. Has good approaches, but the overall sight on the technology is broken somewhere.
> 
> I am not sure how you come to the conclusion that it renders most of its stuff in software.  It renders quite a lot in hardware.  Even the example here of using snapshot to optimize a blurred background - the rendering of the scene is done in hw.  It is only copied to main memory because the API requires a persistent image.  If you render that image to the screen it is copied back into a texture and reused from that texture unless we run low on vram.  There is no rendering in software there, only use of a heap buffer for persistent storage...

Your point is that it uses hardware to render and it just uses snapshot to satisfy the API? 


My current understaning is ... what happens in our blurred-effect case?

1. the application constructs a JavaFX node tree. 
2. the node tree is rendered mostly in hardware (shader effects and so on on top) to the main framebuffer
3. Snapshot calls ReadPixels (or whatever it is called on the particular platform) 
4. JavaFX encapsulates this "new" image with Object
5. We draw the "new" image with effects to the OGL context with the same node-tree API as we did before on top of the first node-tree.


That works, but there is too much CPU and memcpy involved for my believe. Furthermore ReadPixels takes forever in
an OpenGL perspective. 
Since this is just one very simple effect, it is actually not good to spend more than approx. 20% CPU (i7) load on it.
I expect 0% (not noticable) for this blurry effect.



The ideal implementation from my perspective would be:
1. the application constructs a JavaFX node tree. 
2. the node tree is rendered mostly in hardware (shader effects and so on on top) to a virtual framebuffer in the GPU space
3. The virtual framebuffer is drawn by a simple drawVert-call for the background
4. The virtual framebuffer is drawn once again shaped and shaded (blurry filter) by another drawVert-call

This would reduce CPU usage by a very big amount of cycles. 

(From an Shader perspective we could reduce GPU load one more if we merge point 3 and 4 and run a partially shader to blur just the upper corner.
Well this is much more hand-coding and not good integratable in a bigger API.)


Even tough these ReadPixel-Functions to retrieve GPU rendered images to the CPU space are not very fast.



I have to correct my assumption that JavaFX renders most of its stuff in software a bit. The wording "rendering" was not correct.
This feeling comes probaly from the massive use of Snapshot. I have not understood the entire core of Prism until now. 
In some implementation of snapshot it seems to do the rendering entirely in software. That might only the fallback, but the
ReadPixels-Stuff is still valid.
Additionally to that I have the feeling that not all effects can be translated to Shaders and so it falls back pretty fast to software 
rendering. The main problem here is that as a developer we cannot see, where we are and what is done fast and what is not. Is there 
a better way to debug performance leaks in JavaFX than guessing? OpenGL tools don't help much, since they are soo far away, that it is 
pretty hard to get the right way.



Tiny-Offtopic: I already meantioned on this list that JFXPanel (Swing) is way below it's expectations. The main reason there is also the use 
of the snapshot function instead of letting render JavaFX in it's own heavy weight window. In our small test case a simple List
displayed via JFXPanel was not even to render more than 1 fps and it slowed down the entire Swing-UI. Since I know jogle and it's canvas 
implementation, would it be good to have a similar heavy weight JFX canvas in Swing? This would let JFX use it's full hardware rendered power.




Matthias