Blur effect on live scene?

[Jim Graham]

On 8/13/2015 1:29 AM, Matthias Hänel wrote:
>>>> I'd argue that we sort of do have something like that - it is the cache flag.  If a Node is cached then we do have a copy of it in a texture and that can help make the Blur Effect work more efficiently, but there may be some additional copies between textures if everything isn't set up right.  Still, that is an avenue for someone to check to see if there isn't a better way to achieve this effect in the short term...
>>>
>>> I am not pretty sure what cache does. Probably some hasmap that holds objects and they are not instantly destroyed in the graphics RAM?
>>
>> It is not a hashmap.
>>
>> It is a hint to save the rendering of that node in a buffer:
>>
>> https://docs.oracle.com/javase/8/javafx/api/javafx/scene/Node.html#cacheProperty
>>
>> I don't like the way that this doc comment is worded as it implies that using it on a node that is blurred is unwise, but if the node is animated over even GPU acceleration of the rendering and blurring operations" are going to have some cost that it could save.
>
> That documentation says nodes are cached as Bitmaps. In GPU or in CPU space? It is not clear here. I suspect it is in CPU RAM space.
> That will just lead to more copy-tasks from CPU to GPU or not? If optimized there is no benefit at all as stated in the docu "note that on some platforms such as GPU accelerated platforms there is little benefit".

As I said, the doc comments there are confusing.  "bitmap" here, despite 
any capitalization, is a general term for "some type of pixel store". 
On GPU accelerated platforms, this is a vram texture/FBO.

> Actually, I would expect that renered Nodes are textures in GPU-VRAM to get the most performance out of it.

Yes, nodes with the cache hint are rendered into VRAM textures and 
reused for subsequent frames if the cache hint is set - subject to the 
policies of the cacheHint property if any (mainly transform) properties 
of the node are changed in the meantime.

>>>  From my current point the major problem with JavaFX is still the same.
>>>
>>> 1. Has a good API
>>> 2. renders most of its stuff in software, hence does not run performant
>>> 3. Has good approaches, but the overall sight on the technology is broken somewhere.
>>
>> I am not sure how you come to the conclusion that it renders most of its stuff in software.  It renders quite a lot in hardware.  Even the example here of using snapshot to optimize a blurred background - the rendering of the scene is done in hw.  It is only copied to main memory because the API requires a persistent image.  If you render that image to the screen it is copied back into a texture and reused from that texture unless we run low on vram.  There is no rendering in software there, only use of a heap buffer for persistent storage...
>
> Your point is that it uses hardware to render and it just uses snapshot to satisfy the API?

What is "it"?

JavaFX doesn't "use" snapshot anywhere.  It provides the snapshot API so 
the developer can use snapshot where the developer wants to, but JavaFX 
doesn't invoke the snapshot API on its own.

In particular, the node cache hint does not use the snapshot API under 
the covers.

> My current understaning is ... what happens in our blurred-effect case?

To be clear, you are describing what happens in a particular 
implementation of the blurred-effect case.  The implementation that uses 
snapshot, which may be the popular technique in use at this time.

> 1. the application constructs a JavaFX node tree.
> 2. the node tree is rendered mostly in hardware (shader effects and so on on top) to the main framebuffer

Correct

> 3. Snapshot calls ReadPixels (or whatever it is called on the particular platform)
> 4. JavaFX encapsulates this "new" image with Object

Technically, snapshot does its own (hardware accelerated) rendering into 
a separate GPU texture/FBO and does a read pixels on that texture.  It 
then encapsulates those pixels into an Image object.

This snapshot operation need only be done once if the underlying node 
tree is static.

> 5. We draw the "new" image with effects to the OGL context with the same node-tree API as we did before on top of the first node-tree.

Correct.  Note that when you render that Image object its pixels are 
cached in vram and that vram copy is reused from frame to frame.  So, if 
you reuse the one snapshot then there was only one trip from vram to 
memory and back to vram on the first frame you did this and then 
everything should be done in vram for subsequent frames reusing the same 
snapshot.

> That works, but there is too much CPU and memcpy involved for my believe. Furthermore ReadPixels takes forever in
> an OpenGL perspective.

If the underlying tree is static, though, that readpixels operation only 
happens once at the start of the operation, but you are correct that it 
would be much better if it didn't need to happen at all.

> Since this is just one very simple effect, it is actually not good to spend more than approx. 20% CPU (i7) load on it.
> I expect 0% (not noticable) for this blurry effect.

That would be ideal.  We may need new API to get there, but there are 
also options to consider that may get us there in the short term.  In 
particular, cached nodes - which are already present in the API - may 
get us closer to that goal.

> The ideal implementation from my perspective would be:
> 1. the application constructs a JavaFX node tree.
> 2. the node tree is rendered mostly in hardware (shader effects and so on on top) to a virtual framebuffer in the GPU space
> 3. The virtual framebuffer is drawn by a simple drawVert-call for the background
> 4. The virtual framebuffer is drawn once again shaped and shaded (blurry filter) by another drawVert-call

If you set the cache hint to true on the underlying tree then this may 
be approximately what happens in our current implementation.  The part I 
would still need to investigate would be how well the effects machinery 
(called Decora) can reuse the cached version of the nodes.  If it 
doesn't attempt to reuse the cached version of the nodes then it might 
end up re-rendering the tree.

> I have to correct my assumption that JavaFX renders most of its stuff in software a bit. The wording "rendering" was not correct.
> This feeling comes probaly from the massive use of Snapshot. I have not understood the entire core of Prism until now.
> In some implementation of snapshot it seems to do the rendering entirely in software. That might only the fallback, but the
> ReadPixels-Stuff is still valid.

To be clear, snapshot was mainly created for its namesake - to produce 
static copies of the scene data to be saved as images for putting into 
documentation.  Alternate uses that were considered were for generating 
thumbnails of scenes for an application that could open/close various 
scene graph panes.  One wouldn't use the Windows "print screen" API in a 
performance intensive part of one's application either.  It is great 
that it can be used for this particular result, but it was not designed 
to be performant in that respect.  Note that the primary snapshot API is 
the asynchronous version with the callback to deliver the data.  The 
non-asynchronous version is mostly just a helper around that, but it 
stalls the rendering pipeline to complete its work.  None of that was 
designed for insertion into a running animation technique.

> Tiny-Offtopic: I already meantioned on this list that JFXPanel (Swing) is way below it's expectations. The main reason there is also the use
> of the snapshot function instead of letting render JavaFX in it's own heavy weight window. In our small test case a simple List
> displayed via JFXPanel was not even to render more than 1 fps and it slowed down the entire Swing-UI. Since I know jogle and it's canvas
> implementation, would it be good to have a similar heavy weight JFX canvas in Swing? This would let JFX use it's full hardware rendered power.

It doesn't use snapshot per-se, but it does do something similar. 
Sharing contexts and GPU resources with AWT/Java2D is on a wish list, 
but architecturally we aren't there.  Kevin would know more about our 
challenges on that front than me...

			...jim