Canvas memory issues and performance

[joe andresen]

The issue with canvas doing non desirable things when not visible is 
indeed something we have been looking at.

It is up to the user to stop sending commands, and this is something 
that is and isn't an easy thing to do depending on the scenario the user 
is dealing with.

The issue is, we define canvas as persistent, that is, whatever you tell 
it to do, the results will be processed and the canvas updated. We 
cannot discard the commands you send to the canvas for obvious reasons, 
and we certainly do not want to process them when the canvas is not 
visible. So one of the best things we can do is make sure our commands 
are not huge in terms of memory, even this is not great because Effects 
can get big.

I do not know much about your app but it seems Canvas' persistent nature 
VS your apps non persistent video needs (ie all you care about is the 
latest "frame" command and not any of the previous) sounds like the 
fundamental issue at hand.

There are cases where we can disregard previous commands, for example if 
you clear the entire canvas or draw a shape the covers the entire canvas.

-Joe
JavaFX Graphics Team

On 1/2/2013 8:45 AM, Scott Palmer wrote:
> Interesting, this made me take another look at my implementation…
>
> I discovered I am already using the pre-multiplied BGRA PixelFormat.  The GUI in the application slows down quite a bit without an HD video preview running.  Just by updating Paths as the user drags a node around (see http://javafx-jira.kenai.com/browse/RT-25166).  Add HD-sized video at 60fps and you have a lot of garbage to process as well.
>
> Anything that stalls the thread that is processing the Canvas command pipe causes a backup of video frames to fill the heap - that leads to GC issues and it makes the problem worse.  I still have doubts over the usability of Canvas for realtime video in a non-trivial sample app.  But it gets worse...
>
> There is also this issue http://javafx-jira.kenai.com/browse/RT-24903 that I filed some time ago (complete with reproducible test case).  Basically Canvas stops processing it's command queue when it is not visible.  But it could be as simple as being temporarily scrolled out of view.  It is unreasonable to detect this at stop sending it frame buffers.  So Canvas is most definitely not usable for my situation.
>
> I was thinking of getting access to a native surface (Direct3D, or OpenGL texture?) and rendering directly into it.  But I appreciate the complexity that may be involved with such an implementation.
>
> Scott
>
> On 2012-12-21, at 3:13 PM, John Smith<John_Smith at symantec.com>  wrote:
>
>>> Canvas is sort of close to what is needed, but it fails completely for my use case and simply crashes the app because it runs out of memory.  The FX Thread doesn't service the changes to the canvas fast enough for me to paint live video into the canvas.
>> I used Canvas (and WritableImage) for doing 1024x1024px animations at 60fps and it did not crash or run out of memory.  The FX Thread serviced the changes fast enough for smooth animation.  Others have displayed smooth 1920x1080 full screen video using Canvas (see https://forums.oracle.com/forums/thread.jspa?threadID=2434712).  The PixelFormat used for the WritableImage can make a huge difference to performance.