Canvas memory issues and performance

[Scott Palmer]

The thing I'm not clear on is this: "we certainly do not want to process them when the canvas is not visible"
In the suggested case where the app doesn't send commands while the Canvas isn't visible there won't be anything to process anyway.  So far the alternative is to consume memory until a crash.  So why not process the commands?  What is the harm?   I understand the persistent nature of Canvas, but is there no way to set a limit on the command queue such that you force a flush (i.e. by processing the commands) if you hit it?


Regarding my non-persistent video… 
True, in my case I will always be drawing an image that covers the entire canvas, though because it is BGRA there is no way for Canvas to know that all of the pixels are 100% opaque without actually looking at the alpha value of each pixel.  So it seems some sort of hint is required regardless.

I could render to a WriteableImage and place that in an ImageView… but is that any different?  
I wanted to avoid constructing 1920x1080 image objects at 60fps so as not to stress the GC.   In my Swing app I used JAWT to get the window handle of a heavy-weight component and then use DirectDraw to blit the image into it.  I'm open to suggestions of how to deal with the same situation in JavaFX.

Anticipating this day would come, I filed a request four years ago for improved support in the media playing abilities of JavaFX:
http://javafx-jira.kenai.com/browse/RT-2684

Scott



On 2013-01-02, at 1:19 PM, joe andresen <joseph.andresen at oracle.com> wrote:

> The issue with canvas doing non desirable things when not visible is indeed something we have been looking at.
> 
> It is up to the user to stop sending commands, and this is something that is and isn't an easy thing to do depending on the scenario the user is dealing with.
> 
> The issue is, we define canvas as persistent, that is, whatever you tell it to do, the results will be processed and the canvas updated. We cannot discard the commands you send to the canvas for obvious reasons, and we certainly do not want to process them when the canvas is not visible. So one of the best things we can do is make sure our commands are not huge in terms of memory, even this is not great because Effects can get big.
> 
> I do not know much about your app but it seems Canvas' persistent nature VS your apps non persistent video needs (ie all you care about is the latest "frame" command and not any of the previous) sounds like the fundamental issue at hand.
> 
> There are cases where we can disregard previous commands, for example if you clear the entire canvas or draw a shape the covers the entire canvas.
> 
> -Joe
> JavaFX Graphics Team
> 
> On 1/2/2013 8:45 AM, Scott Palmer wrote:
>> Interesting, this made me take another look at my implementation…
>> 
>> I discovered I am already using the pre-multiplied BGRA PixelFormat.  The GUI in the application slows down quite a bit without an HD video preview running.  Just by updating Paths as the user drags a node around (see http://javafx-jira.kenai.com/browse/RT-25166).  Add HD-sized video at 60fps and you have a lot of garbage to process as well.
>> 
>> Anything that stalls the thread that is processing the Canvas command pipe causes a backup of video frames to fill the heap - that leads to GC issues and it makes the problem worse.  I still have doubts over the usability of Canvas for realtime video in a non-trivial sample app.  But it gets worse...
>> 
>> There is also this issue http://javafx-jira.kenai.com/browse/RT-24903 that I filed some time ago (complete with reproducible test case).  Basically Canvas stops processing it's command queue when it is not visible.  But it could be as simple as being temporarily scrolled out of view.  It is unreasonable to detect this at stop sending it frame buffers.  So Canvas is most definitely not usable for my situation.
>> 
>> I was thinking of getting access to a native surface (Direct3D, or OpenGL texture?) and rendering directly into it.  But I appreciate the complexity that may be involved with such an implementation.
>> 
>> Scott
>> 
>> On 2012-12-21, at 3:13 PM, John Smith<John_Smith at symantec.com>  wrote:
>> 
>>>> Canvas is sort of close to what is needed, but it fails completely for my use case and simply crashes the app because it runs out of memory.  The FX Thread doesn't service the changes to the canvas fast enough for me to paint live video into the canvas.
>>> I used Canvas (and WritableImage) for doing 1024x1024px animations at 60fps and it did not crash or run out of memory.  The FX Thread serviced the changes fast enough for smooth animation.  Others have displayed smooth 1920x1080 full screen video using Canvas (see https://forums.oracle.com/forums/thread.jspa?threadID=2434712).  The PixelFormat used for the WritableImage can make a huge difference to performance.
>