RFR: 8352209: NPE com.sun.prism.d3d.D3DTextureData.getContext [v2]

[Lukasz Kostyra]

On Thu, 30 Oct 2025 08:12:10 GMT, John Hendrikx <jhendrikx at openjdk.org> wrote:

>> Lukasz Kostyra has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   D3DTextureResource: Set resource to null after disposing
>
> I've added this fix to my application, and I don't know exactly how the resources are now managed, but it seems that it does solve the problem and the program no longer gets stuck in an NPE loop, even with more large images showing than usual.
> 
> Very happy with the fix, it was something I already looked into but didn't have time to further investigate :)

@hjohn I wanted to do a more thorough check before making a response to your first comment.

The problem here is a use-after-free situation. `D3DVramPool` can prune and unlink a bunch of Textures (with `-Dprism.pooldebug=true` you can see a bunch of `unlinking: null (size=...)` messages). Right after that Prism tries to create a new RTT for rendering the next frame and `D3DResourceFactory` attempts to initialize it by first creating a temporary Graphics object and then calling `clear()` on it. This can cause a flush of pre-existing rendering data and can potentially trigger an update on a mask Texture that has just been pruned by the Pool. Here's the stack trace of this happening:

LKDEBUG Tried to update invalid resource at:
java.lang.Exception: Stack trace
        at java.base/java.lang.Thread.dumpStack(Thread.java:1991)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:147)
        at javafx.graphics at 26-internal/com.sun.prism.impl.BaseContext.flushMask(BaseContext.java:115)
        at javafx.graphics at 26-internal/com.sun.prism.impl.BaseContext.drawQuads(BaseContext.java:124)
        at javafx.graphics at 26-internal/com.sun.prism.impl.VertexBuffer.flush(VertexBuffer.java:98)
        at javafx.graphics at 26-internal/com.sun.prism.impl.BaseContext.flushVertexBuffer(BaseContext.java:107)
        at javafx.graphics at 26-internal/com.sun.prism.impl.ps.BaseShaderContext.setRenderTarget(BaseShaderContext.java:791)
        at javafx.graphics at 26-internal/com.sun.prism.impl.BaseContext.setRenderTarget(BaseContext.java:149)
        at javafx.graphics at 26-internal/com.sun.prism.impl.BaseGraphics.<init>(BaseGraphics.java:107)
        at javafx.graphics at 26-internal/com.sun.prism.impl.ps.BaseShaderGraphics.<init>(BaseShaderGraphics.java:84)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DGraphics.<init>(D3DGraphics.java:40)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DGraphics.create(D3DGraphics.java:63)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DRTTexture.createGraphics(D3DRTTexture.java:80)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:360)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:301)
        at javafx.graphics at 26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:61)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSDrawable.create(PPSDrawable.java:59)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:218)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:67)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.ImagePool.checkOut(ImagePool.java:178)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.Renderer.getCompatibleImage(Renderer.java:120)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.getCompatibleImage(PPSRenderer.java:226)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSTwoSamplerPeer.filterImpl(PPSTwoSamplerPeer.java:62)
        at javafx.graphics at 26-internal/com.sun.scenario.effect.impl.prism.ps.PPSEffectPeer.filter(PPSEffectPeer.java:54)
        ...


The mask texture has been considered not needed anymore by other parts of Prism, but its reference still lingers in the Context. Since the rendering code considers the mask Texture as unnecessary, skipping that update has no effect on the rendering loop or the output.

Ideally I would expect to have the mask Texture detached from the Context when it gets considered unnecessary, but after a brief look into common Prism code (unless I missed something) I think it might not be as easy as it sounds. Mask Texture is created by `BaseContext` which will also locally store its reference for the purpose of potentially flushing it later - since that mask Texture is returned to other parts of Prism, those can eventually consider it unneeded and the VramPool will prune/unlink it, but the existing reference will occasionally still linger in `BaseContext`. I agree the check in this change is not ideal, but we would probably have to rework this code a bit to properly solve it. I feel like this area is also very fragile and any small change can do a butterfly effect thing and trigger some other special edge-case scenario...

An alternative solution I saw just now would be to put a check directly in `BaseContext.flushMask()` to verify if maskTex is valid before doing the lock-update-unlock sequence - then we will be sure no backend enters the update method when it would be done on an already freed object. I might add that check and do some extra testing before integrating this change.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1951#issuecomment-3467741288