RFR: 6507038: Memory Leak in JTree / BasicTreeUI

[Alexey Ivanov]

On Wed, 17 Jan 2024 07:19:21 GMT, Prasanta Sadhukhan <psadhukhan at openjdk.org> wrote:

> When using a TreeCellRenterer which creates new components in getTreeCellRendererComponent() in a JTree that is not visible, changes to the nodes cause a memory leak.
> When a node is changed, the Method getNodeDimensions() is called to calculate the new dimensions for the node. In this method, getTreeCellRendererComponent() is called to obtain the renderer component (what else...) and [this component is added to rendererPane](https://github.com/openjdk/jdk/blob/36f4b34f1953af736706ec67192204727808bc6c/src/java.desktop/share/classes/javax/swing/plaf/basic/BasicTreeUI.java#L3283-L3284). It is not removed from the rendererPane afterwards. 
> Only when the tree is painted, the paint() method does a removeAll on the rendererPane [in this code](https://github.com/openjdk/jdk/blob/36f4b34f1953af736706ec67192204727808bc6c/src/java.desktop/share/classes/javax/swing/plaf/basic/BasicTreeUI.java#L1500)
> 
> FIx is added to remove the components from rendererPane when the JTree UI is changed/uninstalled only when tree is not visible since they are already removed when tree is painted in paint() method..

> > can we trigger this memory leak by some new or existed tests?
>
> I cant see any nor can I think of any..you have any idea?

As far as I understand, the renderer component is leaked.

Usually, the renderer isn't re-created each time `getTreeCellRendererComponent()` is called. So, we can create a renderer component and store a weak/phantom reference to it.

After `tree.uninstallUI` is called, the renderer component should be freed. You can use [`ForceGC`](https://github.com/openjdk/jdk/blob/master/test/lib/jdk/test/lib/util/ForceGC.java) class for it. A sample usage of `ForceGC` can be found in [`AwtListGarbageCollectionTest.java`](https://github.com/openjdk/jdk/blob/master/test/jdk/java/awt/List/ListGarbageCollectionTest/AwtListGarbageCollectionTest.java) or other tests.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/17458#issuecomment-1904580067