RFR: 8297638: Memory leak in case of many started-dead threads

[Chris Plummer]

On Wed, 18 Jan 2023 20:30:47 GMT, Alan Bateman <alanb at openjdk.org> wrote:

> Maybe you've tried this already but if listeners were changed to be a WeakHashMap<VMListener, Boolean> then this would expunge the stable entries. This would also avoid the O(n) search in removeUnreachableListeners and simplify many other operations. But maybe you are concerned about preserving the insertion order of the listeners?

I did read your JBS comment with this suggestion and considered it, but decided to keep the changes simple and stick with something I knew was working and would not risk introducing other issues. I'm also not so sure it would help performance much:

When running the test case, IIRC I would see the listeners grow to about 2500 before a GC would trigger about 2/3 of them being released in a batch. So w.r.t. `removeUnreachableListeners()`, I'm not too concerned about performance since it's iterating over the listeners is only triggered after a GC, and most are getting removed. It's not having to iterate over the list for each listener removed, but instead iterates once for each new batch of unreachable listeners.

It looks like listeners are also removed with `iter.remove()` in `processVMAction()`. This is probably the more common path to removal when not creating the special situation in the debugger described in this CR. However, it will also remove a batch of listeners with one iteration over the list.

I just noticed that that `removeListener()` is never called. I'll file an RFE to remove it. In any case, we don't need to worry about its performance.

-------------

PR: https://git.openjdk.org/jdk/pull/12081