Promptly freeing the per-thread cached direct buffers when a thread exits

[Tony Printezis]

Hi all,

We recently hit another interesting issue with the NIO thread-local
DirectByteBuffer cache. One of our services seems to create and drop
threads at regular intervals. I assume this is due to a thread pool
dynamically resizing itself.

Let's say a thread starts, lives long enough for its Thread object to be
promoted to the old gen (along with its cached direct buffer), then exits.
This will result in its cached direct buffer(s) being unreachable in the
old gen and will only be reclaimed after the next full GC / concurrent GC
cycle.

Interestingly, the service in question does concurrent GC cycles really
infrequently (one every few days) as it has a really low promotion rate.
This results in the JVM's total direct size constantly increasing (which is
effectively a native buffer leak).

Has anyone come across this issue before?

There are a few obvious ways to mitigate this, e.g., cause a Full GC /
concurrent GC cycle at regular intervals. However, the best solution IMHO
is to explicitly free any direct buffers that are still in the cache when a
thread exits.

I'll be happy to implement this and test it internally at Twitter, if it's
not on anyone else's radar. However, to do what I'm proposing I need some
sort of thread exit hook. Unfortunately, there doesn't seem to be one.

Would proposing to introduce thread exit hooks be reasonable? Or is
everyone going to freak out? :-) The hooks can be either per-Thread or even
per-ThreadLocal. And it's OK if the hooks can only be used within java.base.

FWIW, I did a simple prototype of this (I call the hooks from Thread::exit)
and it seems to work as expected.

Any thoughts / feedback on this will be very appreciated.

Thanks,

Tony



—————
Tony Printezis | @TonyPrintezis | tprintezis at twitter.com