Promptly freeing the per-thread cached direct buffers when a thread exits

[David Holmes]

Hi Tony,

On 6/04/2018 7:45 AM, Tony Printezis wrote:
> Hi all,
> 
> We recently hit another interesting issue with the NIO thread-local
> DirectByteBuffer cache. One of our services seems to create and drop

If it's in a ThreadLocal then aren't you just asking for thread-locals 
to be cleared at thread exit?

Cheers,
David

> threads at regular intervals. I assume this is due to a thread pool
> dynamically resizing itself.
> 
> Let's say a thread starts, lives long enough for its Thread object to be
> promoted to the old gen (along with its cached direct buffer), then exits.
> This will result in its cached direct buffer(s) being unreachable in the
> old gen and will only be reclaimed after the next full GC / concurrent GC
> cycle.
> 
> Interestingly, the service in question does concurrent GC cycles really
> infrequently (one every few days) as it has a really low promotion rate.
> This results in the JVM's total direct size constantly increasing (which is
> effectively a native buffer leak).
> 
> Has anyone come across this issue before?
> 
> There are a few obvious ways to mitigate this, e.g., cause a Full GC /
> concurrent GC cycle at regular intervals. However, the best solution IMHO
> is to explicitly free any direct buffers that are still in the cache when a
> thread exits.
> 
> I'll be happy to implement this and test it internally at Twitter, if it's
> not on anyone else's radar. However, to do what I'm proposing I need some
> sort of thread exit hook. Unfortunately, there doesn't seem to be one.
> 
> Would proposing to introduce thread exit hooks be reasonable? Or is
> everyone going to freak out? :-) The hooks can be either per-Thread or even
> per-ThreadLocal. And it's OK if the hooks can only be used within java.base.
> 
> FWIW, I did a simple prototype of this (I call the hooks from Thread::exit)
> and it seems to work as expected.
> 
> Any thoughts / feedback on this will be very appreciated.
> 
> Thanks,
> 
> Tony
> 
> 
> 
> —————
> Tony Printezis | @TonyPrintezis | tprintezis at twitter.com
>