RFR: 8284632: runtime/Thread/StopAtExit.java possibly leaking memory again

[David Holmes]

On Mon, 25 Apr 2022 22:12:38 GMT, Daniel D. Daugherty <dcubed at openjdk.org> wrote:

> Throwing async exceptions at exiting JavaThreads can leak the exception:
> 
> 1) HandshakeOperation::do_handshake() recognizes that the target thread
>     might be terminated, but needs to do some cleanup when the operation
>     is one that installs an async exception.
> 
> 2) JavaThread::exit() uses a NoAsyncExceptionDeliveryMark to protect the
>     VM's call out to Thread::exit() from async exceptions, but it needs to do
>     some cleanup when those async exceptions cannot be delivered because
>     the target thread has logically exited.
> 
> Also tweaked runtime/Thread/StopAtExit.java to alternate between throwing
> RuntimeException and ThreadDeath. The async exception queuing code accepts
> any exception, but it recognizes ThreadDeath as special. When a target thread
> already has a queued async ThreadDeath exception, then another exception will
> not be queued. So the test now throws RuntimeException on odd iterations and
> ThreadDeath on even iterations and the target thread will have at most two async
> exceptions queued up.
> 
> This fix has been tested with Mach5 Tier[1-7] (about to start a Tier8) and I also ran
> my StressWrapper_StopAtExit.java test using {release, fastdebug, and slowdebug}
> bits for 12 hours per config. No leaks detected. Previously, release bits would
> fail with OutOfMemoryException in ~2 minutes with StressWrapper_StopAtExit.java.

I think this overlaps somewhat with what @pchilano said but it seems to me that we are missing a cleanup of the handshake operation queue when a thread terminates - regardless of the nature of the op. Once we will no longer process handshake requests the queue should be cleared and each op deleted so its necessary cleanup is done.

-------------

PR: https://git.openjdk.java.net/jdk/pull/8388