RFR: 8274298: JFR Thread Sampler thread must not acquire malloc lock after suspending a thread because of possible deadlock [v3]

[Markus Grönlund]

> Greetings,
> 
> [JDK-8233705](https://bugs.openjdk.java.net/browse/JDK-8233705) introduced a means to only iterate the incrementally tagged Klasses, instead of the entire set of Klasses in the JVM, as part of serializing metadata information. The newly tagged Klasses are enqueued onto a thread-local buffer using a load barrier. If the buffer runs out of space, a new buffer is accommodated, either from a free list or via a new allocation.
> 
> This is problematic for the JfrThreadSampler because the thread it has suspended could hold the malloc lock. For example, this could be the case if the suspended thread was in the process of deoptimization. The Deoptimization::UnrollBlock is a CHeapObj, and the vframes arrays use the NEW_C_HEAP_ARRAY macro.
> 
> The solution to this problem is to explicitly monitor the size of the thread-local buffer of the JfrThreadSampler thread, and pre-emptively renew it before thread suspension.
> 
> Testing: jdk_jfr, stress testing
> 
> Thanks
> Markus

Markus Grönlund has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:

  Extensible API for JfrEpochQueue

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/5977/files
  - new: https://git.openjdk.java.net/jdk/pull/5977/files/fc3dbb61..a0cc0b42

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=02
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=01-02

  Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5977.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5977/head:pull/5977

PR: https://git.openjdk.java.net/jdk/pull/5977