RFR: 8274298: JFR Thread Sampler thread must not acquire malloc lock after suspending a thread because of possible deadlock [v2]

[Markus Grönlund]

> Greetings,
> 
> [JDK-8233705](https://bugs.openjdk.java.net/browse/JDK-8233705) introduced a means to only iterate the incrementally tagged Klasses, instead of the entire set of Klasses in the JVM, as part of serializing metadata information. The newly tagged Klasses are enqueued onto a thread-local buffer using a load barrier. If the buffer runs out of space, a new buffer is accommodated, either from a free list or via a new allocation.
> 
> This is problematic for the JfrThreadSampler because the thread it has suspended could hold the malloc lock. For example, this could be the case if the suspended thread was in the process of deoptimization. The Deoptimization::UnrollBlock is a CHeapObj, and the vframes arrays use the NEW_C_HEAP_ARRAY macro.
> 
> The solution to this problem is to explicitly monitor the size of the thread-local buffer of the JfrThreadSampler thread, and pre-emptively renew it before thread suspension.
> 
> Testing: jdk_jfr, stress testing
> 
> Thanks
> Markus

Markus Grönlund has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision:

 - Extensible API for JfrEpochQueue
 - Merge branch '8274298' of github.com:mgronlun/jdk into 8274298
 - newline
 - no malloc by JfrThreadSampler
 - newline
 - no malloc by JfrThreadSampler

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/5977/files
  - new: https://git.openjdk.java.net/jdk/pull/5977/files/8b31f3a9..fc3dbb61

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=00-01

  Stats: 950819 lines in 2703 files changed: 500220 ins; 435031 del; 15568 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5977.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5977/head:pull/5977

PR: https://git.openjdk.java.net/jdk/pull/5977