RFR: 8274298: JFR Thread Sampler thread must not acquire malloc lock after suspending a thread because of possible deadlock [v4]

[Markus Grönlund]

> Greetings,
> 
> [JDK-8233705](https://bugs.openjdk.java.net/browse/JDK-8233705) introduced a means to only iterate the incrementally tagged Klasses, instead of the entire set of Klasses in the JVM, as part of serializing metadata information. The newly tagged Klasses are enqueued onto a thread-local buffer using a load barrier. If the buffer runs out of space, a new buffer is accommodated, either from a free list or via a new allocation.
> 
> This is problematic for the JfrThreadSampler because the thread it has suspended could hold the malloc lock. For example, this could be the case if the suspended thread was in the process of deoptimization. The Deoptimization::UnrollBlock is a CHeapObj, and the vframes arrays use the NEW_C_HEAP_ARRAY macro.
> 
> The solution to this problem is to explicitly monitor the size of the thread-local buffer of the JfrThreadSampler thread, and pre-emptively renew it before thread suspension.
> 
> Testing: jdk_jfr, stress testing
> 
> Thanks
> Markus

Markus Grönlund has updated the pull request incrementally with one additional commit since the last revision:

  hoist outside Threads_lock

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/5977/files
  - new: https://git.openjdk.java.net/jdk/pull/5977/files/a0cc0b42..1d638b70

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=03
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=02-03

  Stats: 17 lines in 1 file changed: 7 ins; 9 del; 1 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5977.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5977/head:pull/5977

PR: https://git.openjdk.java.net/jdk/pull/5977