RFR: 8274298: JFR Thread Sampler thread must not acquire malloc lock after suspending a thread because of possible deadlock

[Markus Grönlund]

Greetings,

[JDK-8233705](https://bugs.openjdk.java.net/browse/JDK-8233705) introduced a means to only iterate the incrementally tagged Klasses, instead of the entire set of Klasses in the JVM, as part of serializing metadata information. The newly tagged Klasses are enqueued onto a thread-local buffer using a load barrier. If the buffer runs out of space, a new buffer is accommodated, either from a free list or via a new allocation.

This is problematic for the JfrThreadSampler because the thread it has suspended could hold the malloc lock. For example, this could be the case if the suspended thread was in the process of deoptimization. The Deoptimization::UnrollBlock is a CHeapObj, and the vframes arrays use the NEW_C_HEAP_ARRAY macro.

The solution to this problem is to explicitly monitor the size of the thread-local buffer of the JfrThreadSampler thread, and pre-emptively renew it before thread suspension.

Testing: jdk_jfr, stress testing

Thanks
Markus

-------------

Commit messages:
 - newline
 - no malloc by JfrThreadSampler

Changes: https://git.openjdk.java.net/jdk/pull/5977/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5977&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8274298
  Stats: 93 lines in 9 files changed: 84 ins; 2 del; 7 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5977.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5977/head:pull/5977

PR: https://git.openjdk.java.net/jdk/pull/5977