RFR: 8287982: Concurrent implicit attach from native threads crashes VM

[Alan Bateman]

If several native threads attach to the VM at around the same time, and before any threads get an automatically generated name are created, then the VM may crash attempting to access the thread status. The issue exists for native threads that attach explicitly with JNI AttachCurrentThread without a thread name, or native threads that attach implicitly by using a function pointer to do an up call.

The issue that raises its head periodically is that native threads that JNI attach do the initializaiton of the Thread object in the context of the attaching thread. Great care must be taken because Java code is executing in the context of a Thread that is not fully initialized. The right thing is probably to create the Thread object in another thread, using the service thread has been mentioned. The issue at this time arises when two or more native threads attempt to attach without thread names at around the same time. The first thread that needs an automatically generated name triggers the loading and initialization of a helper class.  If there are other threads attaching at the same time then they may have to wait on the monitor which can trigger the crash because the field holder with the thread status is not created at this time. Crashes in monitor enter and notify have been observed. Coleen has changed this code so that linking and initialization uses a mutex (JDK-8288064) so t
 his specific crash doesn't duplicate in the main line. The short term fix for openjdk/jdk19 is to reorder the initialization so that field holder with the thread status is created before setting the name.

Creating a jtreg test with the conditions to duplicate this issue is complicated. The jtreg main wrapper creates the main thread with an automatically generated thread name before it runs the test main method. This is the reason that the test needs to launch a new VM with the right setup to exercise both explicit and implicit attach.

-------------

Commit messages:
 - Move -ljvm to LIBS
 - Swap link options
 - Update test
 - Cleanup
 - Merge
 - Initial version

Changes: https://git.openjdk.org/jdk19/pull/28/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk19&pr=28&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8287982
  Stats: 402 lines in 7 files changed: 364 ins; 21 del; 17 mod
  Patch: https://git.openjdk.org/jdk19/pull/28.diff
  Fetch: git fetch https://git.openjdk.org/jdk19 pull/28/head:pull/28

PR: https://git.openjdk.org/jdk19/pull/28