RFR: 8277444: Race condition on Instrumentation.retransformClasses() and class linking

[David Holmes]

On Thu, 21 Aug 2025 23:34:13 GMT, Evgeny Astigeevich <eastigeevich at openjdk.org> wrote:

> Class.forName does not link bigClass. 

Just to be clear `Class.forName(name, false, loader)` would not initialize the class; the default form would initialize and thus link the class. The `Class.forName` spec does not state whether or not it actually performs full linking in the absence of initialization, though the hotspot implementation does not. Note that the preparation phase of linking must have occurred to get the Class object.

The retransformation API's are somewhat vague about the exact state of a class to be retransformed - the class must be "modifiable" but that seems to be treated as a static rather than dynamic property ie. primitives, arrays, hidden classes are not modifiable - any other type of class instance is.

Bottom line is that I don't think any of the specifications help us here, so we need to look at implementation practicalities.

The linking code, uses the `init_lock` and assumes it cannot be interfered with in any way. I don't know either pieces of code well enough to say whether we could devise a lock-free protocol that would handle the current case; or whether we could use a VM mutex around the critical part of the linking code?

I don't like JVM TI having to know anything about the `init_lock` and I don't like seeing another place where we acquire the `init_lock` via an `ObjectLocker`, as that does not play nicely with the work to avoid pinning for virtual threads.

I'd like to hear from others more knowledgeable than myself in this area, unfortunately @coleenp  is on vacation and won't be back till next week.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26863#issuecomment-3219033001