RFR: 8277444: Race condition on Instrumentation.retransformClasses() and class linking

[Evgeny Astigeevich]

On Wed, 20 Aug 2025 15:43:03 GMT, Evgeny Astigeevich <eastigeevich at openjdk.org> wrote:

>> There is a race between `JvmtiClassFileReconstituter::copy_bytecodes` and `InstanceKlass::link_class_impl`.  `InstanceKlass::link_class_impl` can be rewriting bytecodes. `JvmtiClassFileReconstituter::copy_bytecodes` will not restore them to the original ones because the flag `rewritten` is `false`. This will result in invalid bytecode.
>> 
>> This PR adds a lock (`init_lock`) to the `copy_bytecodes` method to prevent reading bytecodes while they are being rewritten during class linking.
>> 
>> Tested fastdebug and release builds: Linux x86_64 and arm64
>> - The reproducer from JDK-8277444 passed.
>> - Tier1 - tier3 passed.
>
> Hi @coleenp,
> Could you please take a look?

> @eastig I am not sure about this one. Can you clarify please how you can be transforming a class that has not yet been linked? If this is possible then it seems to me we are missing a call to ensure linkage.

Hi @dholmes-ora,

I checked what was happening.

The reproducer from JDK-8277444 simplified:
- Thread 1 does:

bigClass = Class.forName();
consumer.accept(bigClass); // Puts bigClass in QUEUE
final Object o = bigClass.getConstructor().newInstance();
System.out.println(o.hashCode());


- Thread 2 does:

final Class<?> aClass = QUEUE.take();
Instrumentation.retransformClasses(aClass); 


`Class.forName` does not link `bigClass`. So an unlinked class is put in a queue. The linking process starts when we start using `bigClass`.
Thread 2 gets unlinked `bigClass` from the queue. It can request to retransform  before we start using it and the linking process starts.

So we can have the linking process and the retransforming process running in parallel. There is no synchronization between them. We get a race condition. `bigClass` is big enough to make the linking process running long.

I think `Class.forName` does not do linking intentionally, for performance reasons.

I hope I've got everything correctly from logs and sources.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26863#issuecomment-3212401838