RFR: 8277444: Race condition on Instrumentation.retransformClasses() and class linking

[David Holmes]

On Thu, 21 Aug 2025 23:34:13 GMT, Evgeny Astigeevich <eastigeevich at openjdk.org> wrote:

>> Hi @coleenp,
>> Could you please take a look?
>
>> @eastig I am not sure about this one. Can you clarify please how you can be transforming a class that has not yet been linked? If this is possible then it seems to me we are missing a call to ensure linkage.
> 
> Hi @dholmes-ora,
> 
> I checked what was happening.
> 
> The reproducer from JDK-8277444 simplified:
> - Thread 1 does:
> 
> bigClass = Class.forName();
> consumer.accept(bigClass); // Puts bigClass in QUEUE
> final Object o = bigClass.getConstructor().newInstance();
> System.out.println(o.hashCode());
> 
> 
> - Thread 2 does:
> 
> final Class<?> aClass = QUEUE.take();
> Instrumentation.retransformClasses(aClass); 
> 
> 
> `Class.forName` does not link `bigClass`. So an unlinked class is put in a queue. The linking process starts when we start using `bigClass`.
> Thread 2 gets unlinked `bigClass` from the queue. It can request to retransform  before we start using it and the linking process starts.
> 
> So we can have the linking process and the retransforming process running in parallel. There is no synchronization between them. We get a race condition. `bigClass` is big enough to make the linking process running long.
> 
> I think `Class.forName` does not do linking intentionally, for performance reasons.
> 
> I hope I've got everything correctly from logs and sources.

@eastig Thank you very much for that detailed analysis. An interesting scenario. I still find it somewhat suspect that we can transform an unlinked class and wonder if we should instead ensure it is linked first, rather than trying to coordinate the two pieces of code via the use of the init_lock. ?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26863#issuecomment-3213233511