RFR: 8308762: Metaspace leak with Instrumentation.retransform

[Jean-Philippe Bempel]

On Fri, 7 Jul 2023 00:34:59 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Fix a small leak in constant pool merging during retransformation of a class. If this class has a catch block with `Throwable`, the class `Throwable` is pre-resolved in the constant pool, while all the other classes are in a unresolved state. So the constant pool merging process was considering the entry with pre-resolved class as different compared to the destination and create a new entry. We now try to consider it as equal specially for Methodref/Fieldref.
>
> src/hotspot/share/oops/constantPool.cpp line 1361:
> 
>> 1359:     int recur2 = cp2->uncached_klass_ref_index_at(index2);
>> 1360:     bool match = compare_entry_to(recur1, cp2, recur2);
>> 1361:     match |= is_unresolved_class_mismatch(recur1, cp2, recur2);
> 
> This is changing the definition of a "match" such that in all cases an unresolved class entry is considered equal to a resolved class entry - but only for these Field and MethodRefs. I don't see how this connects back to the original problem. Nor do I see why this is alway a correct thing to do.

I agree that it may be not the right place to call is_unresolved_class_mismatch method because we capture all cases calling compare_entry_to. For a JVM_CONSTANT_Class, compare_entry_to is recovered by the call to is_unresolved_class_mismatch in VM_RedefineClasses::merge_constant_pools. The idea is to do the same for other reference to the class in FieldRefs and (Interface)MethodRefs because otherwise the entry does not match with pre-resolved Throwable class.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14780#discussion_r1261120394