RFR: 8308762: Metaspace leak with Instrumentation.retransform

[Jean-Philippe Bempel]

On Fri, 7 Jul 2023 00:46:09 GMT, David Holmes <dholmes at openjdk.org> wrote:

> these comments both pertain to the problem at hand: the merge reverts the resolved class entry to be unresolved; unresolved entries are not considered matches for resolved ones, hence we grow the constant pool.

And If if I am reading this correctly from `VM_RedefineClasses::merge_constant_pools`:


      bool match = scratch_cp->compare_entry_to(scratch_i, *merge_cp_p, scratch_i);
      if (match) {
        // found a match at the same index so nothing more to do
        continue;
      } else if (is_unresolved_class_mismatch(scratch_cp, scratch_i,
                                              *merge_cp_p, scratch_i)) {
        // The mismatch in compare_entry_to() above is because of a
        // resolved versus unresolved class entry at the same index
        // with the same string value. Since Pass 0 reverted any
        // class entries to unresolved class entries in *merge_cp_p,
        // we go with the unresolved class entry.
        continue;
      }

In a case of `JVM_CONSTANT_Class`/`JVM_CONSTANT_UnresolvedClass` this is handled specially with is_unresolved_class_mismatch.

But the same problem rises again when dealing with `JVM_CONSTANT_Methodref` and al. because the unresolved state contains also unresolved class for the scratch while for old_cp it is a resolved one.
The idea behind my fix proposal is to do the same as for `JVM_CONSTANT_Class`.
But I can understand your concern that introducing this in a method that could be used outside of the code of class redefintion is worrisome.
I can try to find another way to do the same but specifically in the context of class redefinition.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14780#issuecomment-1632587401