RFR: 7124710: interleaved RedefineClasses() and RetransformClasses() calls may have a problem [v2]

[Serguei Spitsyn]

On Wed, 31 Aug 2022 21:28:24 GMT, Alex Menkov <amenkov at openjdk.org> wrote:

>> The problem is RedefineClasses does not update cached_class_bytes, so subsequent RetransformClasses gets obsolete class bytes (this are testcases 3-6 from the new test)
>> 
>> cached_class_bytes are set when an agent instruments the class from ClassFileLoadHook.
>> After successful RedefineClasses it should be reset.
>> The fix updates ClassFileLoadHook caller to not use old cached_class_bytes with RedefineClasses (if some agent instruments the class, new cached_class_bytes are allocated for scratch_class) and updates cached_class_bytes after successful RedefineClasses or RetransformClasses.
>
> Alex Menkov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   updated comments

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/libRedefineRetransform.cpp line 179:

> 177:     if (err != JVMTI_ERROR_NONE) {
> 178:         _log("nRedefine: SetEventNotificationMode(JVMTI_DISABLE) error %d\n", err);
> 179:     }

It makes sense to introduce an utility function which does SetEventNotificationMode.
It can be two separate functions to enable and disable or one single function can support both cases.

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/libRedefineRetransform.cpp line 195:

> 193:         _log("nRedefine: classLoadHookSavedClassBytes is NULL\n");
> 194:         return nullptr;
> 195:     }

The checks 181-195 seems to be the same in functions nRedefine and nRetransform,
so one utility function can be used in both cases.

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/libRedefineRetransform.cpp line 274:

> 272:     classLoadHookSavedClassBytes = nullptr;
> 273: 
> 274:     return result;

The fragments 197-214 and 256-274 do the same.
I'd suggest to define and use a function that does this post-processing.

-------------

PR: https://git.openjdk.org/jdk/pull/10032