RFR: 7124710: interleaved RedefineClasses() and RetransformClasses() calls may have a problem [v5]

[Serguei Spitsyn]

On Mon, 12 Sep 2022 20:30:04 GMT, Alex Menkov <amenkov at openjdk.org> wrote:

>> The problem is RedefineClasses does not update cached_class_bytes, so subsequent RetransformClasses gets obsolete class bytes (this are testcases 3-6 from the new test)
>> 
>> cached_class_bytes are set when an agent instruments the class from ClassFileLoadHook.
>> After successful RedefineClasses it should be reset.
>> The fix updates ClassFileLoadHook caller to not use old cached_class_bytes with RedefineClasses (if some agent instruments the class, new cached_class_bytes are allocated for scratch_class) and updates cached_class_bytes after successful RedefineClasses or RetransformClasses.
>
> Alex Menkov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   s/verity/verify/

I've posted one minor comment.
Otherwise, it is good.
Thanks,
Serguei

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/libRedefineRetransform.cpp line 222:

> 220:     caps.can_retransform_classes = 1;
> 221:     res = jvmti->AddCapabilities(&caps);
> 222:     if (res != JVMTI_ERROR_NONE) {

The type of res is jint. It is better to use a variable of type jvmtiError. I've missed this initially.

-------------

Marked as reviewed by sspitsyn (Reviewer).

PR: https://git.openjdk.org/jdk/pull/10032