RFR: 7124710: interleaved RedefineClasses() and RetransformClasses() calls may have a problem [v2]

[Serguei Spitsyn]

On Wed, 31 Aug 2022 21:28:24 GMT, Alex Menkov <amenkov at openjdk.org> wrote:

>> The problem is RedefineClasses does not update cached_class_bytes, so subsequent RetransformClasses gets obsolete class bytes (this are testcases 3-6 from the new test)
>> 
>> cached_class_bytes are set when an agent instruments the class from ClassFileLoadHook.
>> After successful RedefineClasses it should be reset.
>> The fix updates ClassFileLoadHook caller to not use old cached_class_bytes with RedefineClasses (if some agent instruments the class, new cached_class_bytes are allocated for scratch_class) and updates cached_class_bytes after successful RedefineClasses or RetransformClasses.
>
> Alex Menkov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   updated comments

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/RedefineRetransform.java line 76:

> 74:     private static byte[] initialClassBytes;
> 75: 
> 76:     private static class VersionScanner extends ClassVisitor {

This class needs some explanation.
It is unclear how does it work.
Could you, please, add relevant comments where needed?
For instance, how are the class file bytes constructed and what role does the version play.

test/hotspot/jtreg/serviceability/jvmti/RedefineClasses/RedefineRetransform/RedefineRetransform.java line 236:

> 234:                 redefine(2, 5);    // updates cached class bytes
> 235:                 retransform(3, 2);
> 236:                 retransform(4, 2);

The comments for all test cases need to be aligned.
Also, it is better to comment all redefine/retransform cases.

-------------

PR: https://git.openjdk.org/jdk/pull/10032