RFR: 7124710: interleaved RedefineClasses() and RetransformClasses() calls may have a problem [v2]

[Serguei Spitsyn]

On Wed, 31 Aug 2022 21:28:24 GMT, Alex Menkov <amenkov at openjdk.org> wrote:

>> The problem is RedefineClasses does not update cached_class_bytes, so subsequent RetransformClasses gets obsolete class bytes (this are testcases 3-6 from the new test)
>> 
>> cached_class_bytes are set when an agent instruments the class from ClassFileLoadHook.
>> After successful RedefineClasses it should be reset.
>> The fix updates ClassFileLoadHook caller to not use old cached_class_bytes with RedefineClasses (if some agent instruments the class, new cached_class_bytes are allocated for scratch_class) and updates cached_class_bytes after successful RedefineClasses or RetransformClasses.
>
> Alex Menkov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   updated comments

Alex,
My apology for latency with review.
The fix itself looks good to me.
Thank you for your good analysis and fix!
I've posted several comments on new test.
It feels like more comments are needed.
Could you, please, add them where you consider it relevant?
Also, I'm curious how did you verify that no regressions have been introduced?

Thanks,
Serguei

-------------

Marked as reviewed by sspitsyn (Reviewer).

PR: https://git.openjdk.org/jdk/pull/10032