ClassFileTransformer does not apply to anonymous classes

[Vladimir Ivanov]

Rafael,

What you are seeing are just consequences. My impression is that VM 
anonymous class redefinition/retransformation works mostly by accident.

The real problem is that current API (both JVMTI and 
java.lang.instrument) doesn't serve for them well.

When VM anonymous class is defined, a user provides both bytecode and CP 
patches:
     public native Class<?> defineAnonymousClass(Class<?> hostClass, 
byte[] data, Object[] cpPatches);

Unless there's a way to adjust CP patches along with the bytecode, 
patching bytecode is very fragile (e.g., no way to add new patches, 
should keep patched CP entry indexes intact).

The root cause is Unsafe.defineAnonymousClass is part of 
implementation-specific private API, so no way to expose it a public 
APIs (special entry points in JVMTI or java.lang.instrument).

Also, though the concept of VM anonymous class was a step in the right 
direction, it isn't good enough to be standardized. The ultimate goal is 
to get lightweight code loading mechanism, but VM anonymous classes are 
still loaded with class semantics.

So, I don't think we can do much w.r.t. VM anonymous classes. I'd prefer 
JVM to skip agent notifications or completely forbidding VM anonymous 
class retransformation/redefinition, but I haven't thought through 
compatibility consequences.

Best regards,
Vladimir Ivanov

On 1/22/16 5:04 PM, Rafael Winterhalter wrote:
> Hi Coleen, thanks for looking into this. My original mail was the following:
>
> Hello everybody,
>
> classes that are loaded via Unsafe::defineAnonymousClass are not
> transformed by a registered ClassFileTransformer. At the same time, it
> is possible to retransform / redefine such an anonymous classes using
> the instrumentation API.
>
> Here is a rather confusing bug that I encountered when working with an
> internally used Java agent after upgrading a code base to Java 8:
>
> The Java agent attaches at runtime (agentmain). It then registers a
> ClassFileTransformer that also applies for retransformation.
> Afterwards, all loaded classes that fullfil a given condition are
> explicitly registered to be retransformed by the agent. (Doing so,
> anonymous classes are returned by
> Instrumentation::getAllLoadedClasses.) This resulted in the following
> behavior of the agent:
>
> a) If the agent was attached after an anonymous class was loaded, the
> retransformation would apply to an anonymous class.
> b) If the agent was attached "too early", the (non-re-)transformation
> would not apply to an anonymous class.
>
> I wonder if it is intentional that a ClassFileTransformer is not
> applied when an anonymous class is loaded. Personally, I find this
> counter-intuitive, especially when converting anonymous inner classes
> to lambda expressions where many users of instrumentation do not
> forsee the behavioral change. It also puts a very unforseeable limit
> to the instrumentation API. I would therefore like to suggest that
> ClassFileTransformers are also applied to anonymous classes when
> Unsafe::defineAnonymousClass is called just as when going via
> ClassLoader::defineClass.
>
> I can tell that this behavior has not only affected me as I had this
> question comming up by multiple users of my open-source code
> generation library.
>
> What is your view on this?
>
> Thank you for your feedback!
> Best regards, Rafael
>
> 2016-01-22 15:00 GMT+01:00 Coleen Phillimore <coleen.phillimore at oracle.com>:
>>
>>
>> On 1/22/16 4:11 AM, Andrew Dinn wrote:
>>>
>>> On 21/01/16 22:14, Rafael Winterhalter wrote:
>>>>
>>>> Hi Andrew,
>>>> if there is any update on the matter, I would of course love to hear
>>>> about it.
>>>> Unfortunately, I never received an answer to my question, but maybe I
>>>> picked the wrong list.
>>>> Thank you for your support on this issue!
>>>
>>> I'm pretty sure this is the right list. But I don't know who cold answer
>>> the question. I know that Coleen Phillmore (added explicitly to CC)
>>> often deals with ClassFileTransformer issues. Coleen, can you answer?
>>
>>
>> Can you send the question again?  I didn't see it.  I also mostly look at
>> JVM code and rarely deal with the Java side.
>>
>> I'm also adding Dan, Serguei and Markus.
>>
>> Thanks,
>> Coleen
>>
>>>
>>> regards,
>>>
>>>
>>> Andrew Dinn
>>> -----------
>>> Senior Principal Software Engineer
>>> Red Hat UK Ltd
>>> Registered in UK and Wales under Company Registration No. 3798903
>>> Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul
>>> Argiry (US)
>>
>>