ClassFileTransformer does not apply to anonymous classes

Tue Jan 26 12:31:13 UTC 2016

Hi Remi,

this is what I am doing. At least using Byte Buddy, you turn a switch
such that Byte Buddy redefines the LambdaMetafactory to spin custom
classes where Byte Buddy also applies a ClassFileTransformer.
Currently this results in the (almost) exact class file as if the
default LambdaMetafactory had applied this transformation. Therefore,
I do not understand why it would be more harmful as a solution if this
was standard behavior.

As mentioned, one problem that I see with my solution is that I have
to code against the Unsafe API what I would like to avoid. The class
is surely being removed at some point and then I would already need to
implement a version sensitive instrumentation what, in the long run,
might result in unstable behavior. At the same time, I could not fall
back to public API class loading as it is impossible to regularly load
a class that invokes the private method containing the lambda
expression's method body.

Also, spinning your own LambdaMetafactory is quite an effort if other
people would want to do it without the help of Byte Buddy. Especially
exception handling is difficult to get right (and to test) and
supporting lambda serialization is another bigger task to get done
correctly. In my view, it would therefore be desireable if the JVM
could natively handle instrumentation (as I would also expect it from
the contract of the Instrumentation API that does not even mention
this).

Best regards, Rafael

2016-01-26 13:03 GMT+01:00  <forax at univ-mlv.fr>:
> Hi Rafael,
> why not providing a LambdaInterceptor as asked ?
>
> My point is that a lambda should be considered as a feature by itself instead as a way to create a class that implement an interface at runtime.
> If you do that, your code will still work when future release (maybe a point release), the implementation of the lambda proxy class will change.
>
> BTW, i maybe wrong, but as far as remember, the first release of Java 8 was not using vm anonymous classes for lambdas.
>
> regards,
> Rémi
>
> ----- Mail original -----
>> De: "Rafael Winterhalter" <rafael.wth at gmail.com>
>> À: "Rémi Forax" <forax at univ-mlv.fr>
>> Cc: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>, "Coleen Phillimore" <coleen.phillimore at oracle.com>,
>> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com Spitsyn" <serguei.spitsyn at oracle.com>, "Daniel
>> Daugherty" <daniel.daugherty at oracle.com>
>> Envoyé: Mardi 26 Janvier 2016 10:30:54
>> Objet: Re: ClassFileTransformer does not apply to anonymous classes
>>
>> Maybe this is a more graphic example of a problem that end-users
>> currently face:
>> http://stackoverflow.com/questions/33912026/intercepting-calls-to-java-8-lambda-expressions-using-byte-buddy
>>
>> 2016-01-26 10:02 GMT+01:00  <forax at univ-mlv.fr>:
>> > Hello,
>> >
>> > ----- Mail original -----
>> >> De: "Rafael Winterhalter" <rafael.wth at gmail.com>
>> >> À: "Remi Forax" <forax at univ-mlv.fr>
>> >> Cc: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>, "Coleen Phillimore"
>> >> <coleen.phillimore at oracle.com>,
>> >> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com Spitsyn"
>> >> <serguei.spitsyn at oracle.com>, "Daniel
>> >> Daugherty" <daniel.daugherty at oracle.com>
>> >> Envoyé: Lundi 25 Janvier 2016 20:24:01
>> >> Objet: Re: ClassFileTransformer does not apply to anonymous classes
>> >>
>> >> Hi Vladmir, hello Remi,
>> >>
>> >> what bothers me about instrumenting a lambda expression's target
>> >> method is the difficulty of locating the method that contains the code
>> >> and setting it into context. This is a lot of work to do since one
>> >> needs to first locate any invokedynamic call sites and interpret the
>> >> connection via the LambdaMetafactory. This is difficult to instrument
>> >> without greater efforts and does not feel like a clean solution.
>> >>
>> >> Many real life instrumentations are based on the assumption of being
>> >> able to instrument any class of a given subtype or other any other
>> >> class property. For example, one would want to instrument any class
>> >> that implements InputStream to discover a resource leak. If the
>> >> interface is however functional and implemented as a lambda
>> >> expression, a tool that manipulating all class files of this type
>> >> stops working. At the same time, end users do not really understand
>> >> the difference of their former anonymous class that is now expressed a
>> >> lambda expression and perceive the beaviour as a regression.
>> >
>> > The lambda body is analyzed because the body is translated to a static
>> > method (which can be non static BTW) which is part of the class that uses
>> > the lambda.
>> > The lambda proxy is a runtime artifact and the way it is currently
>> > implemented will likely to change in the future, using a vm anonymous
>> > class is a workaround of the fact that there is currently no way to load a
>> > function/method without a class. You should not try to analyze the lambda
>> > proxy, it doesn't contain codes but it's a straw glue between the
>> > interface and the body of the lambda.
>> >
>> >>
>> >> I have myself implemented a custom solution for my code generation
>> >> library that instruments the LambdaMetafactory to apply class file
>> >> transformers to these classes manually. This does however show two
>> >> disadvantages:
>> >>
>> >> 1. javac creates the target method of a lambda expression as a private
>> >> method. The only way to implement a lambda expression legally in byte
>> >> code is by loading the generated implementation anonymously what
>> >> requires non-public API. While it is possible to call
>> >> Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory,
>> >> it is again required to use non-standardized APIs which might fail on
>> >> differing implementations of the JVM.
>> >
>> > The invokedynamic corresponding to a lambda creation contains a link to the
>> > private method, it's i believe all you need, but i'm not sure because what
>> > you're doing exactly is fuzy for me.
>> >
>> >>
>> >> 2. At the same time, there is no standardized way to receive all
>> >> ClassFileTransformers that are currently registered on the VM. this
>> >> also requires calls into internal APIs that are not necessarily
>> >> supported on other platforms.
>> >>
>> >> I fully understand the hesitation to support this from a technical
>> >> point of view but in reality, people are already dependant on this
>> >> feature and disallowing the instrumentation of lambda classes will
>> >> only inspire work-arrounds that reduce the stability of software using
>> >> this APIs that does currently work without problems for non-lambda
>> >> classes.
>> >
>> > again, there is no such thing as a lambda class, trying to see a lambda as
>> > an anonymous class will never fully work.
>> > a lambda is a lambda, you have the link between the functional interface
>> > and the lambda body encoded into the invokedynamic which is used to create
>> > the lambda, you should not try to get more information, because as i said,
>> > it may change.
>> >
>> >>
>> >> Best regards, Rafael
>> >>
>> >> PS: While implementing my solution, I found that the LambdaMetafactory
>> >> of the Open JDK creates private, final methods for the serialization
>> >> bits. The methods should however not be final as they are already
>> >> private.
>> >
>> > yes, private is enough but it's not a big deal.
>> >
>> >>
>> >
>> > regards,
>> > Rémi
>> >
>> >>
>> >> 2016-01-23 13:55 GMT+01:00 Remi Forax <forax at univ-mlv.fr>:
>> >> > I agree with Vladimir,
>> >> >
>> >> > You should not be able to transform/redefine a VM anonymous class
>> >> > because
>> >> > the transformer will certainly mess up with the order of the constant
>> >> > pool
>> >> > entries.
>> >> >
>> >> > Slightly off-topic, about ASM, when you create a ClassWriter [1], you
>> >> > can
>> >> > pass a ClassReader of an existing class, in that case ASM copy the
>> >> > constant pool from the class reader to the class writer so the constant
>> >> > pool is preserved.
>> >> >
>> >> > Rémi
>> >> >
>> >> > [1]
>> >> > http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29
>> >> >
>> >> > ----- Mail original -----
>> >> >> De: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>
>> >> >> À: "Rafael Winterhalter" <rafael.wth at gmail.com>
>> >> >> Cc: "Coleen Phillimore" <coleen.phillimore at oracle.com>,
>> >> >> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com
>> >> >> Spitsyn" <serguei.spitsyn at oracle.com>, "Daniel Daugherty"
>> >> >> <daniel.daugherty at oracle.com>
>> >> >> Envoyé: Vendredi 22 Janvier 2016 18:47:31
>> >> >> Objet: Re: ClassFileTransformer does not apply to anonymous classes
>> >> >>
>> >> >> Rafael,
>> >> >>
>> >> >> First of all, I'd like to agree on the terminology. There's some
>> >> >> confusion there. Anonymous term is ambiguous in Java. There are
>> >> >> anonymous classes on language level and there's
>> >> >> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
>> >> >> classes. I prefer to call them VM anonymous classes to avoid confusion.
>> >> >>
>> >> >> I assume that whenever you use anonymous you assume "VM anonymous".
>> >> >>
>> >> >> > thank you for your response. While I completely understand your view
>> >> >> > from a VM implementor's point of view, as a practicioner I would
>> >> >> > recommend against it. Not being able to instrument lambda expressions
>> >> >> > puts a severe limitation onto using the instrumentation API
>> >> >> > alltogether. For example, a monitoring application that promises to
>> >> >> > record all invocations of a method of a certain interface that only
>> >> >> > works 95% of the time is not 5% less usefull but might no longer be
>> >> >> > useful at all. People have build large applications based on the
>> >> >> > assumption that all user application code can be instrumented and
>> >> >> > such
>> >> >> > a regression would hurt them. For example, until today, over 30
>> >> >> > people
>> >> >> > that use my code generation framework reached out to me and reported
>> >> >> > the reported behavior as a bug in my library and people are only
>> >> >> > starting to migrate their applications to Java 8. The outcome is
>> >> >> > simply not intuitive as using a lambda expression over an anonyous
>> >> >> > inner class should not change an application's behavior.
>> >> >> Can you elaborate on that point, please?
>> >> >>
>> >> >> I don't see any problems with instrumenting user code. Javac represents
>> >> >> lambda expression body as a private static method of the enclosing
>> >> >> class, which can be instrumented. VM anonymous classes are used only as
>> >> >> a mechanism to glue functional interfaces and lambda expressions
>> >> >> together at runtime.
>> >> >>
>> >> >> For example:
>> >> >>     static void f(Runnable r) { r.run(); }
>> >> >>     f(() -> {});
>> >> >>
>> >> >> Test::f (7 bytes)
>> >> >>    @ 1   Test$$Lambda$1/791452441::run (4 bytes)   inline (hot)
>> >> >>      @ 0   Test::lambda$main$0 (1 bytes)   inline (hot)
>> >> >>
>> >> >> Why do you need to instrument Test$$Lambda$1/... and not
>> >> >> Test::lambda$main$0?
>> >> >>
>> >> >> > The currently used workaround that people use is to instrument the
>> >> >> > LambdaMetaFactory class itself to apply the transformer manually when
>> >> >> > it is being created. This solution is spreading as a standard
>> >> >> > approach
>> >> >> > and I am sure that forbidding a redefinition alltogether would only
>> >> >> > inspire to other workarrounds for being able to migrate running code
>> >> >> > to the next Java version.
>> >> >> It doesn't sound like a workaround, but as a solution for a different
>> >> >> problem.
>> >> >>
>> >> >> If there's a need to gather profile for every lambda expression
>> >> >> instantiation site (indy call), then redefining bootstrap method is the
>> >> >> right way to go IMO. It allows to intercept and customize indy call
>> >> >> site
>> >> >> binding.
>> >> >>
>> >> >> What does sound as a workaround is an attempt to instrument classes
>> >> >> produced by LambdaMetaFactory. Right now, it generates a fresh VM
>> >> >> anonymous class on every request, but it is an implementation detail
>> >> >> and
>> >> >> not a requirement.
>> >> >>
>> >> >> For example, LambdaMetaFactory can reuse wrappers on per-functional
>> >> >> interface basis. It would provide significant footprint savings for
>> >> >> lambda expression-rich code bases (usually, there are much more
>> >> >> instantiations than functional interface flavors).
>> >> >>
>> >> >> > Furthermore, I do not think that not being able to patch constant
>> >> >> > pool
>> >> >> > indices in the generated code introduces any problems in practice.
>> >> >> > Most real-world instrumentation only appends new constant pool
>> >> >> > entries
>> >> >> > without interfering with the existant pool. Rather, I would like to
>> >> >> > see an exception being raised if one attempts to change the original
>> >> >> > constant pool without me being able to consider this from a technical
>> >> >> > perspective. I think this would serve to be sufficient for most
>> >> >> > people.
>> >> >> If you care only about lambda expressions, then constant pool patching
>> >> >> shouldn't be a problem (AFAIR it isn't used for lambda expressions).
>> >> >> But
>> >> >> in a more generic case (even for java.lang.invoke purposes), the
>> >> >> coupling between class file and CP patches is very tight and can be
>> >> >> easily broken with benign class file transformations. Considering ASM
>> >> >> library, I'm not sure that even simple instrumentations preserve
>> >> >> original constant pool structure.
>> >> >>
>> >> >> Probably, VM can do some structural checks on CP to forbid any
>> >> >> modifications except appends, but there are still aliasing problems -
>> >> >> sharing doesn't work (even for string constants), so new code shouldn't
>> >> >> use original CP entries.
>> >> >>
>> >> >> I'd prefer to avoid such increase in complexity in VM implementation.
>> >> >>
>> >> >> > I really hope that there is a way to allow for patching anonymously
>> >> >> > loaded classes even without exposing the constant pool patches.
>> >> >>  From VM implementation perspective I don't see any problems with
>> >> >> allowing class file redefinition for VM anonymous classes. What I want
>> >> >> to do is to discourage from doing so. Again - it is very fragile and
>> >> >> there's little we can do to fix that.
>> >> >>
>> >> >> > So far, thank you for looking into this. I am sure this a more
>> >> >> > complex
>> >> >> > matter than what I am able to comprehend. I appreciate that you are
>> >> >> > taking the time to consider my opinion!
>> >> >>
>> >> >> Best regards,
>> >> >> Vladimir Ivanov
>> >> >>
>> >>
>>