ClassFileTransformer does not apply to anonymous classes

[forax at univ-mlv.fr]

Hi Rafael,
why not providing a LambdaInterceptor as asked ?

My point is that a lambda should be considered as a feature by itself instead as a way to create a class that implement an interface at runtime.  
If you do that, your code will still work when future release (maybe a point release), the implementation of the lambda proxy class will change.  

BTW, i maybe wrong, but as far as remember, the first release of Java 8 was not using vm anonymous classes for lambdas.

regards,
Rémi

----- Mail original -----
> De: "Rafael Winterhalter" <rafael.wth at gmail.com>
> À: "Rémi Forax" <forax at univ-mlv.fr>
> Cc: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>, "Coleen Phillimore" <coleen.phillimore at oracle.com>,
> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com Spitsyn" <serguei.spitsyn at oracle.com>, "Daniel
> Daugherty" <daniel.daugherty at oracle.com>
> Envoyé: Mardi 26 Janvier 2016 10:30:54
> Objet: Re: ClassFileTransformer does not apply to anonymous classes
> 
> Maybe this is a more graphic example of a problem that end-users
> currently face:
> http://stackoverflow.com/questions/33912026/intercepting-calls-to-java-8-lambda-expressions-using-byte-buddy
> 
> 2016-01-26 10:02 GMT+01:00  <forax at univ-mlv.fr>:
> > Hello,
> >
> > ----- Mail original -----
> >> De: "Rafael Winterhalter" <rafael.wth at gmail.com>
> >> À: "Remi Forax" <forax at univ-mlv.fr>
> >> Cc: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>, "Coleen Phillimore"
> >> <coleen.phillimore at oracle.com>,
> >> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com Spitsyn"
> >> <serguei.spitsyn at oracle.com>, "Daniel
> >> Daugherty" <daniel.daugherty at oracle.com>
> >> Envoyé: Lundi 25 Janvier 2016 20:24:01
> >> Objet: Re: ClassFileTransformer does not apply to anonymous classes
> >>
> >> Hi Vladmir, hello Remi,
> >>
> >> what bothers me about instrumenting a lambda expression's target
> >> method is the difficulty of locating the method that contains the code
> >> and setting it into context. This is a lot of work to do since one
> >> needs to first locate any invokedynamic call sites and interpret the
> >> connection via the LambdaMetafactory. This is difficult to instrument
> >> without greater efforts and does not feel like a clean solution.
> >>
> >> Many real life instrumentations are based on the assumption of being
> >> able to instrument any class of a given subtype or other any other
> >> class property. For example, one would want to instrument any class
> >> that implements InputStream to discover a resource leak. If the
> >> interface is however functional and implemented as a lambda
> >> expression, a tool that manipulating all class files of this type
> >> stops working. At the same time, end users do not really understand
> >> the difference of their former anonymous class that is now expressed a
> >> lambda expression and perceive the beaviour as a regression.
> >
> > The lambda body is analyzed because the body is translated to a static
> > method (which can be non static BTW) which is part of the class that uses
> > the lambda.
> > The lambda proxy is a runtime artifact and the way it is currently
> > implemented will likely to change in the future, using a vm anonymous
> > class is a workaround of the fact that there is currently no way to load a
> > function/method without a class. You should not try to analyze the lambda
> > proxy, it doesn't contain codes but it's a straw glue between the
> > interface and the body of the lambda.
> >
> >>
> >> I have myself implemented a custom solution for my code generation
> >> library that instruments the LambdaMetafactory to apply class file
> >> transformers to these classes manually. This does however show two
> >> disadvantages:
> >>
> >> 1. javac creates the target method of a lambda expression as a private
> >> method. The only way to implement a lambda expression legally in byte
> >> code is by loading the generated implementation anonymously what
> >> requires non-public API. While it is possible to call
> >> Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory,
> >> it is again required to use non-standardized APIs which might fail on
> >> differing implementations of the JVM.
> >
> > The invokedynamic corresponding to a lambda creation contains a link to the
> > private method, it's i believe all you need, but i'm not sure because what
> > you're doing exactly is fuzy for me.
> >
> >>
> >> 2. At the same time, there is no standardized way to receive all
> >> ClassFileTransformers that are currently registered on the VM. this
> >> also requires calls into internal APIs that are not necessarily
> >> supported on other platforms.
> >>
> >> I fully understand the hesitation to support this from a technical
> >> point of view but in reality, people are already dependant on this
> >> feature and disallowing the instrumentation of lambda classes will
> >> only inspire work-arrounds that reduce the stability of software using
> >> this APIs that does currently work without problems for non-lambda
> >> classes.
> >
> > again, there is no such thing as a lambda class, trying to see a lambda as
> > an anonymous class will never fully work.
> > a lambda is a lambda, you have the link between the functional interface
> > and the lambda body encoded into the invokedynamic which is used to create
> > the lambda, you should not try to get more information, because as i said,
> > it may change.
> >
> >>
> >> Best regards, Rafael
> >>
> >> PS: While implementing my solution, I found that the LambdaMetafactory
> >> of the Open JDK creates private, final methods for the serialization
> >> bits. The methods should however not be final as they are already
> >> private.
> >
> > yes, private is enough but it's not a big deal.
> >
> >>
> >
> > regards,
> > Rémi
> >
> >>
> >> 2016-01-23 13:55 GMT+01:00 Remi Forax <forax at univ-mlv.fr>:
> >> > I agree with Vladimir,
> >> >
> >> > You should not be able to transform/redefine a VM anonymous class
> >> > because
> >> > the transformer will certainly mess up with the order of the constant
> >> > pool
> >> > entries.
> >> >
> >> > Slightly off-topic, about ASM, when you create a ClassWriter [1], you
> >> > can
> >> > pass a ClassReader of an existing class, in that case ASM copy the
> >> > constant pool from the class reader to the class writer so the constant
> >> > pool is preserved.
> >> >
> >> > Rémi
> >> >
> >> > [1]
> >> > http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29
> >> >
> >> > ----- Mail original -----
> >> >> De: "Vladimir Ivanov" <vladimir.x.ivanov at oracle.com>
> >> >> À: "Rafael Winterhalter" <rafael.wth at gmail.com>
> >> >> Cc: "Coleen Phillimore" <coleen.phillimore at oracle.com>,
> >> >> core-libs-dev at openjdk.java.net, "serguei.spitsyn at oracle.com
> >> >> Spitsyn" <serguei.spitsyn at oracle.com>, "Daniel Daugherty"
> >> >> <daniel.daugherty at oracle.com>
> >> >> Envoyé: Vendredi 22 Janvier 2016 18:47:31
> >> >> Objet: Re: ClassFileTransformer does not apply to anonymous classes
> >> >>
> >> >> Rafael,
> >> >>
> >> >> First of all, I'd like to agree on the terminology. There's some
> >> >> confusion there. Anonymous term is ambiguous in Java. There are
> >> >> anonymous classes on language level and there's
> >> >> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
> >> >> classes. I prefer to call them VM anonymous classes to avoid confusion.
> >> >>
> >> >> I assume that whenever you use anonymous you assume "VM anonymous".
> >> >>
> >> >> > thank you for your response. While I completely understand your view
> >> >> > from a VM implementor's point of view, as a practicioner I would
> >> >> > recommend against it. Not being able to instrument lambda expressions
> >> >> > puts a severe limitation onto using the instrumentation API
> >> >> > alltogether. For example, a monitoring application that promises to
> >> >> > record all invocations of a method of a certain interface that only
> >> >> > works 95% of the time is not 5% less usefull but might no longer be
> >> >> > useful at all. People have build large applications based on the
> >> >> > assumption that all user application code can be instrumented and
> >> >> > such
> >> >> > a regression would hurt them. For example, until today, over 30
> >> >> > people
> >> >> > that use my code generation framework reached out to me and reported
> >> >> > the reported behavior as a bug in my library and people are only
> >> >> > starting to migrate their applications to Java 8. The outcome is
> >> >> > simply not intuitive as using a lambda expression over an anonyous
> >> >> > inner class should not change an application's behavior.
> >> >> Can you elaborate on that point, please?
> >> >>
> >> >> I don't see any problems with instrumenting user code. Javac represents
> >> >> lambda expression body as a private static method of the enclosing
> >> >> class, which can be instrumented. VM anonymous classes are used only as
> >> >> a mechanism to glue functional interfaces and lambda expressions
> >> >> together at runtime.
> >> >>
> >> >> For example:
> >> >>     static void f(Runnable r) { r.run(); }
> >> >>     f(() -> {});
> >> >>
> >> >> Test::f (7 bytes)
> >> >>    @ 1   Test$$Lambda$1/791452441::run (4 bytes)   inline (hot)
> >> >>      @ 0   Test::lambda$main$0 (1 bytes)   inline (hot)
> >> >>
> >> >> Why do you need to instrument Test$$Lambda$1/... and not
> >> >> Test::lambda$main$0?
> >> >>
> >> >> > The currently used workaround that people use is to instrument the
> >> >> > LambdaMetaFactory class itself to apply the transformer manually when
> >> >> > it is being created. This solution is spreading as a standard
> >> >> > approach
> >> >> > and I am sure that forbidding a redefinition alltogether would only
> >> >> > inspire to other workarrounds for being able to migrate running code
> >> >> > to the next Java version.
> >> >> It doesn't sound like a workaround, but as a solution for a different
> >> >> problem.
> >> >>
> >> >> If there's a need to gather profile for every lambda expression
> >> >> instantiation site (indy call), then redefining bootstrap method is the
> >> >> right way to go IMO. It allows to intercept and customize indy call
> >> >> site
> >> >> binding.
> >> >>
> >> >> What does sound as a workaround is an attempt to instrument classes
> >> >> produced by LambdaMetaFactory. Right now, it generates a fresh VM
> >> >> anonymous class on every request, but it is an implementation detail
> >> >> and
> >> >> not a requirement.
> >> >>
> >> >> For example, LambdaMetaFactory can reuse wrappers on per-functional
> >> >> interface basis. It would provide significant footprint savings for
> >> >> lambda expression-rich code bases (usually, there are much more
> >> >> instantiations than functional interface flavors).
> >> >>
> >> >> > Furthermore, I do not think that not being able to patch constant
> >> >> > pool
> >> >> > indices in the generated code introduces any problems in practice.
> >> >> > Most real-world instrumentation only appends new constant pool
> >> >> > entries
> >> >> > without interfering with the existant pool. Rather, I would like to
> >> >> > see an exception being raised if one attempts to change the original
> >> >> > constant pool without me being able to consider this from a technical
> >> >> > perspective. I think this would serve to be sufficient for most
> >> >> > people.
> >> >> If you care only about lambda expressions, then constant pool patching
> >> >> shouldn't be a problem (AFAIR it isn't used for lambda expressions).
> >> >> But
> >> >> in a more generic case (even for java.lang.invoke purposes), the
> >> >> coupling between class file and CP patches is very tight and can be
> >> >> easily broken with benign class file transformations. Considering ASM
> >> >> library, I'm not sure that even simple instrumentations preserve
> >> >> original constant pool structure.
> >> >>
> >> >> Probably, VM can do some structural checks on CP to forbid any
> >> >> modifications except appends, but there are still aliasing problems -
> >> >> sharing doesn't work (even for string constants), so new code shouldn't
> >> >> use original CP entries.
> >> >>
> >> >> I'd prefer to avoid such increase in complexity in VM implementation.
> >> >>
> >> >> > I really hope that there is a way to allow for patching anonymously
> >> >> > loaded classes even without exposing the constant pool patches.
> >> >>  From VM implementation perspective I don't see any problems with
> >> >> allowing class file redefinition for VM anonymous classes. What I want
> >> >> to do is to discourage from doing so. Again - it is very fragile and
> >> >> there's little we can do to fix that.
> >> >>
> >> >> > So far, thank you for looking into this. I am sure this a more
> >> >> > complex
> >> >> > matter than what I am able to comprehend. I appreciate that you are
> >> >> > taking the time to consider my opinion!
> >> >>
> >> >> Best regards,
> >> >> Vladimir Ivanov
> >> >>
> >>
>