Invoking code generated by an annotation processor

Jonathan Gibbons jonathan.gibbons at oracle.com
Wed Feb 17 19:45:51 UTC 2016



On 02/17/2016 11:36 AM, Stéphane NICOLAS wrote:
> Thx Jon.
>
> I would say there are 2 levels to answer this :
> 1) in case the error seems non recoverable, javac should just not list 
> both recoverable and non recoverable errors. Only the non recoverable 
> ones.
> We would be happy with that.

We could *maybe* prioritize the messages, but I think it would be wrong 
to only show non-recoverable errors.  Just because an error is 
recoverable doesn't mean that an anno processor will generate code to 
make the error go away.



> 2) I am not sure this difference really holds in the case of "B 
> extends A and A is generated", what would be marked as non recoverable 
> is actually also recoverable.

Generally, recoverable errors are those of the form "name not found", in 
the sense that "name" could maybe be found if an anno processor 
generates the code.


-- Jon


>
> S.
>
> 2016-02-17 11:00 GMT-08:00 Jonathan Gibbons 
> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>>:
>
>     Stéphane,
>
>     Internally, during the period before anno processing, javac
>     categorizes errors as recoverable or not, where "recoverable"
>     means "could this error conceivably be fixed by running annotation
>     processors, which may generate code."
>
>     It may be that we have to examine the scenario you describe to
>     decide if the error should be marked as recoverable.
>
>     -- Jon
>
>
>     On 02/17/2016 10:53 AM, Stéphane NICOLAS wrote:
>>
>>     Hi Jon,
>>
>>
>>     Here is a summary about the bug, it should be enough for you to
>>     understand what is going wrong. Later, we will provide you with
>>     more details on an example you can use to reproduce the bug.
>>
>>
>>     Summary
>>
>>
>>     The problem is about static importsand invocation of generated
>>     codefrom main code.
>>
>>
>>     In our main code, we invoke some code that is generated by an
>>     annotation processor. This scenario is fully supported by javac.
>>     However, javac reports the calls to generated code as errors,
>>     when it should not, in a very specific scenario.
>>
>>
>>     In our main code, if we have an erroneous static import to a non
>>     existing member in an existing class, then the annotation
>>     processors are not triggered and all calls to the generated code
>>     will be marked, unduly, as errors.
>>
>>     → For us this is a very inconvenient bug because it floods
>>     developers with too many errors and it becomes very hard to
>>     pinpoint the only actual error in the code (the wrong static
>>     import statement). This error can actually be triggered very
>>     easily when refactoring.
>>
>>
>>     Please note that if we have an equally erroneous static import to
>>     a non existing class, then the bug doesn’t happen and only this
>>     static import will be marked as error.
>>
>>
>>     Extra thoughts
>>
>>     We realized that we are kind of asking implicitly to distinguish
>>     static import errors to existing classes and non existing
>>     classes. But this may not be the right approach. It would not
>>     accommodate the hacky but possible scenario when you define class
>>     B extends Ain your main code, and generate the class A. In that
>>     case, if A defines a public static field Foo, B would inherit it,
>>     and though B exists, B.Foo would not exist before annotation
>>     processing completes.
>>     The best approach for us seems to be to always run annotation
>>     processors even though there are some static imports problems and
>>     to resolve them after code generation = annotation processing.
>>     Static imports shouldn’t fail so fast..
>>
>>
>>     Details
>>
>>
>>     We tested with the following version of Javac :
>>
>>      *
>>
>>         javac 1.7.0_79
>>
>>      *
>>
>>         open JDK javac 1.7.0_79
>>
>>      *
>>
>>         javac 1.8.0_60
>>
>>
>>     here is our example : sorry, it will be a bit long to read, but
>>     it comes with a full explanation.
>>
>>     We first give you easy repro steps and then an overview of the
>>     library.
>>
>>
>>     Javac issue and reproduction steps
>>
>>     We have 3 branches to illustrate the problem :
>>
>>
>>     1. Branch :
>>     Masterhttps://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>>     This branch works fine, there is no compilation error. See below
>>     to compile it.
>>
>>
>>     To compile it quickly :
>>
>>     1.
>>
>>         git clone git at github.com
>>         <mailto:git at github.com>:f2prateek/dart.git
>>
>>     2.
>>
>>         cd dart
>>
>>     3.
>>
>>         mvn clean install
>>
>>
>>
>>     2. Branch : henson-error-linkage
>>
>>     https://github.com/f2prateek/dart/blob/henson-error-linkage/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>>     import staticcom.f2prateek.dart.example.SampleActivity.UNEXISTING;
>>     is a static import to an existing class but a non existing
>>     member. It will reproduce the bug of javac, the compiler will
>>     return 2 errors. The call to generated code will also be marked
>>     as error :
>>
>>     [INFO] -------------------------------------------------------------
>>
>>     [ERROR] COMPILATION ERROR :
>>
>>     [INFO] -------------------------------------------------------------
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>>     cannot find symbol
>>
>>      symbol:   static UNEXISTING
>>
>>      location: class
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[39,21]
>>     cannot find symbol
>>
>>      symbol:   variable Henson
>>
>>      location: class com.f2prateek.dart.example.MainActivity
>>
>>
>>     To compile it quickly :
>>
>>     1.
>>
>>         git clone git at github.com
>>         <mailto:git at github.com>:f2prateek/dart.git
>>
>>     2.
>>
>>         cd dart
>>
>>     3.
>>
>>         git checkout origin/henson-error-linkage
>>
>>     4.
>>
>>         mvn clean install
>>
>>
>>
>>     3. Branch :
>>     henson-error-linkage-goodhttps://github.com/f2prateek/dart/blob/henson-error-linkage-good/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>>     import staticcom.f2prateek.dart.example.UnExistingClass.UNEXISTING;
>>     is a static import to a non-existing class. It will not reproduce
>>     the bug of javac. We get only errors related to static import.
>>     (You can see that there is a minor glitch, errors are duplicated,
>>     but that is not our main concern).
>>
>>     [INFO] -------------------------------------------------------------
>>
>>     [ERROR] COMPILATION ERROR :
>>
>>     [INFO] -------------------------------------------------------------
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
>>     cannot find symbol
>>
>>      symbol:   class UnExistingClass
>>
>>      location: package com.f2prateek.dart.example
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>>     static import only from classes and interfaces
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
>>     cannot find symbol
>>
>>      symbol:   class UnExistingClass
>>
>>      location: package com.f2prateek.dart.example
>>
>>     [ERROR]
>>     /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>>     static import only from classes and interfaces
>>
>>
>>
>>     To compile it quickly :
>>
>>     1.
>>
>>         git clone git at github.com
>>         <mailto:git at github.com>:f2prateek/dart.git
>>
>>     2.
>>
>>         cd dart
>>
>>     3.
>>
>>         git checkout origin/henson-error-linkage-good
>>
>>     4.
>>
>>         mvn clean install
>>
>>
>>     Library overview
>>
>>     Sample: https://github.com/f2prateek/dart/blob/master/dart-sample
>>
>>     This project is the sample of our annotation processor based
>>     library for Android.
>>
>>     Just to give you a little background, the lib has 2 parts : Dart&
>>     Henson.
>>
>>
>>     When creating an activity (~ a screen / JWindow) on Android, you
>>     can be passed a set of arguments via a Bundle. A bundle is a
>>     serialized container that contains serialized arguments.
>>
>>     A bundle is like a map, parameters have a name (called key) and a
>>     value.
>>
>>
>>     The library is all about navigating from a source activity to a
>>     target activity and passing arguments via the bundle in a more
>>     convenient way. When using Dart & Henson together, we completely
>>     abstract the Bundle usage.
>>
>>
>>     ----
>>
>>
>>     Dartallows to annotate fields members in a target screen and to
>>     assign values to these fields based on the content of the Bundle
>>     you receive :
>>
>>
>>     //from SampleActivity on GH
>>     <https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/SampleActivity.java>
>>
>>     public class SampleActivity {
>>
>>       @InjectExtra(“key1”) String s1;  //required key
>>
>>       @InjectExtra(“key2”) @Nullable int i1;  //optional key
>>
>>       @Override
>>       protected void onCreate(Bundle bundle) {
>>
>>          Dart.inject(this);
>>
>>          // after this call the field s1 is assigned the value
>>          // of the entry key1 of the bundle
>>
>>          // it is equivalent to : s = getIntent().getStringExtra(“key1”)
>>
>>          // s1 is a required field in the bundle, whereas i1 is
>>     optional.
>>
>>          // If an entry key2 is present, i2 will be assigned to it.
>>
>>       }
>>
>>     }
>>
>>     ----
>>
>>     Henson allows to build the Bundle in the source activity to reach
>>     the given target activity. Henson creates a small DSL to build
>>     the Bundle, with a little state machine to emphasize the required
>>     and optional natures of parameters.
>>
>>
>>     //from MainActivity on GH
>>     <https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java>
>>
>>     public class MainActivity extends Activity {
>>
>>      public void launchTargetActivity() {
>>
>>        Intent intent = Henson.with(this)
>>
>>            .gotoSampleActivity()    // here is the generated DSL call
>>
>>            .key1("foo")
>>
>>            .key2(4)
>>
>>            .build();
>>
>>
>>        startActivity(intent);
>>
>>      }
>>
>>     }
>>
>>
>>     ----
>>
>>
>>     Both Dart &Hensonuse annotation processing, which is a very
>>     common way to do things on Android, mostly for speed reasons
>>     (reflection is slow like hell, it is not JITed unlike on a
>>     standard JVM). Dart does not involve calling generated called
>>     directly, the code will be called via a little reflection. Henson
>>     is all about calling generated code. In the sample above, Henson
>>     class is generated and offers the DSL to build the bundle.
>>     Nice, no ? :)
>>
>>
>>     Thanks a lot guys, and sorry for this far too long but report,
>>
>>     Daniel & Stephane
>>
>>
>>     2016-02-11 11:03 GMT-08:00 Jonathan Gibbons
>>     <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>>:
>>
>>         Stéphane,
>>
>>         Can you reduce the test case down to a simple one(s) that you
>>         can post here?
>>
>>         -- Jon
>>
>>
>>
>>         On 02/11/2016 10:44 AM, Stéphane NICOLAS wrote:
>>>         Hey Jon,
>>>
>>>         we are going to create an example based on our annotation
>>>         processor. It will come in the form of a github repo, 1
>>>         branch that compiles fine, 2 branches to illustrate
>>>         different compile problems. Everything is mavenized so it
>>>         should work out of the box for you. We will also tell you
>>>         what error messages we have on the 2 failing branches,
>>>         actual and expected behavior, and provide you with our javac
>>>         version(s).
>>>
>>>         We hope this can help, thank you very much for your answers.
>>>
>>>         Stéphane
>>>
>>>
>>>         2016-02-10 13:25 GMT-08:00 Jonathan Gibbons
>>>         <jonathan.gibbons at oracle.com
>>>         <mailto:jonathan.gibbons at oracle.com>>:
>>>
>>>
>>>
>>>             On 02/10/2016 12:59 PM, Stéphane NICOLAS wrote:
>>>>             Thanks Jon for your fast answer. It's quite
>>>>             disappointing but it's better to know it early.
>>>>             It's sad to see how much effort we have put in our
>>>>             annotation processor, that it's a good idea but fails
>>>>             because of some general java ecosystem issue.
>>>>
>>>>             Though, talking of the issue with team mates, it became
>>>>             more clear. Let me try to rephrase them more accurately
>>>>             than in my first mail.
>>>>
>>>>             It seems to me that the design of javac could be
>>>>             improved to accommodate these 2 scenarios :
>>>>
>>>>              1. in case of a parse error (for instance a missing ;
>>>>                 a bad brace, etc.) , javac should stop everything
>>>>                 and report the actual problem, without getting any
>>>>                 further into the analysis and not report any
>>>>                 linkage problem (because javac would know that
>>>>                 annotation processors ain't gonna run and that can
>>>>                 create false positive about missing linkages. All
>>>>                 other parse errors should be reported.). It seems
>>>>                 to me this would be a healthy "fail early" design.
>>>>
>>>
>>>             This is the intent. Generally, javac is supposed to fail
>>>             reasonably fast in general, and in that case in
>>>             particular.     If you are seeing otherwise, please
>>>             provide a test case, including the version of javac you
>>>             are using.
>>>
>>>
>>>
>>>>              1. in case of a linkage error (for instance having a
>>>>                 non existing import / calling a non existing
>>>>                 method, etc.) then we should run the annotation
>>>>                 processors on the partial AST. This mechanism is
>>>>                 already in place to accomodate code generation.
>>>>                 Then the linkage problems should be resolved and
>>>>                 take into account the generated code, and only the
>>>>                 remaining problems should show up. This is where
>>>>                 javac actually fails because it reports all
>>>>                 problems, including those that can be resolved by
>>>>                 the code that is actually generated. Javac reports
>>>>                 problem from a previous internal stage, it doesn't
>>>>                 match the latest state it could reach internally
>>>>                 where only 1 problem could not be resolved, the one
>>>>                 that was not related to invoking generated code, it
>>>>                 reports all linkages problem exactly like if no
>>>>                 code was generated and no annotation processor had run.
>>>>
>>>
>>>             That is the intent.
>>>
>>>>             Really, there is no way for javac to support those 2
>>>>             scenarios ? I got the strong feeling that it should but
>>>>             I understand my knowledge of javac has huge holes. I
>>>>             have been surprised by the complexity of the compiler's
>>>>             code.
>>>>
>>>>             Thx again for your answer, it's been tremendously
>>>>             appreciated; there are just a handful people would
>>>>             could have been able to answer my question I am very
>>>>             happy someone actually did take the time to do it.
>>>>
>>>>             Stephane
>>>>
>>>>             2016-02-10 11:41 GMT-08:00 Jonathan Gibbons
>>>>             <jonathan.gibbons at oracle.com
>>>>             <mailto:jonathan.gibbons at oracle.com>>:
>>>>
>>>>
>>>>
>>>>                 On 02/10/2016 09:09 AM, Stéphane NICOLAS wrote:
>>>>>                 I am facing a problem with Javac : I got a sweet
>>>>>                 annotation processor that generates a DSL and I
>>>>>                 invoke the generated code from my main code.
>>>>>
>>>>>                 Everything works fine except when there is an
>>>>>                 error in my main code : if I got a compile error
>>>>>                 in the main code, not related to the generated
>>>>>                 code, like a missing comma (parsing problem), or
>>>>>                 calling an inexisting method (linkage problem),
>>>>>                 then all calls to the annotation processor are
>>>>>                 marked as errors by the compiler.
>>>>>
>>>>>                 Ideally I am just looking for a way to get rid of
>>>>>                 those error messages, and keep only the original
>>>>>                 one that fails to compile.
>>>>>
>>>>>                 AFAIK, javac should support this scenario but it
>>>>>                 doesn't support it in a very convenient way and I
>>>>>                 don't understand exactly why..
>>>>>
>>>>>                 After reafing the javac code, open jdk 7 & 8, I
>>>>>                 realized that :
>>>>>
>>>>>                 Annotation processors are not called if there is
>>>>>                 any error to compile the code. As the annotation
>>>>>                 processor are not triggered, the calls to
>>>>>                 generated code get mark as an error.
>>>>>
>>>>>                 However, I could, by playing with shouldStopPolicy
>>>>>                 (+if error and if no error) trigger the annotation
>>>>>                 processor and generate the code. But still, I
>>>>>                 would receive in this case the errors related to
>>>>>                 calling the generated code. Like if the generated
>>>>>                 code was not taken into account, like if I was
>>>>>                 receiving errors from a previous compilation steps.
>>>>>
>>>>>                 I understand that javac parses the code, if it
>>>>>                 fails there, I will receive only an error about
>>>>>                 the actual problem. And this is great. I just want
>>>>>                 this error to show up, not any other, and it works
>>>>>                 fine. (Though I am wondering if this is always
>>>>>                 true even I clean my build & generated files
>>>>>                 before compiling)
>>>>>
>>>>>                 If there is a linkage problem in my original code,
>>>>>                 like calling an non-existent method, javac will
>>>>>                 relax the error and trigger the annotation
>>>>>                 processors and try to let them generate the code
>>>>>                 to see if the generated code that satisfies the
>>>>>                 linkage problem. But then, though there is a state
>>>>>                 in which javac will successfully satisfy all
>>>>>                 linkage related to generated code but not the
>>>>>                 other linkage error, I get all calls to generated
>>>>>                 code marked as errors. This is my problem. It
>>>>>                 looks like this internal state of the compiler is
>>>>>                 never accessible and errors returned do not match
>>>>>                 this state but more probably a previous internal
>>>>>                 state in which all linkage errors are aggregated
>>>>>                 and returned.
>>>>>
>>>>>                 I think there is a solution to achieve what I want
>>>>>                 but I couldn't find one. And the situation is bad
>>>>>                 : it may actually cancel the development of the
>>>>>                 annotation processor lib I am developing. Just
>>>>>                 because the errors are flooding the actual problem
>>>>>                 in the code.
>>>>>
>>>>>                 Would you have some time to advise me, and help me
>>>>>                 to make the lib work as expected.
>>>>>                 My goal is just to get only errors that are not
>>>>>                 related to calling generated code if they are not
>>>>>                 those who fail the compilation. I don't want to
>>>>>                 see my errors polluted by those calls if they are
>>>>>                 not responsible for breaking the compilation.
>>>>>
>>>>>                 Stephane
>>>>
>>>>                 Stephane,
>>>>
>>>>                 In general, there is no convenient solution for you
>>>>                 here, because there is no way for javac to predict
>>>>                 what unresolved symbols may or may not be generated
>>>>                 by the annotation processor that has yet to be run.
>>>>                 And in general, it is not desirable to run
>>>>                 annotation processors if the input source code has
>>>>                 significant errors, such as syntax errors.
>>>>
>>>>                 -- Jon
>>>>
>>>>
>>>
>>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/compiler-dev/attachments/20160217/5358bc34/attachment-0001.html>


More information about the compiler-dev mailing list