Invoking code generated by an annotation processor
Stéphane NICOLAS
steff.nicolas at gmail.com
Wed Feb 17 20:21:21 UTC 2016
2016-02-17 11:45 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>:
>
>
> On 02/17/2016 11:36 AM, Stéphane NICOLAS wrote:
>
> Thx Jon.
>
> I would say there are 2 levels to answer this :
> 1) in case the error seems non recoverable, javac should just not list
> both recoverable and non recoverable errors. Only the non recoverable ones.
> We would be happy with that.
>
>
> We could *maybe* prioritize the messages, but I think it would be wrong to
> only show non-recoverable errors. Just because an error is recoverable
> doesn't mean that an anno processor will generate code to make the error go
> away.
>
> No, it doesn't mean that, but in our case it is what happens and as the
> annotation processor is not given the chance to run and resolve those
> recoverable errors, they should not be listed.
>
> 2) I am not sure this difference really holds in the case of "B extends A
> and A is generated", what would be marked as non recoverable is actually
> also recoverable.
>
>
> Generally, recoverable errors are those of the form "name not found", in
> the sense that "name" could maybe be found if an anno processor generates
> the code.
>
So we were wrong in our understanding as we understood that they were
different levels of "names not found" errors. It looks like when the class
is found and a field not found, the error is ranked as non-recoverable. But
really this case is further to ours, we just added an extra thought so that
bug resolution takes into account the scenario, but it's not our main
concern.
>
>
> -- Jon
>
>
>
>
> S.
>
> 2016-02-17 11:00 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>:
>
>> Stéphane,
>>
>> Internally, during the period before anno processing, javac categorizes
>> errors as recoverable or not, where "recoverable" means "could this error
>> conceivably be fixed by running annotation processors, which may generate
>> code."
>>
>> It may be that we have to examine the scenario you describe to decide if
>> the error should be marked as recoverable.
>>
>> -- Jon
>>
>>
>> On 02/17/2016 10:53 AM, Stéphane NICOLAS wrote:
>>
>> Hi Jon,
>>
>> Here is a summary about the bug, it should be enough for you to
>> understand what is going wrong. Later, we will provide you with more
>> details on an example you can use to reproduce the bug.
>>
>> Summary
>>
>> The problem is about static imports and invocation of generated code
>> from main code.
>>
>> In our main code, we invoke some code that is generated by an annotation
>> processor. This scenario is fully supported by javac. However, javac
>> reports the calls to generated code as errors, when it should not, in a
>> very specific scenario.
>>
>> In our main code, if we have an erroneous static import to a non existing
>> member in an existing class, then the annotation processors are not
>> triggered and all calls to the generated code will be marked, unduly, as
>> errors.
>>
>> → For us this is a very inconvenient bug because it floods developers
>> with too many errors and it becomes very hard to pinpoint the only actual
>> error in the code (the wrong static import statement). This error can
>> actually be triggered very easily when refactoring.
>>
>> Please note that if we have an equally erroneous static import to a non
>> existing class, then the bug doesn’t happen and only this static import
>> will be marked as error.
>>
>> Extra thoughts
>>
>> We realized that we are kind of asking implicitly to distinguish static
>> import errors to existing classes and non existing classes. But this may
>> not be the right approach. It would not accommodate the hacky but possible
>> scenario when you define class B extends A in your main code, and
>> generate the class A. In that case, if A defines a public static field Foo,
>> B would inherit it, and though B exists, B.Foo would not exist before
>> annotation processing completes.
>> The best approach for us seems to be to always run annotation processors
>> even though there are some static imports problems and to resolve them
>> after code generation = annotation processing. Static imports shouldn’t
>> fail so fast..
>>
>> Details
>>
>> We tested with the following version of Javac :
>>
>> -
>>
>> javac 1.7.0_79
>> -
>>
>> open JDK javac 1.7.0_79
>> -
>>
>> javac 1.8.0_60
>>
>>
>> here is our example : sorry, it will be a bit long to read, but it comes
>> with a full explanation.
>>
>> We first give you easy repro steps and then an overview of the library.
>>
>> Javac issue and reproduction steps
>>
>> We have 3 branches to illustrate the problem :
>>
>> 1. Branch : Master
>> https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>> This branch works fine, there is no compilation error. See below to
>> compile it.
>>
>> To compile it quickly :
>>
>> 1.
>>
>> git clone git at github.com:f2prateek/dart.git
>> 2.
>>
>> cd dart
>> 3.
>>
>> mvn clean install
>>
>>
>>
>> 2. Branch : henson-error-linkage
>>
>>
>> https://github.com/f2prateek/dart/blob/henson-error-linkage/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>> import static com.f2prateek.dart.example.SampleActivity.UNEXISTING;
>> is a static import to an existing class but a non existing member. It
>> will reproduce the bug of javac, the compiler will return 2 errors. The
>> call to generated code will also be marked as error :
>>
>> [INFO] -------------------------------------------------------------
>>
>> [ERROR] COMPILATION ERROR :
>>
>> [INFO] -------------------------------------------------------------
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>> cannot find symbol
>>
>> symbol: static UNEXISTING
>>
>> location: class
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[39,21]
>> cannot find symbol
>>
>> symbol: variable Henson
>>
>> location: class com.f2prateek.dart.example.MainActivity
>>
>> To compile it quickly :
>>
>> 1.
>>
>> git clone git at github.com:f2prateek/dart.git
>> 2.
>>
>> cd dart
>> 3.
>>
>> git checkout origin/henson-error-linkage
>> 4.
>>
>> mvn clean install
>>
>>
>>
>> 3. Branch : henson-error-linkage-good
>> https://github.com/f2prateek/dart/blob/henson-error-linkage-good/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
>> import static com.f2prateek.dart.example.UnExistingClass.UNEXISTING;
>> is a static import to a non-existing class. It will not reproduce the bug
>> of javac. We get only errors related to static import. (You can see that
>> there is a minor glitch, errors are duplicated, but that is not our main
>> concern).
>>
>> [INFO] -------------------------------------------------------------
>>
>> [ERROR] COMPILATION ERROR :
>>
>> [INFO] -------------------------------------------------------------
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
>> cannot find symbol
>>
>> symbol: class UnExistingClass
>>
>> location: package com.f2prateek.dart.example
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>> static import only from classes and interfaces
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
>> cannot find symbol
>>
>> symbol: class UnExistingClass
>>
>> location: package com.f2prateek.dart.example
>>
>> [ERROR]
>> /Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
>> static import only from classes and interfaces
>>
>>
>> To compile it quickly :
>>
>> 1.
>>
>> git clone git at github.com:f2prateek/dart.git
>> 2.
>>
>> cd dart
>> 3.
>>
>> git checkout origin/henson-error-linkage-good
>> 4.
>>
>> mvn clean install
>>
>>
>> Library overview
>>
>> Sample: https://github.com/f2prateek/dart/blob/master/dart-sample
>>
>> This project is the sample of our annotation processor based library for
>> Android.
>>
>> Just to give you a little background, the lib has 2 parts : Dart & Henson.
>>
>>
>> When creating an activity (~ a screen / JWindow) on Android, you can be
>> passed a set of arguments via a Bundle. A bundle is a serialized container
>> that contains serialized arguments.
>>
>> A bundle is like a map, parameters have a name (called key) and a value.
>>
>> The library is all about navigating from a source activity to a target
>> activity and passing arguments via the bundle in a more convenient way.
>> When using Dart & Henson together, we completely abstract the Bundle usage.
>>
>> ----
>>
>> Dart allows to annotate fields members in a target screen and to assign
>> values to these fields based on the content of the Bundle you receive :
>>
>> //from SampleActivity on GH
>> <https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/SampleActivity.java>
>>
>> public class SampleActivity {
>>
>> @InjectExtra(“key1”) String s1; //required key
>>
>> @InjectExtra(“key2”) @Nullable int i1; //optional key
>>
>> @Override
>> protected void onCreate(Bundle bundle) {
>>
>> Dart.inject(this);
>>
>> // after this call the field s1 is assigned the value
>> // of the entry key1 of the bundle
>>
>> // it is equivalent to : s = getIntent().getStringExtra(“key1”)
>>
>> // s1 is a required field in the bundle, whereas i1 is optional.
>>
>> // If an entry key2 is present, i2 will be assigned to it.
>>
>> }
>>
>> }
>>
>> ----
>>
>> Henson allows to build the Bundle in the source activity to reach the
>> given target activity. Henson creates a small DSL to build the Bundle, with
>> a little state machine to emphasize the required and optional natures of
>> parameters.
>>
>> //from MainActivity on GH
>> <https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java>
>>
>> public class MainActivity extends Activity {
>>
>> public void launchTargetActivity() {
>>
>> Intent intent = Henson.with(this)
>>
>> .gotoSampleActivity() // here is the generated DSL call
>>
>> .key1("foo")
>>
>> .key2(4)
>>
>> .build();
>>
>> startActivity(intent);
>>
>> }
>>
>> }
>>
>> ----
>>
>> Both Dart & Henson use annotation processing, which is a very common way
>> to do things on Android, mostly for speed reasons (reflection is slow like
>> hell, it is not JITed unlike on a standard JVM). Dart does not involve
>> calling generated called directly, the code will be called via a little
>> reflection. Henson is all about calling generated code. In the sample
>> above, Henson class is generated and offers the DSL to build the bundle.
>> Nice, no ? :)
>>
>> Thanks a lot guys, and sorry for this far too long but report,
>>
>> Daniel & Stephane
>>
>> 2016-02-11 11:03 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>
>> :
>>
>>> Stéphane,
>>>
>>> Can you reduce the test case down to a simple one(s) that you can post
>>> here?
>>>
>>> -- Jon
>>>
>>>
>>>
>>> On 02/11/2016 10:44 AM, Stéphane NICOLAS wrote:
>>>
>>> Hey Jon,
>>>
>>> we are going to create an example based on our annotation processor. It
>>> will come in the form of a github repo, 1 branch that compiles fine, 2
>>> branches to illustrate different compile problems. Everything is mavenized
>>> so it should work out of the box for you. We will also tell you what error
>>> messages we have on the 2 failing branches, actual and expected behavior,
>>> and provide you with our javac version(s).
>>>
>>> We hope this can help, thank you very much for your answers.
>>>
>>> Stéphane
>>>
>>>
>>> 2016-02-10 13:25 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com
>>> >:
>>>
>>>>
>>>>
>>>> On 02/10/2016 12:59 PM, Stéphane NICOLAS wrote:
>>>>
>>>> Thanks Jon for your fast answer. It's quite disappointing but it's
>>>> better to know it early.
>>>> It's sad to see how much effort we have put in our annotation
>>>> processor, that it's a good idea but fails because of some general java
>>>> ecosystem issue.
>>>>
>>>> Though, talking of the issue with team mates, it became more clear. Let
>>>> me try to rephrase them more accurately than in my first mail.
>>>>
>>>> It seems to me that the design of javac could be improved to
>>>> accommodate these 2 scenarios :
>>>>
>>>> 1. in case of a parse error (for instance a missing ; a bad brace,
>>>> etc.) , javac should stop everything and report the actual problem, without
>>>> getting any further into the analysis and not report any linkage problem
>>>> (because javac would know that annotation processors ain't gonna run and
>>>> that can create false positive about missing linkages. All other parse
>>>> errors should be reported.). It seems to me this would be a healthy "fail
>>>> early" design.
>>>>
>>>>
>>>> This is the intent. Generally, javac is supposed to fail reasonably
>>>> fast in general, and in that case in particular. If you are seeing
>>>> otherwise, please provide a test case, including the version of javac you
>>>> are using.
>>>>
>>>>
>>>>
>>>>
>>>> 1. in case of a linkage error (for instance having a non existing
>>>> import / calling a non existing method, etc.) then we should run the
>>>> annotation processors on the partial AST. This mechanism is already in
>>>> place to accomodate code generation.
>>>> Then the linkage problems should be resolved and take into account
>>>> the generated code, and only the remaining problems should show up. This is
>>>> where javac actually fails because it reports all problems, including those
>>>> that can be resolved by the code that is actually generated. Javac reports
>>>> problem from a previous internal stage, it doesn't match the latest state
>>>> it could reach internally where only 1 problem could not be resolved, the
>>>> one that was not related to invoking generated code, it reports all
>>>> linkages problem exactly like if no code was generated and no annotation
>>>> processor had run.
>>>>
>>>>
>>>> That is the intent.
>>>>
>>>> Really, there is no way for javac to support those 2 scenarios ? I got
>>>> the strong feeling that it should but I understand my knowledge of javac
>>>> has huge holes. I have been surprised by the complexity of the compiler's
>>>> code.
>>>>
>>>> Thx again for your answer, it's been tremendously appreciated; there
>>>> are just a handful people would could have been able to answer my question
>>>> I am very happy someone actually did take the time to do it.
>>>>
>>>> Stephane
>>>>
>>>> 2016-02-10 11:41 GMT-08:00 Jonathan Gibbons <
>>>> jonathan.gibbons at oracle.com>:
>>>>
>>>>>
>>>>>
>>>>> On 02/10/2016 09:09 AM, Stéphane NICOLAS wrote:
>>>>>
>>>>> I am facing a problem with Javac : I got a sweet annotation processor
>>>>> that generates a DSL and I invoke the generated code from my main code.
>>>>>
>>>>> Everything works fine except when there is an error in my main code :
>>>>> if I got a compile error in the main code, not related to the generated
>>>>> code, like a missing comma (parsing problem), or calling an inexisting
>>>>> method (linkage problem), then all calls to the annotation processor are
>>>>> marked as errors by the compiler.
>>>>>
>>>>> Ideally I am just looking for a way to get rid of those error
>>>>> messages, and keep only the original one that fails to compile.
>>>>>
>>>>> AFAIK, javac should support this scenario but it doesn't support it in
>>>>> a very convenient way and I don't understand exactly why..
>>>>>
>>>>> After reafing the javac code, open jdk 7 & 8, I realized that :
>>>>>
>>>>> Annotation processors are not called if there is any error to compile
>>>>> the code. As the annotation processor are not triggered, the calls to
>>>>> generated code get mark as an error.
>>>>>
>>>>> However, I could, by playing with shouldStopPolicy (+if error and if
>>>>> no error) trigger the annotation processor and generate the code. But
>>>>> still, I would receive in this case the errors related to calling the
>>>>> generated code. Like if the generated code was not taken into account, like
>>>>> if I was receiving errors from a previous compilation steps.
>>>>>
>>>>> I understand that javac parses the code, if it fails there, I will
>>>>> receive only an error about the actual problem. And this is great. I just
>>>>> want this error to show up, not any other, and it works fine. (Though I am
>>>>> wondering if this is always true even I clean my build & generated files
>>>>> before compiling)
>>>>>
>>>>> If there is a linkage problem in my original code, like calling an
>>>>> non-existent method, javac will relax the error and trigger the annotation
>>>>> processors and try to let them generate the code to see if the generated
>>>>> code that satisfies the linkage problem. But then, though there is a state
>>>>> in which javac will successfully satisfy all linkage related to generated
>>>>> code but not the other linkage error, I get all calls to generated code
>>>>> marked as errors. This is my problem. It looks like this internal state of
>>>>> the compiler is never accessible and errors returned do not match this
>>>>> state but more probably a previous internal state in which all linkage
>>>>> errors are aggregated and returned.
>>>>>
>>>>> I think there is a solution to achieve what I want but I couldn't find
>>>>> one. And the situation is bad : it may actually cancel the development of
>>>>> the annotation processor lib I am developing. Just because the errors are
>>>>> flooding the actual problem in the code.
>>>>>
>>>>> Would you have some time to advise me, and help me to make the lib
>>>>> work as expected.
>>>>> My goal is just to get only errors that are not related to calling
>>>>> generated code if they are not those who fail the compilation. I don't want
>>>>> to see my errors polluted by those calls if they are not responsible for
>>>>> breaking the compilation.
>>>>>
>>>>> Stephane
>>>>>
>>>>>
>>>>> Stephane,
>>>>>
>>>>> In general, there is no convenient solution for you here, because
>>>>> there is no way for javac to predict what unresolved symbols may or may not
>>>>> be generated by the annotation processor that has yet to be run. And in
>>>>> general, it is not desirable to run annotation processors if the input
>>>>> source code has significant errors, such as syntax errors.
>>>>>
>>>>> -- Jon
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/compiler-dev/attachments/20160217/b0c41874/attachment-0001.html>
More information about the compiler-dev
mailing list