Invoking code generated by an annotation processor
Stéphane NICOLAS
steff.nicolas at gmail.com
Wed Feb 17 18:53:02 UTC 2016
Hi Jon,
Here is a summary about the bug, it should be enough for you to understand
what is going wrong. Later, we will provide you with more details on an
example you can use to reproduce the bug.
Summary
The problem is about static imports and invocation of generated code from
main code.
In our main code, we invoke some code that is generated by an annotation
processor. This scenario is fully supported by javac. However, javac
reports the calls to generated code as errors, when it should not, in a
very specific scenario.
In our main code, if we have an erroneous static import to a non existing
member in an existing class, then the annotation processors are not
triggered and all calls to the generated code will be marked, unduly, as
errors.
→ For us this is a very inconvenient bug because it floods developers with
too many errors and it becomes very hard to pinpoint the only actual error
in the code (the wrong static import statement). This error can actually be
triggered very easily when refactoring.
Please note that if we have an equally erroneous static import to a non
existing class, then the bug doesn’t happen and only this static import
will be marked as error.
Extra thoughts
We realized that we are kind of asking implicitly to distinguish static
import errors to existing classes and non existing classes. But this may
not be the right approach. It would not accommodate the hacky but possible
scenario when you define class B extends A in your main code, and generate
the class A. In that case, if A defines a public static field Foo, B would
inherit it, and though B exists, B.Foo would not exist before annotation
processing completes.
The best approach for us seems to be to always run annotation processors
even though there are some static imports problems and to resolve them
after code generation = annotation processing. Static imports shouldn’t
fail so fast..
Details
We tested with the following version of Javac :
-
javac 1.7.0_79
-
open JDK javac 1.7.0_79
-
javac 1.8.0_60
here is our example : sorry, it will be a bit long to read, but it comes
with a full explanation.
We first give you easy repro steps and then an overview of the library.
Javac issue and reproduction steps
We have 3 branches to illustrate the problem :
1. Branch : Master
https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
This branch works fine, there is no compilation error. See below to compile
it.
To compile it quickly :
1.
git clone git at github.com:f2prateek/dart.git
2.
cd dart
3.
mvn clean install
2. Branch : henson-error-linkage
https://github.com/f2prateek/dart/blob/henson-error-linkage/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
import static com.f2prateek.dart.example.SampleActivity.UNEXISTING;
is a static import to an existing class but a non existing member. It will
reproduce the bug of javac, the compiler will return 2 errors. The call to
generated code will also be marked as error :
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
cannot find symbol
symbol: static UNEXISTING
location: class
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[39,21]
cannot find symbol
symbol: variable Henson
location: class com.f2prateek.dart.example.MainActivity
To compile it quickly :
1.
git clone git at github.com:f2prateek/dart.git
2.
cd dart
3.
git checkout origin/henson-error-linkage
4.
mvn clean install
3. Branch : henson-error-linkage-good
https://github.com/f2prateek/dart/blob/henson-error-linkage-good/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java
import static com.f2prateek.dart.example.UnExistingClass.UNEXISTING;
is a static import to a non-existing class. It will not reproduce the bug
of javac. We get only errors related to static import. (You can see that
there is a minor glitch, errors are duplicated, but that is not our main
concern).
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
cannot find symbol
symbol: class UnExistingClass
location: package com.f2prateek.dart.example
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
static import only from classes and interfaces
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,41]
cannot find symbol
symbol: class UnExistingClass
location: package com.f2prateek.dart.example
[ERROR]
/Users/dmolinero/Development/dart/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java:[26,1]
static import only from classes and interfaces
To compile it quickly :
1.
git clone git at github.com:f2prateek/dart.git
2.
cd dart
3.
git checkout origin/henson-error-linkage-good
4.
mvn clean install
Library overview
Sample: https://github.com/f2prateek/dart/blob/master/dart-sample
This project is the sample of our annotation processor based library for
Android.
Just to give you a little background, the lib has 2 parts : Dart & Henson.
When creating an activity (~ a screen / JWindow) on Android, you can be
passed a set of arguments via a Bundle. A bundle is a serialized container
that contains serialized arguments.
A bundle is like a map, parameters have a name (called key) and a value.
The library is all about navigating from a source activity to a target
activity and passing arguments via the bundle in a more convenient way.
When using Dart & Henson together, we completely abstract the Bundle usage.
----
Dart allows to annotate fields members in a target screen and to assign
values to these fields based on the content of the Bundle you receive :
//from SampleActivity on GH
<https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/SampleActivity.java>
public class SampleActivity {
@InjectExtra(“key1”) String s1; //required key
@InjectExtra(“key2”) @Nullable int i1; //optional key
@Override
protected void onCreate(Bundle bundle) {
Dart.inject(this);
// after this call the field s1 is assigned the value
// of the entry key1 of the bundle
// it is equivalent to : s = getIntent().getStringExtra(“key1”)
// s1 is a required field in the bundle, whereas i1 is optional.
// If an entry key2 is present, i2 will be assigned to it.
}
}
----
Henson allows to build the Bundle in the source activity to reach the given
target activity. Henson creates a small DSL to build the Bundle, with a
little state machine to emphasize the required and optional natures of
parameters.
//from MainActivity on GH
<https://github.com/f2prateek/dart/blob/master/dart-sample/src/main/java/com/f2prateek/dart/example/MainActivity.java>
public class MainActivity extends Activity {
public void launchTargetActivity() {
Intent intent = Henson.with(this)
.gotoSampleActivity() // here is the generated DSL call
.key1("foo")
.key2(4)
.build();
startActivity(intent);
}
}
----
Both Dart & Henson use annotation processing, which is a very common way to
do things on Android, mostly for speed reasons (reflection is slow like
hell, it is not JITed unlike on a standard JVM). Dart does not involve
calling generated called directly, the code will be called via a little
reflection. Henson is all about calling generated code. In the sample
above, Henson class is generated and offers the DSL to build the bundle.
Nice, no ? :)
Thanks a lot guys, and sorry for this far too long but report,
Daniel & Stephane
2016-02-11 11:03 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>:
> Stéphane,
>
> Can you reduce the test case down to a simple one(s) that you can post
> here?
>
> -- Jon
>
>
>
> On 02/11/2016 10:44 AM, Stéphane NICOLAS wrote:
>
> Hey Jon,
>
> we are going to create an example based on our annotation processor. It
> will come in the form of a github repo, 1 branch that compiles fine, 2
> branches to illustrate different compile problems. Everything is mavenized
> so it should work out of the box for you. We will also tell you what error
> messages we have on the 2 failing branches, actual and expected behavior,
> and provide you with our javac version(s).
>
> We hope this can help, thank you very much for your answers.
>
> Stéphane
>
>
> 2016-02-10 13:25 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>:
>
>>
>>
>> On 02/10/2016 12:59 PM, Stéphane NICOLAS wrote:
>>
>> Thanks Jon for your fast answer. It's quite disappointing but it's better
>> to know it early.
>> It's sad to see how much effort we have put in our annotation processor,
>> that it's a good idea but fails because of some general java ecosystem
>> issue.
>>
>> Though, talking of the issue with team mates, it became more clear. Let
>> me try to rephrase them more accurately than in my first mail.
>>
>> It seems to me that the design of javac could be improved to accommodate
>> these 2 scenarios :
>>
>> 1. in case of a parse error (for instance a missing ; a bad brace,
>> etc.) , javac should stop everything and report the actual problem, without
>> getting any further into the analysis and not report any linkage problem
>> (because javac would know that annotation processors ain't gonna run and
>> that can create false positive about missing linkages. All other parse
>> errors should be reported.). It seems to me this would be a healthy "fail
>> early" design.
>>
>>
>> This is the intent. Generally, javac is supposed to fail reasonably
>> fast in general, and in that case in particular. If you are seeing
>> otherwise, please provide a test case, including the version of javac you
>> are using.
>>
>>
>>
>>
>> 1. in case of a linkage error (for instance having a non existing
>> import / calling a non existing method, etc.) then we should run the
>> annotation processors on the partial AST. This mechanism is already in
>> place to accomodate code generation.
>> Then the linkage problems should be resolved and take into account
>> the generated code, and only the remaining problems should show up. This is
>> where javac actually fails because it reports all problems, including those
>> that can be resolved by the code that is actually generated. Javac reports
>> problem from a previous internal stage, it doesn't match the latest state
>> it could reach internally where only 1 problem could not be resolved, the
>> one that was not related to invoking generated code, it reports all
>> linkages problem exactly like if no code was generated and no annotation
>> processor had run.
>>
>>
>> That is the intent.
>>
>> Really, there is no way for javac to support those 2 scenarios ? I got
>> the strong feeling that it should but I understand my knowledge of javac
>> has huge holes. I have been surprised by the complexity of the compiler's
>> code.
>>
>> Thx again for your answer, it's been tremendously appreciated; there are
>> just a handful people would could have been able to answer my question I am
>> very happy someone actually did take the time to do it.
>>
>> Stephane
>>
>> 2016-02-10 11:41 GMT-08:00 Jonathan Gibbons <jonathan.gibbons at oracle.com>
>> :
>>
>>>
>>>
>>> On 02/10/2016 09:09 AM, Stéphane NICOLAS wrote:
>>>
>>> I am facing a problem with Javac : I got a sweet annotation processor
>>> that generates a DSL and I invoke the generated code from my main code.
>>>
>>> Everything works fine except when there is an error in my main code : if
>>> I got a compile error in the main code, not related to the generated code,
>>> like a missing comma (parsing problem), or calling an inexisting method
>>> (linkage problem), then all calls to the annotation processor are marked as
>>> errors by the compiler.
>>>
>>> Ideally I am just looking for a way to get rid of those error messages,
>>> and keep only the original one that fails to compile.
>>>
>>> AFAIK, javac should support this scenario but it doesn't support it in a
>>> very convenient way and I don't understand exactly why..
>>>
>>> After reafing the javac code, open jdk 7 & 8, I realized that :
>>>
>>> Annotation processors are not called if there is any error to compile
>>> the code. As the annotation processor are not triggered, the calls to
>>> generated code get mark as an error.
>>>
>>> However, I could, by playing with shouldStopPolicy (+if error and if no
>>> error) trigger the annotation processor and generate the code. But still, I
>>> would receive in this case the errors related to calling the generated
>>> code. Like if the generated code was not taken into account, like if I was
>>> receiving errors from a previous compilation steps.
>>>
>>> I understand that javac parses the code, if it fails there, I will
>>> receive only an error about the actual problem. And this is great. I just
>>> want this error to show up, not any other, and it works fine. (Though I am
>>> wondering if this is always true even I clean my build & generated files
>>> before compiling)
>>>
>>> If there is a linkage problem in my original code, like calling an
>>> non-existent method, javac will relax the error and trigger the annotation
>>> processors and try to let them generate the code to see if the generated
>>> code that satisfies the linkage problem. But then, though there is a state
>>> in which javac will successfully satisfy all linkage related to generated
>>> code but not the other linkage error, I get all calls to generated code
>>> marked as errors. This is my problem. It looks like this internal state of
>>> the compiler is never accessible and errors returned do not match this
>>> state but more probably a previous internal state in which all linkage
>>> errors are aggregated and returned.
>>>
>>> I think there is a solution to achieve what I want but I couldn't find
>>> one. And the situation is bad : it may actually cancel the development of
>>> the annotation processor lib I am developing. Just because the errors are
>>> flooding the actual problem in the code.
>>>
>>> Would you have some time to advise me, and help me to make the lib work
>>> as expected.
>>> My goal is just to get only errors that are not related to calling
>>> generated code if they are not those who fail the compilation. I don't want
>>> to see my errors polluted by those calls if they are not responsible for
>>> breaking the compilation.
>>>
>>> Stephane
>>>
>>>
>>> Stephane,
>>>
>>> In general, there is no convenient solution for you here, because there
>>> is no way for javac to predict what unresolved symbols may or may not be
>>> generated by the annotation processor that has yet to be run. And in
>>> general, it is not desirable to run annotation processors if the input
>>> source code has significant errors, such as syntax errors.
>>>
>>> -- Jon
>>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/compiler-dev/attachments/20160217/c171659c/attachment-0001.html>
More information about the compiler-dev
mailing list