[foreign] RFR: improved macro parser

[Maurizio Cimadamore]

On 08/06/18 02:35, Samuel Audet wrote:
> Hi, Maurizio,
>
> Does that mean that the original expression of the constant is removed 
> from the Java class entirely? A macro definition like the one below 
> `#define MIN (-MAX)` is often considered to be part of the 
> documentation of C/C++ libraries. JavaCPP tries to keep them, it 
> doesn't always work out, but I still think it would be important to 
> try to keep them...
Yeah, the current approach (which is not different from what was there 
before), would not retain this info. That said, the tokens are there at 
extraction time, so it is definitively doable to add them into some 
annotation (we already have one annotation that is used to describe info 
'as it was present in the header'). But, note that even if jextract 
generated something like this:

interface MyHeader {

     @C("'define MIN (-MAX)")
     default int MIN { return -256; }

}

It would still quite hard for the programmer to see that extra bit of 
annotation - while Java IDEs can enable some kind of pop up that shows 
annotations in the APIs they are calling, this support is typically very 
spotty, and quite often turned off by default and/or hidden under 
advanced settings.

So, I'm all for keeping C stuff somewhere in some debugging annotation, 
but we have to be realistic about how accessible that extra bit of info 
will be.

Cheers
Maurizio
>
> Samuel
>
> On 06/07/2018 10:23 PM, Maurizio Cimadamore wrote:
>> Hi,
>> please find a second revision of the parser. This has a quite 
>> different design, and I've ditched use of jshell API in favor of a 
>> more direct approach using compiler API. The reason is that jshell 
>> API doesn't expose ASTs, so it is hard to filter out stuff we don't 
>> want - which means we can accidentally pull in Java constants (e.g. 
>> Integer.MAX_VALUE).
>>
>> The new approach is much more straightforward (and faster too, since 
>> bits are not executed). It leans heavily on javac's constant folding 
>> - so macro are expanded, and given to the compiler which type-checks 
>> them; if all is a constant, javac will set the constant value on the 
>> variable - which is then accessible through the VariableElement API. 
>> If it's not constant, we should not bother with it in the first place.
>>
>> A small visitor makes sure that we can only accept the right kind of 
>> trees - e.g. unary/binary expessions, parenthesis, literals.
>>
>> The parser is using an internal API point - namely JavacTaskPool, 
>> which is used to share compilation context among multiple runs, and 
>> greatly speed up the type-checking machinery (otherwise we have to 
>> init the symbol tables at every macro evaluation, which is very 
>> costly). This is a standard trick we do in both compiler combo tests 
>> and jshell.
>>
>> Webrev:
>>
>> http://cr.openjdk.java.net/~mcimadamore/panama/macro_parser_v2/
>>
>> Cheers
>> Maurizio
>>
>>
>> On 01/06/18 22:21, Maurizio Cimadamore wrote:
>>> Hi,
>>> this webrev adds a new way of parsing object-like macros which 
>>> relies on the jshell API; defined macros are turned into jshell var 
>>> declaration - e.g.
>>>
>>> #define FOO 12
>>>
>>> is turned into
>>>
>>> 'var FOO = 12;'
>>>
>>> All macros are entered into a jshell context, and this allows to 
>>> 'resolve' macro names so that we can handle common cases like:
>>>
>>> #define MAX 43
>>> #define MIN (-MAX)
>>>
>>> Thanks to the jshell API the new code is not too complex, and it 
>>> handles the vast majority of constant-like macros. Of course we need 
>>> a more general fallback for macros (esp. function-like), e.g. I'm 
>>> under no illusion that this will work in all cases, but this looks 
>>> like a good improvement over the current strategy that blows up as 
>>> soon as there are more than two tokens in the macro.
>>>
>>> I tested with a bunch of system headers and results seemed 
>>> significantly better than before.
>>>
>>> Webrev:
>>>
>>> http://cr.openjdk.java.net/~mcimadamore/panama/macro_parser/
>>>
>>> Maurizio
>>>
>>>
>>>
>>
>