[foreign] RFR 8219470: Use clang API to parse macros

[Maurizio Cimadamore]

Hi Sundar,
thanks for the detailed review - new webrev below:

http://cr.openjdk.java.net/~mcimadamore/panama/8219470_v3/

Maurizio

On 22/02/2019 05:25, Sundararajan Athijegannathan wrote:
> * module-info.java of jdk.jextract requires jdk.compiler which can be 
> removed now.
>
> * Cursor's eval0() native method could be private.
>
> * "ptr" field of EvalResult can be final (and private too?)
>
> * erroneous static field of EvalResult can be final
>
> * native methods of EvalResult can be private
>
> * LibClang.createIndex's boolean parameter can have a comment.
>
> I built & tested this patch on Mac (with jdk.compiler dependency 
> removal suggested above). Also ran all samples on Mac. All fine.
>
> -Sundar
>
> On 22/02/19, 12:34 AM, Maurizio Cimadamore wrote:
>> Hi,
>> updated version of the webrev:
>>
>> http://cr.openjdk.java.net/~mcimadamore/panama/8219470_v2/
>>
>> This passes all tests on all platforms; I've added changes to the 
>> clang/FFI test, and also fixed few issues that were showing up on 
>> windows:
>>
>> * a missing cast to 'long' was giving a warning which caused build 
>> failure
>> * now that we use clang to parse constants, things like '4L' 
>> translates into 'int' carrier on Windows - as long is just 4 bytes 
>> there, making the ConstantsTest fail. I fixed by casting to (long long).
>>
>> Maurizio
>>
>> On 20/02/2019 18:56, Maurizio Cimadamore wrote:
>>> Hi,
>>> macro support in jextract was added some time ago, using javac's 
>>> constant folding support to evaluate expressions. While clever, that 
>>> approach has some limitations - namely it is not possible for it to 
>>> understand types that belong to the C language. This will make 
>>> eventually impossible to support constants such as this:
>>>
>>> #define PTR (void*)0
>>>
>>> Clang offers an 'evaluation' API [1], but unfortunately this API 
>>> inexplicably doesn't work on macros. But it does work on regular 
>>> variable declarations. So here's an idea - given a macro of the kind:
>>>
>>> #define NAME VALUE
>>>
>>> let's generate a snippet like this:
>>>
>>> __auto_type jextract$NAME = NAME;
>>>
>>> and see what comes out of clang. The __auto_type extension is a GNU 
>>> extension which is also supported by clang [2]; this is rather handy 
>>> because it allows us to rely on clang to do type inference too!
>>>
>>> The problem with this approach is, of course, to speed up the 
>>> snippet recompilation enough - to this extent three measures were 
>>> taken:
>>>
>>> * instead of generating a snippet with an #include - using clang API 
>>> [3] we save the jextract translation unit onto a precompiled header 
>>> - these headers won't change anyway
>>>
>>> * we then parse the snippet with -import-pch <precompiler header>; 
>>> this allows to skip all symbols that are defined outside the snippet 
>>> (to do this you have to create a 'local' Index - that's why I 
>>> exposed that part of the clang API)
>>>
>>> * instead of writing onto a file over and over, we make use of 
>>> clang's in-memory file support [4]. This allows us to create an 
>>> empty file once, and to keep passing snippets as strings in memory.
>>>
>>> The result is quite pleasing - not only we now parse macros 'the 
>>> right way' but performances got a significant bump; on my machine 
>>> (before/after):
>>>
>>> Opengl 5s/3s
>>> Python 6s/3.7s
>>> Ncurses 3s/1.5s
>>>
>>> Almost 2x boost - not bad. On top of that, by diffing the --log FINE 
>>> output it seems like the new implementation is able to pick up an 
>>> handful of constants that were left out in the previous implementation.
>>>
>>> Note that, this patch retains the previous optimization for special 
>>> casing simple numeric #define - where we just try to parse the 
>>> number in Java. For API such as OpenGL with loads of constants, this 
>>> is an essential optimization.
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~mcimadamore/panama/8219470/
>>>
>>> Cheers
>>> Maurizio
>>>
>>> [1] - 
>>> https://clang.llvm.org/doxygen/group__CINDEX__MISC.html#ga6be809ca82538f4a610d9a5b18a10ccb
>>> [2] - https://reviews.llvm.org/D12686
>>> [3] - 
>>> https://clang.llvm.org/doxygen/group__CINDEX__TRANSLATION__UNIT.html#ga3abe9df81f9fef269d737d82720c1d33
>>> [4] - https://clang.llvm.org/doxygen/structCXUnsavedFile.html
>>>
>>>
>>>