[foreign] RFR 8219470: Use clang API to parse macros

[Sundararajan Athijegannathan]

Looks good.

-Sundar

On 25/02/19, 5:35 PM, Maurizio Cimadamore wrote:
> Hi Sundar,
> thanks for the detailed review - new webrev below:
>
> http://cr.openjdk.java.net/~mcimadamore/panama/8219470_v3/
>
> Maurizio
>
> On 22/02/2019 05:25, Sundararajan Athijegannathan wrote:
>> * module-info.java of jdk.jextract requires jdk.compiler which can be 
>> removed now.
>>
>> * Cursor's eval0() native method could be private.
>>
>> * "ptr" field of EvalResult can be final (and private too?)
>>
>> * erroneous static field of EvalResult can be final
>>
>> * native methods of EvalResult can be private
>>
>> * LibClang.createIndex's boolean parameter can have a comment.
>>
>> I built & tested this patch on Mac (with jdk.compiler dependency 
>> removal suggested above). Also ran all samples on Mac. All fine.
>>
>> -Sundar
>>
>> On 22/02/19, 12:34 AM, Maurizio Cimadamore wrote:
>>> Hi,
>>> updated version of the webrev:
>>>
>>> http://cr.openjdk.java.net/~mcimadamore/panama/8219470_v2/
>>>
>>> This passes all tests on all platforms; I've added changes to the 
>>> clang/FFI test, and also fixed few issues that were showing up on 
>>> windows:
>>>
>>> * a missing cast to 'long' was giving a warning which caused build 
>>> failure
>>> * now that we use clang to parse constants, things like '4L' 
>>> translates into 'int' carrier on Windows - as long is just 4 bytes 
>>> there, making the ConstantsTest fail. I fixed by casting to (long 
>>> long).
>>>
>>> Maurizio
>>>
>>> On 20/02/2019 18:56, Maurizio Cimadamore wrote:
>>>> Hi,
>>>> macro support in jextract was added some time ago, using javac's 
>>>> constant folding support to evaluate expressions. While clever, 
>>>> that approach has some limitations - namely it is not possible for 
>>>> it to understand types that belong to the C language. This will 
>>>> make eventually impossible to support constants such as this:
>>>>
>>>> #define PTR (void*)0
>>>>
>>>> Clang offers an 'evaluation' API [1], but unfortunately this API 
>>>> inexplicably doesn't work on macros. But it does work on regular 
>>>> variable declarations. So here's an idea - given a macro of the kind:
>>>>
>>>> #define NAME VALUE
>>>>
>>>> let's generate a snippet like this:
>>>>
>>>> __auto_type jextract$NAME = NAME;
>>>>
>>>> and see what comes out of clang. The __auto_type extension is a GNU 
>>>> extension which is also supported by clang [2]; this is rather 
>>>> handy because it allows us to rely on clang to do type inference too!
>>>>
>>>> The problem with this approach is, of course, to speed up the 
>>>> snippet recompilation enough - to this extent three measures were 
>>>> taken:
>>>>
>>>> * instead of generating a snippet with an #include - using clang 
>>>> API [3] we save the jextract translation unit onto a precompiled 
>>>> header - these headers won't change anyway
>>>>
>>>> * we then parse the snippet with -import-pch <precompiler header>; 
>>>> this allows to skip all symbols that are defined outside the 
>>>> snippet (to do this you have to create a 'local' Index - that's why 
>>>> I exposed that part of the clang API)
>>>>
>>>> * instead of writing onto a file over and over, we make use of 
>>>> clang's in-memory file support [4]. This allows us to create an 
>>>> empty file once, and to keep passing snippets as strings in memory.
>>>>
>>>> The result is quite pleasing - not only we now parse macros 'the 
>>>> right way' but performances got a significant bump; on my machine 
>>>> (before/after):
>>>>
>>>> Opengl 5s/3s
>>>> Python 6s/3.7s
>>>> Ncurses 3s/1.5s
>>>>
>>>> Almost 2x boost - not bad. On top of that, by diffing the --log 
>>>> FINE output it seems like the new implementation is able to pick up 
>>>> an handful of constants that were left out in the previous 
>>>> implementation.
>>>>
>>>> Note that, this patch retains the previous optimization for special 
>>>> casing simple numeric #define - where we just try to parse the 
>>>> number in Java. For API such as OpenGL with loads of constants, 
>>>> this is an essential optimization.
>>>>
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~mcimadamore/panama/8219470/
>>>>
>>>> Cheers
>>>> Maurizio
>>>>
>>>> [1] - 
>>>> https://clang.llvm.org/doxygen/group__CINDEX__MISC.html#ga6be809ca82538f4a610d9a5b18a10ccb
>>>> [2] - https://reviews.llvm.org/D12686
>>>> [3] - 
>>>> https://clang.llvm.org/doxygen/group__CINDEX__TRANSLATION__UNIT.html#ga3abe9df81f9fef269d737d82720c1d33
>>>> [4] - https://clang.llvm.org/doxygen/structCXUnsavedFile.html
>>>>
>>>>
>>>>