static inline and jextract
Maurizio Cimadamore
maurizio.cimadamore at oracle.com
Tue Sep 6 17:36:18 UTC 2022
On 06/09/2022 18:14, Manuel Bleichenbacher wrote:
> I generally agree that it should not be opinionated. But you could
> easily end up writing a C to Java compiler to make it happen.
>
> It starts with simple things. How do you deal with the below variable
> (which is exportable according to your definition)?
>
> int __declspec(selectany) myvar = 4;
>
> If you allocate native memory, initialize it with 4 and generate a
> getter: great. But an "int" variable is just the start. You also have
> to do it for complex, nested structs and unions including
> initialization with designators like:
I don't get what you mean here at all. Jextract won't _allocate_
anything. It would just look for "myvar", which will be either there, or
not there, depending on how the lib was compiled. If the symbol is not
there, that will result in a runtime error when looking it up. I think
you read way too much into what I said.
>
> This is a global function definition. In a header file for consumers
> of a library, it's nonsense, like "int x = 4;". How do you treat it in
> a non-opinionated way? Will you translate the function to Java code?
> Will you also do it for any arbitrarily complex function? Where does
> this end? Can we then trick jextract to translate all C code to Java
> if we just put it in a header file?
This is absolutely not what I was implying. Jextract should not
"simulate" or "translate" anything. The question is much simpler, and is
whether, for a given definition seen in an header, jextract should
generate lookup logic and accessors or not.
>
> Whatever you do, make jextract more transparent. And this is probably
> the biggest complaint about the tool in its current state. When I
> started using it, I initially didn't succeed at all. In most cases,
> the tool didn't generate the function or struct I was looking for and,
> worst of all, didn't report anything.
>
> * If you explicitly specify "--include-var myar" and jextract skips
> it because it's static, it should tell so.
> * If you explicitly specify "--include-struct mystruct" and jextract
> skips it because it's a typedef, it should tell so.
> * If you explicitly specify "--include-var eDefault" and jextract
> skips it because it's an enum constant and enum constants are only
> included if you use "--include-macro", it should tell so.
> * If you explicitly specify "--include-macro XY" and jextract skips
> it because it's function like, it should tell so.
> * If you explicitly specify "--include-function myfunc" and jextract
> can't find such a function, it should tell so.
>
> These are many of the things I had to learn the hard way, and spent
> (or wasted) time for experiments to derive jextract's behavior.
Fair enough. For the records, should you have any patch which add extra
diagnostics we'd be happy to accept the contribution!
Note that, as documented in the jextract readme, the option
"--dump-includes <String>" can be used to dump whatever jextract found
in the header files. From there you can easily see if the symbol you are
looking for is there, what is its header, and what the symbol kind is.
Of course, it's no substitution for better diagnostics, but I feel it
would minimize the experimentation.
>
> So before putting any serious effort into implementing non-opinionated
> treatment of things that shouldn't have been in header files in the
> first place, I propose to put the main effort into making jextract
> more transparent. In particular, it should report whenever something
> was requested but skipped or not found. And it should provide the
> reason, possibly indicating near misses like "struct" instead of
> "typedef".
Again, the suggestion in my email was to stop at what we did here:
https://git.openjdk.org/jextract/pull/70
No extra crazy logic to be implemented.
Maurizio
> On Tue, Sep 6, 2022 at 6:14 PM Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com> wrote:
>
> Thinking about some more (after some offline internal discussion).
>
> I don't think jextract should be "opinionated" as to how global
> variables are declared in headers. Is there a "correct" way to add
> global variables to a library header? Yes, by using "extern". Does
> that mean that all other declarations should be skipped? That's
> where I think this is taking us too far.
>
> There might be libraries built in non-idiomatic ways, which expect
> developers that use them to do other tweaks. Does it mean that
> jextract should ignore those? That seems harsh.
>
> If Jextract finds a global variable definition in a header file,
> then it should just treat it at face value, w/o asking whether it
> makes sense or not for the header file to be the way it is. After
> all, that's why we also have filters, so that users can exclude
> symbols based on additional domain-level knowledge.
>
> So, I think unless a symbol is explicitly marked as non-exported
> (e.g. because it's static, or inline, as per [1]), then jextract
> should act on it.
>
> Maurizio
>
> On 06/09/2022 16:08, Manuel Bleichenbacher wrote:
>> I mostly agree except for:
>>
>> It seems to me that all this boil to two rules:
>>
>> * we should only generate global var accessor for "extern"
>> variables
>> * we should skip static symbols (this is covered by [1])
>>
>> This opens space for a third case: neither "extern" nor "static".
>> I think it boils down to a single rule:
>>
>> * For variables: generate code if declared as "extern", skip
>> otherwise
>>
>>
>> On Tue, Sep 6, 2022 at 4:37 PM Maurizio Cimadamore
>> <maurizio.cimadamore at oracle.com> wrote:
>>
>> Some replies inline:
>>
>>
>> On 06/09/2022 15:08, Manuel Bleichenbacher wrote:
>>> I think there are only a few cases:
>>>
>>> Case 1: extern int i;
>>> It is a declaration and should generate the lookup for a
>>> symbol "i" (plus a method to get the value). It is already
>>> implemented like this.
>> I agree this is the most idiomatic way to do things.
>>>
>>> Case 2: int i = 4;
>>> It can be skipped as it never appears in a header file that
>>> can be consumed by multiple compilation units (as it would
>>> generate an error). It's simply not relevant for jextract,
>>> which generates code to access an existing library.
>>>
>>> Case 3: int i;
>>> It is the same as case 2 (implicitly initializes to 0).
>>
>> I also agree that these two seem suspicious. While in a
>> standalone application, an header containg symbols like these
>> might make sense, in a library, this most likely does not
>> make any sense, as it will create storage in all clients
>> importing the headerfile.
>>
>>
>>>
>>> Case 4: int __declspec(selectany) i = 4;
>>> This is a special case *and most likely Microsoft specific).
>>> In theory, the behavior could be replicated, i.e. a variable
>>> or constant is allocated in native memory and properly
>>> initialized. Since it's a rare case, I'd rather skip it.
>>
>> Yes, this seems to be special and unique to Windows. But, I'd
>> rather gate on presence of "extern", than to obscure __declspec.
>>
>>
>>
>>>
>>> Case 5: static int i = 4;
>>> It is a mixture between case 2 and 4. It's unlikely to
>>> appear in header files although it does not generate an
>>> error. The behavior could be replicated but I would spend
>>> time on more important things and skip it for now.
>>>
>>> Case 6: static int i;
>>> It is the same as case 5 (implicitly initializes to 0).
>>
>> These seem easier, since declaration says "static".
>>
>>
>> It seems to me that all this boil to two rules:
>>
>> * we should only generate global var accessor for "extern"
>> variables
>> * we should skip static symbols (this is covered by [1])
>>
>> Maurizio
>>
>> [1] - https://github.com/openjdk/jextract/pull/70
>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract/pull/70__;!!ACWV5N9M2RV99hQ!Lbol2roLw9oo_l0GVBCHT9dbxHa0KQGBPdKqbBD18RKaDeuM_ajfqaRi1j0hXELnSrtW6HNUS1X8hErjVv5juTzsQlwnawetxw$>
>>
>>
>>>
>>> On Tue, Sep 6, 2022 at 3:29 PM Maurizio Cimadamore
>>> <maurizio.cimadamore at oracle.com> wrote:
>>>
>>> Hi,
>>> I think there's more to it.
>>>
>>> On Linux, if I do the following:
>>>
>>> ```
>>> $ cat foo.h
>>> int i = 4;
>>> ```
>>>
>>> and:
>>>
>>> ```
>>> $ cat foo.c
>>> #include "foo.h"
>>> ```
>>>
>>> And then I compile into a shared lib, the obtained
>>> foo.so does have a symbol for "i" (rightfully so):
>>>
>>> ```
>>> $ objdump -T foo.so
>>>
>>> foo.so: file format elf64-x86-64
>>>
>>> DYNAMIC SYMBOL TABLE:
>>> 0000000000000000 w D *UND* 0000000000000000
>>> __cxa_finalize
>>> 0000000000000000 w D *UND* 0000000000000000
>>> _ITM_registerTMCloneTable
>>> 0000000000000000 w D *UND* 0000000000000000
>>> _ITM_deregisterTMCloneTable
>>> 0000000000000000 w D *UND* 0000000000000000
>>> __gmon_start__
>>> 0000000000004020 g DO .data 0000000000000004 i
>>>
>>> ```
>>>
>>> E.g. the above header contains a *definition*, so the
>>> compiler will create storage for it (and add it to the
>>> shared library), and "i" can be looked up in the shared
>>> lib. Conversely, a global marked as "extern" might or
>>> might not be present in the shared library (depending on
>>> whether it's defined in one of the associated C/CPP files).
>>>
>>> Maurizio
>>>
>>>
>>> On 06/09/2022 13:57, Manuel Bleichenbacher wrote:
>>>> There are two indications that it's definition and not
>>>> a declaration:
>>>>
>>>> - absence of "extern" keyword
>>>> - presence of initialization ( = { 0x... )
>>>>
>>>> If it was compiled as C/C++ code, it would allocate
>>>> memory in the current compilation unit while a
>>>> declaration would just refer to something outside the
>>>> compilation unit.
>>>>
>>>> Definitions are rare in header files as they usually
>>>> lead to duplicate symbol errors at link time. But
>>>> through the magic of __declspec(selectany), this is
>>>> avoided.
>>>>
>>>> On Tue, Sep 6, 2022 at 12:11 PM Maurizio Cimadamore
>>>> <maurizio.cimadamore at oracle.com> wrote:
>>>>
>>>>
>>>> On 05/09/2022 16:32, Manuel Bleichenbacher wrote:
>>>> > extern "C" const GUID __declspec(selectany)
>>>> GUID_DEVINTERFACE_USB_DEVICE
>>>> > = { 0xA5DCBF10L, 0x6530, 0x11D2, { 0x90,
>>>> 0x1F, 0x00, 0xC0,
>>>> > 0x4F, 0xB9, 0x51, 0xED } };
>>>> >
>>>> I guess the problem here is the lack of
>>>> "dllexport", right?
>>>>
>>>> But, while dllexport is common, some libraries can
>>>> still export symbols
>>>> using a .def file [1].
>>>>
>>>> So, I'm not sure this belongs in the same category
>>>> as "static inline",
>>>> as it is not possible, just by looking at the
>>>> header, to understand
>>>> whether the symbol will be present or not?
>>>>
>>>> [1] -
>>>> https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170
>>>> <https://urldefense.com/v3/__https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170__;!!ACWV5N9M2RV99hQ!IZ9OZ1GPWyZCLxdiYrnGJewYmhC_BCpH8sxH4Lo2jBtSsuMJPH_j3-vIXRO1emIKo2SMrT3Sj1AqI3daPLx2VOq6oGrpPgb-Xw$>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220906/0fc40acb/attachment-0001.htm>
More information about the jextract-dev
mailing list