static inline and jextract

Tue Sep 6 17:14:59 UTC 2022

I generally agree that it should not be opinionated. But you could easily
end up writing a C to Java compiler to make it happen.

It starts with simple things. How do you deal with the below variable
(which is exportable according to your definition)?

int __declspec(selectany) myvar = 4;

If you allocate native memory, initialize it with 4 and generate a getter:
great. But an "int" variable is just the start. You also have to do it for
complex, nested structs and unions including initialization with
designators like:

struct { int sec, min, hour, day, mon, year; } z = { .day=31, 12, 2014,
.sec=30, 15, 17 };

And what if the header files contains:

int func(int a, int b) {
    return a + b;
}

This is a global function definition. In a header file for consumers of a
library, it's nonsense, like "int x = 4;". How do you treat it in a
non-opinionated way? Will you translate the function to Java code? Will you
also do it for any arbitrarily complex function? Where does this end? Can
we then trick jextract to translate all C code to Java if we just put it in
a header file?

Whatever you do, make jextract more transparent. And this is probably the
biggest complaint about the tool in its current state. When I started using
it, I initially didn't succeed at all. In most cases, the tool didn't
generate the function or struct I was looking for and, worst of all, didn't
report anything.

   - If you explicitly specify "--include-var myar" and jextract skips it
   because it's static, it should tell so.
   - If you explicitly specify "--include-struct mystruct" and jextract
   skips it because it's a typedef, it should tell so.
   - If you explicitly specify "--include-var eDefault" and jextract skips
   it because it's an enum constant and enum constants are only included if
   you use "--include-macro", it should tell so.
   - If you explicitly specify "--include-macro XY" and jextract skips it
   because it's function like, it should tell so.
   - If you explicitly specify "--include-function myfunc" and jextract
   can't find such a function, it should tell so.

These are many of the things I had to learn the hard way, and spent (or
wasted) time for experiments to derive jextract's behavior.

So before putting any serious effort into implementing non-opinionated
treatment of things that shouldn't have been in header files in the first
place, I propose to put the main effort into making jextract more
transparent. In particular, it should report whenever something was
requested but skipped or not found. And it should provide the reason,
possibly indicating near misses like "struct" instead of "typedef".

The case of "int __declspec(selectany) myvar = 4" and similar rare cases
could then be postponed. jextract would simply report "No code can be
generated for the variable 'myvar' as it uses a storage-class that has not
been implemented yet".

This is a non-opinionated approach except that the order of implementation
is prioritized. And priorities are just a part of life if something needs
to be accomplished.

On Tue, Sep 6, 2022 at 6:14 PM Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

> Thinking about some more (after some offline internal discussion).
>
> I don't think jextract should be "opinionated" as to how global variables
> are declared in headers. Is there a "correct" way to add global variables
> to a library header? Yes, by using "extern". Does that mean that all other
> declarations should be skipped? That's where I think this is taking us too
> far.
>
> There might be libraries built in non-idiomatic ways, which expect
> developers that use them to do other tweaks. Does it mean that jextract
> should ignore those? That seems harsh.
>
> If Jextract finds a global variable definition in a header file, then it
> should just treat it at face value, w/o asking whether it makes sense or
> not for the header file to be the way it is. After all, that's why we also
> have filters, so that users can exclude symbols based on additional
> domain-level knowledge.
>
> So, I think unless a symbol is explicitly marked as non-exported (e.g.
> because it's static, or inline, as per [1]), then jextract should act on it.
>
> Maurizio
> On 06/09/2022 16:08, Manuel Bleichenbacher wrote:
>
> I mostly agree except for:
>
>> It seems to me that all this boil to two rules:
>>
>> * we should only generate global var accessor for "extern" variables
>> * we should skip static symbols (this is covered by [1])
>>
> This opens space for a third case: neither "extern" nor "static". I think
> it boils down to a single rule:
>
>
>    - For variables: generate code if declared as "extern", skip otherwise
>
>
> On Tue, Sep 6, 2022 at 4:37 PM Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com> wrote:
>
>> Some replies inline:
>>
>>
>> On 06/09/2022 15:08, Manuel Bleichenbacher wrote:
>>
>> I think there are only a few cases:
>>
>> Case 1: extern int i;
>> It is a declaration and should generate the lookup for a symbol "i" (plus
>> a method to get the value). It is already implemented like this.
>>
>> I agree this is the most idiomatic way to do things.
>>
>>
>> Case 2: int i = 4;
>> It can be skipped as it never appears in a header file that can be
>> consumed by multiple compilation units (as it would generate an error).
>> It's simply not relevant for jextract, which generates code to access an
>> existing library.
>>
>> Case 3: int i;
>> It is the same as case 2 (implicitly initializes to 0).
>>
>> I also agree that these two seem suspicious. While in a standalone
>> application, an header containg symbols like these might make sense, in a
>> library, this most likely does not make any sense, as it will create
>> storage in all clients importing the headerfile.
>>
>>
>>
>> Case 4: int __declspec(selectany) i = 4;
>> This is a special case *and most likely Microsoft specific). In theory,
>> the behavior could be replicated, i.e. a variable or constant is allocated
>> in native memory and properly initialized. Since it's a rare case, I'd
>> rather skip it.
>>
>> Yes, this seems to be special and unique to Windows. But, I'd rather gate
>> on presence of "extern", than to obscure __declspec.
>>
>>
>>
>>
>> Case 5: static int i = 4;
>> It is a mixture between case 2 and 4. It's unlikely to appear in header
>> files although it does not generate an error. The behavior could be
>> replicated but I would spend time on more important things and skip it for
>> now.
>>
>> Case 6: static int i;
>> It is the same as case 5 (implicitly initializes to 0).
>>
>> These seem easier, since declaration says "static".
>>
>>
>> It seems to me that all this boil to two rules:
>>
>> * we should only generate global var accessor for "extern" variables
>> * we should skip static symbols (this is covered by [1])
>>
>> Maurizio
>>
>> [1] - https://github.com/openjdk/jextract/pull/70
>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract/pull/70__;!!ACWV5N9M2RV99hQ!Lbol2roLw9oo_l0GVBCHT9dbxHa0KQGBPdKqbBD18RKaDeuM_ajfqaRi1j0hXELnSrtW6HNUS1X8hErjVv5juTzsQlwnawetxw$>
>>
>>
>>
>> On Tue, Sep 6, 2022 at 3:29 PM Maurizio Cimadamore <
>> maurizio.cimadamore at oracle.com> wrote:
>>
>>> Hi,
>>> I think there's more to it.
>>>
>>> On Linux, if I do the following:
>>>
>>> ```
>>> $ cat foo.h
>>> int i = 4;
>>> ```
>>>
>>> and:
>>>
>>> ```
>>> $ cat foo.c
>>> #include "foo.h"
>>> ```
>>>
>>> And then I compile into a shared lib, the obtained foo.so does have a
>>> symbol for "i" (rightfully so):
>>>
>>> ```
>>> $ objdump -T foo.so
>>>
>>> foo.so:     file format elf64-x86-64
>>>
>>> DYNAMIC SYMBOL TABLE:
>>> 0000000000000000  w   D  *UND*    0000000000000000 __cxa_finalize
>>> 0000000000000000  w   D  *UND*    0000000000000000
>>> _ITM_registerTMCloneTable
>>> 0000000000000000  w   D  *UND*    0000000000000000
>>> _ITM_deregisterTMCloneTable
>>> 0000000000000000  w   D  *UND*    0000000000000000 __gmon_start__
>>> 0000000000004020 g    DO .data    0000000000000004 i
>>>
>>> ```
>>>
>>> E.g. the above header contains a *definition*, so the compiler will
>>> create storage for it (and add it to the shared library), and "i" can be
>>> looked up in the shared lib. Conversely, a global marked as "extern" might
>>> or might not be present in the shared library (depending on whether it's
>>> defined in one of the associated C/CPP files).
>>>
>>> Maurizio
>>>
>>>
>>> On 06/09/2022 13:57, Manuel Bleichenbacher wrote:
>>>
>>> There are two indications that it's definition and not a declaration:
>>>
>>> - absence of "extern" keyword
>>> - presence of initialization ( = { 0x... )
>>>
>>> If it was compiled as C/C++ code, it would allocate memory in the
>>> current compilation unit while a declaration would just refer to something
>>> outside the compilation unit.
>>>
>>> Definitions are rare in header files as they usually lead to duplicate
>>> symbol errors at link time. But through the magic of __declspec(selectany),
>>> this is avoided.
>>>
>>> On Tue, Sep 6, 2022 at 12:11 PM Maurizio Cimadamore <
>>> maurizio.cimadamore at oracle.com> wrote:
>>>
>>>>
>>>> On 05/09/2022 16:32, Manuel Bleichenbacher wrote:
>>>> > extern "C" const GUID __declspec(selectany)
>>>> GUID_DEVINTERFACE_USB_DEVICE
>>>> >     = { 0xA5DCBF10L, 0x6530, 0x11D2, { 0x90, 0x1F,  0x00,  0xC0,
>>>> >  0x4F,  0xB9,  0x51,  0xED } };
>>>> >
>>>> I guess the problem here is the lack of "dllexport", right?
>>>>
>>>> But, while dllexport is common, some libraries can still export symbols
>>>> using a .def file [1].
>>>>
>>>> So, I'm not sure this belongs in the same category as "static inline",
>>>> as it is not possible, just by looking at the header, to understand
>>>> whether the symbol will be present or not?
>>>>
>>>> [1] -
>>>>
>>>> https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170
>>>> <https://urldefense.com/v3/__https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170__;!!ACWV5N9M2RV99hQ!IZ9OZ1GPWyZCLxdiYrnGJewYmhC_BCpH8sxH4Lo2jBtSsuMJPH_j3-vIXRO1emIKo2SMrT3Sj1AqI3daPLx2VOq6oGrpPgb-Xw$>
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220906/6a3a2136/attachment-0001.htm>