static inline and jextract

Tue Sep 6 20:48:02 UTC 2022

It definitely looks as if I misunderstood you. So what's the plan now for
things like:

int i = 4;
int __declspec(selectany) myvar = 4;

Would they all generate code with a symbol lookup?

If so, who would benefit? The code is 99% likely to fail at run-time. It
would just make it harder to use the tool.

On Tue, Sep 6, 2022 at 7:36 PM Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

>
> On 06/09/2022 18:14, Manuel Bleichenbacher wrote:
>
> I generally agree that it should not be opinionated. But you could easily
> end up writing a C to Java compiler to make it happen.
>
> It starts with simple things. How do you deal with the below variable
> (which is exportable according to your definition)?
>
> int __declspec(selectany) myvar = 4;
>
> If you allocate native memory, initialize it with 4 and generate a getter:
> great. But an "int" variable is just the start. You also have to do it for
> complex, nested structs and unions including initialization with
> designators like:
>
> I don't get what you mean here at all. Jextract won't _allocate_ anything.
> It would just look for "myvar", which will be either there, or not there,
> depending on how the lib was compiled. If the symbol is not there, that
> will result in a runtime error when looking it up. I think you read way too
> much into what I said.
>
>
> This is a global function definition. In a header file for consumers of a
> library, it's nonsense, like "int x = 4;". How do you treat it in a
> non-opinionated way? Will you translate the function to Java code? Will you
> also do it for any arbitrarily complex function? Where does this end? Can
> we then trick jextract to translate all C code to Java if we just put it in
> a header file?
>
> This is absolutely not what I was implying. Jextract should not "simulate"
> or "translate" anything. The question is much simpler, and is whether, for
> a given definition seen in an header, jextract should generate lookup logic
> and accessors or not.
>
>
> Whatever you do, make jextract more transparent. And this is probably the
> biggest complaint about the tool in its current state. When I started using
> it, I initially didn't succeed at all. In most cases, the tool didn't
> generate the function or struct I was looking for and, worst of all, didn't
> report anything.
>
>    - If you explicitly specify "--include-var myar" and jextract skips it
>    because it's static, it should tell so.
>    - If you explicitly specify "--include-struct mystruct" and jextract
>    skips it because it's a typedef, it should tell so.
>    - If you explicitly specify "--include-var eDefault" and jextract
>    skips it because it's an enum constant and enum constants are only included
>    if you use "--include-macro", it should tell so.
>    - If you explicitly specify "--include-macro XY" and jextract skips it
>    because it's function like, it should tell so.
>    - If you explicitly specify "--include-function myfunc" and jextract
>    can't find such a function, it should tell so.
>
> These are many of the things I had to learn the hard way, and spent (or
> wasted) time for experiments to derive jextract's behavior.
>
> Fair enough. For the records, should you have any patch which add extra
> diagnostics we'd be happy to accept the contribution!
>
> Note that, as documented in the jextract readme, the option
> "--dump-includes <String>" can be used to dump whatever jextract found in
> the header files. From there you can easily see if the symbol you are
> looking for is there, what is its header, and what the symbol kind is. Of
> course, it's no substitution for better diagnostics, but I feel it would
> minimize the experimentation.
>
>
> So before putting any serious effort into implementing non-opinionated
> treatment of things that shouldn't have been in header files in the first
> place, I propose to put the main effort into making jextract more
> transparent. In particular, it should report whenever something was
> requested but skipped or not found. And it should provide the reason,
> possibly indicating near misses like "struct" instead of "typedef".
>
> Again, the suggestion in my email was to stop at what we did here:
>
> https://git.openjdk.org/jextract/pull/70
>
> No extra crazy logic to be implemented.
>
> Maurizio
>
> On Tue, Sep 6, 2022 at 6:14 PM Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com> wrote:
>
>> Thinking about some more (after some offline internal discussion).
>>
>> I don't think jextract should be "opinionated" as to how global variables
>> are declared in headers. Is there a "correct" way to add global variables
>> to a library header? Yes, by using "extern". Does that mean that all other
>> declarations should be skipped? That's where I think this is taking us too
>> far.
>>
>> There might be libraries built in non-idiomatic ways, which expect
>> developers that use them to do other tweaks. Does it mean that jextract
>> should ignore those? That seems harsh.
>>
>> If Jextract finds a global variable definition in a header file, then it
>> should just treat it at face value, w/o asking whether it makes sense or
>> not for the header file to be the way it is. After all, that's why we also
>> have filters, so that users can exclude symbols based on additional
>> domain-level knowledge.
>>
>> So, I think unless a symbol is explicitly marked as non-exported (e.g.
>> because it's static, or inline, as per [1]), then jextract should act on it.
>>
>> Maurizio
>> On 06/09/2022 16:08, Manuel Bleichenbacher wrote:
>>
>> I mostly agree except for:
>>
>>> It seems to me that all this boil to two rules:
>>>
>>> * we should only generate global var accessor for "extern" variables
>>> * we should skip static symbols (this is covered by [1])
>>>
>> This opens space for a third case: neither "extern" nor "static". I think
>> it boils down to a single rule:
>>
>>
>>    - For variables: generate code if declared as "extern", skip otherwise
>>
>>
>> On Tue, Sep 6, 2022 at 4:37 PM Maurizio Cimadamore <
>> maurizio.cimadamore at oracle.com> wrote:
>>
>>> Some replies inline:
>>>
>>>
>>> On 06/09/2022 15:08, Manuel Bleichenbacher wrote:
>>>
>>> I think there are only a few cases:
>>>
>>> Case 1: extern int i;
>>> It is a declaration and should generate the lookup for a symbol "i"
>>> (plus a method to get the value). It is already implemented like this.
>>>
>>> I agree this is the most idiomatic way to do things.
>>>
>>>
>>> Case 2: int i = 4;
>>> It can be skipped as it never appears in a header file that can be
>>> consumed by multiple compilation units (as it would generate an error).
>>> It's simply not relevant for jextract, which generates code to access an
>>> existing library.
>>>
>>> Case 3: int i;
>>> It is the same as case 2 (implicitly initializes to 0).
>>>
>>> I also agree that these two seem suspicious. While in a standalone
>>> application, an header containg symbols like these might make sense, in a
>>> library, this most likely does not make any sense, as it will create
>>> storage in all clients importing the headerfile.
>>>
>>>
>>>
>>> Case 4: int __declspec(selectany) i = 4;
>>> This is a special case *and most likely Microsoft specific). In theory,
>>> the behavior could be replicated, i.e. a variable or constant is allocated
>>> in native memory and properly initialized. Since it's a rare case, I'd
>>> rather skip it.
>>>
>>> Yes, this seems to be special and unique to Windows. But, I'd rather
>>> gate on presence of "extern", than to obscure __declspec.
>>>
>>>
>>>
>>>
>>> Case 5: static int i = 4;
>>> It is a mixture between case 2 and 4. It's unlikely to appear in header
>>> files although it does not generate an error. The behavior could be
>>> replicated but I would spend time on more important things and skip it for
>>> now.
>>>
>>> Case 6: static int i;
>>> It is the same as case 5 (implicitly initializes to 0).
>>>
>>> These seem easier, since declaration says "static".
>>>
>>>
>>> It seems to me that all this boil to two rules:
>>>
>>> * we should only generate global var accessor for "extern" variables
>>> * we should skip static symbols (this is covered by [1])
>>>
>>> Maurizio
>>>
>>> [1] - https://github.com/openjdk/jextract/pull/70
>>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract/pull/70__;!!ACWV5N9M2RV99hQ!Lbol2roLw9oo_l0GVBCHT9dbxHa0KQGBPdKqbBD18RKaDeuM_ajfqaRi1j0hXELnSrtW6HNUS1X8hErjVv5juTzsQlwnawetxw$>
>>>
>>>
>>>
>>> On Tue, Sep 6, 2022 at 3:29 PM Maurizio Cimadamore <
>>> maurizio.cimadamore at oracle.com> wrote:
>>>
>>>> Hi,
>>>> I think there's more to it.
>>>>
>>>> On Linux, if I do the following:
>>>>
>>>> ```
>>>> $ cat foo.h
>>>> int i = 4;
>>>> ```
>>>>
>>>> and:
>>>>
>>>> ```
>>>> $ cat foo.c
>>>> #include "foo.h"
>>>> ```
>>>>
>>>> And then I compile into a shared lib, the obtained foo.so does have a
>>>> symbol for "i" (rightfully so):
>>>>
>>>> ```
>>>> $ objdump -T foo.so
>>>>
>>>> foo.so:     file format elf64-x86-64
>>>>
>>>> DYNAMIC SYMBOL TABLE:
>>>> 0000000000000000  w   D  *UND*    0000000000000000 __cxa_finalize
>>>> 0000000000000000  w   D  *UND*    0000000000000000
>>>> _ITM_registerTMCloneTable
>>>> 0000000000000000  w   D  *UND*    0000000000000000
>>>> _ITM_deregisterTMCloneTable
>>>> 0000000000000000  w   D  *UND*    0000000000000000 __gmon_start__
>>>> 0000000000004020 g    DO .data    0000000000000004 i
>>>>
>>>> ```
>>>>
>>>> E.g. the above header contains a *definition*, so the compiler will
>>>> create storage for it (and add it to the shared library), and "i" can be
>>>> looked up in the shared lib. Conversely, a global marked as "extern" might
>>>> or might not be present in the shared library (depending on whether it's
>>>> defined in one of the associated C/CPP files).
>>>>
>>>> Maurizio
>>>>
>>>>
>>>> On 06/09/2022 13:57, Manuel Bleichenbacher wrote:
>>>>
>>>> There are two indications that it's definition and not a declaration:
>>>>
>>>> - absence of "extern" keyword
>>>> - presence of initialization ( = { 0x... )
>>>>
>>>> If it was compiled as C/C++ code, it would allocate memory in the
>>>> current compilation unit while a declaration would just refer to something
>>>> outside the compilation unit.
>>>>
>>>> Definitions are rare in header files as they usually lead to duplicate
>>>> symbol errors at link time. But through the magic of __declspec(selectany),
>>>> this is avoided.
>>>>
>>>> On Tue, Sep 6, 2022 at 12:11 PM Maurizio Cimadamore <
>>>> maurizio.cimadamore at oracle.com> wrote:
>>>>
>>>>>
>>>>> On 05/09/2022 16:32, Manuel Bleichenbacher wrote:
>>>>> > extern "C" const GUID __declspec(selectany)
>>>>> GUID_DEVINTERFACE_USB_DEVICE
>>>>> >     = { 0xA5DCBF10L, 0x6530, 0x11D2, { 0x90, 0x1F,  0x00,  0xC0,
>>>>> >  0x4F,  0xB9,  0x51,  0xED } };
>>>>> >
>>>>> I guess the problem here is the lack of "dllexport", right?
>>>>>
>>>>> But, while dllexport is common, some libraries can still export
>>>>> symbols
>>>>> using a .def file [1].
>>>>>
>>>>> So, I'm not sure this belongs in the same category as "static inline",
>>>>> as it is not possible, just by looking at the header, to understand
>>>>> whether the symbol will be present or not?
>>>>>
>>>>> [1] -
>>>>>
>>>>> https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170
>>>>> <https://urldefense.com/v3/__https://docs.microsoft.com/en-us/cpp/build/exporting-from-a-dll-using-def-files?view=msvc-170__;!!ACWV5N9M2RV99hQ!IZ9OZ1GPWyZCLxdiYrnGJewYmhC_BCpH8sxH4Lo2jBtSsuMJPH_j3-vIXRO1emIKo2SMrT3Sj1AqI3daPLx2VOq6oGrpPgb-Xw$>
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220906/ed1a19fe/attachment-0001.htm>