Provide API points for implementing linkers with non-standard calling conventions

Fri Oct 11 16:19:05 UTC 2024

There seems to be an even stranger example: running riscv64 code on 32-bit
x86. I'm not sure about this - the documentation says that the library is
intended for riscv64 code on x86_64, but... If you search a little, x86
seems to be supported too

https://android.googlesource.com/platform/frameworks/libs/binary_translation/+/refs/heads/main/

пт, 11 окт. 2024 г., 21:50 Владимир Козелков <vova20037878 at gmail.com>:

> > As far as I'm aware, there are no implementations that allow you to load
> a 32-bit library into a 64-bit process (and vice versa of course).
>
> I don't program for PC very often, I need this functionality to implement
> Panama on Android. In the implementation of SymbolLookup.loaderLookup(), I
> scan all libraries connected to the current class loader. And some of them
> may actually have a different ABI - via the native bridge mechanism.
>
>
> https://android.googlesource.com/platform/art/+/refs/heads/main/libnativebridge/
>
> пт, 11 окт. 2024 г., 21:41 Jorn Vernee <jorn.vernee at oracle.com>:
>
>> As far as I'm aware, there are no implementations that allow you to load
>> a 32-bit library into a 64-bit process (and vice versa of course). Though,
>> it is possible to run 32-bit processes on 64-bit machines through emulation
>> (e.g. Wow64 on Windows). Do you have an example of loading 32-bit libraries
>> into 64-bit processes?
>>
>> Jorn
>> On 11-10-2024 15:46, Владимир Козелков wrote:
>>
>> Thanks for the answer.
>>
>> At the moment, support on 64-bit architectures, their 32-bit variants are
>> very difficult, and I see several problems with this.
>>
>> It seems to me that the main problem is in considering addresses outside
>> of Linker and the existence of the ValueLayout.ADDRESS constant. All
>> ValueLayout.JAVA_* constants have the same size, alignment and byte order
>> on all platforms - this is determined by the Java platform itself. All
>> native layouts are inside Linker.canonicalLayouts(), except for addresses
>> (which are *always *platform-dependent). Why?
>>
>> If we really want to support multiple calling conventions for ABIs with
>> different bit depths (and this is the most common case of different ABIs on
>> the same platform), we will also need to add support for AddressLayouts not
>> only of different alignments, but also of different sizes, which will
>> require non-trivial handling in VarHandles and some other places. In this
>> case, we also need to say that ValueLayout.ADDRESS refers to
>> Linker.nativeLayout(), but there could be others...
>>
>> Unfortunately, there are problems not only with layouts, but also with
>> memory segments. 32-bit ABIs only support 32-bit addresses, as funny as it
>> may sound. So standard memory segments are unlikely to be used with 32-bit
>> ABIs - you need a linker-dependent way to allocate memory, for example only
>> in the first four gigabytes of process memory (I know for sure that Linux
>> supports this)
>>
>> All of this needs to be carefully thought out and reflected in the
>> documentation, which can require a lot of work. This seems like a pretty
>> big and radical change, but it is possible.
>>
>> Cheers
>> Vladimir
>>
>>
>> пт, 11 окт. 2024 г. в 16:46, Maurizio Cimadamore <
>> maurizio.cimadamore at oracle.com>:
>>
>>> Hi,
>>> Having the ability to select different calling conventions (or, more
>>> accurately, completely different ABIs) is a powerful trick. It comes in
>>> especially handy in cases that I'd call foreign^2 - that is, when you want
>>> to talk to some native function that adopts calling conventions that are
>>> not first-class on that particular system. I view x86 on x64 and x64 on
>>> arm64 as largely similar in spirit.
>>>
>>> That is in contrast, IMHO with the situation we had with x86 - where
>>> multiple competing calling conventions often existed within the same system
>>> (sometimes with the intent of providing better performances in certain
>>> contexts). Windows x86 supports six (!!) calling conventions [1]. By
>>> contrast, on Windows x64 there's only two (__vectorcall is apparently still
>>> around, although I don't know how widely used). Other platforms followed a
>>> similar evolution.
>>>
>>> The cross-architecture-compatibility use case you mention is an emerging
>>> important one, so we will keep an eye in this space for sure.
>>>
>>> Maurizio
>>>
>>> [1] -
>>> https://learn.microsoft.com/en-us/cpp/cpp/argument-passing-and-naming-conventions?view=msvc-170
>>> On 11/10/2024 01:48, Владимир Козелков wrote:
>>>
>>> I think the main use of alternative linkers is to reflect the existing
>>> ability of systems to run binaries from other platforms.
>>>
>>> In my example, it was possible to use old binaries for 32-bin systems on
>>> 64-bit systems. But platforms are not limited to this. You were wrong when
>>> you said about the unified calling convention on new architectures - just
>>> look at the ARM64EC calling convention - it allows an application to have
>>> both aarch64 and x86_64 binaries in the process!
>>>
>>> Also... I'm confused by the existence of Wine on Linux - it provides a
>>> platform for running binaries of the same architecture, but of a different
>>> operating system (Windows). Unfortunately, I don't know if it has the
>>> ability to have a process with mixed binaries and how this relates to Java,
>>> but this is also an interesting example.
>>>
>>> пт, 11 окт. 2024 г., 4:04 Maurizio Cimadamore <
>>> maurizio.cimadamore at oracle.com>:
>>>
>>>> Hi,
>>>> as you noticed, while the Linker javadoc alludes at the fact that there
>>>> might be other calling conventions supported in the future, at the
>>>> moment there's no API to expose this. What we had in mind the last time
>>>> we discussed this was not too dissimilar to what you propose here -
>>>> basically just keep calling convention open, by using strings, and then
>>>> allow the "nativeLinker" factory to accept a calling convention string.
>>>>
>>>> Another possibility would be to use linker options - e.g. have a
>>>> CallingConvention linker option that can be passed to
>>>> downcallHandle/upcallStub. This would allow to keep a single linker,
>>>> but
>>>> to support downcalls with different calling conventions. Both
>>>> approaches
>>>> are equally expressive, at least in terms of allowing to call functions
>>>> using different argument shuffling. That said, on some platforms, like
>>>> PowerPC support for instance different kind of endianness. So perhaps
>>>> it
>>>> would be good to have a way to ask for the "big endian" Linker, whose
>>>> canonical layouts will be... big endian. That is, a Linker is about
>>>> functions as much as it is about the definition of fundamental data
>>>> types. So, perhaps when adding support for different Linker "flavors"
>>>> it
>>>> would be good to keep this in mind.
>>>>
>>>> The reason we left this out in 22 was that we wanted to learn more use
>>>> cases where this was useful. For instance, while it's true that x86
>>>> supported several calling conventions, modern systems seems to have
>>>> evolved a bit, so that each major platform tend to gravitate towards
>>>> one
>>>> main set of calling convention, typically specified in that platform's
>>>> ABI (e.g. SysV for Linux). It seems to me that even in your case, the
>>>> main driver for selecting an alternate calling convention is x86
>>>> really.
>>>> So I'm still not 100% sure that this is something worth pursuing. I
>>>> would feel more at ease if we had more cases where this was useful.
>>>>
>>>> Cheers
>>>> Maurizio
>>>>
>>>>
>>>> On 10/10/2024 20:14, Владимир Козелков wrote:
>>>> > Greetings,
>>>> >
>>>> > The documentation for the Linker.nativeLinker() method says: "It is
>>>> > not currently possible to obtain a linker for a different combination
>>>> > of OS and processor."
>>>> >
>>>> > This is indeed true for hotspot, but what if another implementation
>>>> > could provide the ability to create a linker for a different calling
>>>> > convention? Even if the implementation wanted to do this, it would
>>>> > fail because the API does not provide any points through which this
>>>> > could be done.
>>>> >
>>>> > As an example - android allows us to use binaries for arm in aarch64
>>>> > and for x86 in x86_64 with JNI. In the current implementation, I have
>>>> > to filter the output of SymbolLookup.loaderLookup() so that the user
>>>> > does not get symbols with a different calling convention, although
>>>> the
>>>> > platform really allows to use them.
>>>> >
>>>> > Additionally, I would like to note that the x86 and x86_64 platforms
>>>> > have several "native" calling conventions, such as cdecl (which is
>>>> > actually used now), fastcall, vectorcall, etc. Even if a hotspot does
>>>> > not allow these calling conventions, it would be useful to have at
>>>> > least the potential to implement them.
>>>> >
>>>> > I can suggest a not very good and naive method for solving the
>>>> problem
>>>> > - it is inspired by target-triple from LLVM:
>>>> >
>>>> > interface Linker ... {
>>>> >     static List<String> supportedConventions() {return ... ;}
>>>> >     static String defaultConvention() {return ... ;}
>>>> >     static boolean isSupportedConvention(String convention) {return
>>>> ... ;}
>>>> >     static Linker linkerForConvention(String  convention) {return ...
>>>> ;}
>>>> >     static Linker nativeLinker() {
>>>> >         return linkerForConvention(defaultConvention());
>>>> >     }
>>>> > }
>>>> >
>>>> > For android aarch64 defaultConvention() will return something like
>>>> > "aarch64-android-cdecl"
>>>> >
>>>> > Thanks for reading
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20241011/64702124/attachment-0001.htm>