Provide API points for implementing linkers with non-standard calling conventions

Fri Oct 11 14:50:50 UTC 2024

> As far as I'm aware, there are no implementations that allow you to load
a 32-bit library into a 64-bit process (and vice versa of course).

I don't program for PC very often, I need this functionality to implement
Panama on Android. In the implementation of SymbolLookup.loaderLookup(), I
scan all libraries connected to the current class loader. And some of them
may actually have a different ABI - via the native bridge mechanism.

https://android.googlesource.com/platform/art/+/refs/heads/main/libnativebridge/

пт, 11 окт. 2024 г., 21:41 Jorn Vernee <jorn.vernee at oracle.com>:

> As far as I'm aware, there are no implementations that allow you to load a
> 32-bit library into a 64-bit process (and vice versa of course). Though, it
> is possible to run 32-bit processes on 64-bit machines through emulation
> (e.g. Wow64 on Windows). Do you have an example of loading 32-bit libraries
> into 64-bit processes?
>
> Jorn
> On 11-10-2024 15:46, Владимир Козелков wrote:
>
> Thanks for the answer.
>
> At the moment, support on 64-bit architectures, their 32-bit variants are
> very difficult, and I see several problems with this.
>
> It seems to me that the main problem is in considering addresses outside
> of Linker and the existence of the ValueLayout.ADDRESS constant. All
> ValueLayout.JAVA_* constants have the same size, alignment and byte order
> on all platforms - this is determined by the Java platform itself. All
> native layouts are inside Linker.canonicalLayouts(), except for addresses
> (which are *always *platform-dependent). Why?
>
> If we really want to support multiple calling conventions for ABIs with
> different bit depths (and this is the most common case of different ABIs on
> the same platform), we will also need to add support for AddressLayouts not
> only of different alignments, but also of different sizes, which will
> require non-trivial handling in VarHandles and some other places. In this
> case, we also need to say that ValueLayout.ADDRESS refers to
> Linker.nativeLayout(), but there could be others...
>
> Unfortunately, there are problems not only with layouts, but also with
> memory segments. 32-bit ABIs only support 32-bit addresses, as funny as it
> may sound. So standard memory segments are unlikely to be used with 32-bit
> ABIs - you need a linker-dependent way to allocate memory, for example only
> in the first four gigabytes of process memory (I know for sure that Linux
> supports this)
>
> All of this needs to be carefully thought out and reflected in the
> documentation, which can require a lot of work. This seems like a pretty
> big and radical change, but it is possible.
>
> Cheers
> Vladimir
>
>
> пт, 11 окт. 2024 г. в 16:46, Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com>:
>
>> Hi,
>> Having the ability to select different calling conventions (or, more
>> accurately, completely different ABIs) is a powerful trick. It comes in
>> especially handy in cases that I'd call foreign^2 - that is, when you want
>> to talk to some native function that adopts calling conventions that are
>> not first-class on that particular system. I view x86 on x64 and x64 on
>> arm64 as largely similar in spirit.
>>
>> That is in contrast, IMHO with the situation we had with x86 - where
>> multiple competing calling conventions often existed within the same system
>> (sometimes with the intent of providing better performances in certain
>> contexts). Windows x86 supports six (!!) calling conventions [1]. By
>> contrast, on Windows x64 there's only two (__vectorcall is apparently still
>> around, although I don't know how widely used). Other platforms followed a
>> similar evolution.
>>
>> The cross-architecture-compatibility use case you mention is an emerging
>> important one, so we will keep an eye in this space for sure.
>>
>> Maurizio
>>
>> [1] -
>> https://learn.microsoft.com/en-us/cpp/cpp/argument-passing-and-naming-conventions?view=msvc-170
>> On 11/10/2024 01:48, Владимир Козелков wrote:
>>
>> I think the main use of alternative linkers is to reflect the existing
>> ability of systems to run binaries from other platforms.
>>
>> In my example, it was possible to use old binaries for 32-bin systems on
>> 64-bit systems. But platforms are not limited to this. You were wrong when
>> you said about the unified calling convention on new architectures - just
>> look at the ARM64EC calling convention - it allows an application to have
>> both aarch64 and x86_64 binaries in the process!
>>
>> Also... I'm confused by the existence of Wine on Linux - it provides a
>> platform for running binaries of the same architecture, but of a different
>> operating system (Windows). Unfortunately, I don't know if it has the
>> ability to have a process with mixed binaries and how this relates to Java,
>> but this is also an interesting example.
>>
>> пт, 11 окт. 2024 г., 4:04 Maurizio Cimadamore <
>> maurizio.cimadamore at oracle.com>:
>>
>>> Hi,
>>> as you noticed, while the Linker javadoc alludes at the fact that there
>>> might be other calling conventions supported in the future, at the
>>> moment there's no API to expose this. What we had in mind the last time
>>> we discussed this was not too dissimilar to what you propose here -
>>> basically just keep calling convention open, by using strings, and then
>>> allow the "nativeLinker" factory to accept a calling convention string.
>>>
>>> Another possibility would be to use linker options - e.g. have a
>>> CallingConvention linker option that can be passed to
>>> downcallHandle/upcallStub. This would allow to keep a single linker, but
>>> to support downcalls with different calling conventions. Both approaches
>>> are equally expressive, at least in terms of allowing to call functions
>>> using different argument shuffling. That said, on some platforms, like
>>> PowerPC support for instance different kind of endianness. So perhaps it
>>> would be good to have a way to ask for the "big endian" Linker, whose
>>> canonical layouts will be... big endian. That is, a Linker is about
>>> functions as much as it is about the definition of fundamental data
>>> types. So, perhaps when adding support for different Linker "flavors" it
>>> would be good to keep this in mind.
>>>
>>> The reason we left this out in 22 was that we wanted to learn more use
>>> cases where this was useful. For instance, while it's true that x86
>>> supported several calling conventions, modern systems seems to have
>>> evolved a bit, so that each major platform tend to gravitate towards one
>>> main set of calling convention, typically specified in that platform's
>>> ABI (e.g. SysV for Linux). It seems to me that even in your case, the
>>> main driver for selecting an alternate calling convention is x86 really.
>>> So I'm still not 100% sure that this is something worth pursuing. I
>>> would feel more at ease if we had more cases where this was useful.
>>>
>>> Cheers
>>> Maurizio
>>>
>>>
>>> On 10/10/2024 20:14, Владимир Козелков wrote:
>>> > Greetings,
>>> >
>>> > The documentation for the Linker.nativeLinker() method says: "It is
>>> > not currently possible to obtain a linker for a different combination
>>> > of OS and processor."
>>> >
>>> > This is indeed true for hotspot, but what if another implementation
>>> > could provide the ability to create a linker for a different calling
>>> > convention? Even if the implementation wanted to do this, it would
>>> > fail because the API does not provide any points through which this
>>> > could be done.
>>> >
>>> > As an example - android allows us to use binaries for arm in aarch64
>>> > and for x86 in x86_64 with JNI. In the current implementation, I have
>>> > to filter the output of SymbolLookup.loaderLookup() so that the user
>>> > does not get symbols with a different calling convention, although the
>>> > platform really allows to use them.
>>> >
>>> > Additionally, I would like to note that the x86 and x86_64 platforms
>>> > have several "native" calling conventions, such as cdecl (which is
>>> > actually used now), fastcall, vectorcall, etc. Even if a hotspot does
>>> > not allow these calling conventions, it would be useful to have at
>>> > least the potential to implement them.
>>> >
>>> > I can suggest a not very good and naive method for solving the problem
>>> > - it is inspired by target-triple from LLVM:
>>> >
>>> > interface Linker ... {
>>> >     static List<String> supportedConventions() {return ... ;}
>>> >     static String defaultConvention() {return ... ;}
>>> >     static boolean isSupportedConvention(String convention) {return
>>> ... ;}
>>> >     static Linker linkerForConvention(String  convention) {return ...
>>> ;}
>>> >     static Linker nativeLinker() {
>>> >         return linkerForConvention(defaultConvention());
>>> >     }
>>> > }
>>> >
>>> > For android aarch64 defaultConvention() will return something like
>>> > "aarch64-android-cdecl"
>>> >
>>> > Thanks for reading
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20241011/6d4d5246/attachment-0001.htm>