RFR: 8303524: Check FunctionDescriptor byte order when linking

[Jorn Vernee]

On Fri, 24 Mar 2023 08:02:05 GMT, ExE Boss <duke at openjdk.org> wrote:

>> Linkers are strongly tied to a particular byte order, because they are tied to a particular platform. So, the linker should reject layouts that have another byte order. This patch implements that check.
>
> src/java.base/share/classes/jdk/internal/foreign/abi/AbstractLinker.java line 114:
> 
>> 112:     private void checkLayouts(FunctionDescriptor descriptor) {
>> 113:         descriptor.returnLayout().ifPresent(this::checkLayoutsRecursive);
>> 114:         descriptor.argumentLayouts().forEach(this::checkLayoutsRecursive);
> 
> Storing `this::checkLayoutsRecursive` in a local variable allows the `Consumer` to be reused for the return and argument layouts, thus only one lambda class and instance needs to be allocated instead of two.
> Suggestion:
> 
>         final Consumer<MemoryLayout> checker = this::checkLayoutsRecursive;
>         descriptor.returnLayout().ifPresent(checker);
>         descriptor.argumentLayouts().forEach(checker);

Thanks for the suggestion, but I won't engage in speculative performance engineering. Any attempts to improve performance should be paired with thorough benchmarking to verify that effort is being made in the right area (the `checkLayouts` method barely shows up in profiles) and to validate the changes made actually result in a net improvement. But I think that's outside of the scope of this PR.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13161#discussion_r1147673174