[forein-memaccess] memory layout constants and endianness

[Maurizio Cimadamore]

Hi,
in a recent review [1] I brought up the problem that constants such as 
MemoryLayouts::JAVA_INT, MemoryLayouts::JAVA_FLOAT & friends seem to 
have the _wrong_ endianness; that is they default to big endian (BE), 
while I think that, at least in principle, these constants should really 
return the layout the VM thinks a given primitive type has (hence, they 
should use native endianness (NE)).

There is a tension here: as the memory access API can be used to work 
with both arrays and byte buffers, the semantics for these constants is 
pulled fom different directions. When working with Java arrays, what you 
really want is NE (otherwise you'd be reading and writing with a 
swapping which is the opposite with respect to the one used by the VM). 
Conversely, when working with ByteBuffers, BE seems to be a 
nice-to-have, since most ByteBuffers are created BE (sometimes, even 
unbeknownst to the users).

Fact: no matter what semantics we pick for JAVA_INT, there will be puzzlers.

So, I think I really like the idea that JAVA_INT should eventually move 
to Integer::LAYOUT and just mean "the layout of an int for _this_ VM". 
Which leaves us few options for buffers interop:

(a) provide another set of constants (e.g. BUFFER_INT) which defaults to 
BE and can therefore be used to play with segments which come from 
(some) buffers
(b) do nothing - after all, it is a peculiarity of the ByteBuffer API to 
give special preference to BE polarity. While in some cases this choice 
might result in mismatches when playing with memory segments and 
layouts, such issues can be addressed by flipping the buffer to the 
desired order _before_ accessing it through the segment API.

I'm starting to warm up to (b). I think that having constants which work 
seamlessly with Java arrays is far more important than having constants 
which work well with ByteBuffer. While some users of the ByteBuffer API 
might not be aware of the implicit BE polarity, I do not think that 
something like (a) would provide much extra cover, given that a 
ByteBuffer endianness can always be flipped at any point in time - so 
there's no guarantee that e.g. a BUFFER_INT layout constant (which has 
BE polarity as default) is _really_ going to work.

tl;dr; ByteBuffers are "messy" when it comes to endianness - trying to 
come up with a set of constants which minimize the mismatches when 
working with them from the memory access API is starting to look like a 
siren song.

Maurizio

[1] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2019-November/006735.html