RFR: JDK-8314163: os::print_hex_dump prints incorrectly for big endian platforms and unit sizes larger than 1

[Aleksey Shipilev]

On Sat, 12 Aug 2023 07:55:25 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> See comment. Seen on AIX. Big-endian is a pain.
> 
> Thanks to @MBaesken for testing the patch.

src/hotspot/share/runtime/os.cpp line 968:

> 966:     // bytoff:   0  1  2  3  4  5  6  7
> 967:     // LE bits  0  8  16 24 32 40 48 56
> 968:     // BE bits  56 48 40 32 24 16  8  0

Fixing a bit of misalignment (pun intended):

Suggestion:

    // bytes:   CA FE BA BE DE AD C0 DE
    // bytoff:   0  1  2  3  4  5  6  7
    // LE bits:  0  8 16 24 32 40 48 56
    // BE bits: 56 48 40 32 24 16  8  0

src/hotspot/share/runtime/os.cpp line 972:

> 970:     const int bitoffset =
> 971:       LITTLE_ENDIAN_ONLY(offset * BitsPerByte)
> 972:       BIG_ENDIAN_ONLY((int)((sizeof(intptr_t) - unitsize - offset) * BitsPerByte));

Um, my head hurts a bit thinking about this. So for 64-bit `sizeof(intptr_t)` = 8 and unitsize=8, `bitoffset` is negative? Which we then feed into `bitfield`, which does the shift by the negative value, which is undefined behavior?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/15256#discussion_r1294440066
PR Review Comment: https://git.openjdk.org/jdk/pull/15256#discussion_r1294446709