RFR: 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper()

[David Holmes]

On Tue, 25 Mar 2025 06:07:45 GMT, Dean Long <dlong at openjdk.org> wrote:

>> The `array_layout_helper()` with `jint tag` as its first arg, is called with a `tag` whose sign-bit is always set and considered as negative. This negative value is UB in left-shift operation.  Changing the type to `juint`  fixes this.
>> 
>> Tests:
>> linux-x64-debug tier1 with UBSAN enabled.
>
> src/hotspot/share/oops/klass.hpp line 527:
> 
>> 525:   }
>> 526:   static jint array_layout_helper(jint tag, int hsize, BasicType etype, int log2_esize) {
>> 527:     return ((juint)tag << _lh_array_tag_shift)
> 
> Doesn't this turn the type of the return expression to unsigned, causing an implicit conversion back to signed, which is implementation-defined?  I think that's the reason for the weird-looking reinterpret_cast<> in JAVA_INTEGER_OP.

Note we seem to do this a lot e.g.

> ./share/oops/klass.cpp:  `int  tag   =  isobj ? _lh_array_tag_obj_value : _lh_array_tag_type_value;`

`_lh_array_tag_type_value` is unsigned. So this call sequence goes from unsigned -> signed -> unsigned -> signed

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24184#discussion_r2011428322