RFR: 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper() [v3]

[Dean Long]

On Tue, 13 May 2025 12:54:11 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>> The `array_layout_helper()` with `jint tag` as its first arg, is called with a `tag` whose sign-bit is always set and considered as negative. This negative value is UB in left-shift operation.  Changing the type to `juint`  fixes this.
>> 
>> Tests:
>> linux-x64-debug tier1 with UBSAN enabled.
>
> Afshin Zafari has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - Merge remote-tracking branch 'origin/master' into _8352140_lshift_klass_hpp
>  - minimum change.
>  - Merge remote-tracking branch 'origin/master' into _8352140_lshift_klass_hpp
>  - 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper()

src/hotspot/share/oops/klass.hpp line 526:

> 524:     return l2esz;
> 525:   }
> 526:   static jint array_layout_helper(uint32_t tag, int hsize, BasicType etype, int log2_esize) {

Suggestion:

  static jint array_layout_helper(jint tag, int hsize, BasicType etype, int log2_esize) {
    tag &= right_n_bits(_lh_array_tag_bits);

I'd like to propose a simpler fix that doesn't involve changing to unsigned.  In my experiments this satisfies UBSAN while still generating the same optimized code in product builds.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24184#discussion_r2090835514