RFR: 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper() [v3]

[Afshin Zafari]

On Tue, 13 May 2025 12:54:11 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>> The `array_layout_helper()` with `jint tag` as its first arg, is called with a `tag` whose sign-bit is always set and considered as negative. This negative value is UB in left-shift operation.  Changing the type to `juint`  fixes this.
>> 
>> Tests:
>> linux-x64-debug tier1 with UBSAN enabled.
>
> Afshin Zafari has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - Merge remote-tracking branch 'origin/master' into _8352140_lshift_klass_hpp
>  - minimum change.
>  - Merge remote-tracking branch 'origin/master' into _8352140_lshift_klass_hpp
>  - 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper()

I made the minimum `int->uint32` change because the layout_helper constants are `public` in the `Kalss` class and they are used all around Hotspot code, particularly in architecture dependent codes. 
Changing them breaks the build at SYMBOL generation phase. 
The two most-significant bits of the layout are used for array types and `lh < 0` (or more precisely `lh < _lh_neutral_value`) is used in C++ and assembly language of arch-dep code. 
@kimbarrett , can we limit the change here to `array_layout_helper` or should we proceed to use `uint32_t` instead?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/24184#issuecomment-2879176401