RFR: 8352140: UBSAN: fix the left shift of negative value in klass.hpp, array_layout_helper()

[Afshin Zafari]

On Wed, 26 Mar 2025 18:07:02 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> I had to emphasize that the case shown in the example may happen at run-time where compiler has no chance to warn/avoid/address it. 
>> My concern is that developers should not rely on the compiler to check the validation of left-shift op. They should be aware of the `signed` <-> `unsigned` and `int` <-> `long` <-> `long long` conversions during the left-shift.
>> To find invalid cases of left-shift, UBSAN instruments them with assertions to catch them at run-time. If the assertion raised, good we found the problem. However, if no assertion raised for some left-shift ops, it doesn't mean that they are valid.
>
> Is there a way to tell ubsan that we care about detecting overflows, but we do not care about detecting
> left shift of a negative value?  Not that I can find, but maybe I missed something.  `-fsanitize=shift-base`
> looks like it would check for both overflow and (prior to C++20) negative base.  We could disable
> shift-base checking and do our own overflow assertion.  (Which might want to be packaged up in a helper,
> as discussed in https://github.com/openjdk/jdk/pull/24196.)

For the case in this PR, the left-shift results in overflow, since the operand is either 0xFFFFFFFF or 0xFFFFFFFE. Should we have two versions of `left_shift()` and `left_shift_no_overflow()`.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24184#discussion_r2018650745