RFR: 8358957: [ubsan]: The assert in layout_helper_boolean_diffbit() in klass.hpp needs UB to fail [v4]

[Kim Barrett]

On Mon, 15 Sep 2025 19:10:14 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> Afshin Zafari has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   comments and post-cond
>
> src/hotspot/share/oops/klass.hpp line 518:
> 
>> 516:   static int layout_helper_boolean_diffbit() {
>> 517:     uint zlh = checked_cast<uint>(array_layout_helper(T_BOOLEAN));
>> 518:     uint blh = checked_cast<uint>(array_layout_helper(T_BYTE));
> 
> Use of check_cast is probably wrong. I think an alh is negative. Oops, my mistake. It probably doesn't fail currently because of [JDK-8314258](https://bugs.openjdk.org/browse/JDK-8314258).

Note that by "my mistake" I meant it was a mistake to use `checked_cast` here.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27288#discussion_r2488603639