RFR: 8358957: [ubsan]: The assert in layout_helper_boolean_diffbit() in klass.hpp needs UB to fail

[Kim Barrett]

On Mon, 15 Sep 2025 09:01:56 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

> Avoid using loop and UB in left-shift operation as suggested by Kim's comment in the JBS-issue.
> 
> Tests:
>  mach5 tiers 1-5 {macosx-aarch64, linux-x64, windows-x64} x {debug, product}

src/hotspot/share/oops/klass.hpp line 518:

> 516:   static int layout_helper_boolean_diffbit() {
> 517:     uint zlh = checked_cast<uint>(array_layout_helper(T_BOOLEAN));
> 518:     uint blh = checked_cast<uint>(array_layout_helper(T_BYTE));

Use of check_cast is probably wrong. I think an alh is negative. Oops, my mistake. It probably doesn't fail currently because of [JDK-8314258](https://bugs.openjdk.org/browse/JDK-8314258).

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27288#discussion_r2349874742