RFR: 8358957: [ubsan]: The assert in layout_helper_boolean_diffbit() in klass.hpp needs UB to fail

[Kim Barrett]

On Mon, 15 Sep 2025 09:01:56 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

> Avoid using loop and UB in left-shift operation as suggested by Kim's comment in the JBS-issue.
> 
> Tests:
>  mach5 tiers 1-5 {macosx-aarch64, linux-x64, windows-x64} x {debug, product}

src/hotspot/share/oops/klass.hpp line 515:

> 513: 
> 514:   // Want a pattern to quickly diff against layout header in register
> 515:   // find something less clever!

Comment needs to be updated. It should describe what is being calculated, and the 2nd line
is presumably resolved by this change.

src/hotspot/share/oops/klass.hpp line 526:

> 524:     // So use alternate form of negation to avoid warning.
> 525:     uint result = candidates & (~candidates + 1);
> 526:     return static_cast<int>(result);

Since this is the implementation I suggested in JBS, I'm going to leave it to others to approve or not.
Maybe there should be a post-condition check that the result meets the requirements. That's simpler
than adding a gtest or anything like that.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27288#discussion_r2348529098
PR Review Comment: https://git.openjdk.org/jdk/pull/27288#discussion_r2348546757