RFR: 8371297: C2: assert triggerred in BoolTest::BoolTest

[Hamlin Li]

On Wed, 5 Nov 2025 21:20:46 GMT, Dean Long <dlong at openjdk.org> wrote:

>> Hi,
>> Can you help to review this patch?
>> 
>> Not sure how this one slipped in (https://github.com/openjdk/jdk/pull/28047/), I think it passed my local test and github CI test at that time.
>> Please check https://bugs.openjdk.org/browse/JDK-8371297 for details.
>> 
>> Thanks!
>
> src/hotspot/share/opto/vtransform.cpp line 1427:
> 
>> 1425:   const BoolTest bt(m);
>> 1426:   tty->print(" test=%s", m == _test._mask ? "" : "unsigned ");
>> 1427:   bt.dump_on(tty);
> 
> I was wondering why we pass the raw mask around instead of keeping it encapsulated in a BoolTest object.  Elsewhere I saw code like this: 
> `cond->get_con() & (BoolTest::unsigned_compare - 1)`
> which seems to be making fragile assumptions about BoolTest internals.

@dean-long Yes, I have the same feeling that BoolTest is currently used in a fragile way. The reasons could be, BoolTest itself is by design a struct and expose all its status, and `unsigned_compare` is indeed not supported (well) but needed somewhere e.g. in vector intrinsic, and auto-vectorization (after https://github.com/openjdk/jdk/pull/28047).
I think it's worth to do more investigation about the refactoring of BoolTest, file https://bugs.openjdk.org/browse/JDK-8371396 to track it, feel free to take it if you already have a solution or idea.

This issue (in fact it's https://github.com/openjdk/jdk/pull/27942) blocks several other prs in my backlog for a while, e.g. https://github.com/openjdk/jdk/pull/25336, https://github.com/openjdk/jdk/pull/25341. It helps to resolve this assert in a quick (although ugly) way. Please kindly let me know how you think about it. Thanks!

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28141#discussion_r2498279741