RFR: 8361211: C2: Final graph reshaping generates unencodeable klass constants

[Aleksey Shipilev]

On Mon, 4 Aug 2025 12:34:33 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> See the bug for more investigation. I have tried to come up with an isolated test, but failed. So I am doing this change somewhat blindly, without a clear regression test. The investigation on the CTW points directly to this code, and I believe we should be more conservative in final graph reshaping. [JDK-8343206](https://bugs.openjdk.org/browse/JDK-8343206) added the assert for `ConNKlass`, which somehow does not trigger. I think it is safe to bail out of this transformation.
>> 
>> Also, this only plugs this particular leak. I think we should really be disabling the abstract/interface encoding optimization until C2 does not expose itself to this issue on more paths. There is [JDK-8343218](https://bugs.openjdk.org/browse/JDK-8343218) that we can re-open.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 server fastdebug, a rare CTW failure does not reproduce anymore
>>  - [x] Linux x86_64 server fastdebug, `tier1`
>>  - [x] Linux x86_64 server fastdebug, `all`
>>  - [x] Linux AArch64 server fastdebug, `all`
>
> src/hotspot/share/opto/compile.cpp line 3663:
> 
>> 3661:         n->subsume_by(ConNode::make(t->make_narrowoop()), this);
>> 3662:       } else if (t->isa_klassptr()) {
>> 3663:         ciKlass* klass = t->is_klassptr()->exact_klass();
> 
> This branch means that we are trying to compress a pointer that cannot be compressed. This seems wrong either way.

Right! This replaces (`ConP` -> `EncodeP`) -> `ConN`. While I agree that it should theoretically be handled on `EncodeP` path, it seems sane to gate the conversion here as well. Issues like this is why I want to disable/revert the optimization that puts us into this situation.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26559#discussion_r2251425489