RFR: 8361211: C2: Final graph reshaping generates unencodeable klass constants

[Quan Anh Mai]

On Wed, 30 Jul 2025 16:20:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for more investigation. I have tried to come up with an isolated test, but failed. So I am doing this change somewhat blindly, without a clear regression test. The investigation on the CTW points directly to this code, and I believe we should be more conservative in final graph reshaping. [JDK-8343206](https://bugs.openjdk.org/browse/JDK-8343206) added the assert for `ConNKlass`, which somehow does not trigger. I think it is safe to bail out of this transformation.
> 
> Also, this only plugs this particular leak. I think we should really be disabling the abstract/interface encoding optimization until C2 does not expose itself to this issue on more paths. There is [JDK-8343218](https://bugs.openjdk.org/browse/JDK-8343218) that we can re-open.
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, a rare CTW failure does not reproduce anymore
>  - [x] Linux x86_64 server fastdebug, `tier1`
>  - [x] Linux x86_64 server fastdebug, `all`
>  - [x] Linux AArch64 server fastdebug, `all`

I'm not sure about the fix. Besides the creation of illegal `ConN` which can be caught during compilation, we also face the circumstances when we try to `EncodeP` an uncompressible pointer, which seems very wrong. Maybe the type system needs to keep track of the compressibility of each pointer and disallows ones that may not be compressible to be the input of an `EncodeP`.

src/hotspot/share/opto/compile.cpp line 3663:

> 3661:         n->subsume_by(ConNode::make(t->make_narrowoop()), this);
> 3662:       } else if (t->isa_klassptr()) {
> 3663:         ciKlass* klass = t->is_klassptr()->exact_klass();

This branch means that we are trying to compress a pointer that cannot be compressed. This seems wrong either way.

-------------

PR Review: https://git.openjdk.org/jdk/pull/26559#pullrequestreview-3083998493
PR Review Comment: https://git.openjdk.org/jdk/pull/26559#discussion_r2251354535