RFR: 8361211: C2: Final graph reshaping generates unencodeable klass constants

[Vladimir Kozlov]

On Wed, 30 Jul 2025 16:20:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for more investigation. I have tried to come up with an isolated test, but failed. So I am doing this change somewhat blindly, without a clear regression test. The investigation on the CTW points directly to this code, and I believe we should be more conservative in final graph reshaping. [JDK-8343206](https://bugs.openjdk.org/browse/JDK-8343206) added the assert for `ConNKlass`, which somehow does not trigger. I think it is safe to bail out of this transformation.
> 
> Also, this only plugs this particular leak. I think we should really be disabling the abstract/interface encoding optimization until C2 does not expose itself to this issue on more paths. There is [JDK-8343218](https://bugs.openjdk.org/browse/JDK-8343218) that we can re-open.
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, a rare CTW failure does not reproduce anymore
>  - [x] Linux x86_64 server fastdebug, `tier1`
>  - [ ] Linux x86_64 server fastdebug, `all`
>  - [ ] Linux AArch64 server fastdebug, `all`

Do you know why logic in `CmpPNode::Ideal()` did not work?  That is what @TobiHartmann pointed before.

I have  2 assumptions which could be wrong:

  - We did not call IGVN transform when we do some constant folding and replaced its input with more exact klass which is note encodable.
  - Node's inputs were swapped when `CmpPNode::Ideal()` is called - code assumes that Decode is in(1) and Klass is in(2).
  
May be it is something else.

Would be interesting to track it done because it may cause other issues.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26559#issuecomment-3138073871