RFR: 8257594: C2 compiled checkcast of non-null object triggers endless deoptimization/recompilation cycle

[Tobias Hartmann]

When C2 determines at compile time that there is no subtype relation between the objects involved in a checkcast, it inserts a null assert because the checkcast can only pass if the value is null. This will trigger deoptimization and recompilation if the object is non-null. However, we can do better here if we already know that the object is non-null (i.e. the check always fails) or if we hit `too_many_traps_or_recompiles` (which suggests that the object is often null) to avoid an endless deoptimization/recompilation cycle until we hit the `PerMethodRecompilationCutoff` and starve at lower tiers.

In the former case we can simply throw right away and in the latter case we should fall back to a full check in case there is an exception handler in compiled code that handles the failing (non-null) case. That way we avoid deoptimization.

I've spotted this with our test framework for inline types in project Valhalla, while investigating an intermittent failure where several test methods suddenly became non-compilable at tier 4.

Thanks,
Tobias

-------------

Commit messages:
 - Handle non-constant, non-null value
 - 8257594: C2 compiled checkcast of non-null object triggers endless deoptimization/recompilation cycle

Changes: https://git.openjdk.java.net/jdk/pull/1566/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=1566&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8257594
  Stats: 133 lines in 4 files changed: 131 ins; 0 del; 2 mod
  Patch: https://git.openjdk.java.net/jdk/pull/1566.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/1566/head:pull/1566

PR: https://git.openjdk.java.net/jdk/pull/1566