RFR: 8257594: C2 compiled checkcast of non-null object triggers endless deoptimization/recompilation cycle

[Roland Westrelin]

On Wed, 2 Dec 2020 15:22:57 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

> When C2 determines at compile time that there is no subtype relation between the objects involved in a checkcast, it inserts a null assert because the checkcast can only pass if the value is null. This will trigger deoptimization and recompilation if the object is non-null. However, we can do better here if we already know that the object is non-null (i.e. the check always fails) or if we hit `too_many_traps_or_recompiles` (which suggests that the object is often null) to avoid an endless deoptimization/recompilation cycle until we hit the `PerMethodRecompilationCutoff` and starve at lower tiers.
> 
> In the former case we can simply throw right away and in the latter case we should fall back to a full check in case there is an exception handler in compiled code that handles the failing (non-null) case. That way we avoid deoptimization.
> 
> I've spotted this with our test framework for inline types in project Valhalla, while investigating an intermittent failure where several test methods suddenly became non-compilable at tier 4.
> 
> Thanks,
> Tobias

Looks good to me.

-------------

Marked as reviewed by roland (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/1566