[9, 8u40] RFR (XS): 8068915: uncommon trap w/ Reason_speculate_class_check causes performance regression due to continuous deoptimizations

[Vladimir Ivanov]

http://cr.openjdk.java.net/~vlivanov/8068915/webrev.00
https://bugs.openjdk.java.net/browse/JDK-8068915

The fix for 8063137 [1] (just sent out for review) uncovered another 
issue with deoptimization counts pollution. In some cases, speculative 
traps can stuck continuously deoptimizing and never reach recompilation. 
Usually, it ruins performance if it happens.

When a speculative guard is considered, Compile::too_many_traps() is 
consulted to decide whether to add it or not. But, uncommon trap action 
can be changed to Action_none in GraphKit::uncommon_trap, if 
Compile::too_many_recompiles() fires.

The fix is to (1) forbid changing uncommon trap action under the hood, 
and (2) consult Compile::too_many_recompiles when adding a speculative 
guard.

The fix is based on 8063137 [2] (uncommon_trap_exact is used).

Testing: JPRT, java/lang/invoke tests, nashorn.

Thanks!

Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8063137
[2] http://cr.openjdk.java.net/~vlivanov/8063137/webrev.00/hotspot/