Bug(s) in improved profiled predicates

[Aleksey Shipilev]

Improved profiled predicates were pushed by Roland recently for stabilization in our forest before
upstreaming it:
  http://hg.openjdk.java.net/shenandoah/jdk10/rev/279bd9767680

Below are bugs that seem to reproduce with x86_32 slowdebug build. It crashed during jmod creation
(with G1) with hs_err like this:
  http://cr.openjdk.java.net/~shade/shenandoah/wtf-profiled-predicates/hs_err_pid24920.log

#  Internal Error (/home/user/shenandoah-jdk10/src/hotspot/share/oops/methodData.hpp:2510),
pid=24920, tid=24947
#  assert((uint)reason < _trap_hist_limit) failed: oob

The obvious patch to that is:

diff -r 16f17f40cecf src/hotspot/share/oops/methodData.hpp
--- a/src/hotspot/share/oops/methodData.hpp	Thu Feb 08 13:24:03 2018 +0100
+++ b/src/hotspot/share/oops/methodData.hpp	Fri Feb 09 12:01:48 2018 +0100
@@ -2175,7 +2175,7 @@

   // Whole-method sticky bits and flags
   enum {
-    _trap_hist_limit    = 23 JVMCI_ONLY(+5),   // decoupled from Deoptimization::Reason_LIMIT
+    _trap_hist_limit    = 24 JVMCI_ONLY(+5),   // decoupled from Deoptimization::Reason_LIMIT
     _trap_hist_mask     = max_jubyte,
     _extra_data_count   = 4     // extra DataLayout headers, for trap history
   }; // Public flag values


...and then it starts to fail with:
  http://cr.openjdk.java.net/~shade/shenandoah/wtf-profiled-predicates/hs_err_pid25082.log

#  Internal Error (/home/user/shenandoah-jdk10/src/hotspot/share/runtime/deoptimization.cpp:2103),
pid=25082, tid=25083
#  assert(DS_REASON_MASK >= Reason_RECORDED_LIMIT) failed: enough bits

Roland's original patch changed Reason_RECORDED_LIMIT to be Reason_profile_predicate, from
Reason_bimorphic, but that also captured JVMCI events? Anyhow, this patch failed with the same
assert as well:

diff -r 16f17f40cecf src/hotspot/share/runtime/deoptimization.hpp
--- a/src/hotspot/share/runtime/deoptimization.hpp	Thu Feb 08 13:24:03 2018 +0100
+++ b/src/hotspot/share/runtime/deoptimization.hpp	Fri Feb 09 12:16:39 2018 +0100
@@ -56,14 +56,14 @@
     Reason_intrinsic,             // saw unexpected operand to intrinsic (@bci)
     Reason_bimorphic,             // saw unexpected object class in bimorphic inlining (@bci)

+    Reason_profile_predicate,     // compiler generated predicate moved from frequent branch in a
loop failed
+
 #if INCLUDE_JVMCI
     Reason_unreached0             = Reason_null_assert,
     Reason_type_checked_inlining  = Reason_intrinsic,
     Reason_optimized_type_check   = Reason_bimorphic,
 #endif

-    Reason_profile_predicate,     // compiler generated predicate moved from frequent branch in a
loop failed
-
     // recorded per method
     Reason_unloaded,              // unloaded class or constant pool entry
     Reason_uninitialized,         // bad class state (uninitialized)


-Aleksey