RFR (S): 8011138: C2: stack overflow in compiler thread because of recursive inlining of lambda form methods

[Christian Thalinger]

https://bugs.openjdk.java.net/browse/JDK-8011138
http://cr.openjdk.java.net/~twisti/8011138

8011138: C2: stack overflow in compiler thread because of recursive inlining of lambda form methods
Reviewed-by:

The fix is to walk the JVM states when inlining compiled lambda forms and try to figure out if the "receiver" of the same inlinee (static method in the compiled lambda form) is the same as the current one. If that's the case we have a recursion.

I was tempted to remove the InlineTree::_caller_jvms completely but that might be too intrusive for now.

Tested with the failing test case and Octane on Nashorn to verify that I didn't break anything.