C2 compiler hangs and eats memory in escape analysis

[Fredrik Tolf]

Dear list,

I'm not all too familiar with the OpenJDK project, so if this is not the right 
place to discuss bugs, please tell me off and I'll know better.

Otherwise, I seem to have encountered a bug in the C2 compiler where it appears 
to hang in the optimizer/escapeAnalysis/connectionGraph phase if the 
compilation log is anything to go by.

I have no idea thus far what actually triggers the problem, but I appear to 
have found a reliable (though complex, hard-to-communicate, and without any 
obvious way to reduce to an SSCCE) procedure for reproducing it. When it 
happens, both the C2 compiler threads hang, consuming 100% CPU each and very 
often (but not always) eating all my memory. I've been able to reproduce it on 
three distinct systems, and with both Oracle JRE 8u31 and the Debian-supplied 
OpenJDK-8 JRE, so it seems to be a bug in "stock" Hotspot rather than some 
local configuration issue. All the systems are running x86_64 Linux, but I 
doubt that is related to the problem.

I tried debugging it using GDB but either something confuses GDB, or the debug 
symbols are bogus somehow, or Hotspot uses some very weird flow control that I 
don't understand. As such, I'm at a bit of a loss as to how to debug this 
further, and I'd appreciate any help.

As this seems to be perfectly reproducible, I can provide any number of 
compilation logs, core dumps, stack traces or anything else you might want. To 
begin with, here's a compilation log:

<http://www.dolda2000.com/~fredrik/tmp/hotspot_pid30770.log>

I had this JVM running for some 90 seconds, and triggered the bug at slightly 
under 30. You can see both the C2 compiler threads ending with a "fragment" 
that ends with the connectionGraph phase. It seems common, also, to this 
problem that, when I actually trigger the problem, both threads encounter a 
single compilation with a connectionGraph phase that takes "longer than normal" 
(some 5-10 seconds, where as "normal" appears to be almost instantaneous), and 
only then comes another compilation that never finishes. In the case of this 
log, those "harder" compilations are IDs 4929 and 4930, for each thread.

Now I'm really not sure how accurate this is given how weirdly GDB behaved, but 
just for the record, here's the backtrace of one of the compiler threads at an 
interrupt during this hang, from Debian's OpenJDK-8 JVM:

<http://www.dolda2000.com/~fredrik/tmp/hotspot_bt30770>

Please let me know if I can provide more information. In the meantime, I'll try 
to get acquainted with the build procedure so that I can build my own Hotspot 
with, hopefully, more useful debugging characteristics.

Thanks for reading!

--
Fredrik Tolf