[16] RFR(S): C2: assert(no_dead_loop) failed: dead loop detected

[Christian Hagedorn]

Hi

Please review the following patch:
https://bugs.openjdk.java.net/browse/JDK-8249605
http://cr.openjdk.java.net/~chagedorn/8249605/webrev.00/

There is a dead memory loop detected during IGVN. In the testcase, many 
nodes are dying during IGVN because they are not reachable anymore. In 
this process, a (not yet dead) memory phi node (150 Phi) with two inputs 
is processed (see [1]):
(1) 289 MergeMem, whose base memory is 150 Phi and has one slice for 274 
StoreD which is again an output of 150 Phi
(2) 356 MergeMem, whose base memory is top (i.e. dead and would be 
removed when IGVN processes this node)

In PhiNode::Ideal, we check if a phi node is part of a dead loop where 
all its inputs reference itself over a MergeMemNode input whose base 
memory is the phi node itself again [2]. However, in this check we do 
not account for dead MergeMemNodes (like the input 356 MergeMem of 150 
Phi). Therefore, we do not return top and apply the optimization [3] to 
replace 150 Phi by a new MergeMemNode (380 MergeMem) whose base memory 
is top and now has again one slice which is input and output to 274 
StoreD [4]. This cycle is later detected and the assertion fails.

The fix accounts additionally for dead MergeMemNodes when trying to 
detect dead loops in PhiNode::Ideal to return top instead of a new dead 
MergeMemNode.

Best regards,
Christian


[1] 
https://bugs.openjdk.java.net/secure/attachment/89582/before_PhiNode_Ideal.png
[2] 
http://hg.openjdk.java.net/jdk/jdk/file/8f7ede592c28/src/hotspot/share/opto/cfgnode.cpp#l2234
[3] 
http://hg.openjdk.java.net/jdk/jdk/file/8f7ede592c28/src/hotspot/share/opto/cfgnode.cpp#l2246
[4] 
https://bugs.openjdk.java.net/secure/attachment/89583/after_PhiNode_Ideal.png