RFR(S): 8219517: assert(false) failed: infinite loop in PhaseIterGVN::optimize

[Nils Eliasson]

Hi all,

Background:

We can get stuck in an infinite loop in IGVN. The method reproducing the 
problem is quite a big graph, and after some optimization, a huge loop 
will die. But since it is so big, it takes a while before it has been 
pruned.

In an inner loop there is a phi on memory that gets reduced to a self 
looping heart, with a membar on each in edge. There is also a connected 
region that keeps it alive. (From the start there is other memory state 
coming into this loop, but it gets disconnected early when the loop dies.)

+---+                 +---+
|   v                 v   |
| Membar +-+ +---+ Membar |
|          | |            |
|          v v            |
|          Phi            |
|          + + +          |
|          | | |          |
+----------+ | +----------+
              |
              v
            LoadN

In IGVN, Ideal() will be called on the Load.

On iteration 1 - A split_through_phi on one edge will be performed, 
because we can prove that other edge of the phi is a loop. Now the Load 
hangs of one of the membars.

On iteration 2 - Optimize_memory_chain will suggest the in to the membar 
as a more ideal memory, and then the load get the phi back as the memory 
input.

Repeat.

I have gone great lengths to show that this code is part of a huge loop, 
that is dead, and will be eliminated in due time.

My suggested solution to breaking the infinite loop, is to change the 
first case, by simply not perform the memory replacement when both 
inputs are self loops.

https://bugs.openjdk.java.net/browse/JDK-8219517

http://cr.openjdk.java.net/~neliasso/8219517/webrev.01/

Regards,

Nils