RFR: 8371464: C2: assert(no_dead_loop) failed: dead loop detected

[Roland Westrelin]

Crash occurs because a `MergeMem` node references itself:


 608  MergeMem  === _ 1 608 1 1 1 1 1 1 1 1 1 1 878  [[ 877 878 608 420 597 ]]  { - - - - - - - - - - N878:java/lang/Throwable (java/io/Serializable)+20 * [narrow] }  Memory: @BotPTR *+bot, idx=Bot; !orig=[524] !jvms: TestDeadLoopAtMergeMem::test @ bci:94 (line 62)
 ```

Before IGVN, that part of the stream is:


 522  Region  === 522 604 521  [[ 522 538 523 524 525 526 527 528 529 530 531 ]] #reducible  !jvms: TestDeadLoopAtMergeMem::test @ bci:75 (line 59)
 524  Phi  === 522 608 464  [[ 588 581 564 546 564 559 ]]  #memory  Memory: @BotPTR *+bot, idx=Bot; !jvms: TestDeadLoopAtMergeMem::test @ bci:75 (line 59)
 
 538  If  === 522 535  [[ 539 540 ]] P=0.999000, C=-1.000000 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)
 539  IfTrue  === 538  [[ 553 547 ]] #1 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)
 540  IfFalse  === 538  [[ 548 546 ]] #0 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)
 553  If  === 539 535  [[ 554 555 ]] P=0.999000, C=-1.000000 !jvms: TestDeadLoopAtMergeMem::test @ bci:82 (line 59)
 554  IfTrue  === 553  [[ 562 560 ]] #1 !jvms: TestDeadLoopAtMergeMem::test @ bci:82 (line 59)
 555  IfFalse  === 553  [[ 548 559 ]] #0 !jvms: TestDeadLoopAtMergeMem::test @ bci:82 (line 59)
 
 548  Region  === 548 _ 540 555  [[ 548 562 561 563 564 565 566 567 568 569 570 571 572 573 574 575 576 ]] #reducible  !jvms: TestDeadLoopAtMergeMem::test @ bci:88 (line 60)
 564  Phi  === 548 _ 524 524  [[ 581 ]]  #memory  Memory: @BotPTR *+bot, idx=Bot; !jvms: TestDeadLoopAtMergeMem::test @ bci:85 (line 61)

562  Region  === 562 548 554  [[ 562 600 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 596 ]] #reducible  !jvms: TestDeadLoopAtMergeMem::test @ bci:90 (line 62)
581  Phi  === 562 564 524  [[ 420 597 610 608 ]]  #memory  Memory: @BotPTR *+bot, idx=Bot; !jvms: TestDeadLoopAtMergeMem::test @ bci:90 (line 62)

608  MergeMem  === _ 1 581 1 1 1 1 1 1 1 1 1 1 588  [[ 524 ]]  { - - - - - - - - - - N588:java/lang/Throwable (java/io/Serializable)+20 * [narrow] }  Memory: @BotPTR *+bot, idx=Bot; !jvms: TestDeadLoopAtMergeMem::test @ bci:94 (line 62)


522 is a loop head, 604 is the backedge. The loop becomes unreachable
during IGVN. The loop body above is transformed to:


 538  If  === 604 535  [[ 539 540 ]] P=0.999000, C=-1.000000 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)
 539  IfTrue  === 538  [[ 562 547 560 ]] #1 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)
 540  IfFalse  === 538  [[ 562 ]] #0 !jvms: TestDeadLoopAtMergeMem::test @ bci:79 (line 59)                                                                                                                                                                                                                                                                                  
 562  Region  === 562 540 539  [[ 562 600 596 878 592 876 ]] #reducible  !jvms: TestDeadLoopAtMergeMem::test @ bci:90 (line 62)
 876  Phi  === 562 608 876  [[ 877 876 597 420 608 ]]  #memory  Memory: @BotPTR *+bot, idx=Bot; !orig=[581] !jvms: TestDeadLoopAtMergeMem::test @ bci:90 (line 62)
 
 608  MergeMem  === _ 1 876 1 1 1 1 1 1 1 1 1 1 878  [[ 876 878 ]]  { - - - - - - - - - - N878:java/lang/Throwable (java/io/Serializable)+20 * [narrow] }  Memory: @BotPTR *+bot, idx=Bot; !orig=[524] !jvms: TestDeadLoopAtMergeMem::test @ bci:94 (line 62)
 ```

That part of the graph is dead but still being updated. It is matched
by `PhiNode::try_clean_memory_phi()` and the result is that 608
reference itself.

I think all that is missing here is a check that if the `Phi` is
transformed a self loop is not created. Other `Phi` transformations do
that.

-------------

Commit messages:
 - test
 - fix

Changes: https://git.openjdk.org/jdk/pull/28554/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=28554&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8371464
  Stats: 90 lines in 3 files changed: 83 ins; 0 del; 7 mod
  Patch: https://git.openjdk.org/jdk/pull/28554.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/28554/head:pull/28554

PR: https://git.openjdk.org/jdk/pull/28554