RFR: 8260650: test failed with "assert(false) failed: infinite loop in PhaseIterGVN::optimize"

[Igor Veresov]

The test constructs a large tree of `AddINode`s that add constants to a variable. The tree is completely constant-foldable. Depending of the processing order the folding process may happen in the loop in `PhaseIterGVN::transform_old()` instead of going through the worklist. That blows through the loop limit, which triggers assert. The solution is to make the loop limit proportional to the number of live nodes. I chose it to be `K*live_nodes()`, which is already the limit in `PhaseIterGVN::optimize()`. I also noticed the same problem in `PhaseGVN::transform_no_reclaim()` and adjusted the code accordingly.

-------------

Commit messages:
 - Adjust loop limit in PhaseIterGVN::transform_old() and PhaseGVN::transform_no_reclaim().

Changes: https://git.openjdk.java.net/jdk/pull/3022/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3022&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8260650
  Stats: 51 lines in 2 files changed: 24 ins; 6 del; 21 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3022.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3022/head:pull/3022

PR: https://git.openjdk.java.net/jdk/pull/3022