RFR: 8362394: C2: Repeated stacked string concatenation fails with "Hit MemLimit" and other resourcing errors

[Daniel Skantz]

This PR addresses a bug in the stringopts phase. During string concatenation, repeated stacking of concatenations can lead to excessive compilation resource use and generation of questionable code as the merging of two StringBuilder-append-toString links sc1 and sc2 can result in a new StringBuilder with the size sc1->num_arguments() * sc2->num_arguments().

In the attached test, the size of the successively merged StringBuilder doubles on each merge -- there's 24 of them -- as the toString result of the first component is used twice in the second component [1], etc. Not only does the compiler hang on this test case, but the string concat optimization seems to give an arbitrary amount of back-to-back stores in the generated code depending on the number of stacked concatenations.

The proposed solution is to put an upper bound on the size of a merged concatenation, which guards against this case of repeated concatenations on the same string variable, and potentially other edge cases. 100 seems like a generous limit, and higher limits could be insufficient as each argument corresponds to about 20 new nodes later in replace_string_concat [2].

[1] https://github.com/openjdk/jdk/blob/0ceb366dc26e2e4f6252da9dd8930b016a5d46ba/src/hotspot/share/opto/stringopts.cpp#L303

[2] https://github.com/openjdk/jdk/blob/0ceb366dc26e2e4f6252da9dd8930b016a5d46ba/src/hotspot/share/opto/stringopts.cpp#L1806

Testing: T1-4.

Extra testing: verified that no method in T1-4 is being compiled with a merged concat candidate exceeding the suggested limit of 100 aguments, regardless of whether or not the later checks verify_control_flow() and verify_mem_flow pass.

-------------

Commit messages:
 - PL
 - fix

Changes: https://git.openjdk.org/jdk/pull/26685/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=26685&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8362394
  Stats: 95 lines in 3 files changed: 94 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/26685.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/26685/head:pull/26685

PR: https://git.openjdk.org/jdk/pull/26685