RFR (S) 8158851: MH.publicLookup() init circularity, triggered by custom SecurityManager with String concat and -limitmods java.base

[Aleksey Shipilev]

Hi,

Please review a fix for a MH.publicLookup() circularity, which is
triggered if you run existing String concat tests with -limitmods java.base:
  https://bugs.openjdk.java.net/browse/JDK-8158851

Webrev:
  http://cr.openjdk.java.net/~shade/8158851/webrev.01/

Testing: java/lang/ jtreg tests.

Alas, the previous fix for String concat circularity [1] does help only
so much, because it only covers the transitional state. Without
-limitmods java.base the test passes with first (new SM + setSM + new
CL) block and fails on second try. With -limitmods java.base it fails on
the first block.

In both cases, it fails because StringConcatFactory is still subject to
MH.publicLookup() init circularity: MH_INLINE_SIZED_EXACT strategy
lazily looks up some methods, which forces MH.publicLookup()
initialization, which goes to custom SM, which has String concat, which
goes all the way back to SCF, and fails.

It would have manifested as StackOverflowError, but it is
short-circuited earlier by CHM cache that detects recursive update in
computeIfAbsent.

This not the only instance of circularity manifested with lazy
MH.publicLookup() init [2], and so I think it should be fixed once and
for all by eagerly initializing publicLookup() at initPhase3(), as Alan
suggested earlier.

Thanks,
-Aleksey

[1] https://bugs.openjdk.java.net/browse/JDK-8155090
[2] https://bugs.openjdk.java.net/browse/JDK-8156930