RFR: 8222895: StackOverflowError in custom security manager that relies on ClassSpecializer

[Alan Bateman]

On 24/04/2019 13:49, Claes Redestad wrote:
> Hi,
>
> recent changes to the String concatenation bootstrap sequence[1]
> accidentally uncovered an issue introduced earlier by changing the
> ClassSpecializer to use Lookup.defineClass[2].
>
> The issue with this is the introduction of a call to SM.checkPermission
> deep inside the ClassSpecializer code used when bootstrapping certain
> String concatenation expressions. When triggered, this causes a
> recursive bootstrap cycle and a crash with a StackOverflowError.
>
> This is related to earlier bootstrapping issues in the area[3], and
> would have been caught by the regression test added then if not for the
> fact that we've been rather successful in avoiding the use of
> ClassSpecializer during bootstrap of commonly used concatenation shapes.
>
> Providing a more contorted concatenation shape in the test ensures we
> drop into the ClassSpecializer code path where the recursive
> checkPermission call happens. The updated test passes before
> JDK-8181443, fails since, and passes with the changes proposed in this
> patch.
>
> Webrev: http://cr.openjdk.java.net/~redestad/8222895/open.00/
> Bug:    https://bugs.openjdk.java.net/browse/JDK-8222895
This looks okay as a temporary solution to fix the regression. It will 
probably need a bit of thought to figure how to get the specializer to 
work when there is a security manager, something for another issue.

-Alan.