RFR: 8261395: C1 crash "cannot make java calls from the native compiler" [v2]

[David Holmes]

On Sun, 9 May 2021 22:10:24 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> If a nest host and a nest member are associated with different protection domains it can lead to execution of Java code (to validate the "new" protection domain) during a nestmate access check, if nest membership verification has not yet been performed. This will cause assertion or guarantee failures if executed by a JIT compiler thread during access checks.
>> 
>> After much discussion and trying different solutions it was decided that the existing logic for nest membership validation unnecessarily tries to resolve constant-pool entries, when it suffices that the symbolic entry in the constant-pool has the same name as the class being checked. Given this check occurs after we have verified the nest host and the purported member are loaded by the same classloader and in the same runtime package, there can only be one class with the name of the member, and that is the member class. Hence resolution of the constant-pool entry serves no purpose but introduces the complexity of dealing with exceptions and avoiding Java code execution in compiler threads.
>> 
>> @iklam contributed to an earlier version of the fix, and devised the initial testcase approach.
>> @coleenp also contributed to an earlier version of the fix. 
>> 
>> Thanks to both Coleen and Ioi for their insights, discussions and contributions. 
>> 
>> Testing:
>> - the new test
>> - tiers 1-3
>> 
>> Thanks,
>> David
>
> David Holmes has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
> 
>  - Update test per Coleen's comments:
>    - need to add java.security.manager=allow just to be safe
>    - simplify logic to read classfile from disk (ref ClassUnloadCommon.getClassData())
>  - Merge branch 'master' into 8261395-nestmember
>  - Per Coleen: Reverse arguments order for has_nest_member so we can revert the introduction of "current".
>  - Fix typo
>  - Completely simpified the has_nest_member check to only use names.
>    This avoids any possibility of class loading or executing any Java
>    code and removes all possibility of exceptions in this part of the
>    nestmate verification process.
>  - Add test
>  - Tweak logging message
>  - 8261395: C1 crash "cannot make java calls from the native compiler"
>  - First cleanup: when we have a resolved klass we don't need to use klass_at but can use resolved_klass_at. Now exceptions are truly impossible.

Thanks for the reviews Coleen, Ioi and Harold.

Minor updates in place based on Coleen's feedback and suggestions. Please re-review.

@coleenp we wouldn't have arrived at this simplified name-check-only version without all of the discussions and investigations of how to fix the more complex version. So credit remains where credit is due. :)

Thanks,
David

-------------

PR: https://git.openjdk.java.net/jdk/pull/3913