RFR: 8261395: C1 crash "cannot make java calls from the native compiler" [v2]

[David Holmes]

> If a nest host and a nest member are associated with different protection domains it can lead to execution of Java code (to validate the "new" protection domain) during a nestmate access check, if nest membership verification has not yet been performed. This will cause assertion or guarantee failures if executed by a JIT compiler thread during access checks.
> 
> After much discussion and trying different solutions it was decided that the existing logic for nest membership validation unnecessarily tries to resolve constant-pool entries, when it suffices that the symbolic entry in the constant-pool has the same name as the class being checked. Given this check occurs after we have verified the nest host and the purported member are loaded by the same classloader and in the same runtime package, there can only be one class with the name of the member, and that is the member class. Hence resolution of the constant-pool entry serves no purpose but introduces the complexity of dealing with exceptions and avoiding Java code execution in compiler threads.
> 
> @iklam contributed to an earlier version of the fix, and devised the initial testcase approach.
> @coleenp also contributed to an earlier version of the fix. 
> 
> Thanks to both Coleen and Ioi for their insights, discussions and contributions. 
> 
> Testing:
> - the new test
> - tiers 1-3
> 
> Thanks,
> David

David Holmes has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:

 - Update test per Coleen's comments:
   - need to add java.security.manager=allow just to be safe
   - simplify logic to read classfile from disk (ref ClassUnloadCommon.getClassData())
 - Merge branch 'master' into 8261395-nestmember
 - Per Coleen: Reverse arguments order for has_nest_member so we can revert the introduction of "current".
 - Fix typo
 - Completely simpified the has_nest_member check to only use names.
   This avoids any possibility of class loading or executing any Java
   code and removes all possibility of exceptions in this part of the
   nestmate verification process.
 - Add test
 - Tweak logging message
 - 8261395: C1 crash "cannot make java calls from the native compiler"
 - First cleanup: when we have a resolved klass we don't need to use klass_at but can use resolved_klass_at. Now exceptions are truly impossible.

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/3913/files
  - new: https://git.openjdk.java.net/jdk/pull/3913/files/0025ac9d..353ac945

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3913&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3913&range=00-01

  Stats: 9985 lines in 251 files changed: 6367 ins; 2386 del; 1232 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3913.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3913/head:pull/3913

PR: https://git.openjdk.java.net/jdk/pull/3913