RFR: 8285914: AppCDS crash when using shared archive with old class file from JUnit

[Ioi Lam]

On Thu, 5 May 2022 04:41:04 GMT, Calvin Cheung <ccheung at openjdk.org> wrote:

> Hi David,
> 
> The `ChildOldInf` class of the test case has an inner class `InnerChild`. A lambda proxy class will be generated if the `doTest` method is called. The nest host of the lambda proxy class is `ChildOldInf` which extends an old interface `OldInf`. So the nest host is considered old. The test case is just a simplified version of the test in the bug report.
> 
> The `ChildOldInf` corresponds to the `JsaCrash` in the bug report test case; the `ChildOldInf$InnerChild` corresponds to the `JsaCrash$Builder`.
> 
> `public class JsaCrash implements TestRule`
> 
> `TestRule` is an old class.
> 
> Thanks, Calvin

I think the PR description should be updated. The nest host `JsaCrash` itself is not an old class. Rather, it implements an interface `TestRule` which is an old class. 

Due to limitations in CDS (*1), `JsaCrash` cannot be archived, so it will be filtered by `SystemDictionaryShared::check_for_exclusion_impl()`. As a result, Lambda proxy classes of  `JsaCrash` cannot be archived, because such classes contain a direct pointer reference to `JsaCrash`, and such pointers will be set to NULL during CDS dump.

There are actually multiple reasons that a class could be excluded from the archive. `!nest_host->can_be_verified_at_dumptime()` is just one of them. To check for all cases, instead of adding the check in `SystemDictionaryShared::add_lambda_proxy_class()`, I think it's better to add a check like this in `SystemDictionaryShared::check_for_exclusion_impl()`:


    if (is_registered_lambda_proxy_class(k)) {
      DumpTimeClassInfo* info = _dumptime_table->get(k);
      assert(info != NULL, "sanity");
     if (!check_for_exclusion(info->nest_host(), NULL) {
      return true;
    }


(*1) `TestRule` is verified by the old verifier, whose results cannot be saved in CDS. As a result, `TestRule` (and  `JsaCrash`) must be re-verified at runtime. However, verification runs during class linking. If we want to re-verify `JsaCrash` at runtime, we must revert it back to the `loaded` state, which is non trivial because we need to undo the results of bytecode rewriting.

-------------

PR: https://git.openjdk.java.net/jdk/pull/8540