RFR: 8285914: AppCDS crash when using shared archive with old class file [v2]

[Ioi Lam]

On Thu, 5 May 2022 22:55:05 GMT, Calvin Cheung <ccheung at openjdk.org> wrote:

>> Please review this change for an additional check before archiving a lambda proxy class. If the nest host of an lambda proxy class implements an old (major version < JDK_6 (50)) interface, the lambda proxy class should not be archived. This avoids accessing a null nest host during runtime while loading the lambda proxy class from the CDS archive.
>> 
>> Passed the test in the bug report.
>> 
>> Passed CI tiers 1 - 4 testing (including the new tests).
>
> Calvin Cheung has updated the pull request incrementally with one additional commit since the last revision:
> 
>   @iklam comments and also rename the test classes

I was curious about how the lambda proxy classes are removed from the archive, so I applied you patch and traced inside gdb. The removal is done in `CleanupDumpTimeLambdaProxyClassTable`, which will remove a proxy class if its `caller_ik` is excluded.

I think it makes sense to put the `nest_host` check in the same place. So instead of my previous suggestion, this seems to be a better way to handle it:


class CleanupDumpTimeLambdaProxyClassTable: StackObj {
 public:
  bool do_entry(LambdaProxyClassKey& key, DumpTimeLambdaProxyClassInfo& info) {
    assert_lock_strong(DumpTimeTable_lock);
    InstanceKlass* caller_ik = key.caller_ik();
    InstanceKlass* nest_host = caller_ik->nest_host_not_null();

    // If the caller class and/or nest_host are excluded, the associated lambda proxy
    // must also be excluded.
    bool always_exclude = SystemDictionaryShared::check_for_exclusion(caller_ik, NULL) ||
                          SystemDictionaryShared::check_for_exclusion(nest_host, NULL);

    for (int i = info._proxy_klasses->length() - 1; i >= 0; i--) {
      InstanceKlass* ik = info._proxy_klasses->at(i);
      if (always_exclude || SystemDictionaryShared::check_for_exclusion(ik, NULL)) {
        SystemDictionaryShared::reset_registered_lambda_proxy_class(ik);
        info._proxy_klasses->remove_at(i);
      }
    }
    return info._proxy_klasses->length() == 0 ? true /* delete the node*/ : false;
  }
};

... add add this to instanceKlass.hpp

  // Call this only if you know that the nest host has been initialized.
  InstanceKlass* nest_host_not_null() {
    assert(_nest_host != NULL, "must be");
    return _nest_host;
  }

-------------

PR: https://git.openjdk.java.net/jdk/pull/8540