RFR: 8285914: AppCDS crash when using shared archive with old class file [v2]

[Calvin Cheung]

On Fri, 6 May 2022 04:50:06 GMT, Ioi Lam <iklam at openjdk.org> wrote:

> I was curious about how the lambda proxy classes are removed from the archive, so I applied you patch and traced inside gdb. The removal is done in `CleanupDumpTimeLambdaProxyClassTable`, which will remove a proxy class if its `caller_ik` is excluded.
> 
> I think it makes sense to put the `nest_host` check in the same place. So instead of my previous suggestion, this seems to be a better way to handle it:
> 
> ```
> class CleanupDumpTimeLambdaProxyClassTable: StackObj {
>  public:
>   bool do_entry(LambdaProxyClassKey& key, DumpTimeLambdaProxyClassInfo& info) {
>     assert_lock_strong(DumpTimeTable_lock);
>     InstanceKlass* caller_ik = key.caller_ik();
>     InstanceKlass* nest_host = caller_ik->nest_host_not_null();
> 
>     // If the caller class and/or nest_host are excluded, the associated lambda proxy
>     // must also be excluded.
>     bool always_exclude = SystemDictionaryShared::check_for_exclusion(caller_ik, NULL) ||
>                           SystemDictionaryShared::check_for_exclusion(nest_host, NULL);
> 
>     for (int i = info._proxy_klasses->length() - 1; i >= 0; i--) {
>       InstanceKlass* ik = info._proxy_klasses->at(i);
>       if (always_exclude || SystemDictionaryShared::check_for_exclusion(ik, NULL)) {
>         SystemDictionaryShared::reset_registered_lambda_proxy_class(ik);
>         info._proxy_klasses->remove_at(i);
>       }
>     }
>     return info._proxy_klasses->length() == 0 ? true /* delete the node*/ : false;
>   }
> };
> 
> ... add add this to instanceKlass.hpp
> 
>   // Call this only if you know that the nest host has been initialized.
>   InstanceKlass* nest_host_not_null() {
>     assert(_nest_host != NULL, "must be");
>     return _nest_host;
>   }
> ```

Ok. I've made the above change. Re-running tiers 1 and 2 tests.

-------------

PR: https://git.openjdk.java.net/jdk/pull/8540