RFR: 8368727: CDS custom loader support causes asserts during class unloading

[Ioi Lam]

On Fri, 26 Sep 2025 17:32:19 GMT, Coleen Phillimore <coleenp at openjdk.org> wrote:

>> When loading a class `k` from the CDS archive on behalf of a custom class loader, we were calling `loader_data->add_class(k)` too early. If the loading of `k` fails, it may be in `loader_data->_klasses`, but `k->init_state()` will remain `allocated`. This causes an assert during class unloading:
>> 
>> 
>> # assert(ik->is_loaded()) failed: class should be loaded 0x000000000b518eb8
>> V [libjvm.so+0xfa6bd5] InstanceKlass::unload_class(InstanceKlass*)+0x555 (instanceKlass.cpp:2870)
>> V [libjvm.so+0xa790e3] ClassLoaderData::classes_do(void (*)(InstanceKlass*))+0xc3 (classLoaderData.cpp:441
>> 
>> 
>> The fix is to move the  `loader_data->add_class(k)` call to `k->Klass::restore_unshareable_info()`. This is the same location as if `k` were loaded by the 3 built-in class loaders.
>> 
>> This was discovered when running some JCK tests in AOT mode. I've added a reproducer as a jtreg test case.
>
> Looks good.

Thanks @coleenp @dholmes-ora for the review

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27511#issuecomment-3362729718