RFR(XS) 8213250 CDS archive creation aborts due to metaspace object allocation failure

[Jiangli Zhou]

Hi Ioi,

- src/hotspot/share/classfile/verifier.cpp

You moved 
SystemDictionaryShared::finalize_verification_constraints(_klass) to 
ClassVerifier::verify_class(). We should be very careful in this area. 
Have you tested with cases where classes fail verification? I haven't 
ran your code yet, we should avoid copying the constraints for classes 
that fails verification at dump time. I also need to think through any 
potential impact on dynamic archiving (possibly none).

- src/hotspot/share/memory/metaspace.cpp

-    if (is_init_completed() && !(DumpSharedSpaces && 
THREAD->is_VM_thread())) {
+    if (is_init_completed()) {

Please also add an assert to make sure we don't allocate metaspace 
objects from the VM_thread at dump time now.

Thanks,

Jiangli


On 11/1/18 9:47 PM, Ioi Lam wrote:
> Hi,
>
> I've revised the webrev to get to the the root of the problem -- we 
> shouldn't allocate metaspace objects from within the VM Thread!
>
> http://cr.openjdk.java.net/~iklam/jdk12/8213250-cds-dump-run-out-of-metaspace.v02/ 
>
>
> The original bug that caused the "!(DumpSharedSpaces && 
> THREAD->is_VM_thread())" check to be added in Metaspace::allocate() was
>
>     https://bugs.openjdk.java.net/browse/JDK-8196626
>     UseAppCDS.java crashes with "VM thread using lock Heap_lock (not 
> allowed to block on)"
>
> The fix in JDK-8196626 is just a band-aid solution -- avoiding the GC. 
> The proper fix is not to do any allocation inside the VM thread. So 
> instead of calling finalize_verification_constraints, which allocates 
> MetaspaceObjs, very late in the CDS dumping process, now we call be as 
> soon as a class has been verified.
>
> Thanks
> - Ioi
>
>
>
>
>
> On 11/1/18 11:07 AM, Ioi Lam wrote:
>> https://bugs.openjdk.java.net/browse/JDK-8213250
>> http://cr.openjdk.java.net/~iklam/jdk12/8213250-cds-dump-run-out-of-metaspace.v01/ 
>>
>>
>> When Metaspace::allocate() fails, usually we first try to GC to see if
>> some MetaspaceObjs can be freed. This is an optimization to avoid 
>> prematurely
>> expanding the metaspace.
>>
>> However, GC cannot run in the last phase of CDS which runs inside the 
>> VM thread.
>> The old code just aborts the CDS dump. The fix is to unconditionally 
>> expand the
>> metaspace without doing a GC.
>>
>> (During GC dump time, no classes are unloaded, so even if we force a 
>> GC, it would
>> not free up any metaspace anyway).
>>
>> Testing:
>>
>> Before the fix, the test case (jruby) would fail to dump once a 
>> couple of times. Now
>> it has dumped hundreds of times without failure.
>>
>> I am also running hs-tier 1/2
>>
>> Thanks
>> - Ioi
>