Improving AppCDS for Custom Loaders

[Volker Simonis]

On Sat, May 12, 2018 at 7:43 AM, Ioi Lam <ioi.lam at oracle.com> wrote:
>
>
> On 5/10/18 4:05 PM, Ioi Lam wrote:
>>
>>
>>
>> On 5/10/18 3:34 PM, Volker Simonis wrote:
>>>
>>>
>>> yumin qi <yumin.qi at gmail.com <mailto:yumin.qi at gmail.com>> schrieb am Do.
>>> 10. Mai 2018 um 22:20:
>>>
>>>     Hi, Jiangli
>>>
>>>       Thanks for the info.
>>>
>>>     On Thu, May 10, 2018 at 11:39 AM, Jiangli Zhou
>>>     <jiangli.zhou at oracle.com <mailto:jiangli.zhou at oracle.com>> wrote:
>>>
>>>         Hi Yumin and Volker,
>>>
>>>         I’ve create an umbrella RFE JDK-8202854 to keep track this
>>>         effort. New sub-RFEs will be created and targeted for specific
>>>         JDK releases when each sub-item matures. Please refer to
>>>         JDK-8202854 for future progress.
>>>
>>>         Yumin, please see below for comments/questions.
>>>
>>>         > Agree not including objects in java heap at now --- thanks
>>> Jiangli for answering my question
>>>         in other thread.
>>>
>>>         I conclude from above that you are not using java heap object
>>>         archiving due to different GC algorithm being used(?). I
>>>         would like to understand more about your GC choices. Could you
>>>         please provide more information on that? G1 GC is the default
>>>         GC in JDK. Thanks to our GC team, the performance of G1 has
>>>         been consistently improving over the recent JDK releases,
>>>         especially in JDK 9 and 10. I strongly encourage you to do
>>>         performance measurement/comparison and reevaluate your GC
>>>         choice if G1 is not used.
>>>
>>>
>>>     Most of our applications still using CMS, we are pushing to switch
>>>     to G1GC at this time but still most of them still with CMS. I know
>>>     CMS will be removed so it will not be your focus.
>>>
>>>
>>>         >   I think for #1, it does not conflict with the two layer
>>>         archive solution. We can skip the classes from base CDS
>>>         archive, only dump the non-base loaded classes into second
>>>         archive, this gives one more option for user to choose. Also
>>>         it will be good after dump archive to let the application
>>>         continue to run.
>>>         >
>>>         >   Can we do a Full GC before dump at exit? it may unload not
>>>         referenced classes, or it is not necessary since CDS wants to
>>>         resolve the startup performance for all loaded classes?
>>>
>>>         For classes loaded by the builtin loaders, they can be kept
>>>         alive when archiving phase start. For classes loaded by user
>>>         defined class loaders, we also need to prevent them from being
>>>         unloaded before archiving. I have a preliminary idea for how
>>>         to do that. I can share more information on that when it matures.
>>>
>>>
>>>     Yes, if classes with custom class loader cleaned, the class loader
>>>     itself will be purged. Full GC before exit may not be a good idea,
>>>     but we can skip unloading classes --- in case dump objects in java
>>>     heap.
>>>
>>>
>>> Why not dump the classes iteratively, while they are loaded. Class
>>> loading time would be the most natural time for dumping a class, and it‘s
>>> the time when we know the most about a class (i.e. full class bytes).
>>>
>>
>> The archive is divided into RO/RW sections. Mutable metadata such as
>> InstanceKlasses are stored in RW, while non-mutable ones such as ConstMethod
>> and Symbol are stored into the RO section. In order to have a single section
>> each for RW and RO, currently we load all the classes first, and then
>> calculate the sizes of the 2 sections.
>>
>>
> One possible solution is to copy the loaded classes into a temp memory
> buffer, as the classes are being loaded. Then, when we decide to dump the
> archive, we can iterate over all the classes in the temp memory buffer.
>

This sounds like a good a good approach. I wanted to propose something
similar but hadn't time to look at the possible implementation details
more closely until now :)

> - Ioi
>
>
>>>     Thanks
>>>     Yumin
>>>
>>>
>>
>