RFR: 8020962: dump loaded java classes when vm crash

[Christian Thalinger]

On Aug 12, 2013, at 2:43 PM, Coleen Phillimore <coleen.phillimore at oracle.com> wrote:

> 
> Yumin,
> 
> I don't think this change should be added to the JVM for the following reasons.

This change is something that I (kind of) requested from Yumin since it would help to reproduce crashes in the compilers.  Getting replay data after the fact can be very tricky (given all the system library interactions or a broken SA; see below).

> 
> 1. Error handling should only contain safe actions.   We have concerns that the SA is not that stable

…which is a problem, I agree.  To make it more stable we need to use it so we can detect problems early.  

Having the SA dump data when we crash in the compiler can be one of these usages.  The SA logic would be exercised during our internal testing and could be fixed before the release.  Right now the situation is we see a customer crash, we want to run the SA and notice that it's broken.  That's too late.

> and would prevent getting a real core file in many error situations.   You couldn't have tested all error situations because these are usually the things we don't know about.   You also mention that there are currently bugs preventing this feature from working in your first email.
> 
> 2. I am not convinced that having a separate jar file with loaded classes (is it a list that SA generates?) would be collected by an error reporter.   If it's collected, I don't see how helpful it would be.

As mentioned above, it's required to replay compilations.

>  Also a customer may not want their classes disclosed in error reporting.

They don't have to if they don't want to.

-- Chris

> 
> 3.  This doesn't seem important enough to include as a new feature in jdk8 release, which is feature-complete.   I don't see a customer request for it.
> 
> Coleen
> 
> On 08/12/2013 01:00 PM, Yumin Qi wrote:
>> Chris, David
>> 
>>  Yes. This can be after crash with core file, but some time no core file generated (also if the error could not repeat, we will never got information again), so dumping  target process is also a choice. To avoid more confusion, this step was launched at the very last moment, just before raise abort to exit. At this moment, if client process could not attach to the target process it will exit right away.
>> 
>> Answers to David:
>> 
>> Note that the SA is only present in a full JDK, not a JRE (full or compact profile).
>>  Yes, in code, if no sa-jdi.jar found, skip fork.
>> 
>> - What mechanism will the SA try to use to query the VM?
>> 
>>  After successfully attached to target process, SA will read via proc APIs and vmStructs (named TYPEDB) to iterate  memory of system dictionary read (only)  from target process to dump class information.
>> 
>> - What if the state of the crashed VM stops the SA from being able to attach properly (ie both processes hang)?
>> 
>>  That should be an attaching API problem. We haven't watched such case happened so far.
>> 
>> - What if it the SA also crashes, will it launch a third VM then a fourth etc?
>> 
>>  Definitely don't want to see this happened in a chain. The solution may use a property such as sun.jvm.hotspot.DumpLoadedClasses.dumpingInProcess=true to pass into SA process, at launching call, check if the property set, if set, do not fork. When SA process died, it will generate core file first, note the target process still waiting for its exit, so when target exit, the core file (if both use default core as name) will be override by target. The SA process will only leave a hs_err_pid*.log file. (? read such property in handler is possible?)
>> 
>> Also what is the nature of this dump? How big is it? Where will it go?
>> The jars includes *app.jar and *boot.jar, the later usually can be ignored (rt.jar etc system jars) unless it's rewritten by JVMTI agent. The app.jar will contain all classes by customer, so we can do whatever we can to the jar.
>> 
>> 
>> Thanks
>> Yumin
>> 
>> 
>> On 8/12/2013 5:51 AM, Christian Tornqvist wrote:
>>> Hi Yumin,
>>> 
>>> The idea is to do as little as possible in the VM error handler, since we've crashed for some reason we don't know what state the process is in and we have to be extremely careful in when we're gathering the information. This seems like a step that is risky for all of the reasons David mentioned below.
>>> 
>>> It's also information that can easily be extracted post-mortem from the core/mdmp using the method you described for OSX, so gathering this at the time of a crash seems like an unnecessary risk.
>>> 
>>> Thanks,
>>> Christian
>>> 
>>> -----Original Message-----
>>> From: hotspot-compiler-dev-bounces at openjdk.java.net [mailto:hotspot-compiler-dev-bounces at openjdk.java.net] On Behalf Of David Holmes
>>> Sent: Monday, August 12, 2013 1:56 AM
>>> To: Yumin Qi
>>> Cc: hotspot compiler; hotspot-runtime-dev at openjdk.java.net
>>> Subject: Re: RFR: 8020962: dump loaded java classes when vm crash
>>> 
>>> Hi Yumin,
>>> 
>>> Note that the SA is only present in a full JDK, not a JRE (full or compact profile).
>>> 
>>> I have quite a few concerns with this:
>>> 
>>> We already do a lot of things that are not valid to be done from a signal handling context but this really takes that to an extreme. Doing fork-exec from a signal handler seems like a recipe for disaster (Note the existing onError facility is typically used for synchronous failures.)
>>> 
>>> The idea of launching a second VM to try and query a VM that has crashed also seems somewhat problematic:
>>> - What mechanism will the SA try to use to query the VM?
>>> - What if the state of the crashed VM stops the SA from being able to attach properly (ie both processes hang)?
>>> - What if it the SA also crashes, will it launch a third VM then a fourth etc?
>>> 
>>> Also what is the nature of this dump? How big is it? Where will it go?
>>> 
>>> Thanks,
>>> David
>>> 
>>> On 12/08/2013 9:36 AM, Yumin Qi wrote:
>>>> Hi, all
>>>> 
>>>>    I would like to have your review for
>>>> 
>>>> http://cr.openjdk.java.net/~minqi/8020962/webrev0/
>>>> <http://cr.openjdk.java.net/%7Eminqi/8020962/webrev0/>
>>>> 
>>>> Description: When JVM crashed, we also want to check the application
>>>> class files especially we got core file from customers.  The aftermath
>>>> analysis will benefit from all loaded java classes available. In this
>>>> change, spawn another process running SA to do the job when JVM
>>>> crashes, this way also avoid further messing up with the error report
>>>> which already in signal handler.
>>>> 
>>>> Note: The test has done with following two bugs worked around:
>>>>    8022655: ClassDump ignored jarStream setting (This will be fixed
>>>> and integrated by Kevin Walls soon)
>>>>    8011888: sa.js: TypeError: [object JSAdapter] has no such function
>>>> "__has__" (Not know when it will be integrated)
>>>>    That is, without those two fixed, the jars of loaded classes will
>>>> not be successfully dumped.
>>>>    Also, on MacOS it requires security access permission to attach to
>>>> another process, so omit doing so. To get loaded jar file s, with core
>>>> file available (on all platforms), one can (only after this
>>>> change) do
>>>> 
>>>>    $JAVA_HOME/bin/java [-d64] -cp $JAVA_HOME/lib/sa-jdi.jar
>>>> sun.jvm.hotspot.DumpLoadedClasses $JAVA_HOME/bin/java corefile
>>>> 
>>>> 
>>>> Thanks
>>>> Yumin
>> 
>