RFR: 8020962: dump loaded java classes when vm crash

Tue Aug 13 03:57:14 PDT 2013

I agree with those skeptical to this change. The correct approach here is to make sure we get core files when crashes occur and then process that core with SA. And I agree that SA needs to be stabilized and could use a lot more testing.

/Staffan

On 13 aug 2013, at 07:17, Christian Tornqvist <christian.tornqvist at oracle.com> wrote:

> Hi Chris,
> 
>> The SA logic would be exercised during our internal testing and could be
> fixed before the release.  Right now the situation is we see a >customer
> crash, we want to run the SA and notice that it's broken.  That's too late.
> 
> This is an issue with our current testing of the SA. This should be
> addressed by making sure our test works and covers the functionality of the
> SA, not by making the JVM dependent on the SA.
> 
> Thanks,
> Christian
> 
> -----Original Message-----
> From: hotspot-runtime-dev-bounces at openjdk.java.net
> [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of Christian
> Thalinger
> Sent: Monday, August 12, 2013 9:12 PM
> To: Coleen Phillimore
> Cc: serviceability-dev at openjdk.java.net serviceability-dev at openjdk.java.net;
> 'hotspot compiler'; hotspot-runtime-dev at openjdk.java.net; 'David Holmes'
> Subject: Re: RFR: 8020962: dump loaded java classes when vm crash
> 
> 
> On Aug 12, 2013, at 2:43 PM, Coleen Phillimore
> <coleen.phillimore at oracle.com> wrote:
> 
>> 
>> Yumin,
>> 
>> I don't think this change should be added to the JVM for the following
> reasons.
> 
> This change is something that I (kind of) requested from Yumin since it
> would help to reproduce crashes in the compilers.  Getting replay data after
> the fact can be very tricky (given all the system library interactions or a
> broken SA; see below).
> 
>> 
>> 1. Error handling should only contain safe actions.   We have concerns
> that the SA is not that stable
> 
> .which is a problem, I agree.  To make it more stable we need to use it so
> we can detect problems early.  
> 
> Having the SA dump data when we crash in the compiler can be one of these
> usages.  The SA logic would be exercised during our internal testing and
> could be fixed before the release.  Right now the situation is we see a
> customer crash, we want to run the SA and notice that it's broken.  That's
> too late.
> 
>> and would prevent getting a real core file in many error situations.   You
> couldn't have tested all error situations because these are usually the
> things we don't know about.   You also mention that there are currently bugs
> preventing this feature from working in your first email.
>> 
>> 2. I am not convinced that having a separate jar file with loaded classes
> (is it a list that SA generates?) would be collected by an error reporter.
> If it's collected, I don't see how helpful it would be.
> 
> As mentioned above, it's required to replay compilations.
> 
>> Also a customer may not want their classes disclosed in error reporting.
> 
> They don't have to if they don't want to.
> 
> -- Chris
> 
>> 
>> 3.  This doesn't seem important enough to include as a new feature in jdk8
> release, which is feature-complete.   I don't see a customer request for it.
>> 
>> Coleen
>> 
>> On 08/12/2013 01:00 PM, Yumin Qi wrote:
>>> Chris, David
>>> 
>>> Yes. This can be after crash with core file, but some time no core file
> generated (also if the error could not repeat, we will never got information
> again), so dumping  target process is also a choice. To avoid more
> confusion, this step was launched at the very last moment, just before raise
> abort to exit. At this moment, if client process could not attach to the
> target process it will exit right away.
>>> 
>>> Answers to David:
>>> 
>>> Note that the SA is only present in a full JDK, not a JRE (full or
> compact profile).
>>> Yes, in code, if no sa-jdi.jar found, skip fork.
>>> 
>>> - What mechanism will the SA try to use to query the VM?
>>> 
>>> After successfully attached to target process, SA will read via proc
> APIs and vmStructs (named TYPEDB) to iterate  memory of system dictionary
> read (only)  from target process to dump class information.
>>> 
>>> - What if the state of the crashed VM stops the SA from being able to
> attach properly (ie both processes hang)?
>>> 
>>> That should be an attaching API problem. We haven't watched such case
> happened so far.
>>> 
>>> - What if it the SA also crashes, will it launch a third VM then a fourth
> etc?
>>> 
>>> Definitely don't want to see this happened in a chain. The solution 
>>> may use a property such as 
>>> sun.jvm.hotspot.DumpLoadedClasses.dumpingInProcess=true to pass into 
>>> SA process, at launching call, check if the property set, if set, do 
>>> not fork. When SA process died, it will generate core file first, 
>>> note the target process still waiting for its exit, so when target 
>>> exit, the core file (if both use default core as name) will be 
>>> override by target. The SA process will only leave a hs_err_pid*.log 
>>> file. (? read such property in handler is possible?)
>>> 
>>> Also what is the nature of this dump? How big is it? Where will it go?
>>> The jars includes *app.jar and *boot.jar, the later usually can be
> ignored (rt.jar etc system jars) unless it's rewritten by JVMTI agent. The
> app.jar will contain all classes by customer, so we can do whatever we can
> to the jar.
>>> 
>>> 
>>> Thanks
>>> Yumin
>>> 
>>> 
>>> On 8/12/2013 5:51 AM, Christian Tornqvist wrote:
>>>> Hi Yumin,
>>>> 
>>>> The idea is to do as little as possible in the VM error handler, since
> we've crashed for some reason we don't know what state the process is in and
> we have to be extremely careful in when we're gathering the information.
> This seems like a step that is risky for all of the reasons David mentioned
> below.
>>>> 
>>>> It's also information that can easily be extracted post-mortem from the
> core/mdmp using the method you described for OSX, so gathering this at the
> time of a crash seems like an unnecessary risk.
>>>> 
>>>> Thanks,
>>>> Christian
>>>> 
>>>> -----Original Message-----
>>>> From: hotspot-compiler-dev-bounces at openjdk.java.net 
>>>> [mailto:hotspot-compiler-dev-bounces at openjdk.java.net] On Behalf Of 
>>>> David Holmes
>>>> Sent: Monday, August 12, 2013 1:56 AM
>>>> To: Yumin Qi
>>>> Cc: hotspot compiler; hotspot-runtime-dev at openjdk.java.net
>>>> Subject: Re: RFR: 8020962: dump loaded java classes when vm crash
>>>> 
>>>> Hi Yumin,
>>>> 
>>>> Note that the SA is only present in a full JDK, not a JRE (full or
> compact profile).
>>>> 
>>>> I have quite a few concerns with this:
>>>> 
>>>> We already do a lot of things that are not valid to be done from a 
>>>> signal handling context but this really takes that to an extreme. 
>>>> Doing fork-exec from a signal handler seems like a recipe for 
>>>> disaster (Note the existing onError facility is typically used for 
>>>> synchronous failures.)
>>>> 
>>>> The idea of launching a second VM to try and query a VM that has crashed
> also seems somewhat problematic:
>>>> - What mechanism will the SA try to use to query the VM?
>>>> - What if the state of the crashed VM stops the SA from being able to
> attach properly (ie both processes hang)?
>>>> - What if it the SA also crashes, will it launch a third VM then a
> fourth etc?
>>>> 
>>>> Also what is the nature of this dump? How big is it? Where will it go?
>>>> 
>>>> Thanks,
>>>> David
>>>> 
>>>> On 12/08/2013 9:36 AM, Yumin Qi wrote:
>>>>> Hi, all
>>>>> 
>>>>>   I would like to have your review for
>>>>> 
>>>>> http://cr.openjdk.java.net/~minqi/8020962/webrev0/
>>>>> <http://cr.openjdk.java.net/%7Eminqi/8020962/webrev0/>
>>>>> 
>>>>> Description: When JVM crashed, we also want to check the 
>>>>> application class files especially we got core file from customers.  
>>>>> The aftermath analysis will benefit from all loaded java classes 
>>>>> available. In this change, spawn another process running SA to do 
>>>>> the job when JVM crashes, this way also avoid further messing up 
>>>>> with the error report which already in signal handler.
>>>>> 
>>>>> Note: The test has done with following two bugs worked around:
>>>>>   8022655: ClassDump ignored jarStream setting (This will be fixed 
>>>>> and integrated by Kevin Walls soon)
>>>>>   8011888: sa.js: TypeError: [object JSAdapter] has no such 
>>>>> function "__has__" (Not know when it will be integrated)
>>>>>   That is, without those two fixed, the jars of loaded classes 
>>>>> will not be successfully dumped.
>>>>>   Also, on MacOS it requires security access permission to attach 
>>>>> to another process, so omit doing so. To get loaded jar file s, 
>>>>> with core file available (on all platforms), one can (only after 
>>>>> this
>>>>> change) do
>>>>> 
>>>>>   $JAVA_HOME/bin/java [-d64] -cp $JAVA_HOME/lib/sa-jdi.jar 
>>>>> sun.jvm.hotspot.DumpLoadedClasses $JAVA_HOME/bin/java corefile
>>>>> 
>>>>> 
>>>>> Thanks
>>>>> Yumin
>>> 
>> 
> 
> 