RFR: 8020962: dump loaded java classes when vm crash

Tue Aug 13 09:04:28 PDT 2013

To summarize the discussion of this change:

1) Launching SA process at the last stage of error handling caused much 
concern;
2) SA is not stable and need stabilization in future enhancement, which 
looks like a long term task;
3) With SA and available core file, we can get jar files for aftermath 
analysis but you need to become familiar with SA;

So I will closed this bug as 'no fix'.
BTW, when working on this bug, filed other two bugs which should be 
fixed to dump classes.

Thanks for the inputs from all of you.

Yumin

On 8/13/2013 3:57 AM, Staffan Larsen wrote:
> I agree with those skeptical to this change. The correct approach here is to make sure we get core files when crashes occur and then process that core with SA. And I agree that SA needs to be stabilized and could use a lot more testing.
>
> /Staffan
>
> On 13 aug 2013, at 07:17, Christian Tornqvist <christian.tornqvist at oracle.com> wrote:
>
>> Hi Chris,
>>
>>> The SA logic would be exercised during our internal testing and could be
>> fixed before the release.  Right now the situation is we see a >customer
>> crash, we want to run the SA and notice that it's broken.  That's too late.
>>
>> This is an issue with our current testing of the SA. This should be
>> addressed by making sure our test works and covers the functionality of the
>> SA, not by making the JVM dependent on the SA.
>>
>> Thanks,
>> Christian
>>
>> -----Original Message-----
>> From: hotspot-runtime-dev-bounces at openjdk.java.net
>> [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of Christian
>> Thalinger
>> Sent: Monday, August 12, 2013 9:12 PM
>> To: Coleen Phillimore
>> Cc: serviceability-dev at openjdk.java.net serviceability-dev at openjdk.java.net;
>> 'hotspot compiler'; hotspot-runtime-dev at openjdk.java.net; 'David Holmes'
>> Subject: Re: RFR: 8020962: dump loaded java classes when vm crash
>>
>>
>> On Aug 12, 2013, at 2:43 PM, Coleen Phillimore
>> <coleen.phillimore at oracle.com> wrote:
>>
>>> Yumin,
>>>
>>> I don't think this change should be added to the JVM for the following
>> reasons.
>>
>> This change is something that I (kind of) requested from Yumin since it
>> would help to reproduce crashes in the compilers.  Getting replay data after
>> the fact can be very tricky (given all the system library interactions or a
>> broken SA; see below).
>>
>>> 1. Error handling should only contain safe actions.   We have concerns
>> that the SA is not that stable
>>
>> .which is a problem, I agree.  To make it more stable we need to use it so
>> we can detect problems early.
>>
>> Having the SA dump data when we crash in the compiler can be one of these
>> usages.  The SA logic would be exercised during our internal testing and
>> could be fixed before the release.  Right now the situation is we see a
>> customer crash, we want to run the SA and notice that it's broken.  That's
>> too late.
>>
>>> and would prevent getting a real core file in many error situations.   You
>> couldn't have tested all error situations because these are usually the
>> things we don't know about.   You also mention that there are currently bugs
>> preventing this feature from working in your first email.
>>> 2. I am not convinced that having a separate jar file with loaded classes
>> (is it a list that SA generates?) would be collected by an error reporter.
>> If it's collected, I don't see how helpful it would be.
>>
>> As mentioned above, it's required to replay compilations.
>>
>>> Also a customer may not want their classes disclosed in error reporting.
>> They don't have to if they don't want to.
>>
>> -- Chris
>>
>>> 3.  This doesn't seem important enough to include as a new feature in jdk8
>> release, which is feature-complete.   I don't see a customer request for it.
>>> Coleen
>>>
>>> On 08/12/2013 01:00 PM, Yumin Qi wrote:
>>>> Chris, David
>>>>
>>>> Yes. This can be after crash with core file, but some time no core file
>> generated (also if the error could not repeat, we will never got information
>> again), so dumping  target process is also a choice. To avoid more
>> confusion, this step was launched at the very last moment, just before raise
>> abort to exit. At this moment, if client process could not attach to the
>> target process it will exit right away.
>>>> Answers to David:
>>>>
>>>> Note that the SA is only present in a full JDK, not a JRE (full or
>> compact profile).
>>>> Yes, in code, if no sa-jdi.jar found, skip fork.
>>>>
>>>> - What mechanism will the SA try to use to query the VM?
>>>>
>>>> After successfully attached to target process, SA will read via proc
>> APIs and vmStructs (named TYPEDB) to iterate  memory of system dictionary
>> read (only)  from target process to dump class information.
>>>> - What if the state of the crashed VM stops the SA from being able to
>> attach properly (ie both processes hang)?
>>>> That should be an attaching API problem. We haven't watched such case
>> happened so far.
>>>> - What if it the SA also crashes, will it launch a third VM then a fourth
>> etc?
>>>> Definitely don't want to see this happened in a chain. The solution
>>>> may use a property such as
>>>> sun.jvm.hotspot.DumpLoadedClasses.dumpingInProcess=true to pass into
>>>> SA process, at launching call, check if the property set, if set, do
>>>> not fork. When SA process died, it will generate core file first,
>>>> note the target process still waiting for its exit, so when target
>>>> exit, the core file (if both use default core as name) will be
>>>> override by target. The SA process will only leave a hs_err_pid*.log
>>>> file. (? read such property in handler is possible?)
>>>>
>>>> Also what is the nature of this dump? How big is it? Where will it go?
>>>> The jars includes *app.jar and *boot.jar, the later usually can be
>> ignored (rt.jar etc system jars) unless it's rewritten by JVMTI agent. The
>> app.jar will contain all classes by customer, so we can do whatever we can
>> to the jar.
>>>>
>>>> Thanks
>>>> Yumin
>>>>
>>>>
>>>> On 8/12/2013 5:51 AM, Christian Tornqvist wrote:
>>>>> Hi Yumin,
>>>>>
>>>>> The idea is to do as little as possible in the VM error handler, since
>> we've crashed for some reason we don't know what state the process is in and
>> we have to be extremely careful in when we're gathering the information.
>> This seems like a step that is risky for all of the reasons David mentioned
>> below.
>>>>> It's also information that can easily be extracted post-mortem from the
>> core/mdmp using the method you described for OSX, so gathering this at the
>> time of a crash seems like an unnecessary risk.
>>>>> Thanks,
>>>>> Christian
>>>>>
>>>>> -----Original Message-----
>>>>> From: hotspot-compiler-dev-bounces at openjdk.java.net
>>>>> [mailto:hotspot-compiler-dev-bounces at openjdk.java.net] On Behalf Of
>>>>> David Holmes
>>>>> Sent: Monday, August 12, 2013 1:56 AM
>>>>> To: Yumin Qi
>>>>> Cc: hotspot compiler; hotspot-runtime-dev at openjdk.java.net
>>>>> Subject: Re: RFR: 8020962: dump loaded java classes when vm crash
>>>>>
>>>>> Hi Yumin,
>>>>>
>>>>> Note that the SA is only present in a full JDK, not a JRE (full or
>> compact profile).
>>>>> I have quite a few concerns with this:
>>>>>
>>>>> We already do a lot of things that are not valid to be done from a
>>>>> signal handling context but this really takes that to an extreme.
>>>>> Doing fork-exec from a signal handler seems like a recipe for
>>>>> disaster (Note the existing onError facility is typically used for
>>>>> synchronous failures.)
>>>>>
>>>>> The idea of launching a second VM to try and query a VM that has crashed
>> also seems somewhat problematic:
>>>>> - What mechanism will the SA try to use to query the VM?
>>>>> - What if the state of the crashed VM stops the SA from being able to
>> attach properly (ie both processes hang)?
>>>>> - What if it the SA also crashes, will it launch a third VM then a
>> fourth etc?
>>>>> Also what is the nature of this dump? How big is it? Where will it go?
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>> On 12/08/2013 9:36 AM, Yumin Qi wrote:
>>>>>> Hi, all
>>>>>>
>>>>>>    I would like to have your review for
>>>>>>
>>>>>> http://cr.openjdk.java.net/~minqi/8020962/webrev0/
>>>>>> <http://cr.openjdk.java.net/%7Eminqi/8020962/webrev0/>
>>>>>>
>>>>>> Description: When JVM crashed, we also want to check the
>>>>>> application class files especially we got core file from customers.
>>>>>> The aftermath analysis will benefit from all loaded java classes
>>>>>> available. In this change, spawn another process running SA to do
>>>>>> the job when JVM crashes, this way also avoid further messing up
>>>>>> with the error report which already in signal handler.
>>>>>>
>>>>>> Note: The test has done with following two bugs worked around:
>>>>>>    8022655: ClassDump ignored jarStream setting (This will be fixed
>>>>>> and integrated by Kevin Walls soon)
>>>>>>    8011888: sa.js: TypeError: [object JSAdapter] has no such
>>>>>> function "__has__" (Not know when it will be integrated)
>>>>>>    That is, without those two fixed, the jars of loaded classes
>>>>>> will not be successfully dumped.
>>>>>>    Also, on MacOS it requires security access permission to attach
>>>>>> to another process, so omit doing so. To get loaded jar file s,
>>>>>> with core file available (on all platforms), one can (only after
>>>>>> this
>>>>>> change) do
>>>>>>
>>>>>>    $JAVA_HOME/bin/java [-d64] -cp $JAVA_HOME/lib/sa-jdi.jar
>>>>>> sun.jvm.hotspot.DumpLoadedClasses $JAVA_HOME/bin/java corefile
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Yumin
>>