RFO: a tool to analyze HotSpot fatal error logs

[Baesken, Matthias]

>You can go to the declaration of the PasswordAuthentication class when looking at things like
>Event: 112.303 loading class java/net/PasswordAuthentication

That sounds good !
A colleague just today (in another context) pointed out the idea to have an option to select all the  hserr event log sections  into a single
Log with chronological order .   That would probably also something this tool could do (or is it already implemented) .

>I haven't thought of opening source files via links from the log itself since I rarely see them there (mostly assertion failures), but it's easy enough to implement.

You see them in the assertion failures or guarantees .  But also in that native stacks  (even with line numbers on some platforms) .
See for example :

#  Internal Error (/openjdk-22u-linux_aarch64-dbg/jdk/src/hotspot/share/prims/jvmtiRawMonitor.cpp:174), pid=3474004, tid=3474024
#  guarantee(w ->_t_state == QNode::TS_ENTER) failed: invariant
#
# JRE version: OpenJDK Runtime Environment (22.0.1) (fastdebug build 22.0.1-internal-adhoc.jenkinsi.jdk)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 22.0.1-internal-adhoc.jenkinsi.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x10d1520]  JvmtiRawMonitor::simple_exit(Thread*)+0x18c
#


---------------  T H R E A D  ---------------

Current thread (0x0000ffff38001310):  JavaThread "JDWP Command Reader" daemon [_thread_in_native, id=3474024, stack(0x0000ffff56d92000,0x0000ffff56f90000) (2040K)]

Stack: [0x0000ffff56d92000,0x0000ffff56f90000],  sp=0x0000ffff56f8e600,  free space=2033k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x10d1520]  JvmtiRawMonitor::simple_exit(Thread*)+0x18c  (jvmtiRawMonitor.cpp:174)
V  [libjvm.so+0x10d2b84]  JvmtiRawMonitor::raw_exit(Thread*)+0x44  (jvmtiRawMonitor.cpp:367)
V  [libjvm.so+0x10993ac]  JvmtiEnv::RawMonitorExit(JvmtiRawMonitor*)+0xf8  (jvmtiEnv.cpp:3629)
C  [libjdwp.so+0x2c418]  debugMonitorExit+0x38  (util.c:1023)
C  [libjdwp.so+0x13840]  reader+0xe0  (debugLoop.c:288)
V  [libjvm.so+0x10c9ee0]  JvmtiAgentThread::call_start_function()+0x60  (jvmtiImpl.cpp:89)
V  [libjvm.so+0xddd6f0]  JavaThread::thread_main_inner()+0xec  (javaThread.cpp:721)
V  [libjvm.so+0x1756bc4]  Thread::call_run()+0xb0  (thread.cpp:221)
V  [libjvm.so+0x13aea68]  thread_native_entry(Thread*)+0x138  (os_linux.cpp:789)
C  [libc.so.6+0x82a38]  start_thread+0x2d4

For the failing asssertions / guarantees it  might also be helpful to augment some info about the assertion.

Best regards, Matthias



From: Maxim Kartashev <maxim.kartashev at jetbrains.com>
Sent: Friday, 12 April 2024 10:48
To: Baesken, Matthias <matthias.baesken at sap.com>
Cc: discuss at openjdk.org; hotspot-dev at openjdk.org; Doerr, Martin <martin.doerr at sap.com>; Langer, Christoph <christoph.langer at sap.com>
Subject: Re: RFO: a tool to analyze HotSpot fatal error logs


> Does the tool  work  both with  hserr files (“HotSpot fatal error logs”)    and also   the  output  of   jcmd  <pid>  VM.info   ?

Yes since the latter is more or less a short version of the former.

> How well can it handle incomplete hserr files  (we sometimes see those in case of bad crashes) ?

As well as can be expected; some of the crashes are naturally redacted because of induced crashes and we're taking whatever information is there. We're also processing crashes from many JVM versions each of which introduces its own variance to the content and format of the log.

There are many heuristics in the parsing so it can yield incorrect results (rarely), but it also makes parsing quite stable in the sense that practically any log a human can read the tool can also read.

> Can the tool  “mix in” / augment   additional information  into the  views of the error log   (like a bit of source code or links into the  stack traces for example) ?

You can go to the declaration of the PasswordAuthentication class when looking at things like
Event: 112.303 loading class java/net/PasswordAuthentication
I haven't thought of opening source files via links from the log itself since I rarely see them there (mostly assertion failures), but it's easy enough to implement.

> I thought about  creating s similar tool myself in the past, but did not happen so far …

I think this is the story for many on the list. I've seen enough people on youtube turning to ad hoc scripts to make sense of addresses in the logs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/discuss/attachments/20240412/99b32864/attachment-0001.htm>