Linux crash native stacks only have 1 line

[Volker Simonis]

On Tue, Apr 7, 2020 at 9:37 PM Alexander Miloslavskiy
<alexandr.miloslavskiy at gmail.com> wrote:
>
>
> On 07.04.2020 19:35, Volker Simonis wrote:
>
> > Sure, why not - you can definitively give it a try. On thing you
> > should be aware of is that this code will be usually called from the
> > signal handler, when the VM is already crashing. This means that the
> > VM may already be in an unstable state and we should be very careful
> > in order to not risk another error. You can start by looking at
> > ElfDecoder/ElfFile in decoder_elf.hpp/elfFile.hpp which already parses
> > an ELF file to get symbol information for a certain address. It has
> > the following comment :)
> >
> > // ElfFile is basically an elf file parser, which can lookup the symbol
> > // that is the nearest to the given address.
> > // Beware, this code is called from vm error reporting code, when vm is already
> > // in "error" state, so there are scenarios, lookup will fail. We want this
> > // part of code to be very defensive, and bait out if anything went wrong.
> >
> > I think what you'll have to do is to find out the frame size of a
> > native function and use that for unwinding if you have no FP. But this
> > can be tricky because glibc sometimes uses frameless functions. I
> > think if you look at gdb's unwinding code you will see that it is
> > quite complex.
> >
> > Please also take into account that using a third party library like
> > libunwind is not an option for OpenJDK because of licensing issues.
>
> I looked into default glibc `backtrace()` implementation and found that
> it loads libgcc and uses `_Unwind_Backtrace` from it. I understand that
> libgcc is expected to be installed on most Linux systems, then.
>
> So it sounds promising to also use `_Unwind_Backtrace` in JDK. What do
> you think?

Libjvm.so is already linked against libgcc anyway (either statically
or dynamically, depending on the "--with-stdc++lib" configure option".
But the man page for "backtrace()" mentions;

These functions make some assumptions about how a function's return
address is stored on the stack.  Note the following:
 * Omission  of  the frame pointers (as implied by any of gcc(1)'s
nonzero optimization levels) may cause these assumptions to be
violated.
 * Inlined functions do not have stack frames.
 * Tail-call optimization causes one stack frame to replace another.

So I'm not sure if the result will be really better.

Also notice that "VMError::print_native_stack()" prints a mixed
Native/Java stack which ideally looks something like:

Stack: [0x00007ffff7ed5000,0x00007ffff7fd5000],
sp=0x00007ffff7fd2a70,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa5038d]  Unsafe_SetInt+0xad
j  sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+0
j  CrashNative.crashIt(Lsun/misc/Unsafe;I)V+53
j  CrashNative.doIt()V+44
v  ~StubRoutines::call_stub
V  [libjvm.so+0x66bafa]  JavaCalls::call_helper(JavaValue*,
methodHandle*, JavaCallArguments*, Thread*)+0xe1a
V  [libjvm.so+0x94cea8]  Reflection::invoke(instanceKlassHandle,
methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle,
bool, Thread*)+0xae8
V  [libjvm.so+0x94e876]  Reflection::invoke_method(oopDesc*, Handle,
objArrayHandle, Thread*)+0x126
V  [libjvm.so+0x6b7915]  JVM_InvokeMethod+0x115
j  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0
j  sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+100
j  sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6
j  java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+56
j  CrashNative.mainJava()V+32
v  ~StubRoutines::call_stub
V  [libjvm.so+0x66bafa]  JavaCalls::call_helper(JavaValue*,
methodHandle*, JavaCallArguments*, Thread*)+0xe1a
V  [libjvm.so+0x68c390]  jni_invoke_static(JNIEnv_*, JavaValue*,
_jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
[clone .isra.92] [clone .constprop.98]+0x1f0
V  [libjvm.so+0x68e979]  jni_CallStaticVoidMethodV+0xd9
C  [libCrashNative.so+0xa81]  JNIEnv_::CallStaticVoidMethod(_jclass*,
_jmethodID*, ...)+0xc7
C  [libCrashNative.so+0xb00]  step3(JNIEnv_*, _jobject*)+0x65
C  [libCrashNative.so+0xb5a]  step2(JNIEnv_*, _jobject*)+0x57
C  [libCrashNative.so+0xb45]  step2(JNIEnv_*, _jobject*)+0x42
C  [libCrashNative.so+0xb45]  step2(JNIEnv_*, _jobject*)+0x42
C  [libCrashNative.so+0xb45]  step2(JNIEnv_*, _jobject*)+0x42
C  [libCrashNative.so+0xb45]  step2(JNIEnv_*, _jobject*)+0x42
C  [libCrashNative.so+0xb45]  step2(JNIEnv_*, _jobject*)+0x42
C  [libCrashNative.so+0xb80]  step1(JNIEnv_*, _jobject*)+0x23
C  [libCrashNative.so+0x94d]  Java_CrashNative_nativeMethod+0x23
j  CrashNative.nativeMethod()V+0
j  CrashNative.main([Ljava/lang/String;)V+23
v  ~StubRoutines::call_stub
V  [libjvm.so+0x66bafa]  JavaCalls::call_helper(JavaValue*,
methodHandle*, JavaCallArguments*, Thread*)+0xe1a
V  [libjvm.so+0x68c390]  jni_invoke_static(JNIEnv_*, JavaValue*,
_jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
[clone .isra.92] [clone .constprop.98]+0x1f0
V  [libjvm.so+0x68e84b]  jni_CallStaticVoidMethod+0x15b
C  [libjli.so+0x88fc]  JavaMain+0xa3c
C  [libpthread.so.0+0x76db]  start_thread+0xdb

If you use a helper function like "backtrace()", it will only unwind
native frames until it reaches the first Java frame because it won't
be able to unwind Java frames. Even gdb can not do that , because it
doesn't know the frame layout of HotSpots JIT compiled or interpreted
frames.

Best regards,
Volker