Debugging segmentation faults in the JVM on linux-powerpc

[Gustavo Romero]

Hello Adrian,

On 09-06-2017 12:02, John Paul Adrian Glaubitz wrote:
> Hi Gustavo!
> 
> On 06/09/2017 04:30 PM, Gustavo Romero wrote:
>> You can attach gdb when the error occurs passing to the JVM:
>>
>> -XX:OnError="gdb %p"
>> -XX:OnOutOfMemoryError="gdb %p"
> 
> Aha, that's a very useful feature. Thanks for the tip.
> 
>> Another thing is that the JVM will use SIGSEGV for some state transitions, hence
>> in gdb I usually let SIGSEGV be passed to the JVM:
>>
>> (gdb) handle SIGSEGV pass noprint nostop
> 
> But it does not mean the segmentation faults I have observed are actually
> intentional, are they?

No, if an intended SIGSEGV is caught in an expected transition the JVM will not
abort, exit, or crash. It will instead treat it and move on. Looks like also
that the segfault you observed happened in a way that did not reach the JVM
signal handler since it looks that not even a hs_err log was generated.


> What confuses me most is that the JVM segfaults during the build but
> bails out with the out-of-memory error when I manually run any of
> the commands after the failed build. It almost looks like I forgot
> to set some environment variables.
> 
>> On PPC64 (not sure what's the current state on PPC32) we can also have SIGTRAP
>> for state transitions and I had some trouble in the past debugging with
>> -XX:+UseSIGTRAP enabled (basically gdb halts on some specific thread types that
>> generates such a type of signal), so I also usually ask to the JVM to not use
>> SIGTRAP and use SIGILL instead, enabling the passthrough of SIGILL:
>>
>> (gdb) handle SIGILL pass noprint nostop
>>
>> and calling the JVM with "-XX:-UseSIGTRAP".
> 
> Good to know. I would have been confused by that behavior for sure.
> 
>> Starting the JVM from gdb it's also fine given that you handle the signals,
>> otherwise all can go well until init_globals() but beyond that, after some
>> threads are created, the debugging can halt for no apparently reason.
> 
> What's the best way to actually start the JVM from gdb? Do I just
> load "java" into gdb and run it with the suggested parameters?

Yes, exactly. Also use a slow/fast debug build, not a release one. With a debug
build you can also use, from the gdb, "call help()" which is very helpful to
walk the native and the JVM stack, for instance:

(gdb) call help()

"Executing help"
basic
  pp(void* p)   - try to make sense of p
  pv(intptr_t p)- ((PrintableResourceObj*) p)->print()
  ps()          - print current thread stack
  pss()         - print all thread stacks
  pm(int pc)    - print Method* given compiled PC
  findm(intptr_t pc) - finds Method*
  find(intptr_t x)   - finds & prints nmethod/stub/bytecode/oop based on pointer into it
  pns(void* sp, void* fp, void* pc)  - print native (i.e. mixed) stack trace. E.g.
                   pns($sp, $rbp, $pc) on Linux/amd64 and Solaris/amd64 or
                   pns($sp, $ebp, $pc) on Linux/x86 or
                   pns($sp, 0, $pc)    on Linux/ppc64 or
                   pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC
                 - in gdb do 'set overload-resolution off' before calling pns()
                 - in dbx do 'frame 1' before calling pns()
misc.
  flush()       - flushes the log file
  events()      - dump events from ring buffers
compiler debugging
  debug()       - to set things up for compiler debugging
  ndebug()      - undo debug

(gdb) call pns($sp, 0, $pc)

"Executing pns"
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
v  ~StubRoutines::SafeFetch32
V  [libjvm.so+0xd0bf9c]  init_globals()+0x19c
V  [libjvm.so+0x15acf04]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x364
V  [libjvm.so+0xdb085c]  JNI_CreateJavaVM+0x10c
C  [libjli.so+0x48c0]  JavaMain+0xd0
C  [libpthread.so.0+0x8070]  start_thread+0xf0
C  [libc.so.6+0x123230]  clone+0x98

Regards,
Gustavo


> Adrian
>