RFR: 8376264: Mixed jstack could not unwind optimized frame

[Yasumasa Suenaga]

[JDK-8374482](https://bugs.openjdk.org/browse/JDK-8374482) has introduced to treat signal trampoline. It realized to unwind problematic frame from signal trampoline.
However it would fail if problematic frame is optimized by compiler - it means FDE (Frame Description Entry) of the function is NOP only.


00000018 0000000000000010 0000001c FDE cie=00000000 pc=0000000000000200..0000000000000206
  DW_CFA_nop
  DW_CFA_nop
  DW_CFA_nop


Actually we can see following stacks and exception as following in TestJhsdbJstackMixedCore.java:


0x00007f14d500204d      os::abort(bool, void const*, void const*) [clone .co
ld] + 0x4
0x00007f14d6bd0001      VMError::report_and_die(int, char const*, char const
*, __va_list_tag*, Thread*, unsigned char*, void const*, void const*, char c
onst*, int, unsigned long) + 0x6f1
0x00007f14d6bd07b9      VMError::report_and_die(Thread*, unsigned int, unsig
ned char*, void const*, void const*, char const*, ...) + 0x88
0x00007f14d6bd0818      VMError::report_and_die(Thread*, unsigned int, unsig
ned char*, void const*, void const*) + 0x58
0x00007f14d68ff1ba      JVM_handle_linux_signal + 0x42a
0x00007f14d7964290      <signal handler called>
0x00007f14d780b3a0      Java_jdk_test_lib_apps_LingeredApp_crash
0x00007f14be7468ba
----------System.err:(20/1429)----------
java.lang.NullPointerException: Cannot invoke "sun.jvm.hotspot.debugger.Address.addOffsetTo(long)" because the return value of "sun.jvm.hotspot.runtime.Frame.getFP()" is null
        at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:255)
        at jdk.hotspot.agent/sun.jvm.hotspot.runtime.amd64.AMD64Frame.addressOfInterpreterFrameMethod(AMD64Frame.java:532)
        at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Frame.getInterpreterFrameMethod(Frame.java:389)
        at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InterpretedVFrame.getMethod(InterpretedVFrame.java:36)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.PStack.getJavaNames(PStack.java:251)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.PStack.run(PStack.java:135)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.PStack.run(PStack.java:65)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.PStack.run(PStack.java:60)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.run(JStack.java:67)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:278)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:241)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:134)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:306)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:507)


This PR is not only the fix, but also treat RBP for unwinder robustness. Thus the patch is large...

-------------

Commit messages:
 - 8376264: Mixed jstack could not unwind optimized frame

Changes: https://git.openjdk.org/jdk/pull/29398/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=29398&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8376264
  Stats: 171 lines in 7 files changed: 46 ins; 74 del; 51 mod
  Patch: https://git.openjdk.org/jdk/pull/29398.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/29398/head:pull/29398

PR: https://git.openjdk.org/jdk/pull/29398