[aarch64-port-dev ] Simple HTTP server app crashes on JDK 11 and 8 on OSv AArch64

Sat Mar 5 05:25:33 UTC 2022

I have been able to fix some issues in the signal handling in OSv and I
think I am getting better error reporting now. I have also tried to run the
same examples on JDK 17.

The crashes look very similar but the one on jdk17 seems to have the most
diagnostic information. So I am going to use this one. The summary looks
like this:
"#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002000082f6db8, pid=2, tid=38
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.2+8 (17.0.2+8)
(build 17.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (17.0.2+8, mixed mode,
sharing, tiered, compressed oops, compressed class ptrs, serial gc,
linux-aarc
h64)
# Problematic frame:
# v  ~BufferBlob::vtable chunks
#
# Core dump will be written. Default location: core.2 (may not exist)
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid2.log
Compiled method (c1)   12323  543       3
com.sun.net.httpserver.Headers::put (52 bytes)
 total in heap  [0x000020000886b890,0x000020000886c5a8] = 3352
 relocation     [0x000020000886b9e8,0x000020000886bad0] = 232
 main code      [0x000020000886bb00,0x000020000886c240] = 1856
 stub code      [0x000020000886c240,0x000020000886c388] = 328
 oops           [0x000020000886c388,0x000020000886c390] = 8
 metadata       [0x000020000886c390,0x000020000886c3f0] = 96
 scopes data    [0x000020000886c3f0,0x000020000886c480] = 144
 scopes pcs     [0x000020000886c480,0x000020000886c580] = 256
 dependencies   [0x000020000886c580,0x000020000886c588] = 8
 nul chk table  [0x000020000886c588,0x000020000886c5a8] = 32"

and backtrace like this:
"Current thread (0xffff90004354d000):  JavaThread "HTTP-Dispatcher"
[_thread_in_Java, id=38, stack(0x0000200019a01000,0x0000200019c00000)]

Stack: [0x0000200019a01000,0x0000200019c00000],  sp=0x0000200019bff3d0,
 free space=2040k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
v  ~BufferBlob::vtable chunks
j
 sun.net.httpserver.UnmodifiableHeaders$$Lambda$8+0x0000000800c0b658.accept(Ljava/lang/Object;Ljava/lang/Object;)V+12
jdk.httpserver at 17.0.2
j  java.util.Map.forEach(Ljava/util/function/BiConsumer;)V+72
java.base at 17.0.2
j
 sun.net.httpserver.UnmodifiableHeaders.<init>(Lcom/sun/net/httpserver/Headers;)V+21
jdk.httpserver at 17.0.2
j
 sun.net.httpserver.ExchangeImpl.<init>(Ljava/lang/String;Ljava/net/URI;Lsun/net/httpserver/Request;JLsun/net/httpserver/HttpConnection;)V+37
jdk.httpserver at 17.0.2
j  sun.net.httpserver.ServerImpl$Exchange.run()V+776 jdk.httpserver at 17.0.2
j
 sun.net.httpserver.ServerImpl$DefaultExecutor.execute(Ljava/lang/Runnable;)V+1
jdk.httpserver at 17.0.2
j
 sun.net.httpserver.ServerImpl$Dispatcher.handle(Ljava/nio/channels/SocketChannel;Lsun/net/httpserver/HttpConnection;)V+29
jdk.httpserver at 17.0.2
j  sun.net.httpserver.ServerImpl$Dispatcher.run()V+418 jdk.httpserver at 17.0.2
j  java.lang.Thread.run()V+11 java.base at 17.0.2
v  ~StubRoutines::call_stub
V  [libjvm.so+0x7da1c0]  JavaCalls::call_helper(JavaValue*, methodHandle
const&, JavaCallArguments*, JavaThread*)+0x308
V  [libjvm.so+0x7db92c]  JavaCalls::call_virtual(JavaValue*, Handle,
Klass*, Symbol*, Symbol*, JavaThread*)+0x19c
V  [libjvm.so+0x88f368]  thread_entry(JavaThread*, JavaThread*)+0x70
V  [libjvm.so+0xde2908]  JavaThread::thread_main_inner()+0x150
V  [libjvm.so+0xde2a4c]  JavaThread::run()+0x124
V  [libjvm.so+0xde5fd4]  Thread::call_run()+0xa4
V  [libjvm.so+0xb5d41c]  thread_native_entry(Thread*)+0xdc
C  0x0000000fc02b8e74
C  0x0000000fc02574c4  thread_main_c+0x24"

The attached file has the state of all registers and dumps of part or
memory but these are key observations I have managed to make:
The pc value - 0x00002000082f6db8 - points to this instruction:

ldp     x16, x12, [x9]

which seems to loads two registers x16 and x12 with data located at the
address pointed by x9 register: 0x0000a00040bbb800 - some bogus address.
Based on the value of x30:

R30=0x000020000886bc3c is at entry_point+316 in (nmethod*)0x000020000886b890

it seems that the caller was "nmethod" (native method?) and this is where
my reasoning hits a roadblock. I do not understand what *BufferBlob::vtable
*chunks class do. Is it part of JIT that translates bytecode into machine
code - nmethod? Why this "nmethod" would put a bogus address in x9
register? Unfortunately, I have very little clue what BufferBlob::vtable
and (generated?) "nmethod" is doing here. What kind of Linux ABI might be
called here? Could it be that OSv does not satisfy some assumptions about
how VMA addresses are generated and something should be at
0x0000a00040bbb800 but is not there?

Any ideas on how to debug it further?

Any help is greatly appreciated,
Waldek

I am attaching the full error log files and the VMA layout to this email.

On Sat, Feb 26, 2022 at 8:04 PM Waldek Kozaczuk <jwkozaczuk at gmail.com>
wrote:

> Hi,
>
> I am one of the committers to the OSv project (http://osv.io/). OSv is is
> a Linux-compatible unikernel. OSv has been able to run unmodified Linux
> version of Java very well since early on including Java 7 all the way to
> Java 14 (or possibly including the latest one) on x86_64. However, I have
> discovered some problems trying to run JVM apps on OSv aarch64 port which
> probably has some gaps compared to x64_64. And I am looking for some hints
> or help, in general, to figure out what exactly might be the cause of this
> issue.
>
> In essence, a simple HTTP server app (see the code in the link -
> https://github.com/cloudius-systems/osv-apps/blob/master/java-httpserver/HttpServerApp.java),
> crashes once I sent some requests using curl. The crash looks like this on
> openjdk11:
>
> OSv v0.55.0-241-g8d4039b3
> eth0: 192.168.122.15
> Booted up in 61.89 ms
> Cmdline: /usr/lib/jvm/java/bin/java -cp /java-httpserver HttpServerApp !
> Listening on port 8000 ...
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47
> #
> # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.10)
> # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10, mixed mode, tiered, compressed oops, serial gc, linux-aarch64)
> # Problematic frame:
> # v  ~BufferBlob::I2C/C2I adapters
> #
> # Core dump will be written. Default location: core.2 (may not exist)
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid2.log
> Could not load hsdis-aarch64.so; library not loadable; PrintAssembly is disabled
> #
> # If you would like to submit a bug report, please visit:
> #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
> #
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47
> #
> # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.10)
> # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10, mixed mode, tiered, compressed oops, serial gc, linux-aarch64)
> # Problematic frame:
> # v  ~BufferBlob::I2C/C2I adapters
> #
> # Core dump will be written. Default location: core.2 (may not exist)
> #
> # If you would like to submit a bug report, please visit:
> #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
> #
>
> ---------------  S U M M A R Y ------------
>
> Command Line: HttpServerApp
>
> Host: AArch64, 1 cores, 1G, Linux
> Time: Sat Apr 24 04:10:05 2021 UTC elapsed time: 13.255518 seconds (0d 0h 0m 13s)
>
> ---------------  T H R E A D  ---------------
>
> Current thread (0xffff900042e8b000):  JavaThread "HTTP-Dispatcher" [_thread_in_Java, id=47, stack(0x000020007a801000,0x000020007aa00000)]
>
> Stack: [0x000020007a801000,0x000020007aa00000],  sp=0x000020007a9fdb40,  free space=2034k
> Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
> v  ~BufferBlob::I2C/C2I adapters
> J 356 c1 java.lang.invoke.InvokerBytecodeGenerator.emitStaticInvoke(Ljava/lang/invoke/MemberName;Ljava/lang/invoke/LambdaForm$Name;)V java.base at 11.0.10 (258 bytes) @ 0x000020000982c21c [0x000020000982be00+0x000000000000041c]
> C  0x0000200078a6de18
>
>
> siginfo: si_signo: 11 (SIGSEGV), si_code: 2057296688 (unknown), si_errno: 8192, si_addr: 0x0000a00042f0f000
>
>
> Please see this issue details for full output including core dump and
> state of registers -
> https://github.com/cloudius-systems/osv/issues/1157#issue-933793577.
>
> It is interesting, that when I disable JIT by adding '-X int' to the java
> command line, the app works flawlessly. That makes me thinks that the
> problem most likely happens somewhere in JVM when bytecode is translated
> into aarch64 machine code to be exxecuted later. I wonder if possibly it
> has to do with ARM I-cache/D-cache synchronization (see
> https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/caches-and-self-modifying-code)
> which somehow is not well supported by OSv.  I actually had to fix a
> similar problem in OSv MMU code (see
> https://github.com/cloudius-systems/osv/commit/71919b9bb50604da56dec0ebe6cd046fc6f2db8a)
> to make sure that any page fault to populate page with executable
> permission calls *__builtin___clear_cache*() which was essential when
> loading any application ELF file in memory.
>
> I am guessing, in case of JIT it must be more complicated but presumably
> JVM also needs to clear I-cache after the machine code is generated and
> loaded into memory before execution. Obviously it is still my speculation.
> But that is where I need more hints and possibly pointers to the JVM code
> to help me understand what is going on. It would be nice if I could
> understand how the JIT works to some extent especially as far as
> interactions with Linux kernel go and what possibly might be wrong/missing
> in OSv aarch64 port.
>
> I am also speculating that JDK 8 crashes on OSv even though looking
> different might be have same root cause:
>
> OSv v0.55.0-266-g5aedb749
> eth0: 192.168.122.15
> Booted up in 92.21 ms
> Cmdline: java.so -cp /java-example Hello
> java.so: Starting JVM app using: io/osv/nonisolated/RunNonIsolatedJvmApp
> java.so: Setting Java system classloader to NonIsolatingOsvSystemClassLoader
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (assembler_aarch64.hpp:237), pid=2, tid=0xffffa0004109c630
> #  guarantee(val < (1U << nbits)) failed: Field too big for insn
> #
> # JRE version:  (8.0_282-b08) (build )
> # Java VM: OpenJDK 64-Bit Server VM (25.282-b08 mixed mode linux-aarch64 compressed oops)
> # Core dump written. Default location: //core or core.2
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid2.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> Aborted
>
> [backtrace]
> 0x000010000097a750 <???+9938768>
> 0x0000100000b90590 <???+12125584>
>
>
> See https://github.com/cloudius-systems/osv/issues/1145 for details.
>
> Any help would be greatly appreciated.
>
> Best regards,
> Waldek
>