[aarch64-port-dev ] Simple HTTP server app crashes on JDK 11 and 8 on OSv AArch64

Sun Feb 27 01:04:15 UTC 2022

Hi,

I am one of the committers to the OSv project (http://osv.io/). OSv is is a
Linux-compatible unikernel. OSv has been able to run unmodified Linux
version of Java very well since early on including Java 7 all the way to
Java 14 (or possibly including the latest one) on x86_64. However, I have
discovered some problems trying to run JVM apps on OSv aarch64 port which
probably has some gaps compared to x64_64. And I am looking for some hints
or help, in general, to figure out what exactly might be the cause of this
issue.

In essence, a simple HTTP server app (see the code in the link -
https://github.com/cloudius-systems/osv-apps/blob/master/java-httpserver/HttpServerApp.java),
crashes once I sent some requests using curl. The crash looks like this on
openjdk11:

OSv v0.55.0-241-g8d4039b3
eth0: 192.168.122.15
Booted up in 61.89 ms
Cmdline: /usr/lib/jvm/java/bin/java -cp /java-httpserver HttpServerApp !
Listening on port 8000 ...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build
11.0.10+9-Ubuntu-0ubuntu1.20.10)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10,
mixed mode, tiered, compressed oops, serial gc, linux-aarch64)
# Problematic frame:
# v  ~BufferBlob::I2C/C2I adapters
#
# Core dump will be written. Default location: core.2 (may not exist)
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid2.log
Could not load hsdis-aarch64.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build
11.0.10+9-Ubuntu-0ubuntu1.20.10)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10,
mixed mode, tiered, compressed oops, serial gc, linux-aarch64)
# Problematic frame:
# v  ~BufferBlob::I2C/C2I adapters
#
# Core dump will be written. Default location: core.2 (may not exist)
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#

---------------  S U M M A R Y ------------

Command Line: HttpServerApp

Host: AArch64, 1 cores, 1G, Linux
Time: Sat Apr 24 04:10:05 2021 UTC elapsed time: 13.255518 seconds (0d
0h 0m 13s)

---------------  T H R E A D  ---------------

Current thread (0xffff900042e8b000):  JavaThread "HTTP-Dispatcher"
[_thread_in_Java, id=47, stack(0x000020007a801000,0x000020007aa00000)]

Stack: [0x000020007a801000,0x000020007aa00000],
sp=0x000020007a9fdb40,  free space=2034k
Native frames: (J=compiled Java code, A=aot compiled Java code,
j=interpreted, Vv=VM code, C=native code)
v  ~BufferBlob::I2C/C2I adapters
J 356 c1 java.lang.invoke.InvokerBytecodeGenerator.emitStaticInvoke(Ljava/lang/invoke/MemberName;Ljava/lang/invoke/LambdaForm$Name;)V
java.base at 11.0.10 (258 bytes) @ 0x000020000982c21c
[0x000020000982be00+0x000000000000041c]
C  0x0000200078a6de18

siginfo: si_signo: 11 (SIGSEGV), si_code: 2057296688 (unknown),
si_errno: 8192, si_addr: 0x0000a00042f0f000

Please see this issue details for full output including core dump and state
of registers -
https://github.com/cloudius-systems/osv/issues/1157#issue-933793577.

It is interesting, that when I disable JIT by adding '-X int' to the java
command line, the app works flawlessly. That makes me thinks that the
problem most likely happens somewhere in JVM when bytecode is translated
into aarch64 machine code to be exxecuted later. I wonder if possibly it
has to do with ARM I-cache/D-cache synchronization (see
https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/caches-and-self-modifying-code)
which somehow is not well supported by OSv.  I actually had to fix a
similar problem in OSv MMU code (see
https://github.com/cloudius-systems/osv/commit/71919b9bb50604da56dec0ebe6cd046fc6f2db8a)
to make sure that any page fault to populate page with executable
permission calls *__builtin___clear_cache*() which was essential when
loading any application ELF file in memory.

I am guessing, in case of JIT it must be more complicated but presumably
JVM also needs to clear I-cache after the machine code is generated and
loaded into memory before execution. Obviously it is still my speculation.
But that is where I need more hints and possibly pointers to the JVM code
to help me understand what is going on. It would be nice if I could
understand how the JIT works to some extent especially as far as
interactions with Linux kernel go and what possibly might be wrong/missing
in OSv aarch64 port.

I am also speculating that JDK 8 crashes on OSv even though looking
different might be have same root cause:

OSv v0.55.0-266-g5aedb749
eth0: 192.168.122.15
Booted up in 92.21 ms
Cmdline: java.so -cp /java-example Hello
java.so: Starting JVM app using: io/osv/nonisolated/RunNonIsolatedJvmApp
java.so: Setting Java system classloader to NonIsolatingOsvSystemClassLoader
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:237), pid=2, tid=0xffffa0004109c630
#  guarantee(val < (1U << nbits)) failed: Field too big for insn
#
# JRE version:  (8.0_282-b08) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.282-b08 mixed mode
linux-aarch64 compressed oops)
# Core dump written. Default location: //core or core.2
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid2.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted

[backtrace]
0x000010000097a750 <???+9938768>
0x0000100000b90590 <???+12125584>

See https://github.com/cloudius-systems/osv/issues/1145 for details.

Any help would be greatly appreciated.

Best regards,
Waldek