March 2022 - aarch64-port-dev

Re: [aarch64-port-dev ] Simple HTTP server app crashes on JDK 11 and 8 on OSv AArch64
by Andrew Haley 07 Mar '22

07 Mar '22

On 3/6/22 05:33, Waldek Kozaczuk wrote: > Thanks for this hint. I think I have solved this puzzle. > > Once I got hold of the JDK debuginfo files and identified the patching code > - MacroAssembler::pd_patch_instruction(), I was able to put a breakpoint in > it and see something very revealing: > > #0 MacroAssembler::pd_patch_instruction_size (branch=0x20000879465c > "\351\377\237\322\351\377\277\362\351\377\337\362\n\243\352\227\037", > target=*0xffffa00042c862e0* "\020zB") at > src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp:75 > #1 0x0000100000bc13cc in MacroAssembler::pd_patch_instruction (file=0x0, > line=0, target=0xffffa00042c862e0 "\020zB", branch=<optimized out>) > at src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp:626 > #2 NativeMovConstReg::set_data (this=0x20000879465c, x=-105551995837728) > at src/hotspot/cpu/aarch64/nativeInst_aarch64.cpp:262 > #3 0x0000100000850bd0 in CompiledIC::set_ic_destination_and_value > (value=0xffffa00042c862e0, > entry_point=0x20000823d290 > "(\b@\271\b\001]\322*\005@\371\037\001\n\353,\001", > <incomplete sequence \371\200>, this=<optimized out>) > at src/hotspot/share/code/compiledIC.hpp:193 > #4 ICStub::finalize (this=<optimized out>) at > src/hotspot/share/code/icBuffer.cpp:91 > #5 ICStubInterface::finalize (this=<optimized out>, self=<optimized out>) > at src/hotspot/share/code/icBuffer.cpp:43 > #6 0x0000100000e30958 in StubQueue::stub_finalize > (this=0xffffa00041555300, s=<optimized out>) at > src/hotspot/share/code/stubs.hpp:168 > #7 StubQueue::remove_first (this=0xffffa00041555300) at > src/hotspot/share/code/stubs.cpp:175 > .... > > The corresponding crash value of X9 was this: > > *0x0000a00042c862e0* > > vs the target argument of pd_patch_instruction() (see above in the > backtrace): > > *0xffffa00042c862e0* > > Now given this comment: > > // Move a constant pointer into r. In AArch64 mode the virtual > // address space is 48 bits in size, so we only need three > // instructions to create a patchable instruction sequence that can > // reach anywhere. > > and this fragment of pd_patch_instruction() - > https://github.com/openjdk/jdk17u/blob/6f0f42630eac1febf562062afc523fdf3d2a… > > it seems that the code to load x8 register with an address gets patched > with 0x0000a00042c862e0 instead of 0xffffa00042c862e0. It is interesting > that this assert - > https://github.com/openjdk/jdk17u/blob/6f0f42630eac1febf562062afc523fdf3d2a… > - does not get hit. It won't unless you run a debug build. If I were you I'd do that soon. > The bottom line is that the valid address 0xffffa00042c862e0 gets truncated > to 0x0000a00042c862e0 I guess based on the assumption that in Linux all > userspace addresses are 48-bits long (see > https://www.kernel.org/doc/html/latest/arm64/memory.html) That's right. > In OSv > unikernel, there is no separation between user space and kernel space, and > it happens that addresses returned by malloc fall into this range: > > 0xffffa00000000000 - 0xffffafffffffffff > > So I guess the only solution to fix it on the OSv side would be to tweak > its virtual memory mapping for mallocs and make sure it never uses virtual > addresses > 48-bits. Is that possible? AFAIK AArch64 supports two VA subranges, • The bottom VA range runs up from address 0x0000_0000_0000_0000 . With the maximum address width of 48 bits this gives a VA range of 0x0000_0000_0000_0000 to 0x0000_FFFF_FFFF_FFFF . • The top VA subrange runs up to address 0xFFFF_FFFF_FFFF_FFFF . With the maximum address width of 48 bits this gives a VA range of 0xFFFF_0000_0000_0000 to 0xFFFF_FFFF_FFFF_FFFF . Do you support both subranges simultaneously in OSy? > Could you confirm my analysis? If you can. AArch64 OpenJDK uses 48-bit userspace. We did have a patch to allow 64-bit addresses, but there wasn't any point at the time. But in any case, it'd be a really good idea to allow apps designed for 48-bit userspace to work on OSy. This might be a good time for us to resurrect the 64-bit address patch, but that might not help you much, because I guess you'll want older Java versions to work. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

2 3

Re: [aarch64-port-dev ] Simple HTTP server app crashes on JDK 11 and 8 on OSv AArch64
by Andrew Haley 05 Mar '22

05 Mar '22

On 3/5/22 10:08, Andrew Haley wrote: > On 3/5/22 05:25, Waldek Kozaczuk wrote: >> Any ideas on how to debug it further? >> >> Any help is greatly appreciated, >> Waldek > > Is is possible to attach gdb? Is it possible to get a core dump? > > I see no attached file. Please put it on a file sharing system so we > can see it. Having said that, it looks like there may be a problem when patching inline cache, and that may be due to a problem with cache flushing. Maybe __builtin___clear_cache is flaky. The caller is here: address MacroAssembler::ic_call(address entry, jint method_index) { RelocationHolder rh = virtual_call_Relocation::spec(pc(), method_index); movptr(rscratch2, (uintptr_t)Universe::non_oop_word()); return trampoline_call(Address(entry, rh)); } This generates a sequence that gets patched at runtime when a call is resolved. The compiled sequence initially looks like this: 0xffffa5be35e0: mov x9, #0xffff // #65535 0xffffa5be35e4: movk x9, #0xffff, lsl #16 0xffffa5be35e8: movk x9, #0xffff, lsl #32 0xffffa5be35ec: bl 0xffffa5bd23c0 then it gets patched to look like this: 0xffffa5be35e0: mov x9, #0x1b58 // #7000 0xffffa5be35e4: movk x9, #0x4, lsl #16 0xffffa5be35e8: movk x9, #0x8, lsl #32 0xffffa5be35ec: bl 0xffffa5be36e0 It looks to me as though some threads aren't seeing the patched code, and this could well be a kernel problem. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

1 0

Simple HTTP server app crashes on JDK 11 and 8 on OSv AArch64
by Waldek Kozaczuk 05 Mar '22

05 Mar '22

Hi, I am one of the committers to the OSv project (http://osv.io/) OSv is is a Linux-compatible unikernel. OSv has been able to run unmodified Linux version of Java very well since early on including Java 7 all the way to Java 14 (or possibly including the latest one) on x86_64. However, I have discovered some problems trying to run JVM apps on OSv aarch64 port which probably has some gaps compared to x64_64. And I am looking for some hints or help, in general, to figure out what exactly might be the cause of this issue. In essence, a simple HTTP server app (see the code in the link - https://github.com/cloudius-systems/osv-apps/blob/master/java-httpserver/Ht…) crashes once I sent some requests using curl. The crash looks like this on openjdk11: OSv v0.55.0-241-g8d4039b3 eth0: 192.168.122.15 Booted up in 61.89 ms Cmdline: /usr/lib/jvm/java/bin/java -cp /java-httpserver HttpServerApp ! Listening on port 8000 ... # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.10) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10, mixed mode, tiered, compressed oops, serial gc, linux-aarch64) # Problematic frame: # v ~BufferBlob::I2C/C2I adapters # # Core dump will be written. Default location: core.2 (may not exist) # # An error report file with more information is saved as: # /tmp/hs_err_pid2.log Could not load hsdis-aarch64.so; library not loadable; PrintAssembly is disabled # # If you would like to submit a bug report, please visit: # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts # # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000200009248830, pid=2, tid=47 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.10) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.10, mixed mode, tiered, compressed oops, serial gc, linux-aarch64) # Problematic frame: # v ~BufferBlob::I2C/C2I adapters # # Core dump will be written. Default location: core.2 (may not exist) # # If you would like to submit a bug report, please visit: # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts # --------------- S U M M A R Y ------------ Command Line: HttpServerApp Host: AArch64, 1 cores, 1G, Linux Time: Sat Apr 24 04:10:05 2021 UTC elapsed time: 13.255518 seconds (0d 0h 0m 13s) --------------- T H R E A D --------------- Current thread (0xffff900042e8b000): JavaThread "HTTP-Dispatcher" [_thread_in_Java, id=47, stack(0x000020007a801000,0x000020007aa00000)] Stack: [0x000020007a801000,0x000020007aa00000], sp=0x000020007a9fdb40, free space=2034k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) v ~BufferBlob::I2C/C2I adapters J 356 c1 java.lang.invoke.InvokerBytecodeGenerator.emitStaticInvoke(Ljava/lang/invoke/MemberName;Ljava/lang/invoke/LambdaForm$Name;)V java.base(a)11.0.10 (258 bytes) @ 0x000020000982c21c [0x000020000982be00+0x000000000000041c] C 0x0000200078a6de18 siginfo: si_signo: 11 (SIGSEGV), si_code: 2057296688 (unknown), si_errno: 8192, si_addr: 0x0000a00042f0f000 Please see this issue details for full output including core dump and state of registers - https://github.com/cloudius-systems/osv/issues/1157#issue-933793577. It is interesting, that when I disable JIT by adding '-X int' to the java command line, the app works flawlessly. That makes me thinks that the problem most likely happens somewhere in JVM when bytecode is translated into aarch64 machine code to be exxecuted later. I wonder if possibly it has to do with ARM I-cache/D-cache synchronization (see https://community.arm.com/developer/ip-products/processors/b/processors-ip-…) which somehow is not well supported by OSv. I actually had to fix a similar problem in OSv MMU code (see https://github.com/cloudius-systems/osv/commit/71919b9bb50604da56dec0ebe6cd…) to make sure that any page fault to populate page with executable permission calls *__builtin___clear_cache*() which was essential when loading any application ELF file in memory. I am guessing, in case of JIT it must be more complicated but presumably JVM also needs to clear I-cache after the machine code is generated and loaded into memory before execution. Obviously it is still my speculation. But that is where I need more hints and possibly pointers to the JVM code to help me understand what is going on. It would be nice if I could understand how the JIT works to some extent especially as far as interactions with Linux kernel go and what possibly might be wrong/missing in OSv aarch64 port. I am also speculating that JDK 8 crashes on OSv even though looking different might be have same root cause: OSv v0.55.0-266-g5aedb749 eth0: 192.168.122.15 Booted up in 92.21 ms Cmdline: java.so -cp /java-example Hello java.so: Starting JVM app using: io/osv/nonisolated/RunNonIsolatedJvmApp java.so: Setting Java system classloader to NonIsolatingOsvSystemClassLoader # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (assembler_aarch64.hpp:237), pid=2, tid=0xffffa0004109c630 # guarantee(val < (1U << nbits)) failed: Field too big for insn # # JRE version: (8.0_282-b08) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.282-b08 mixed mode linux-aarch64 compressed oops) # Core dump written. Default location: //core or core.2 # # An error report file with more information is saved as: # /tmp/hs_err_pid2.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted [backtrace] 0x000010000097a750 <???+9938768> 0x0000100000b90590 <???+12125584> See https://github.com/cloudius-systems/osv/issues/1145 for details. Any help would be greatly appreciated. Best regards, Waldek

2 3