<div dir="ltr"><div>Hi Jan,</div><div><br></div><div>are you sure this crash is related to JDK-8169373?</div><div><br></div><div>As far as I remember that bug resulted in threads possibly failing to start because the thread stack size specified was too small; I don't see how it can cause crashes like the one you describe. Once the thread had been started successfully, things should work, no?<br></div><div><br></div><div>Thanks, Thomas<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 6, 2023 at 5:07 PM Jan Kratochvil (Azul) <<a href="mailto:jkratochvil@azul.com">jkratochvil@azul.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
I got a crash report for an OpenJDK-8 derivation (Azul company's build Zulu-8)<br>
on aarch64. Details are at the end. I believe it could be fixed by some<br>
backports below but the resulting patch gets too big (>10k LoC = Lines of<br>
Code).<br>
<br>
Do you have an idea how to make the backport feasible for JDK-8?<br>
<br>
JDK-8169373: Work around linux NPTL stack guard error<br>
13 files changed, 165 insertions(+), 425 deletions(-)<br>
<br>
Unfortunately this patch depends on a lot of other code from JDK-9 and<br>
I believe these fixes are related anyway so they also need a backport:<br>
<br>
JDK-8159335: Fix problems with stack overflow handling<br>
33 files changed, 183 insertions(+), 236 deletions(-)<br>
JDK-8139864: Improve handling of stack protection zones<br>
43 files changed, 312 insertions(+), 226 deletions(-)<br>
<br>
To satisfy dependencies of the patches above I had to backport also:<br>
8078513: [linux] Clean up code relevant to LinuxThreads implementation<br>
8080298: Clean up os::...::supports_variable_stack_size()<br>
8037842: Failing to allocate MethodCounters and MDO causes a serious performance drop<br>
8059847: complement JDK-8055286 and JDK-8056964 changes<br>
8059606: Enable per-method usage of CompileThresholdScaling (per-method compilation thresholds)<br>
8074119: [AARCH64] stage repo misses fixes from several Hotspot changes<br>
8013393: Merge template interpreter files for x86 _32 and _64<br>
8122937: [JEP 245] Validate JVM Command-Line Flag Arguments<br>
8078556: Runtime: implement ranges (optionally constraints) for those flags that have them missing<br>
8048241: Introduce umbrella header os.inline.hpp and clean up includes<br>
^^^ 8139864: Improve handling of stack protection zones<br>
^^^ 8159335: Fix problems with stack overflow handling<br>
8140520: segfault on solaris-amd64 with "-XX:VMThreadStackSize=1" option<br>
^^^ 8169373: Work around linux NPTL stack guard error<br>
8049325: Introduce and clean up umbrella headers for the files in the cpu subdirectories<br>
8064611: AARCH64: Changes to HotSpot shared code<br>
8160189: Fix for 8159335 breaks AArch64<br>
8130858: CICompilerCount=1 when tiered is off is not allowed any more<br>
8072931: JEP-JDK-8059557: Test task: test framework development<br>
228 files changed, 13311 insertions(+), 8831 deletions(-)<br>
<br>
Which is just a too big patch for a backport into JDK-8. There are some<br>
possibilities for minor reduction of the whole patch but it will be still<br>
around 10k LoC. None of the patches above have been backported to JDK-8.<br>
<br>
Unfortunately due to time constraints I do not yet have confirmed these<br>
backports really fix this crash. It needs to be tested at a customer as I do<br>
not have a reproducer (and it is even difficult to reproduce on the target<br>
system).<br>
<br>
I can publish the whole patchset above but I would need to rebase it first<br>
from the Zulu-8 derivation to plain OpenJDK-8.<br>
<br>
<br>
Thanks for your opinion,<br>
Jan Kratochvil<br>
<br>
------------------------------------------------------------------------------<br>
<br>
The crashing memory access of __resp is in TLS (thread-local storage) which<br>
points to an unmapped memory during very early thread startup still in glibc:<br>
<br>
#1 <signal handler called><br>
#2 start_thread (arg=0x7ef47cd160) at /usr/src/debug/glibc/2.23-r0/git/nptl/pthread_create.c:265<br>
265 __resp = &pd->res;<br>
=> 0x0000007f836e7f1c <+36>: str x0, [x2, x1]<br>
<br>
Program Headers:<br>
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align<br>
LOAD 0x10e75000 0x0000007ef45ce000 0x0000000000000000 0x1ff000 0x1ff000 RW 0x1000<br>
0x7ef47cd000=end<br>
7ef47cd850=accessed memory __resp<br>
LOAD 0x11074000 0x0000007ef47ce000 0x0000000000000000 0x003000 0x003000 0x1000<br>
<br>
A different thread was creating the thread above:<br>
<br>
[Current thread is 13 (LWP 834)]<br>
#5 0x0000007f811b83a4 in os::create_thread (thread=0x7f20010420, thr_type=<optimized out>, stack_size=<optimized out>)<br>
at zulu8-arm64-dev/hotspot/src/os/linux/vm/os_linux.cpp:939<br>
tid = 545267700064 = 0x7ef47cd160<br>
<br>
Someone did unmap the page where TLS of the new thread is being located before<br>
the thread really started.<br>
<br>
Given there is always gap 0x3000 between the mappings - it should be the guard<br>
pages.<br>
<br>
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align<br>
LOAD 0x10e75000 0x0000007ef45ce000 0x0000000000000000 0x1ff000 0x1ff000 RW 0x1000<br>
0x7ef47cb6c0=$sp of Thread 1 (LWP 1358)<br>
0x7ef47cd000=mapped area end<br>
0x7ef47cd160=thread tid<br>
0x7ef47cd850=accessed memory __resp<br>
LOAD 0x11074000 0x0000007ef47ce000 0x0000000000000000 0x003000 0x003000 0x1000<br>
LOAD 0x11077000 0x0000007ef47d1000 0x0000000000000000 0x1fd000 0x1fd000 RW 0x1000<br>
0x7ef49cc240=$sp of Thread 37 (LWP 1349)<br>
LOAD 0x11274000 0x0000007ef49ce000 0x0000000000000000 0x003000 0x003000 0x1000<br>
LOAD 0x11277000 0x0000007ef49d1000 0x0000000000000000 0x1fd000 0x1fd000 RW 0x1000<br>
0x7ef4bc8fa0=$sp of Thread 36 (LWP 1290)<br>
LOAD 0x11474000 0x0000007ef4bce000 0x0000000000000000 0x000000 0x003000 0x1000<br>
</blockquote></div>