build openjdk with fsanitizer=thread
Dmitry Vyukov
dvyukov at google.com
Tue Mar 10 15:24:06 UTC 2020
On Tue, Mar 10, 2020 at 12:11 PM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> Yes, I noticed the highlight line before.
> As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
> But I can't find a case to describe the model,
> Could you give me a case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race.
I don't understand the questions. Maybe you could rephrase.
A data race is not possible on atomic operations by definition.
>
> Thank u
> Jie He
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, March 10, 2020 6:51 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> ,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Dmitry
> >
> > From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> > simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
> >
> > I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> > Finally, if two memory accesses are both read or both atomic, it's not a race, right?
> >
> > Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> > Thanks
> >
> > Jie He
>
>
> Hi Jie
>
> Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
> A non-deterministic behavior, or race condition, is not a bug per se.
> It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.
>
> As you noted 2 atomic operations never constitute a race and tsan respects that.
> But it does use memory order for memory model modelling as well, e.g.
> load with memory_order_acquire constraint will acquire visibility:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
> Store may release:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
> and RMWs can do both based on the mo:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L294
>
>
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Wednesday, February 26, 2020 7:39 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net;
> > nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
> >
> > Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
> >
> > Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
> >
> > Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
> >
> > FTR the other sources are here:
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/m
> > utex.cpp
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/ar
> > ena.cpp
> >
> >
> > On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi Arthur
> > > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > > But firstly I’m just wondering in the current case, why tsan couldn’t restore the stack?
> > >
> > > The thread looks like a normal thread created by pthread_create with
> > > a thread func thread_native_entry and a pointer parameter “thread”,
> > >
> > > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> > >
> > > The process is clear and all in c++ env. I’m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Jie He
> > >
> > > From: Arthur Eubanks <aeubanks at google.com>
> > > Sent: Wednesday, February 26, 2020 2:48 AM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net;
> > > nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > >
> > >
> > > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> > >
> > >
> > >
> > > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9770)
> > > Read of size 8 at 0x7b7c00002360 by thread T1:
> > > #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> > > #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> > > #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> > > #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > > #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > > #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > > #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > > #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > > #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > > #9 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > > Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > > [failed to restore the stack]
> > >
> > > Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > > #0 malloc <null> (java+0x421ee7)
> > > #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > > #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > > #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > > #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > Thread T1 (tid=9772, running) created by main thread at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > > Thread T6 (tid=9778, running) created by thread T1 at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > > #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > > #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > > #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 34
> > > 3:3 in Mutex::assert_owner(Thread*)
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: Jie He <Jie.He at arm.com>
> > > Sent: Tuesday, February 25, 2020 4:09 PM
> > > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> > >
> > > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9787)
> > > Read of size 8 at 0x7b7c00002360 by thread T1:
> > > #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> > > #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> > > #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> > > #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> > > #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > > #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > > #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > > #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > > #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > > #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > > #10 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > > Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > > [failed to restore the stack]
> > >
> > > Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > > #0 malloc <null> (java+0x421ee7)
> > > #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > > #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > > #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > > #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > Thread T1 (tid=9789, running) created by main thread at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > > Thread T6 (tid=9795, running) created by thread T1 at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > > #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > > #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > > #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 30
> > > 1:10 in Mutex::owned_by_self() const
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of Jie
> > > He
> > > Sent: Tuesday, February 25, 2020 2:19 PM
> > > To: Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> > >
> > > Currently, I'm not sure if the second thread is JIT thread.
> > > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> > >
> > > like the following tsan reports, I have changed the history_size to 4:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9726)
> > > Write of size 8 at 0x7b1800003ab0 by thread T1:
> > > #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> > > #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> > > #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> > > #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> > > #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> > > #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> > > #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> > > #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> > > #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> > > #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> > > #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> > > #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> > > #12 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x67e9)
> > >
> > > Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> > > [failed to restore the stack]
> > >
> > > Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> > > #0 malloc <null> (java+0x421ee7)
> > > #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> > > #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> > > #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> > > #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> > > #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> > > #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> > > #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #11 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > Thread T1 (tid=9728, running) created by main thread at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > > Thread T14 (tid=9742, running) created by thread T1 at:
> > > #0 pthread_create <null> (java+0x4233d5)
> > > #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> > > #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> > > #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> > > #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > #7 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > > 16 in ChunkPool::free(Chunk*)
> > >
> > >
> > > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> > >
> > > // Return a chunk to the pool
> > > void free(Chunk* chunk) {
> > > assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> > > ThreadCritical tc;
> > > _num_used--;
> > >
> > > // Add chunk to list
> > > chunk->set_next(_first);
> > > 92: _first = chunk;
> > > 93: _num_chunks++;
> > > }
> > >
> > >
> > > -----Original Message-----
> > > From: Dmitry Vyukov <dvyukov at google.com>
> > > Sent: Tuesday, February 25, 2020 1:30 PM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Hi
> > > >
> > > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > > >
> > > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > > >
> > > >
> > > > 1. Benign races, commonly, there is a comment to indicate why it is safe in MP.
> > >
> > > +thread-sanitizer mailing list
> > >
> > > Hi Jie,
> > >
> > > C++ standard still calls this data race and renders behavior of the
> > > program as undefined. Comments don't fix bugs ;)
> > >
> > > > 2. Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> > >
> > > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> > >
> > > > 3. Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > > >
> > > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > > >
> > > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> > >
> > > Yes, tsan understands pthread mutex natively, no annotations required.
> > > You may try to increase history_size flag to get the second stack:
> > > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > > "failed to restore stack trace" should not lead to false positives either.
> > >
> > > Please post several full tsan reports and links to the corresponding source code.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.
More information about the tsan-dev
mailing list