From Jie.He at arm.com  Tue Mar 10 10:21:38 2020
From: Jie.He at arm.com (Jie He)
Date: Tue, 10 Mar 2020 10:21:38 +0000
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
Message-ID: <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>

Hi Dmitry

From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.

I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
Finally, if two memory accesses are both read or both atomic, it's not a race, right?

Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
Thanks

Jie He

-----Original Message-----
From: Dmitry Vyukov <dvyukov at google.com> 
Sent: Wednesday, February 26, 2020 7:39 PM
To: Jie He <Jie.He at arm.com>
Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
Subject: Re: build openjdk with fsanitizer=thread

Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.

Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).

Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.

Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.

FTR the other sources are here:
https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/mutex.cpp
https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/arena.cpp


On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
>
> Hi Arthur
> Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
>
> The thread looks like a normal thread created by pthread_create with a 
> thread func thread_native_entry and  a pointer parameter ?thread?,
>
> then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
>
> The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
>
>
>
> Thanks
>
> Jie He
>
> From: Arthur Eubanks <aeubanks at google.com>
> Sent: Wednesday, February 26, 2020 2:48 AM
> To: Jie He <Jie.He at arm.com>
> Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net; nd 
> <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
>
>
> It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
>
>
>
> IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
>
>
>
>
>
>
>
> On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
>
> Add more report for the previous case, false race between javamain thread and g1 sampling thread.
>
> WARNING: ThreadSanitizer: data race (pid=9770)
>   Read of size 8 at 0x7b7c00002360 by thread T1:
>     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
>     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
>     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
>     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
>     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
>     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
>     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
>     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
>     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
>     #9 JavaMain 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:560:5 (libjli.so+0x681b)
>
>   Previous write of size 8 at 0x7b7c00002360 by thread T6:
>     [failed to restore the stack]
>
>   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
>     #0 malloc <null> (java+0x421ee7)
>     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
>     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
>     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
>     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
>     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
>     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
>     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
>     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
>     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
>     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
>     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #13 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
>   Thread T1 (tid=9772, running) created by main thread at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 CallJavaMainInNewThread 
> /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> md_solinux.c:754:9 (libjli.so+0xb53f)
>
>   Thread T6 (tid=9778, running) created by thread T1 at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
>     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
>     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
>     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
>     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
>     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
>     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
>     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
>     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
>     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #12 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
> SUMMARY: ThreadSanitizer: data race 
> /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:34
> 3:3 in Mutex::assert_owner(Thread*)
>
>
> B.R
> Jie He
>
> -----Original Message-----
> From: Jie He <Jie.He at arm.com>
> Sent: Tuesday, February 25, 2020 4:09 PM
> To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> Cc: nd <nd at arm.com>; thread-sanitizer 
> <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> Subject: RE: build openjdk with fsanitizer=thread
>
> Hi Dmitry
>
> another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
>
> See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
>
> WARNING: ThreadSanitizer: data race (pid=9787)
>   Read of size 8 at 0x7b7c00002360 by thread T1:
>     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
>     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
>     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
>     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
>     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
>     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
>     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
>     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
>     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
>     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
>     #10 JavaMain 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:560:5 (libjli.so+0x681b)
>
>   Previous write of size 8 at 0x7b7c00002360 by thread T6:
>     [failed to restore the stack]
>
>   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
>     #0 malloc <null> (java+0x421ee7)
>     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
>     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
>     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
>     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
>     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
>     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
>     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
>     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
>     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
>     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
>     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #13 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
>   Thread T1 (tid=9789, running) created by main thread at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 CallJavaMainInNewThread 
> /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> md_solinux.c:754:9 (libjli.so+0xb53f)
>
>   Thread T6 (tid=9795, running) created by thread T1 at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
>     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
>     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
>     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
>     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
>     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
>     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
>     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
>     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
>     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #12 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
> SUMMARY: ThreadSanitizer: data race 
> /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:30
> 1:10 in Mutex::owned_by_self() const
>
>
> B.R
> Jie He
>
> -----Original Message-----
> From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of Jie He
> Sent: Tuesday, February 25, 2020 2:19 PM
> To: Dmitry Vyukov <dvyukov at google.com>
> Cc: nd <nd at arm.com>; thread-sanitizer 
> <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> Subject: RE: build openjdk with fsanitizer=thread
>
> Hi Dmitry
>
> Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
>
> Currently, I'm not sure if the second thread is JIT thread.
> But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
>
> like the following tsan reports, I have changed the history_size to 4:
>
> WARNING: ThreadSanitizer: data race (pid=9726)
>   Write of size 8 at 0x7b1800003ab0 by thread T1:
>     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
>     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
>     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
>     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
>     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
>     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
>     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
>     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
>     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
>     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
>     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
>     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
>     #12 JavaMain 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:560:5 (libjli.so+0x67e9)
>
>   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
>     [failed to restore the stack]
>
>   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
>     #0 malloc <null> (java+0x421ee7)
>     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
>     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
>     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
>     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
>     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
>     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
>     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
>     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
>     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #11 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
>   Thread T1 (tid=9728, running) created by main thread at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 CallJavaMainInNewThread 
> /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> md_solinux.c:754:9 (libjli.so+0xb53f)
>
>   Thread T14 (tid=9742, running) created by thread T1 at:
>     #0 pthread_create <null> (java+0x4233d5)
>     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
>     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
>     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
>     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
>     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
>     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
>     #7 InitializeJVM 
> /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> .c:1538:9 (libjli.so+0x6974)
>
> SUMMARY: ThreadSanitizer: data race 
> /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> 16 in ChunkPool::free(Chunk*)
>
>
> And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
>
>       // Return a chunk to the pool
>       void free(Chunk* chunk) {
>          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
>          ThreadCritical tc;
>          _num_used--;
>
>           // Add chunk to list
>           chunk->set_next(_first);
> 92:    _first = chunk;
> 93:    _num_chunks++;
>   }
>
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, February 25, 2020 1:30 PM
> To: Jie He <Jie.He at arm.com>
> Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer 
> <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi
> >
> > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> >
> > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> >
> >
> >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
>
> +thread-sanitizer mailing list
>
> Hi Jie,
>
> C++ standard still calls this data race and renders behavior of the
> program as undefined. Comments don't fix bugs ;)
>
> >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
>
> I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
>
> >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> >
> > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> >
> > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
>
> Yes, tsan understands pthread mutex natively, no annotations required.
> You may try to increase history_size flag to get the second stack:
> https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> "failed to restore stack trace" should not lead to false positives either.
>
> Please post several full tsan reports and links to the corresponding source code.

From dvyukov at google.com  Tue Mar 10 10:50:40 2020
From: dvyukov at google.com (Dmitry Vyukov)
Date: Tue, 10 Mar 2020 11:50:40 +0100
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
Message-ID: <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>

,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
>
> I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> Finally, if two memory accesses are both read or both atomic, it's not a race, right?
>
> Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> Thanks
>
> Jie He


Hi Jie

Tsan detects data races, as defined by C/C++ language. A data race is
always a bug as it renders behavior of the program as undefined.
A non-deterministic behavior, or race condition, is not a bug per se.
It's just multiple allowed executions. Tsan does not detect these, and
in most cases it's not useful. I am pretty sure JVM is full of
non-deterministic behavior all over the place.

As you noted 2 atomic operations never constitute a race and tsan respects that.
But it does use memory order for memory model modelling as well, e.g.
load with memory_order_acquire constraint will acquire visibility:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
Store may release:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
and RMWs can do both based on the mo:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L294


>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Wednesday, February 26, 2020 7:39 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
>
> Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
>
> Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
>
> Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
>
> FTR the other sources are here:
> https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/mutex.cpp
> https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/arena.cpp
>
>
> On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Arthur
> > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> >
> > The thread looks like a normal thread created by pthread_create with a
> > thread func thread_native_entry and  a pointer parameter ?thread?,
> >
> > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> >
> > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> >
> >
> >
> > Thanks
> >
> > Jie He
> >
> > From: Arthur Eubanks <aeubanks at google.com>
> > Sent: Wednesday, February 26, 2020 2:48 AM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net; nd
> > <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> >
> >
> > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> >
> >
> >
> > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> >
> > WARNING: ThreadSanitizer: data race (pid=9770)
> >   Read of size 8 at 0x7b7c00002360 by thread T1:
> >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> >     #9 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:560:5 (libjli.so+0x681b)
> >
> >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #13 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9772, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T6 (tid=9778, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #12 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:34
> > 3:3 in Mutex::assert_owner(Thread*)
> >
> >
> > B.R
> > Jie He
> >
> > -----Original Message-----
> > From: Jie He <Jie.He at arm.com>
> > Sent: Tuesday, February 25, 2020 4:09 PM
> > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > Cc: nd <nd at arm.com>; thread-sanitizer
> > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > Subject: RE: build openjdk with fsanitizer=thread
> >
> > Hi Dmitry
> >
> > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> >
> > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> >
> > WARNING: ThreadSanitizer: data race (pid=9787)
> >   Read of size 8 at 0x7b7c00002360 by thread T1:
> >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> >     #10 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:560:5 (libjli.so+0x681b)
> >
> >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #13 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9789, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T6 (tid=9795, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #12 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:30
> > 1:10 in Mutex::owned_by_self() const
> >
> >
> > B.R
> > Jie He
> >
> > -----Original Message-----
> > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of Jie He
> > Sent: Tuesday, February 25, 2020 2:19 PM
> > To: Dmitry Vyukov <dvyukov at google.com>
> > Cc: nd <nd at arm.com>; thread-sanitizer
> > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > Subject: RE: build openjdk with fsanitizer=thread
> >
> > Hi Dmitry
> >
> > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> >
> > Currently, I'm not sure if the second thread is JIT thread.
> > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> >
> > like the following tsan reports, I have changed the history_size to 4:
> >
> > WARNING: ThreadSanitizer: data race (pid=9726)
> >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> >     #12 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:560:5 (libjli.so+0x67e9)
> >
> >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #11 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9728, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/java_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T14 (tid=9742, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #7 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/java
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > 16 in ChunkPool::free(Chunk*)
> >
> >
> > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> >
> >       // Return a chunk to the pool
> >       void free(Chunk* chunk) {
> >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> >          ThreadCritical tc;
> >          _num_used--;
> >
> >           // Add chunk to list
> >           chunk->set_next(_first);
> > 92:    _first = chunk;
> > 93:    _num_chunks++;
> >   }
> >
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Tuesday, February 25, 2020 1:30 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer
> > <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi
> > >
> > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > >
> > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > >
> > >
> > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> >
> > +thread-sanitizer mailing list
> >
> > Hi Jie,
> >
> > C++ standard still calls this data race and renders behavior of the
> > program as undefined. Comments don't fix bugs ;)
> >
> > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> >
> > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> >
> > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > >
> > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > >
> > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> >
> > Yes, tsan understands pthread mutex natively, no annotations required.
> > You may try to increase history_size flag to get the second stack:
> > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > "failed to restore stack trace" should not lead to false positives either.
> >
> > Please post several full tsan reports and links to the corresponding source code.
>
> --
> You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From Jie.He at arm.com  Tue Mar 10 11:11:32 2020
From: Jie.He at arm.com (Jie He)
Date: Tue, 10 Mar 2020 11:11:32 +0000
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
Message-ID: <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>

Hi Dmitry

Yes, I noticed the highlight line before. 
As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
But I can't find a case to describe the model,
Could you give me a case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race. 

Thank u
Jie He

-----Original Message-----
From: Dmitry Vyukov <dvyukov at google.com> 
Sent: Tuesday, March 10, 2020 6:51 PM
To: Jie He <Jie.He at arm.com>
Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
Subject: Re: build openjdk with fsanitizer=thread

,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
>
> I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> Finally, if two memory accesses are both read or both atomic, it's not a race, right?
>
> Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> Thanks
>
> Jie He


Hi Jie

Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
A non-deterministic behavior, or race condition, is not a bug per se.
It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.

As you noted 2 atomic operations never constitute a race and tsan respects that.
But it does use memory order for memory model modelling as well, e.g.
load with memory_order_acquire constraint will acquire visibility:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
Store may release:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
and RMWs can do both based on the mo:
https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L294


>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Wednesday, February 26, 2020 7:39 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; 
> nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
>
> Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
>
> Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
>
> Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
>
> FTR the other sources are here:
> https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/m
> utex.cpp 
> https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/ar
> ena.cpp
>
>
> On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Arthur
> > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> >
> > The thread looks like a normal thread created by pthread_create with 
> > a thread func thread_native_entry and  a pointer parameter ?thread?,
> >
> > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> >
> > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> >
> >
> >
> > Thanks
> >
> > Jie He
> >
> > From: Arthur Eubanks <aeubanks at google.com>
> > Sent: Wednesday, February 26, 2020 2:48 AM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net; 
> > nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> >
> >
> > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> >
> >
> >
> > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> >
> > WARNING: ThreadSanitizer: data race (pid=9770)
> >   Read of size 8 at 0x7b7c00002360 by thread T1:
> >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> >     #9 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:560:5 (libjli.so+0x681b)
> >
> >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #13 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9772, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > a_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T6 (tid=9778, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #12 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > 34
> > 3:3 in Mutex::assert_owner(Thread*)
> >
> >
> > B.R
> > Jie He
> >
> > -----Original Message-----
> > From: Jie He <Jie.He at arm.com>
> > Sent: Tuesday, February 25, 2020 4:09 PM
> > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > Cc: nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > Subject: RE: build openjdk with fsanitizer=thread
> >
> > Hi Dmitry
> >
> > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> >
> > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> >
> > WARNING: ThreadSanitizer: data race (pid=9787)
> >   Read of size 8 at 0x7b7c00002360 by thread T1:
> >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> >     #10 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:560:5 (libjli.so+0x681b)
> >
> >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #13 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9789, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > a_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T6 (tid=9795, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #12 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > 30
> > 1:10 in Mutex::owned_by_self() const
> >
> >
> > B.R
> > Jie He
> >
> > -----Original Message-----
> > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of Jie 
> > He
> > Sent: Tuesday, February 25, 2020 2:19 PM
> > To: Dmitry Vyukov <dvyukov at google.com>
> > Cc: nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > Subject: RE: build openjdk with fsanitizer=thread
> >
> > Hi Dmitry
> >
> > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> >
> > Currently, I'm not sure if the second thread is JIT thread.
> > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> >
> > like the following tsan reports, I have changed the history_size to 4:
> >
> > WARNING: ThreadSanitizer: data race (pid=9726)
> >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> >     #12 JavaMain
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:560:5 (libjli.so+0x67e9)
> >
> >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> >     [failed to restore the stack]
> >
> >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> >     #0 malloc <null> (java+0x421ee7)
> >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #11 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> >   Thread T1 (tid=9728, running) created by main thread at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 CallJavaMainInNewThread
> > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > a_
> > md_solinux.c:754:9 (libjli.so+0xb53f)
> >
> >   Thread T14 (tid=9742, running) created by thread T1 at:
> >     #0 pthread_create <null> (java+0x4233d5)
> >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> >     #7 InitializeJVM
> > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > va
> > .c:1538:9 (libjli.so+0x6974)
> >
> > SUMMARY: ThreadSanitizer: data race
> > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > 16 in ChunkPool::free(Chunk*)
> >
> >
> > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> >
> >       // Return a chunk to the pool
> >       void free(Chunk* chunk) {
> >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> >          ThreadCritical tc;
> >          _num_used--;
> >
> >           // Add chunk to list
> >           chunk->set_next(_first);
> > 92:    _first = chunk;
> > 93:    _num_chunks++;
> >   }
> >
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Tuesday, February 25, 2020 1:30 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi
> > >
> > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > >
> > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > >
> > >
> > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> >
> > +thread-sanitizer mailing list
> >
> > Hi Jie,
> >
> > C++ standard still calls this data race and renders behavior of the
> > program as undefined. Comments don't fix bugs ;)
> >
> > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> >
> > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> >
> > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > >
> > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > >
> > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> >
> > Yes, tsan understands pthread mutex natively, no annotations required.
> > You may try to increase history_size flag to get the second stack:
> > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > "failed to restore stack trace" should not lead to false positives either.
> >
> > Please post several full tsan reports and links to the corresponding source code.
>
> --
> You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From dvyukov at google.com  Tue Mar 10 15:24:06 2020
From: dvyukov at google.com (Dmitry Vyukov)
Date: Tue, 10 Mar 2020 16:24:06 +0100
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
 <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
Message-ID: <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>

On Tue, Mar 10, 2020 at 12:11 PM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> Yes, I noticed the highlight line before.
> As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
> But I can't find a case to describe the model,
> Could you give me a case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race.

I don't understand the questions. Maybe you could rephrase.
A data race is not possible on atomic operations by definition.


>
> Thank u
> Jie He
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, March 10, 2020 6:51 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> ,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Dmitry
> >
> > From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> > simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
> >
> > I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> > Finally, if two memory accesses are both read or both atomic, it's not a race, right?
> >
> > Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> > Thanks
> >
> > Jie He
>
>
> Hi Jie
>
> Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
> A non-deterministic behavior, or race condition, is not a bug per se.
> It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.
>
> As you noted 2 atomic operations never constitute a race and tsan respects that.
> But it does use memory order for memory model modelling as well, e.g.
> load with memory_order_acquire constraint will acquire visibility:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
> Store may release:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
> and RMWs can do both based on the mo:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L294
>
>
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Wednesday, February 26, 2020 7:39 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net;
> > nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
> >
> > Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
> >
> > Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
> >
> > Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
> >
> > FTR the other sources are here:
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/m
> > utex.cpp
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/ar
> > ena.cpp
> >
> >
> > On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi Arthur
> > > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> > >
> > > The thread looks like a normal thread created by pthread_create with
> > > a thread func thread_native_entry and  a pointer parameter ?thread?,
> > >
> > > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> > >
> > > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Jie He
> > >
> > > From: Arthur Eubanks <aeubanks at google.com>
> > > Sent: Wednesday, February 26, 2020 2:48 AM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net;
> > > nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > >
> > >
> > > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> > >
> > >
> > >
> > > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9770)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> > >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> > >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> > >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #9 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9772, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9778, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 34
> > > 3:3 in Mutex::assert_owner(Thread*)
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: Jie He <Jie.He at arm.com>
> > > Sent: Tuesday, February 25, 2020 4:09 PM
> > > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> > >
> > > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9787)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> > >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> > >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> > >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> > >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #10 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9789, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9795, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 30
> > > 1:10 in Mutex::owned_by_self() const
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of Jie
> > > He
> > > Sent: Tuesday, February 25, 2020 2:19 PM
> > > To: Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> > >
> > > Currently, I'm not sure if the second thread is JIT thread.
> > > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> > >
> > > like the following tsan reports, I have changed the history_size to 4:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9726)
> > >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> > >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> > >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> > >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> > >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> > >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> > >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> > >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> > >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> > >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> > >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> > >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> > >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> > >     #12 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:560:5 (libjli.so+0x67e9)
> > >
> > >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> > >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> > >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> > >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> > >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> > >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> > >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #11 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9728, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/jav
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T14 (tid=9742, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> > >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> > >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> > >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #7 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > > 16 in ChunkPool::free(Chunk*)
> > >
> > >
> > > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> > >
> > >       // Return a chunk to the pool
> > >       void free(Chunk* chunk) {
> > >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> > >          ThreadCritical tc;
> > >          _num_used--;
> > >
> > >           // Add chunk to list
> > >           chunk->set_next(_first);
> > > 92:    _first = chunk;
> > > 93:    _num_chunks++;
> > >   }
> > >
> > >
> > > -----Original Message-----
> > > From: Dmitry Vyukov <dvyukov at google.com>
> > > Sent: Tuesday, February 25, 2020 1:30 PM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Hi
> > > >
> > > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > > >
> > > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > > >
> > > >
> > > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> > >
> > > +thread-sanitizer mailing list
> > >
> > > Hi Jie,
> > >
> > > C++ standard still calls this data race and renders behavior of the
> > > program as undefined. Comments don't fix bugs ;)
> > >
> > > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> > >
> > > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> > >
> > > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > > >
> > > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > > >
> > > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> > >
> > > Yes, tsan understands pthread mutex natively, no annotations required.
> > > You may try to increase history_size flag to get the second stack:
> > > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > > "failed to restore stack trace" should not lead to false positives either.
> > >
> > > Please post several full tsan reports and links to the corresponding source code.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From Jie.He at arm.com  Wed Mar 11 03:44:06 2020
From: Jie.He at arm.com (Jie He)
Date: Wed, 11 Mar 2020 03:44:06 +0000
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
 <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>
Message-ID: <AM6PR08MB52191C50D5ED7EA0707995EC8EFC0@AM6PR08MB5219.eurprd08.prod.outlook.com>

Sorry, 
Simply, the question is, is there a kind of data race caused by wrong memory order in Atomic function, e.g. if using memory order other than RELAXED, the data race will disappear.

Initially, I think the idea in TSAN to detect data race is based on detecting certain non-deterministic memory access order.
 In a sense, it's sort of similar with the issue caused by weak memory order. Of course, there are more non-deterministic memory visibility order in latter, and most of them are benign, harmless.
Perhaps, a na?ve idea is I could use the idea in TSAN to detect the non-deterministic behavior even though more false-positives exist.
e.g. TSAN could help to find a pair of atomic/non-atomic memory accesses without happens-before existed between them if we ignore the isAtomic flag in shadow. It might be a potential issue, or most likely it's not.
But seems it's impossible to distinguish the false positives and real problem, so maybe I have a misunderstanding about happens-before algorithm.

Thanks
Jie He


-----Original Message-----
From: Dmitry Vyukov <dvyukov at google.com> 
Sent: Tuesday, March 10, 2020 11:24 PM
To: Jie He <Jie.He at arm.com>
Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
Subject: Re: build openjdk with fsanitizer=thread

On Tue, Mar 10, 2020 at 12:11 PM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> Yes, I noticed the highlight line before.
> As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
> But I can't find a case to describe the model, Could you give me a 
> case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race.

I don't understand the questions. Maybe you could rephrase.
A data race is not possible on atomic operations by definition.


>
> Thank u
> Jie He
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, March 10, 2020 6:51 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; 
> nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> ,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Dmitry
> >
> > From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> > simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
> >
> > I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> > Finally, if two memory accesses are both read or both atomic, it's not a race, right?
> >
> > Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> > Thanks
> >
> > Jie He
>
>
> Hi Jie
>
> Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
> A non-deterministic behavior, or race condition, is not a bug per se.
> It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.
>
> As you noted 2 atomic operations never constitute a race and tsan respects that.
> But it does use memory order for memory model modelling as well, e.g.
> load with memory_order_acquire constraint will acquire visibility:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
> Store may release:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
> and RMWs can do both based on the mo:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L
> 294
>
>
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Wednesday, February 26, 2020 7:39 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; 
> > nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
> >
> > Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
> >
> > Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
> >
> > Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
> >
> > FTR the other sources are here:
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime
> > /m
> > utex.cpp
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/
> > ar
> > ena.cpp
> >
> >
> > On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi Arthur
> > > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> > >
> > > The thread looks like a normal thread created by pthread_create 
> > > with a thread func thread_native_entry and  a pointer parameter 
> > > ?thread?,
> > >
> > > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> > >
> > > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Jie He
> > >
> > > From: Arthur Eubanks <aeubanks at google.com>
> > > Sent: Wednesday, February 26, 2020 2:48 AM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net; 
> > > nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > >
> > >
> > > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> > >
> > >
> > >
> > > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9770)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> > >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> > >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> > >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #9 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9772, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9778, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 34
> > > 3:3 in Mutex::assert_owner(Thread*)
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: Jie He <Jie.He at arm.com>
> > > Sent: Tuesday, February 25, 2020 4:09 PM
> > > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> > >
> > > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9787)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> > >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> > >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> > >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> > >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #10 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9789, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9795, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 30
> > > 1:10 in Mutex::owned_by_self() const
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of 
> > > Jie He
> > > Sent: Tuesday, February 25, 2020 2:19 PM
> > > To: Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> > >
> > > Currently, I'm not sure if the second thread is JIT thread.
> > > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> > >
> > > like the following tsan reports, I have changed the history_size to 4:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9726)
> > >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> > >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> > >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> > >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> > >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> > >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> > >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> > >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> > >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> > >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> > >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> > >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> > >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> > >     #12 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x67e9)
> > >
> > >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> > >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> > >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> > >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> > >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> > >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> > >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #11 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9728, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T14 (tid=9742, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> > >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> > >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> > >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #7 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > > 16 in ChunkPool::free(Chunk*)
> > >
> > >
> > > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> > >
> > >       // Return a chunk to the pool
> > >       void free(Chunk* chunk) {
> > >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> > >          ThreadCritical tc;
> > >          _num_used--;
> > >
> > >           // Add chunk to list
> > >           chunk->set_next(_first);
> > > 92:    _first = chunk;
> > > 93:    _num_chunks++;
> > >   }
> > >
> > >
> > > -----Original Message-----
> > > From: Dmitry Vyukov <dvyukov at google.com>
> > > Sent: Tuesday, February 25, 2020 1:30 PM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Hi
> > > >
> > > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > > >
> > > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > > >
> > > >
> > > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> > >
> > > +thread-sanitizer mailing list
> > >
> > > Hi Jie,
> > >
> > > C++ standard still calls this data race and renders behavior of 
> > > C++ the
> > > program as undefined. Comments don't fix bugs ;)
> > >
> > > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> > >
> > > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> > >
> > > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > > >
> > > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > > >
> > > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> > >
> > > Yes, tsan understands pthread mutex natively, no annotations required.
> > > You may try to increase history_size flag to get the second stack:
> > > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > > "failed to restore stack trace" should not lead to false positives either.
> > >
> > > Please post several full tsan reports and links to the corresponding source code.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From Jie.He at arm.com  Wed Mar 11 03:56:14 2020
From: Jie.He at arm.com (Jie He)
Date: Wed, 11 Mar 2020 03:56:14 +0000
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
 <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>
Message-ID: <AM6PR08MB52190DD2B3184F4E5061701B8EFC0@AM6PR08MB5219.eurprd08.prod.outlook.com>

Sorry,
Simply, the question is, is there a kind of data race caused by wrong memory order in Atomic function, e.g. if using memory order other than RELAXED, the data race will disappear.

Initially, I think the idea in TSAN to detect data race is based on detecting certain non-deterministic memory access order.
 In a sense, it's sort of similar with the issue caused by weak memory order. Of course, there are more non-deterministic memory visibility order in latter, and most of them are benign, harmless.
Perhaps, a na?ve idea is I could use the idea in TSAN to detect the non-deterministic behavior even though more false-positives exist.
e.g. TSAN could help to find a pair of atomic/non-atomic memory accesses without happens-before existed between them if we ignore the isAtomic flag in shadow. It might be a potential issue, or most likely it's not.
But seems it's impossible to distinguish the false positives and real problem, so maybe I have a misunderstanding about happens-before algorithm.
In other words, I want to loose the constrains of happens-before algorithm to find more non-deterministic memory accesses, and as a side effect, more false positives,
 but that's opposite of the effort TSan made in the past years.

Thanks
Jie He

-----Original Message-----
From: Dmitry Vyukov <dvyukov at google.com> 
Sent: Tuesday, March 10, 2020 11:24 PM
To: Jie He <Jie.He at arm.com>
Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
Subject: Re: build openjdk with fsanitizer=thread

On Tue, Mar 10, 2020 at 12:11 PM Jie He <Jie.He at arm.com> wrote:
>
> Hi Dmitry
>
> Yes, I noticed the highlight line before.
> As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
> But I can't find a case to describe the model, Could you give me a 
> case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race.

I don't understand the questions. Maybe you could rephrase.
A data race is not possible on atomic operations by definition.


>
> Thank u
> Jie He
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, March 10, 2020 6:51 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; 
> nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> ,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Dmitry
> >
> > From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> > simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
> >
> > I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> > Finally, if two memory accesses are both read or both atomic, it's not a race, right?
> >
> > Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> > Thanks
> >
> > Jie He
>
>
> Hi Jie
>
> Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
> A non-deterministic behavior, or race condition, is not a bug per se.
> It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.
>
> As you noted 2 atomic operations never constitute a race and tsan respects that.
> But it does use memory order for memory model modelling as well, e.g.
> load with memory_order_acquire constraint will acquire visibility:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
> Store may release:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
> and RMWs can do both based on the mo:
> https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L
> 294
>
>
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Wednesday, February 26, 2020 7:39 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; 
> > nd <nd at arm.com>; thread-sanitizer 
> > <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
> >
> > Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
> >
> > Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
> >
> > Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
> >
> > FTR the other sources are here:
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime
> > /m
> > utex.cpp
> > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/
> > ar
> > ena.cpp
> >
> >
> > On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi Arthur
> > > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> > >
> > > The thread looks like a normal thread created by pthread_create 
> > > with a thread func thread_native_entry and  a pointer parameter 
> > > ?thread?,
> > >
> > > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> > >
> > > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Jie He
> > >
> > > From: Arthur Eubanks <aeubanks at google.com>
> > > Sent: Wednesday, February 26, 2020 2:48 AM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net; 
> > > nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > >
> > >
> > > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> > >
> > >
> > >
> > > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9770)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> > >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> > >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> > >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #9 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9772, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9778, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 34
> > > 3:3 in Mutex::assert_owner(Thread*)
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: Jie He <Jie.He at arm.com>
> > > Sent: Tuesday, February 25, 2020 4:09 PM
> > > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> > >
> > > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9787)
> > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> > >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> > >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> > >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> > >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > >     #10 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x681b)
> > >
> > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #13 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9789, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T6 (tid=9795, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #12 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > 30
> > > 1:10 in Mutex::owned_by_self() const
> > >
> > >
> > > B.R
> > > Jie He
> > >
> > > -----Original Message-----
> > > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of 
> > > Jie He
> > > Sent: Tuesday, February 25, 2020 2:19 PM
> > > To: Dmitry Vyukov <dvyukov at google.com>
> > > Cc: nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > Subject: RE: build openjdk with fsanitizer=thread
> > >
> > > Hi Dmitry
> > >
> > > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> > >
> > > Currently, I'm not sure if the second thread is JIT thread.
> > > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> > >
> > > like the following tsan reports, I have changed the history_size to 4:
> > >
> > > WARNING: ThreadSanitizer: data race (pid=9726)
> > >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> > >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> > >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> > >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> > >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> > >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> > >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> > >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> > >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> > >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> > >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> > >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> > >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> > >     #12 JavaMain
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:560:5 (libjli.so+0x67e9)
> > >
> > >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> > >     [failed to restore the stack]
> > >
> > >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> > >     #0 malloc <null> (java+0x421ee7)
> > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> > >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> > >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> > >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> > >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> > >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> > >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #11 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > >   Thread T1 (tid=9728, running) created by main thread at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 CallJavaMainInNewThread
> > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > av
> > > a_
> > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > >
> > >   Thread T14 (tid=9742, running) created by thread T1 at:
> > >     #0 pthread_create <null> (java+0x4233d5)
> > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> > >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> > >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> > >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > >     #7 InitializeJVM
> > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > ja
> > > va
> > > .c:1538:9 (libjli.so+0x6974)
> > >
> > > SUMMARY: ThreadSanitizer: data race
> > > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > > 16 in ChunkPool::free(Chunk*)
> > >
> > >
> > > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> > >
> > >       // Return a chunk to the pool
> > >       void free(Chunk* chunk) {
> > >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> > >          ThreadCritical tc;
> > >          _num_used--;
> > >
> > >           // Add chunk to list
> > >           chunk->set_next(_first);
> > > 92:    _first = chunk;
> > > 93:    _num_chunks++;
> > >   }
> > >
> > >
> > > -----Original Message-----
> > > From: Dmitry Vyukov <dvyukov at google.com>
> > > Sent: Tuesday, February 25, 2020 1:30 PM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer 
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Hi
> > > >
> > > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > > >
> > > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > > >
> > > >
> > > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> > >
> > > +thread-sanitizer mailing list
> > >
> > > Hi Jie,
> > >
> > > C++ standard still calls this data race and renders behavior of 
> > > C++ the
> > > program as undefined. Comments don't fix bugs ;)
> > >
> > > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> > >
> > > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> > >
> > > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > > >
> > > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > > >
> > > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> > >
> > > Yes, tsan understands pthread mutex natively, no annotations required.
> > > You may try to increase history_size flag to get the second stack:
> > > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > > "failed to restore stack trace" should not lead to false positives either.
> > >
> > > Please post several full tsan reports and links to the corresponding source code.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From dvyukov at google.com  Mon Mar 16 12:19:15 2020
From: dvyukov at google.com (Dmitry Vyukov)
Date: Mon, 16 Mar 2020 13:19:15 +0100
Subject: build openjdk with fsanitizer=thread
In-Reply-To: <AM6PR08MB52191C50D5ED7EA0707995EC8EFC0@AM6PR08MB5219.eurprd08.prod.outlook.com>
References: <AM6PR08MB52194FC271149CAD43C6AACC8EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bMwWNOaK+L1DJ2OK8C6yzwK+YW6RyVKyvzFQpX_zDEmw@mail.gmail.com>
 <AM6PR08MB52195A196176F56B1BE15E448EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219F08F18D829771B181D708EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <AM6PR08MB5219A47E24D444002805A8038EED0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sr_eofNBCSB2qO_oTKn8HKGci90tZtZFKm8chx2MFUS5g@mail.gmail.com>
 <AM6PR08MB5219E2DC15754E9321E3A1A78EEA0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+Zfa0hVWJ0YrQysNvHt=MbK-d3jFxDQ4pASb98VYAxuDA@mail.gmail.com>
 <AM6PR08MB5219C70B9E48A24F627584418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bY41h+b1juK8Bd3r9E5Rf2=tvBiQzGZG2j4d1NNhV7Gw@mail.gmail.com>
 <AM6PR08MB52194ED9096217F618CE2A418EFF0@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CACT4Y+bW1gMYhv8918Bi0bV+pCgjHGFAzbLKbpMV9c-fjL31qQ@mail.gmail.com>
 <AM6PR08MB52191C50D5ED7EA0707995EC8EFC0@AM6PR08MB5219.eurprd08.prod.outlook.com>
Message-ID: <CACT4Y+bhAZrD0X_g4VLVOF388Swb1f+82jaSpOy_TsicmVcDug@mail.gmail.com>

On Wed, Mar 11, 2020 at 4:44 AM Jie He <Jie.He at arm.com> wrote:
>
> Sorry,
> Simply, the question is, is there a kind of data race caused by wrong memory order in Atomic function, e.g. if using memory order other than RELAXED, the data race will disappear.

Here is one example:
https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/tsan/atomic_store.cpp
If you replace RELAXED with ACQUIRE I believe that will fix the race.

You may find more tests with atomics in that dir:
https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan



> Initially, I think the idea in TSAN to detect data race is based on detecting certain non-deterministic memory access order.
>  In a sense, it's sort of similar with the issue caused by weak memory order. Of course, there are more non-deterministic memory visibility order in latter, and most of them are benign, harmless.
> Perhaps, a na?ve idea is I could use the idea in TSAN to detect the non-deterministic behavior even though more false-positives exist.
> e.g. TSAN could help to find a pair of atomic/non-atomic memory accesses without happens-before existed between them if we ignore the isAtomic flag in shadow. It might be a potential issue, or most likely it's not.
> But seems it's impossible to distinguish the false positives and real problem, so maybe I have a misunderstanding about happens-before algorithm.

Yes, we could detect non-deterministic behavior on atomic variables by
ignoring isAtomic flag. However, it's the main and the only
determination of atomic operations to participate in non-deterministic
algorithms (any that's deterministic can be replaced with a non-atomic
operation). You mentioned Dekker, all operations there are
non-deterministic. Any mutex - the same. Refcounts - the same.
So if you want to find non-deterministic behavior on atomic operations
-- just grep source code for atomic, really no need for dynamic
analysis :)



> Thanks
> Jie He
>
>
> -----Original Message-----
> From: Dmitry Vyukov <dvyukov at google.com>
> Sent: Tuesday, March 10, 2020 11:24 PM
> To: Jie He <Jie.He at arm.com>
> Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> Subject: Re: build openjdk with fsanitizer=thread
>
> On Tue, Mar 10, 2020 at 12:11 PM Jie He <Jie.He at arm.com> wrote:
> >
> > Hi Dmitry
> >
> > Yes, I noticed the highlight line before.
> > As I understand, tsan is able to detect a kind of lock model, which uses atomic var as a flag to protect the data, so that the memory order in atomic function is useful in such cases.
> > But I can't find a case to describe the model, Could you give me a
> > case that tsan uses the atomic functions with relaxed order to discover the race? and it's better that if with acquire/release order, not a race.
>
> I don't understand the questions. Maybe you could rephrase.
> A data race is not possible on atomic operations by definition.
>
>
> >
> > Thank u
> > Jie He
> >
> > -----Original Message-----
> > From: Dmitry Vyukov <dvyukov at google.com>
> > Sent: Tuesday, March 10, 2020 6:51 PM
> > To: Jie He <Jie.He at arm.com>
> > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net;
> > nd <nd at arm.com>; thread-sanitizer <thread-sanitizer at googlegroups.com>
> > Subject: Re: build openjdk with fsanitizer=thread
> >
> > ,On Tue, Mar 10, 2020 at 11:21 AM Jie He <Jie.He at arm.com> wrote:
> > >
> > > Hi Dmitry
> > >
> > > From the code, I think TSan is unable to detect the race condition caused by nondeterministic behavior, e.g. weak memory order.
> > > simply, a Dekker's test, all atomic load/store with relaxed memory order, it will cause nondeterministic behavior, but tsan doesn't consider it as a race.
> > >
> > > I check the code, tsan will intercept all atomic functions, and save the flag isAtomic into shadow memory.
> > > Finally, if two memory accesses are both read or both atomic, it's not a race, right?
> > >
> > > Seems the parameter "mo" in _tsan_atomic_xx functions just pass to compiler atomic builtin, not useful to determine the memory access visibility in tsan.
> > > Thanks
> > >
> > > Jie He
> >
> >
> > Hi Jie
> >
> > Tsan detects data races, as defined by C/C++ language. A data race is always a bug as it renders behavior of the program as undefined.
> > A non-deterministic behavior, or race condition, is not a bug per se.
> > It's just multiple allowed executions. Tsan does not detect these, and in most cases it's not useful. I am pretty sure JVM is full of non-deterministic behavior all over the place.
> >
> > As you noted 2 atomic operations never constitute a race and tsan respects that.
> > But it does use memory order for memory model modelling as well, e.g.
> > load with memory_order_acquire constraint will acquire visibility:
> > https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> > 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L235
> > Store may release:
> > https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> > 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L275
> > and RMWs can do both based on the mo:
> > https://github.com/llvm/llvm-project/blob/20a3d64c8883c8be550f0759525b
> > 1550b7c2d35f/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp#L289-L
> > 294
> >
> >
> > >
> > > -----Original Message-----
> > > From: Dmitry Vyukov <dvyukov at google.com>
> > > Sent: Wednesday, February 26, 2020 7:39 PM
> > > To: Jie He <Jie.He at arm.com>
> > > Cc: Arthur Eubanks <aeubanks at google.com>; tsan-dev at openjdk.java.net;
> > > nd <nd at arm.com>; thread-sanitizer
> > > <thread-sanitizer at googlegroups.com>
> > > Subject: Re: build openjdk with fsanitizer=thread
> > >
> > > Using history_size=7 may help with [failed to restore the stack] thing. It's unrelated to the way threads are created nor llvm-symbolizer. It's just memorizing all info about all previous memory accesses that ever happened in the program with full stacks is generally impossible, there are implementation limits.
> > >
> > > Tasn understand lock-free synchronization and atomic operations. That should not cause false positives and is in scope of tsan checking. For example, it is effective at detecting missed memory barriers (even if checking is done on strong x86).
> > >
> > > Re the race in Mutex/_owner. Looks like a real bug, owned_by_self does an unsynchronized read concurrently with writes by other threads (if the mutex is not owned by the current thread). C++ declares this as undefined behavior.
> > >
> > > Re ChunkPool::free race. Hard to say without the second stack. Maybe tsan does not understand that ThreadCritical thing. What is it? I can't find the impl.
> > >
> > > FTR the other sources are here:
> > > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime
> > > /m
> > > utex.cpp
> > > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/
> > > ar
> > > ena.cpp
> > >
> > >
> > > On Wed, Feb 26, 2020 at 5:05 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Hi Arthur
> > > > Yes, I agree with you, openjdk with tsan is too far away. There is a lot of work to do.
> > > > But firstly I?m just wondering in the current case, why tsan couldn?t restore the stack?
> > > >
> > > > The thread looks like a normal thread created by pthread_create
> > > > with a thread func thread_native_entry and  a pointer parameter
> > > > ?thread?,
> > > >
> > > > then in function thread_native_entry, it launches the real thread body by calling thread->call_run().
> > > >
> > > > The process is clear and all in c++ env.  I?m not familiar about llvm symbolizer in tsan, but I think it should be handled like other normal thread.
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Jie He
> > > >
> > > > From: Arthur Eubanks <aeubanks at google.com>
> > > > Sent: Wednesday, February 26, 2020 2:48 AM
> > > > To: Jie He <Jie.He at arm.com>
> > > > Cc: Dmitry Vyukov <dvyukov at google.com>; tsan-dev at openjdk.java.net;
> > > > nd <nd at arm.com>; thread-sanitizer
> > > > <thread-sanitizer at googlegroups.com>
> > > > Subject: Re: build openjdk with fsanitizer=thread
> > > >
> > > >
> > > >
> > > > It's going to be a lot of work to get TSAN working with the JVM code. There is lots of synchronization done in JVM that's not visible to TSAN, often through OS-specific mechanisms. You'd have to add TSAN callbacks in the code (what we've done in various places in this project) or change Hotspot to use something that TSAN instruments (as Dmitry suggested). And again as Dmitry pointed out, lots of it is technically incorrect C++ code that happens to work on current C++ compilers (in fact we've run into actual production issues with this where we have a custom memcpy that doesn't always do word-atomic moves even when copying an entire word).
> > > >
> > > >
> > > >
> > > > IMO it's definitely possible to get Hotspot into a state that makes TSAN mostly happy (maybe some branching code paths that only execute when TSAN is turned on), but it'd likely require moving to at least C++11 for portability reasons, then a loooot of investigation into Hotspot's synchronization and TSAN's callbacks. You might be interested in the tsanExternalDecls.hpp file and the corresponding code in LLVM.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Feb 25, 2020 at 12:16 AM Jie He <Jie.He at arm.com> wrote:
> > > >
> > > > Add more report for the previous case, false race between javamain thread and g1 sampling thread.
> > > >
> > > > WARNING: ThreadSanitizer: data race (pid=9770)
> > > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > > >     #0 Mutex::assert_owner(Thread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:343:3 (libjvm.so+0x1924838)
> > > >     #1 Monitor::notify() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:166:3 (libjvm.so+0x1924c1f)
> > > >     #2 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:128:12 (libjvm.so+0x10c50d7)
> > > >     #3 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > > >     #4 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > > >     #5 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > > >     #6 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > > >     #7 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > > >     #8 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > > >     #9 JavaMain
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:560:5 (libjli.so+0x681b)
> > > >
> > > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > > >     [failed to restore the stack]
> > > >
> > > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > > >     #0 malloc <null> (java+0x421ee7)
> > > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #13 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > >   Thread T1 (tid=9772, running) created by main thread at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 CallJavaMainInNewThread
> > > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > > av
> > > > a_
> > > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > > >
> > > >   Thread T6 (tid=9778, running) created by thread T1 at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #12 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > > SUMMARY: ThreadSanitizer: data race
> > > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > > 34
> > > > 3:3 in Mutex::assert_owner(Thread*)
> > > >
> > > >
> > > > B.R
> > > > Jie He
> > > >
> > > > -----Original Message-----
> > > > From: Jie He <Jie.He at arm.com>
> > > > Sent: Tuesday, February 25, 2020 4:09 PM
> > > > To: Jie He <Jie.He at arm.com>; Dmitry Vyukov <dvyukov at google.com>
> > > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > > Subject: RE: build openjdk with fsanitizer=thread
> > > >
> > > > Hi Dmitry
> > > >
> > > > another case that data race exists between javamain thread and G1YoungRemSetSampling Thread, I believe both of them are C++ function threads. But tsan doesn't restore G1 thread stack successfully, and seems to consider there is no lock to protect Mutex's member var _owner.
> > > >
> > > > See the following reports by tsan, code could be found in github https://github.com/openjdk/jdk/tree/master/src/hotspot/share:
> > > >
> > > > WARNING: ThreadSanitizer: data race (pid=9787)
> > > >   Read of size 8 at 0x7b7c00002360 by thread T1:
> > > >     #0 Mutex::owned_by_self() const /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:301:10 (libjvm.so+0x1925966)
> > > >     #1 assert_lock_strong(Mutex const*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.cpp:187:13 (libjvm.so+0x19272fa)
> > > >     #2 MutexLocker::~MutexLocker() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutexLocker.hpp:237:7 (libjvm.so+0x33da8a)
> > > >     #3 G1YoungRemSetSamplingThread::stop_service() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:129:1 (libjvm.so+0x10c50e0)
> > > >     #4 ConcurrentGCThread::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:65:3 (libjvm.so+0xc9fa11)
> > > >     #5 G1CollectedHeap::stop() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1867:31 (libjvm.so+0xfae058)
> > > >     #6 before_exit(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/java.cpp:461:21 (libjvm.so+0x1215b84)
> > > >     #7 Threads::destroy_vm() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4418:3 (libjvm.so+0x1e326ee)
> > > >     #8 jni_DestroyJavaVM_inner(JavaVM_*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3989:7 (libjvm.so+0x137a5eb)
> > > >     #9 jni_DestroyJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4007:14 (libjvm.so+0x137a3ef)
> > > >     #10 JavaMain
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:560:5 (libjli.so+0x681b)
> > > >
> > > >   Previous write of size 8 at 0x7b7c00002360 by thread T6:
> > > >     [failed to restore the stack]
> > > >
> > > >   Location is heap block of size 3168 at 0x7b7c00001c00 allocated by thread T1:
> > > >     #0 malloc <null> (java+0x421ee7)
> > > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > >     #3 Thread::allocate(unsigned long, bool, MemoryType) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:178:43 (libjvm.so+0x1e20a54)
> > > >     #4 Thread::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.hpp:211:52 (libjvm.so+0x86cf52)
> > > >     #5 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:32 (libjvm.so+0xfaca14)
> > > >     #6 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > >     #7 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > >     #8 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > >     #9 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > >     #10 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > >     #11 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #12 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #13 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > >   Thread T1 (tid=9789, running) created by main thread at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 CallJavaMainInNewThread
> > > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > > av
> > > > a_
> > > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > > >
> > > >   Thread T6 (tid=9795, running) created by thread T1 at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > >     #2 ConcurrentGCThread::create_and_start(ThreadPriority) /home/wave/workspace/jdk_master/src/hotspot/share/gc/shared/concurrentGCThread.cpp:37:7 (libjvm.so+0xc9f7d4)
> > > >     #3 G1YoungRemSetSamplingThread::G1YoungRemSetSamplingThread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1YoungRemSetSamplingThread.cpp:47:3 (libjvm.so+0x10c49a5)
> > > >     #4 G1CollectedHeap::initialize_young_gen_sampling_thread() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1642:36 (libjvm.so+0xfaca3a)
> > > >     #5 G1CollectedHeap::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/gc/g1/g1CollectedHeap.cpp:1823:11 (libjvm.so+0xfadd31)
> > > >     #6 Universe::initialize_heap() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:719:33 (libjvm.so+0x1e88c55)
> > > >     #7 universe_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/universe.cpp:653:17 (libjvm.so+0x1e8872b)
> > > >     #8 init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:117:17 (libjvm.so+0x11c9bc1)
> > > >     #9 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3882:17 (libjvm.so+0x1e312a1)
> > > >     #10 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #11 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #12 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > > SUMMARY: ThreadSanitizer: data race
> > > > /home/wave/workspace/jdk_master/src/hotspot/share/runtime/mutex.cpp:
> > > > 30
> > > > 1:10 in Mutex::owned_by_self() const
> > > >
> > > >
> > > > B.R
> > > > Jie He
> > > >
> > > > -----Original Message-----
> > > > From: tsan-dev <tsan-dev-bounces at openjdk.java.net> On Behalf Of
> > > > Jie He
> > > > Sent: Tuesday, February 25, 2020 2:19 PM
> > > > To: Dmitry Vyukov <dvyukov at google.com>
> > > > Cc: nd <nd at arm.com>; thread-sanitizer
> > > > <thread-sanitizer at googlegroups.com>; tsan-dev at openjdk.java.net
> > > > Subject: RE: build openjdk with fsanitizer=thread
> > > >
> > > > Hi Dmitry
> > > >
> > > > Yes, so I don't think the first 2 classes of warnings are data race, they are out of scope of tsan.
> > > >
> > > > Currently, I'm not sure if the second thread is JIT thread.
> > > > But seems the second thread knows a tsan_read8 behavior happened at least in IR level.
> > > >
> > > > like the following tsan reports, I have changed the history_size to 4:
> > > >
> > > > WARNING: ThreadSanitizer: data race (pid=9726)
> > > >   Write of size 8 at 0x7b1800003ab0 by thread T1:
> > > >     #0 ChunkPool::free(Chunk*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:16 (libjvm.so+0x7e6fe0)
> > > >     #1 Chunk::operator delete(void*) /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:207:54 (libjvm.so+0x7e47f4)
> > > >     #2 Chunk::chop() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:225:5 (libjvm.so+0x7e4994)
> > > >     #3 Arena::destruct_contents() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:319:11 (libjvm.so+0x7e5274)
> > > >     #4 Arena::~Arena() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:283:3 (libjvm.so+0x7e52ea)
> > > >     #5 ResourceArea::~ResourceArea() /home/wave/workspace/jdk_master/src/hotspot/share/memory/resourceArea.hpp:44:7 (libjvm.so+0xae9d18)
> > > >     #6 Thread::~Thread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:449:3 (libjvm.so+0x1e2214a)
> > > >     #7 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1903:1 (libjvm.so+0x1e27af4)
> > > >     #8 JavaThread::~JavaThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1856:27 (libjvm.so+0x1e27b3c)
> > > >     #9 ThreadsSMRSupport::smr_delete(JavaThread*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/threadSMR.cpp:1027:3 (libjvm.so+0x1e47408)
> > > >     #10 JavaThread::smr_delete() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:208:5 (libjvm.so+0x1e20e73)
> > > >     #11 jni_DetachCurrentThread /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:4173:11 (libjvm.so+0x137ae7a)
> > > >     #12 JavaMain
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:560:5 (libjli.so+0x67e9)
> > > >
> > > >   Previous read of size 8 at 0x7b1800003ab0 by thread T14:
> > > >     [failed to restore the stack]
> > > >
> > > >   Location is heap block of size 88 at 0x7b1800003a80 allocated by thread T1:
> > > >     #0 malloc <null> (java+0x421ee7)
> > > >     #1 os::malloc(unsigned long, MemoryType, NativeCallStack const&) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/os.cpp:714:18 (libjvm.so+0x19d7e62)
> > > >     #2 AllocateHeap(unsigned long, MemoryType, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:42:21 (libjvm.so+0x7bfc2d)
> > > >     #3 AllocateHeap(unsigned long, MemoryType, AllocFailStrategy::AllocFailEnum) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.cpp:52:10 (libjvm.so+0x7bfd34)
> > > >     #4 CHeapObj<(MemoryType)8>::operator new(unsigned long) /home/wave/workspace/jdk_master/src/hotspot/share/memory/allocation.hpp:193:19 (libjvm.so+0x7e6799)
> > > >     #5 ChunkPool::initialize() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:135 (libjvm.so+0x7e6799)
> > > >     #6 chunkpool_init() /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:154:3 (libjvm.so+0x7e4441)
> > > >     #7 vm_init_globals() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/init.cpp:102:3 (libjvm.so+0x11c9b6a)
> > > >     #8 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:3846:3 (libjvm.so+0x1e3108c)
> > > >     #9 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #10 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #11 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > >   Thread T1 (tid=9728, running) created by main thread at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 CallJavaMainInNewThread
> > > > /home/wave/workspace/jdk_master/src/java.base/unix/native/libjli/j
> > > > av
> > > > a_
> > > > md_solinux.c:754:9 (libjli.so+0xb53f)
> > > >
> > > >   Thread T14 (tid=9742, running) created by thread T1 at:
> > > >     #0 pthread_create <null> (java+0x4233d5)
> > > >     #1 os::create_thread(Thread*, os::ThreadType, unsigned long) /home/wave/workspace/jdk_master/src/hotspot/os/linux/os_linux.cpp:926:15 (libjvm.so+0x19e4413)
> > > >     #2 WatcherThread::WatcherThread() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1375:7 (libjvm.so+0x1e25399)
> > > >     #3 WatcherThread::start() /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:1514:9 (libjvm.so+0x1e2598f)
> > > >     #4 Threads::create_vm(JavaVMInitArgs*, bool*) /home/wave/workspace/jdk_master/src/hotspot/share/runtime/thread.cpp:4105:7 (libjvm.so+0x1e31bb1)
> > > >     #5 JNI_CreateJavaVM_inner(JavaVM_**, void**, void*) /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3852:12 (libjvm.so+0x1379e74)
> > > >     #6 JNI_CreateJavaVM /home/wave/workspace/jdk_master/src/hotspot/share/prims/jni.cpp:3935:14 (libjvm.so+0x1379d0f)
> > > >     #7 InitializeJVM
> > > > /home/wave/workspace/jdk_master/src/java.base/share/native/libjli/
> > > > ja
> > > > va
> > > > .c:1538:9 (libjli.so+0x6974)
> > > >
> > > > SUMMARY: ThreadSanitizer: data race
> > > > /home/wave/workspace/jdk_master/src/hotspot/share/memory/arena.cpp:93:
> > > > 16 in ChunkPool::free(Chunk*)
> > > >
> > > >
> > > > And the openjdk code is below, you could see there is a ThreadCritical which derives from pthread_mutex and implements lock/unlock in the ctor/dtor:
> > > >
> > > >       // Return a chunk to the pool
> > > >       void free(Chunk* chunk) {
> > > >          assert(chunk->length() + Chunk::aligned_overhead_size() == _size, "bad size");
> > > >          ThreadCritical tc;
> > > >          _num_used--;
> > > >
> > > >           // Add chunk to list
> > > >           chunk->set_next(_first);
> > > > 92:    _first = chunk;
> > > > 93:    _num_chunks++;
> > > >   }
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Dmitry Vyukov <dvyukov at google.com>
> > > > Sent: Tuesday, February 25, 2020 1:30 PM
> > > > To: Jie He <Jie.He at arm.com>
> > > > Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>; thread-sanitizer
> > > > <thread-sanitizer at googlegroups.com>
> > > > Subject: Re: build openjdk with fsanitizer=thread
> > > >
> > > > On Tue, Feb 25, 2020 at 6:20 AM Jie He <Jie.He at arm.com> wrote:
> > > > >
> > > > > Hi
> > > > >
> > > > > I built openjdk with enabling fanitizer=thread recently, and got a lot of warning by tsan even a helloworld case.
> > > > >
> > > > > Then I investigated around more than 15 warnings, found they could be divided into 3 classes:
> > > > >
> > > > >
> > > > >   1.  Benign races, commonly, there is a comment to indicate why it is safe in MP.
> > > >
> > > > +thread-sanitizer mailing list
> > > >
> > > > Hi Jie,
> > > >
> > > > C++ standard still calls this data race and renders behavior of
> > > > C++ the
> > > > program as undefined. Comments don't fix bugs ;)
> > > >
> > > > >   2.  Runtime atomic implementation, in x86, the atomic load and store will be translated to platformload/store.
> > > >
> > > > I assume here platformload/store are implemented as plain loads and stores. These may need to be changed at least in tsan build (but maybe in all builds, because see the previous point). I am not aware of the openjdk portability requirements, but today the __atomic_load/store_n intrinsics may be a good choice.
> > > >
> > > > >   3.  Runtime function implement protected by MutexLocker/ThreadCritical, which finally implemented by pthread_mutex.
> > > > >
> > > > > For 3, I couldn't understand why tsan couldn't recognize that it's safe and protected by a lock. In TSAN document, it said pthread functions are supported.
> > > > > So I tried to add annotation(ANNOTATION_RWCLOCK_ACQUIRED/RELEASED) to mark, then got the warning, double of lock, it seems tsan knows MutexLocker is a lock.
> > > > >
> > > > > Or because one of the conflicting threads lost its stack, in this kind of warning, there is one out of the two threads fails to restore its stack.
> > > > > It may result that tsan only knows the thread calls read/write, but doesn't know the memory operation is protected by a lock.
> > > > > These threads couldn't restore the stack are JIT threads/Java threads? I need to fix the tsan symbolizer function first for this situation?
> > > >
> > > > Yes, tsan understands pthread mutex natively, no annotations required.
> > > > You may try to increase history_size flag to get the second stack:
> > > > https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
> > > > "failed to restore stack trace" should not lead to false positives either.
> > > >
> > > > Please post several full tsan reports and links to the corresponding source code.
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe at googlegroups.com.
> > > To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/AM6PR08MB5219C70B9E48A24F627584418EFF0%40AM6PR08MB5219.eurprd08.prod.outlook.com.

From Jie.He at arm.com  Tue Mar 24 11:13:32 2020
From: Jie.He at arm.com (Jie He)
Date: Tue, 24 Mar 2020 11:13:32 +0000
Subject: some questions about project tsan
In-Reply-To: <CAPW48sp2GHNxVFe1H6Hw2v0PLvM=1k3uxQAmOyvFNfpRZ03WHw@mail.gmail.com>
References: <AM6PR08MB5219A4242F9234F49350B9C28E390@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sp2GHNxVFe1H6Hw2v0PLvM=1k3uxQAmOyvFNfpRZ03WHw@mail.gmail.com>
Message-ID: <HE1PR0801MB16436B81220AF535354AAAD18EF10@HE1PR0801MB1643.eurprd08.prod.outlook.com>

Hi Arthur

We plan to add the AARCH64 support in jdk/tsan, probably kick off next month.

Could we commit the patches progressively? each of them will pass parts of the tsan test cases, so that, it?s easier to review and bug tracking than a whole big patch.

B.R
Jie He

From: Arthur Eubanks <aeubanks at google.com>
Sent: Friday, January 10, 2020 1:21 AM
To: Jie He <Jie.He at arm.com>
Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>
Subject: Re: some questions about project tsan



On Thu, Jan 9, 2020 at 1:02 AM Jie He <Jie.He at arm.com<mailto:Jie.He at arm.com>> wrote:
Hi

I'm an engineer from ARM, and noticed this project is helpful to detect data race in Java level.
Thanks for the interest!

I have some questions about this project:

1, how about the current status? All dev jobs have been finished already?
It works and we use this internally at Google. There should be no false positives reported, but not everything is precise and there are false negatives (e.g. final fields, finalizers, etc.). I believe the current implementation catches most races, which is good enough, and extra effort spent making instrumentation more precise would only find a couple more races, which may or may not be worth the effort.
Ideally we'd like to upstream this into mainline.
2, for detection of data race in Java code, I think current implementation only in X86 is enough. A data race in java code will be found whatever the machine platform is.
    So that, porting this project to AARCH64 is meaningless? Is it helpful to find some issues which don't happen in X86?
Yes a data race will likely be found on x86 and it would apply to AARCH64 as well. Porting to AARCH64 could still be interesting and useful for people who only have AARCH64 machines. And maybe running on different architectures will change the timing of concurrent execution, causing races to disappear or appear.
3, Is there a plan to add the support to detect data race in JVM self? E.g. add the instrumentation into the assemble code generated by interpreter, compiler and runtime,
    In some machine platforms which have weak memory model like arm and ppc, engineers often make mistakes when writing data race related code even though they are experienced.
    I think such a tool could help to detect these bugs easily than before.
We've thought about this in the past, but there is a ton of intentionally racy code. You could try adding -fsanitize=thread to the hotspot compilations and running it just to see what would happen.

Thanks
Jie He

From aeubanks at google.com  Tue Mar 24 15:59:21 2020
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 24 Mar 2020 08:59:21 -0700
Subject: some questions about project tsan
In-Reply-To: <HE1PR0801MB16436B81220AF535354AAAD18EF10@HE1PR0801MB1643.eurprd08.prod.outlook.com>
References: <AM6PR08MB5219A4242F9234F49350B9C28E390@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sp2GHNxVFe1H6Hw2v0PLvM=1k3uxQAmOyvFNfpRZ03WHw@mail.gmail.com>
 <HE1PR0801MB16436B81220AF535354AAAD18EF10@HE1PR0801MB1643.eurprd08.prod.outlook.com>
Message-ID: <CAPW48spkmr5FQJVR_BeknPsym1Yju_i--6Cr3d55XTD2bpVw6A@mail.gmail.com>

On Tue, Mar 24, 2020 at 4:13 AM Jie He <Jie.He at arm.com> wrote:

> Hi Arthur
>
>
>
> We plan to add the AARCH64 support in jdk/tsan, probably kick off next
> month.
>
Awesome! Looking forward to it.

>
>
> Could we commit the patches progressively? each of them will pass parts of
> the tsan test cases, so that, it?s easier to review and bug tracking than a
> whole big patch.
>
Yes, incremental patches are greatly preferred.

Maybe you want to first rebase/merge with jdk/jdk? I'm not sure what the
right approach is, I'd prefer rebasing, but maybe merging is the typical
thing to do?
Also, we just got notified that the Project Skara people want to move
jdk/tsan to GitHub so we'll probably be going down that path soon, and code
reviews will be done via GitHub pull requests; keep that in mind.

>
>
> B.R
>
> Jie He
>
>
>
> *From:* Arthur Eubanks <aeubanks at google.com>
> *Sent:* Friday, January 10, 2020 1:21 AM
> *To:* Jie He <Jie.He at arm.com>
> *Cc:* tsan-dev at openjdk.java.net; nd <nd at arm.com>
> *Subject:* Re: some questions about project tsan
>
>
>
>
>
>
>
> On Thu, Jan 9, 2020 at 1:02 AM Jie He <Jie.He at arm.com> wrote:
>
> Hi
>
> I'm an engineer from ARM, and noticed this project is helpful to detect
> data race in Java level.
>
> Thanks for the interest!
>
>
> I have some questions about this project:
>
> 1, how about the current status? All dev jobs have been finished already?
>
> It works and we use this internally at Google. There should be no false
> positives reported, but not everything is precise and there are false
> negatives (e.g. final fields, finalizers, etc.). I believe the current
> implementation catches most races, which is good enough, and extra effort
> spent making instrumentation more precise would only find a couple more
> races, which may or may not be worth the effort.
>
> Ideally we'd like to upstream this into mainline.
>
> 2, for detection of data race in Java code, I think current implementation
> only in X86 is enough. A data race in java code will be found whatever the
> machine platform is.
>     So that, porting this project to AARCH64 is meaningless? Is it helpful
> to find some issues which don't happen in X86?
>
> Yes a data race will likely be found on x86 and it would apply to AARCH64
> as well. Porting to AARCH64 could still be interesting and useful for
> people who only have AARCH64 machines. And maybe running on different
> architectures will change the timing of concurrent execution, causing races
> to disappear or appear.
>
> 3, Is there a plan to add the support to detect data race in JVM self?
> E.g. add the instrumentation into the assemble code generated by
> interpreter, compiler and runtime,
>     In some machine platforms which have weak memory model like arm and
> ppc, engineers often make mistakes when writing data race related code even
> though they are experienced.
>     I think such a tool could help to detect these bugs easily than before.
>
> We've thought about this in the past, but there is a ton of intentionally
> racy code. You could try adding -fsanitize=thread to the hotspot
> compilations and running it just to see what would happen.
>
>
> Thanks
> Jie He
>
>

From Jie.He at arm.com  Wed Mar 25 09:15:31 2020
From: Jie.He at arm.com (Jie He)
Date: Wed, 25 Mar 2020 09:15:31 +0000
Subject: some questions about project tsan
In-Reply-To: <CAPW48spkmr5FQJVR_BeknPsym1Yju_i--6Cr3d55XTD2bpVw6A@mail.gmail.com>
References: <AM6PR08MB5219A4242F9234F49350B9C28E390@AM6PR08MB5219.eurprd08.prod.outlook.com>
 <CAPW48sp2GHNxVFe1H6Hw2v0PLvM=1k3uxQAmOyvFNfpRZ03WHw@mail.gmail.com>
 <HE1PR0801MB16436B81220AF535354AAAD18EF10@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48spkmr5FQJVR_BeknPsym1Yju_i--6Cr3d55XTD2bpVw6A@mail.gmail.com>
Message-ID: <HE1PR0801MB1643D7CDFC96397A974B60398ECE0@HE1PR0801MB1643.eurprd08.prod.outlook.com>

> Yes, incremental patches are greatly preferred.

> Maybe you want to first rebase/merge with jdk/jdk? I'm not sure what the right approach is, I'd prefer rebasing, but maybe merging is the typical thing to do?
[Jie He] I?m not sure how much risk is here. seems no sync with jdk/jdk already 1 year. How big will the changes in master impact the current tsan? if current implementation doesn?t work after rebased, we have to wait until they are fixed. Or, I still do the AARCH64 implementation based on current tsan, in the same time, have a try to rebase with jdk/jdk as well. If latter succeeds smoothly, everything is ok; otherwise, I go back to commit to origin jdk/tsan, then take the rebase when finish AARCH64 implementation even though more merging workload. What do you think?

> Also, we just got notified that the Project Skara people want to move jdk/tsan to GitHub so we'll probably be going down that path soon, and code reviews will be done via GitHub pull requests; keep that in mind.
Got it.

From: Arthur Eubanks <aeubanks at google.com>
Sent: Tuesday, March 24, 2020 11:59 PM
To: Jie He <Jie.He at arm.com>
Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>
Subject: Re: some questions about project tsan



On Tue, Mar 24, 2020 at 4:13 AM Jie He <Jie.He at arm.com<mailto:Jie.He at arm.com>> wrote:
Hi Arthur

We plan to add the AARCH64 support in jdk/tsan, probably kick off next month.
Awesome! Looking forward to it.

Could we commit the patches progressively? each of them will pass parts of the tsan test cases, so that, it?s easier to review and bug tracking than a whole big patch.
Yes, incremental patches are greatly preferred.

Maybe you want to first rebase/merge with jdk/jdk? I'm not sure what the right approach is, I'd prefer rebasing, but maybe merging is the typical thing to do?
Also, we just got notified that the Project Skara people want to move jdk/tsan to GitHub so we'll probably be going down that path soon, and code reviews will be done via GitHub pull requests; keep that in mind.

B.R
Jie He

From: Arthur Eubanks <aeubanks at google.com<mailto:aeubanks at google.com>>
Sent: Friday, January 10, 2020 1:21 AM
To: Jie He <Jie.He at arm.com<mailto:Jie.He at arm.com>>
Cc: tsan-dev at openjdk.java.net<mailto:tsan-dev at openjdk.java.net>; nd <nd at arm.com<mailto:nd at arm.com>>
Subject: Re: some questions about project tsan



On Thu, Jan 9, 2020 at 1:02 AM Jie He <Jie.He at arm.com<mailto:Jie.He at arm.com>> wrote:
Hi

I'm an engineer from ARM, and noticed this project is helpful to detect data race in Java level.
Thanks for the interest!

I have some questions about this project:

1, how about the current status? All dev jobs have been finished already?
It works and we use this internally at Google. There should be no false positives reported, but not everything is precise and there are false negatives (e.g. final fields, finalizers, etc.). I believe the current implementation catches most races, which is good enough, and extra effort spent making instrumentation more precise would only find a couple more races, which may or may not be worth the effort.
Ideally we'd like to upstream this into mainline.
2, for detection of data race in Java code, I think current implementation only in X86 is enough. A data race in java code will be found whatever the machine platform is.
    So that, porting this project to AARCH64 is meaningless? Is it helpful to find some issues which don't happen in X86?
Yes a data race will likely be found on x86 and it would apply to AARCH64 as well. Porting to AARCH64 could still be interesting and useful for people who only have AARCH64 machines. And maybe running on different architectures will change the timing of concurrent execution, causing races to disappear or appear.
3, Is there a plan to add the support to detect data race in JVM self? E.g. add the instrumentation into the assemble code generated by interpreter, compiler and runtime,
    In some machine platforms which have weak memory model like arm and ppc, engineers often make mistakes when writing data race related code even though they are experienced.
    I think such a tool could help to detect these bugs easily than before.
We've thought about this in the past, but there is a ton of intentionally racy code. You could try adding -fsanitize=thread to the hotspot compilations and running it just to see what would happen.

Thanks
Jie He

From Jie.He at arm.com  Fri Mar 27 08:28:38 2020
From: Jie.He at arm.com (Jie He)
Date: Fri, 27 Mar 2020 08:28:38 +0000
Subject: Conflicts exist if move to jdk14
Message-ID: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>

Hi 

I had a quick try to rebase jdk/tsan to jdk14-ga, and found 2 major conflicts:
1, Shenandoah gc had a refactor last year, which implements a concurrent evacuation, seems it impacts current tsan oop map.
2, they limited non-gc to access the reserved region in gc, adding the function back to collectedHeap is a workround.

In addition, a known build error exists in AARCH64 when building openjdk release version with clang 8. changing the -O3 to -Os or -O0 will be ok.

An initial test shows 77 passed of total 79 tsan cases. The failed cases are NonRacyFinalizerLoopTest.java and tsan/NonRacyGarbageCollectionLoopTest.java.

Thanks
Jie He

From jcbeyler at google.com  Fri Mar 27 20:54:16 2020
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 27 Mar 2020 13:54:16 -0700
Subject: Tsan project transition towards Git, GitHub, and Skara
Message-ID: <CAF9BGBwdyyGXBjdP35rfVKmBP8e3sEwDa16JDy4rX4dCmsCerg@mail.gmail.com>

Hi all,

You might have heard that there is an OpenJDK project named Skara
evaluating and trialing using Git for OpenJDK and GitHub for hosting the
repositories [1]. We have been asked to try out Git, GitHub and the
Skara tooling, as a means of validating, and to provide feedback about
the Project Skara effort. There are already a number of OpenJDK projects
using Git, GitHub and Skara:
- Loom
- Panama
- Valhalla
- Metropolis
- Portola
- Lanai
- OpenJFX
- OpenJMC

A transition entails switching to Git as an SCM [2], and the repository
would be hosted on GitHub [3]. After the transition, the current
Mercurial repository would no longer be updated. There is already a
read-only mirror of Tsan in place [4], you can it use to try out Git and
some of the tooling, though submitting pull requests and pushing will
not work (as it is a read-only mirror).

I suggest that project Tsan transition to Git, GitHub and Skara and that
we set the target date for the transition to April 2. I also suggest we
adopt GitHub's pull request workflow together with the bots from project
Skara.

Adopting pull requests and the Skara bots means that the review process
would work partly through GitHub. In particular, all changes must start
out as pull requests. Skara automatically generates webrevs, "RFR"
emails and supports bi-directional syncing between mailing lists and
GitHub for those that prefer using webrev and the mailing lists. If you
are unfamiliar with GitHub and pull requests, I suggest checking out
GitHub's guide [5]. Note that project Skara also provides CLI tools for
interacting with GitHub for those that prefer working from a terminal.
You can find more info about these tools on the Skara wiki page [6].

If you have any comments or questions regarding this transition, please
reply to this thread.

Thanks,
Jc

[1] : https://openjdk.java.net/jeps/369
[2] : https://git-scm.com/
[3] : https://github.com/
[4] : https://github.com/openjdk/tsan
[5] : https://guides.github.com/introduction/flow/
[6] : https://wiki.openjdk.java.net/display/SKARA/Skara

From aeubanks at google.com  Mon Mar 30 16:21:51 2020
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 30 Mar 2020 09:21:51 -0700
Subject: Conflicts exist if move to jdk14
In-Reply-To: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
References: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
Message-ID: <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>

If there are failing tests, we can deal with rebasing later. Applying
patches on top of jdk/tsan should be fine.

On Fri, Mar 27, 2020 at 1:29 AM Jie He <Jie.He at arm.com> wrote:

> Hi
>
> I had a quick try to rebase jdk/tsan to jdk14-ga, and found 2 major
> conflicts:
> 1, Shenandoah gc had a refactor last year, which implements a concurrent
> evacuation, seems it impacts current tsan oop map.
> 2, they limited non-gc to access the reserved region in gc, adding the
> function back to collectedHeap is a workround.
>
> In addition, a known build error exists in AARCH64 when building openjdk
> release version with clang 8. changing the -O3 to -Os or -O0 will be ok.
>
> An initial test shows 77 passed of total 79 tsan cases. The failed cases
> are NonRacyFinalizerLoopTest.java and
> tsan/NonRacyGarbageCollectionLoopTest.java.
>
> Thanks
> Jie He
>

From Jie.He at arm.com  Tue Mar 31 05:51:20 2020
From: Jie.He at arm.com (Jie He)
Date: Tue, 31 Mar 2020 05:51:20 +0000
Subject: Conflicts exist if move to jdk14
In-Reply-To: <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>
References: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>
Message-ID: <HE1PR0801MB1643866774E5111CE46A38428EC80@HE1PR0801MB1643.eurprd08.prod.outlook.com>

Actually, I have already workaround these conflicts,  now all 79 cases passed.

From: Arthur Eubanks <aeubanks at google.com>
Sent: Tuesday, March 31, 2020 12:22 AM
To: Jie He <Jie.He at arm.com>
Cc: tsan-dev at openjdk.java.net; nd <nd at arm.com>
Subject: Re: Conflicts exist if move to jdk14

If there are failing tests, we can deal with rebasing later. Applying patches on top of jdk/tsan should be fine.

On Fri, Mar 27, 2020 at 1:29 AM Jie He <Jie.He at arm.com<mailto:Jie.He at arm.com>> wrote:
Hi

I had a quick try to rebase jdk/tsan to jdk14-ga, and found 2 major conflicts:
1, Shenandoah gc had a refactor last year, which implements a concurrent evacuation, seems it impacts current tsan oop map.
2, they limited non-gc to access the reserved region in gc, adding the function back to collectedHeap is a workround.

In addition, a known build error exists in AARCH64 when building openjdk release version with clang 8. changing the -O3 to -Os or -O0 will be ok.

An initial test shows 77 passed of total 79 tsan cases. The failed cases are NonRacyFinalizerLoopTest.java and tsan/NonRacyGarbageCollectionLoopTest.java.

Thanks
Jie He

From aeubanks at google.com  Tue Mar 31 16:49:01 2020
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 31 Mar 2020 09:49:01 -0700
Subject: Conflicts exist if move to jdk14
In-Reply-To: <HE1PR0801MB1643866774E5111CE46A38428EC80@HE1PR0801MB1643.eurprd08.prod.outlook.com>
References: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>
 <HE1PR0801MB1643866774E5111CE46A38428EC80@HE1PR0801MB1643.eurprd08.prod.outlook.com>
Message-ID: <CAPW48spgEr9vJOCLqa8o0_q1fM_78EsAeJUh_q3U4p3z4E_fEA@mail.gmail.com>

Glad to hear it!
Did you do a merge or a rebase? I'm wondering which one makes more sense.
It'd be nice to keep our patches separate from upstream's development,
meaning I'd prefer a rebase, but I'm not sure if that's feasible in the
long run?

On Mon, Mar 30, 2020 at 10:51 PM Jie He <Jie.He at arm.com> wrote:

> Actually, I have already workaround these conflicts,  now all 79 cases
> passed.
>
>
>
> *From:* Arthur Eubanks <aeubanks at google.com>
> *Sent:* Tuesday, March 31, 2020 12:22 AM
> *To:* Jie He <Jie.He at arm.com>
> *Cc:* tsan-dev at openjdk.java.net; nd <nd at arm.com>
> *Subject:* Re: Conflicts exist if move to jdk14
>
>
>
> If there are failing tests, we can deal with rebasing later. Applying
> patches on top of jdk/tsan should be fine.
>
>
>
> On Fri, Mar 27, 2020 at 1:29 AM Jie He <Jie.He at arm.com> wrote:
>
> Hi
>
> I had a quick try to rebase jdk/tsan to jdk14-ga, and found 2 major
> conflicts:
> 1, Shenandoah gc had a refactor last year, which implements a concurrent
> evacuation, seems it impacts current tsan oop map.
> 2, they limited non-gc to access the reserved region in gc, adding the
> function back to collectedHeap is a workround.
>
> In addition, a known build error exists in AARCH64 when building openjdk
> release version with clang 8. changing the -O3 to -Os or -O0 will be ok.
>
> An initial test shows 77 passed of total 79 tsan cases. The failed cases
> are NonRacyFinalizerLoopTest.java and
> tsan/NonRacyGarbageCollectionLoopTest.java.
>
> Thanks
> Jie He
>
>

From manc at google.com  Tue Mar 31 17:32:43 2020
From: manc at google.com (Man Cao)
Date: Tue, 31 Mar 2020 10:32:43 -0700
Subject: Conflicts exist if move to jdk14
In-Reply-To: <CAPW48spgEr9vJOCLqa8o0_q1fM_78EsAeJUh_q3U4p3z4E_fEA@mail.gmail.com>
References: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>
 <HE1PR0801MB1643866774E5111CE46A38428EC80@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48spgEr9vJOCLqa8o0_q1fM_78EsAeJUh_q3U4p3z4E_fEA@mail.gmail.com>
Message-ID: <CA+w6HxZ-uht9XzeUC4qqsUL-FVZrqzvt-DzT+qdfEybwAsx6nQ@mail.gmail.com>

I just fast forwarded the "default" branch to JDK14-GA.
$ hg up default
$ hg pull -r jdk-14-ga http://hg.openjdk.java.net/jdk/jdk
$ hg push -b default
We are also discussing the proper way to do a rebase/merge.
I also prefer rebase to merge, but it seems OpenJDK projects are mainly
using merge (e.g. loom).

I suppose it is challenging to ask Jie to do the rebase/merge work, unless
Jie becomes a Committer to tsan project first.
I don't know how to make a webrev for the conflict-resolving changes during
the rebase/merge process.

-Man

From manc at google.com  Tue Mar 31 18:39:44 2020
From: manc at google.com (Man Cao)
Date: Tue, 31 Mar 2020 11:39:44 -0700
Subject: Conflicts exist if move to jdk14
In-Reply-To: <CA+w6HxZ-uht9XzeUC4qqsUL-FVZrqzvt-DzT+qdfEybwAsx6nQ@mail.gmail.com>
References: <HE1PR0801MB164304C9D4D22EE99CDE1A2A8ECC0@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48soODuJUSi1CJ4qW7CS=GGkJ+vF_khDcfNxzFaD5g=-w4A@mail.gmail.com>
 <HE1PR0801MB1643866774E5111CE46A38428EC80@HE1PR0801MB1643.eurprd08.prod.outlook.com>
 <CAPW48spgEr9vJOCLqa8o0_q1fM_78EsAeJUh_q3U4p3z4E_fEA@mail.gmail.com>
 <CA+w6HxZ-uht9XzeUC4qqsUL-FVZrqzvt-DzT+qdfEybwAsx6nQ@mail.gmail.com>
Message-ID: <CA+w6HxYydnJ2qd+m_78uBET-LVtmnhwm0=kYRKYmrBHpfzoiiA@mail.gmail.com>

I also looked at how other projects (portola and shenandoah) handle this.
Unlike tsan and loom, which maintain separate development branches,
both portola and shenandoah just maintain a single "default" branch. They
also have separate repositories for JDK11, JDK12, etc.
They also do regular "Merge" commits to sync with jdk/jdk in the master
development repository.
So I think we should go with "hg merge". Mercurial should be able to
differentiate commits between the tsan and jdk/jdk. The "tsan" branch in
our repository would help separate out the changes further.

I guess the main problem with "hg rebase" is that it generates a whole new
set of changeset IDs (hashes). We don't want to change the IDs for
changesets that have already been pushed.

Any more concerns before I try a merge with jdk14-ga?

-Man