From manc at google.com  Wed May  1 01:39:58 2019
From: manc at google.com (Man Cao)
Date: Tue, 30 Apr 2019 18:39:58 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBwe3K1oSwPWq4_Ee325e67VCVrqVxPGzO9xcLChVeBB+Q@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
 <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
 <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>
 <CAPW48srbr7_ZXCpaWXkxeHm9bN6kwQU8zuVk=VyU64cOoXV1sA@mail.gmail.com>
 <CAF9BGBwe3K1oSwPWq4_Ee325e67VCVrqVxPGzO9xcLChVeBB+Q@mail.gmail.com>
Message-ID: <CA+w6HxajaeXDW-Ws=M86NcUmMFMoGV3yZU4_Djfi6XWfdg=pvQ@mail.gmail.com>

Looks good.

On Tue, Apr 30, 2019, 15:25 Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Fixed on my local branch:)
> Jc
>
> On Tue, Apr 30, 2019 at 3:23 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> The comment in src/hotspot/share/prims/jvmtiEnv.cpp still says "below".
>>
>> Otherwise LGTM.
>>
>> On Tue, Apr 30, 2019 at 3:19 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Fair enough :)
>>>
>>> Done here:
>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.07/
>>>
>>> (I left the jvmtiTagMap ones using TSAN_ONLY, as if TSAN is defined,
>>> they call the macros which then use TSAN_RUNTIME_ONLY so that works too)
>>>
>>> Tested that the test still passes :), let me know what you think,
>>> Jc
>>>
>>>
>>> On Tue, Apr 30, 2019 at 2:34 PM Man Cao <manc at google.com> wrote:
>>>
>>>> I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
>>>> tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
>>>> simplify other code.
>>>> The current webrev if left unchanged could cause issues with
>>>> -XX:-ThreadSanitizer.
>>>>
>>>> -Man
>>>>
>>>>
>>>> On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Good point, what do you prefer here? Just leave like this for now
>>>>> since it is a noop? I do :)
>>>>>
>>>>> But I'm biased at preferring forward progress on this webrev :)
>>>>> Jc
>>>>>
>>>>> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>>>>>
>>>>>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer
>>>>>> flag any more.
>>>>>>
>>>>>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in
>>>>>> tsan.hpp? E.g.:
>>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__,
>>>>>> __LINE__, lock); })
>>>>>>
>>>>>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the
>>>>>> following?
>>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>>>>>
>>>>>> In any case, it seems that there is no clean way to guard the
>>>>>> creation of TsanMutexScope objects with the +ThreadSanitizer flag, but it
>>>>>> is probably OK as it is no-op with -ThreadSanitizer.
>>>>>>
>>>>>> -Man
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>>>>>> jcbeyler at google.com> wrote:
>>>>>>
>>>>>>> Fair enough, I forgot you had added it there while I was adding it
>>>>>>> in mine.
>>>>>>>
>>>>>>> I merged it to use yours, and because you don't do the fancy
>>>>>>> do/while in the macro, it works as intended then and we can leave it at
>>>>>>> that then for now :)
>>>>>>>
>>>>>>> New webrev is here:
>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>>>>>
>>>>>>> I'll add in a future webrev the tests for the other VM operations
>>>>>>> and try to see how we can fix the VM operation also in a future webrev; as
>>>>>>> soon as symbolization is up, we should be able to get testing to be a bit
>>>>>>> more reliable :)
>>>>>>>
>>>>>>> Thanks for the reviews!
>>>>>>> Jc
>>>>>>>
>>>>>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe
>>>>>>>> that's helpful?
>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jc
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Wed May  1 02:39:23 2019
From: jcbeyler at google.com (jcbeyler at google.com)
Date: Wed, 01 May 2019 02:39:23 +0000
Subject: hg: tsan/dev: Add JVMTI raw monitor lock/unlock mechanisms
Message-ID: <201905010239.x412dNiU001667@aojmv0008.oracle.com>

Changeset: b540214cea9d
Author:    jcbeyler
Date:      2019-04-30 19:38 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/b540214cea9d

Add JVMTI raw monitor lock/unlock mechanisms

- Added three tests that pass.
- Moved the libAbstractNativeLoop.cpp to C++ because seriously, why stay in C?

Reviewed-by: aeubanks, manc

! src/hotspot/share/prims/jvm.cpp
! src/hotspot/share/prims/jvmtiEnv.cpp
! src/hotspot/share/prims/jvmtiRawMonitor.cpp
! src/hotspot/share/prims/jvmtiTagMap.cpp
! src/hotspot/share/prims/jvmtiTagMap.hpp
! src/hotspot/share/tsan/tsan.cpp
+ src/hotspot/share/tsan/tsan.hpp
! src/hotspot/share/tsan/tsanExternalDecls.hpp
! test/hotspot/jtreg/tsan/AbstractNativeLoop.java
+ test/hotspot/jtreg/tsan/JvmtiTaggerTest.java
+ test/hotspot/jtreg/tsan/NonRawRacyNativeLoopTest.java
+ test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/TsanRunner.java
- test/hotspot/jtreg/tsan/libAbstractNativeLoop.c
+ test/hotspot/jtreg/tsan/libAbstractNativeLoop.cpp


From jcbeyler at google.com  Wed May  1 02:39:35 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 30 Apr 2019 19:39:35 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxajaeXDW-Ws=M86NcUmMFMoGV3yZU4_Djfi6XWfdg=pvQ@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
 <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
 <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>
 <CAPW48srbr7_ZXCpaWXkxeHm9bN6kwQU8zuVk=VyU64cOoXV1sA@mail.gmail.com>
 <CAF9BGBwe3K1oSwPWq4_Ee325e67VCVrqVxPGzO9xcLChVeBB+Q@mail.gmail.com>
 <CA+w6HxajaeXDW-Ws=M86NcUmMFMoGV3yZU4_Djfi6XWfdg=pvQ@mail.gmail.com>
Message-ID: <CAF9BGBzUcuRoYMhTTfKxrSZKJ2FZuGFOON7eCfWP+F5Xg98Dig@mail.gmail.com>

Done, thanks both for the reviews :)
Jc

On Tue, Apr 30, 2019 at 6:40 PM Man Cao <manc at google.com> wrote:

> Looks good.
>
> On Tue, Apr 30, 2019, 15:25 Jean Christophe Beyler <jcbeyler at google.com>
> wrote:
>
>> Fixed on my local branch:)
>> Jc
>>
>> On Tue, Apr 30, 2019 at 3:23 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> The comment in src/hotspot/share/prims/jvmtiEnv.cpp still says "below".
>>>
>>> Otherwise LGTM.
>>>
>>> On Tue, Apr 30, 2019 at 3:19 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Fair enough :)
>>>>
>>>> Done here:
>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.07/
>>>>
>>>> (I left the jvmtiTagMap ones using TSAN_ONLY, as if TSAN is defined,
>>>> they call the macros which then use TSAN_RUNTIME_ONLY so that works too)
>>>>
>>>> Tested that the test still passes :), let me know what you think,
>>>> Jc
>>>>
>>>>
>>>> On Tue, Apr 30, 2019 at 2:34 PM Man Cao <manc at google.com> wrote:
>>>>
>>>>> I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
>>>>> tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
>>>>> simplify other code.
>>>>> The current webrev if left unchanged could cause issues with
>>>>> -XX:-ThreadSanitizer.
>>>>>
>>>>> -Man
>>>>>
>>>>>
>>>>> On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <
>>>>> jcbeyler at google.com> wrote:
>>>>>
>>>>>> Good point, what do you prefer here? Just leave like this for now
>>>>>> since it is a noop? I do :)
>>>>>>
>>>>>> But I'm biased at preferring forward progress on this webrev :)
>>>>>> Jc
>>>>>>
>>>>>> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>>>>>>
>>>>>>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer
>>>>>>> flag any more.
>>>>>>>
>>>>>>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in
>>>>>>> tsan.hpp? E.g.:
>>>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__,
>>>>>>> __LINE__, lock); })
>>>>>>>
>>>>>>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the
>>>>>>> following?
>>>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>>>>>>
>>>>>>> In any case, it seems that there is no clean way to guard the
>>>>>>> creation of TsanMutexScope objects with the +ThreadSanitizer flag, but it
>>>>>>> is probably OK as it is no-op with -ThreadSanitizer.
>>>>>>>
>>>>>>> -Man
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>>>>>>> jcbeyler at google.com> wrote:
>>>>>>>
>>>>>>>> Fair enough, I forgot you had added it there while I was adding it
>>>>>>>> in mine.
>>>>>>>>
>>>>>>>> I merged it to use yours, and because you don't do the fancy
>>>>>>>> do/while in the macro, it works as intended then and we can leave it at
>>>>>>>> that then for now :)
>>>>>>>>
>>>>>>>> New webrev is here:
>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>>>>>>
>>>>>>>> I'll add in a future webrev the tests for the other VM operations
>>>>>>>> and try to see how we can fix the VM operation also in a future webrev; as
>>>>>>>> soon as symbolization is up, we should be able to get testing to be a bit
>>>>>>>> more reliable :)
>>>>>>>>
>>>>>>>> Thanks for the reviews!
>>>>>>>> Jc
>>>>>>>>
>>>>>>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe
>>>>>>>>> that's helpful?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jc
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Jc
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Wed May  1 18:48:45 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 1 May 2019 11:48:45 -0700
Subject: Symbolization
In-Reply-To: <CAPW48sp2H7hMU-2DiJXLA_aL1R3w8zMecQgSZB=68xHu9R=Vvg@mail.gmail.com>
References: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
 <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>
 <CA+w6HxY7YHbvBYnGC2Ort7V1hFBEpVDSTMhcwuAH9a7d9n2Z6g@mail.gmail.com>
 <CAF9BGBwr+Khm0=a2uTvHQ3wEvoUpkJpteTF9sthbK-o7OswUjQ@mail.gmail.com>
 <CAPW48sp2H7hMU-2DiJXLA_aL1R3w8zMecQgSZB=68xHu9R=Vvg@mail.gmail.com>
Message-ID: <CAPW48srYmAhaZSTXv4dpSmQUZSFHxGynOYsqv+qQ7m+YT15caA@mail.gmail.com>

It seems like mixing system libraries and the custom GCC libraries at all
is bad.
With LD_DEBUG=libs, bin/java fails at
calling init: path/to/libstdc++.so.6
whether it's the system libstdc++ or the GCC libstdc++. Even only using the
libtsan.so from the custom GCC and no other libraries results in the same
problem.

I think we need a better solution. Maybe compiling just libtsan.so
ourselves.

On Tue, Apr 30, 2019 at 3:18 PM Arthur Eubanks <aeubanks at google.com> wrote:

> The earliest GCC that uses __tsan_symbolize_external_ex() is GCC 9. So
> using the GCC version supported by OpenJDK (GCC 7) is not feasible.
>
> I tried compiling GCC 9 and using it as the toolchain (install GCC 9 with
> prefix $HOME/bin/gcc9, then configure and build OpenJDK
> --with-toolchain-path=$HOME/bin/gcc9).
>
> Running the TSAN tests, I get
> result: Failed. Unexpected exit from test [exit code: 139]
>
> Doing some searching, it looks like exit code 139 is something to do with
> memory fragmentation, or SIGSEGV, or something like that. So my guess is
> that mixing .so's between the Linux distro and the compiled GCC is not okay.
>
> On Mon, Apr 29, 2019 at 2:48 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> FWIW, it does not really matter to me at this point what we say to do as
>> long as it works for us. At some point, when we get a bit further along, we
>> would want to figure out a "sane" way of doing things; requiring to compile
>> GCC by hand is not sane in my opinion but is ok for now :)
>> Jc
>>
>> On Mon, Apr 29, 2019 at 2:36 PM Man Cao <manc at google.com> wrote:
>>
>>> I think the best way is still to find a way to build libtsan.so from
>>> latest source, with the GCC version supported by OpenJDK:
>>> https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms.
>>> Even if we can get libtsan.so from a newer GCC, I'm not sure if it is
>>> safe to build OpenJDK with the newer GCC, or to use a newer libtsan.so with
>>> an old GCC.
>>>
>>> -Man
>>>
>>>
>>> On Mon, Apr 29, 2019 at 2:18 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Compiling gcc to get libtsan... Is there no way to build libtsan then
>>>> from
>>>> scratch instead?
>>>>
>>>> On Mon, Apr 29, 2019 at 2:00 PM Arthur Eubanks <aeubanks at google.com>
>>>> wrote:
>>>>
>>>> > It looks like the libtsan.so from gcc 8 is out of date. It doesn't
>>>> > contain __tsan_symbolize_external_ex(), only the
>>>> > legacy __tsan_symbolize_external(). This messes with testing the Java
>>>> Tsan
>>>> > patches.
>>>> >
>>>> > $ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym
>>>> >
>>>> > I looked at the latest gcc sources and it does have a version of Tsan
>>>> that
>>>> > includes __tsan_symbolize_external_ex(). We could tell people to
>>>> compile
>>>> > their own gcc and use that libtsan.so.
>>>> > $ apt search gcc
>>>> > only shows up to gcc 8 on my machine.
>>>> >
>>>> > Thoughts?
>>>> >
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From manc at google.com  Mon May  6 16:48:12 2019
From: manc at google.com (Man Cao)
Date: Mon, 6 May 2019 09:48:12 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
Message-ID: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>

Hi all,

Can I have reviews for this bug fix and cleanup?
http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/

Copying commit message below:
Use TSAN_RUNTIME_ONLY for most instrumentation.
This fixes build error using --with-jvm-features=-tsan,
and reduces boilerplate code for instrumentation.
Also simplified oopDesc:is_oop() uses and updated a comment.

-Man

From aeubanks at google.com  Mon May  6 16:57:42 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 6 May 2019 09:57:42 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
In-Reply-To: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
References: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
Message-ID: <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>

Having to type "__FILE__, __LINE__" to every TsanRawLock*() call is
annoying. Is it possible to still keep the TSAN_RAW_LOCK_* macros in
tsan.hpp and call them in a TSAN_RUNTIME_ONLY block? That way they're
stripped out by the preprocessor in non-TSAN builds, and they'll still be
available in TSAN builds.

*From: *Man Cao <manc at google.com>
*Date: *Mon, May 6, 2019 at 9:48 AM
*To: * <tsan-dev at openjdk.java.net>

Hi all,
>
> Can I have reviews for this bug fix and cleanup?
> http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/
>
> Copying commit message below:
> Use TSAN_RUNTIME_ONLY for most instrumentation.
> This fixes build error using --with-jvm-features=-tsan,
> and reduces boilerplate code for instrumentation.
> Also simplified oopDesc:is_oop() uses and updated a comment.
>
> -Man
>

From jcbeyler at google.com  Mon May  6 17:24:37 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 6 May 2019 10:24:37 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
In-Reply-To: <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>
References: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
 <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>
Message-ID: <CAF9BGByzNmwNGYCgaqAjY99eq8jkO2NR5NV73R3tN3DztEnseA@mail.gmail.com>

I agree with Arthur, though nested macros makes me sad, it would be better
than having to do LINE/FILE every time...
Jc

On Mon, May 6, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Having to type "__FILE__, __LINE__" to every TsanRawLock*() call is
> annoying. Is it possible to still keep the TSAN_RAW_LOCK_* macros in
> tsan.hpp and call them in a TSAN_RUNTIME_ONLY block? That way they're
> stripped out by the preprocessor in non-TSAN builds, and they'll still be
> available in TSAN builds.
>
> *From: *Man Cao <manc at google.com>
> *Date: *Mon, May 6, 2019 at 9:48 AM
> *To: * <tsan-dev at openjdk.java.net>
>
> Hi all,
> >
> > Can I have reviews for this bug fix and cleanup?
> > http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/
> >
> > Copying commit message below:
> > Use TSAN_RUNTIME_ONLY for most instrumentation.
> > This fixes build error using --with-jvm-features=-tsan,
> > and reduces boilerplate code for instrumentation.
> > Also simplified oopDesc:is_oop() uses and updated a comment.
> >
> > -Man
> >
>


-- 

Thanks,
Jc

From manc at google.com  Mon May  6 18:56:13 2019
From: manc at google.com (Man Cao)
Date: Mon, 6 May 2019 11:56:13 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
In-Reply-To: <CAF9BGByzNmwNGYCgaqAjY99eq8jkO2NR5NV73R3tN3DztEnseA@mail.gmail.com>
References: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
 <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>
 <CAF9BGByzNmwNGYCgaqAjY99eq8jkO2NR5NV73R3tN3DztEnseA@mail.gmail.com>
Message-ID: <CA+w6HxZf2b=EWq=Tka3L9m1Kx+Uq6yMNRKa6GnT3+WUZRb1aow@mail.gmail.com>

Done. New webrev and incremental webrev:

https://cr.openjdk.java.net/~manc/tsan20190506/webrev.01/
https://cr.openjdk.java.net/~manc/tsan20190506/webrev.diff.00-01/

-Man


*From: *Jean Christophe Beyler <jcbeyler at google.com>
*Date: *Mon, May 6, 2019 at 10:24 AM
*To: *Arthur Eubanks
*Cc: *Man Cao, <tsan-dev at openjdk.java.net>

I agree with Arthur, though nested macros makes me sad, it would be better
> than having to do LINE/FILE every time...
> Jc
>
> On Mon, May 6, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com> wrote:
>
>> Having to type "__FILE__, __LINE__" to every TsanRawLock*() call is
>> annoying. Is it possible to still keep the TSAN_RAW_LOCK_* macros in
>> tsan.hpp and call them in a TSAN_RUNTIME_ONLY block? That way they're
>> stripped out by the preprocessor in non-TSAN builds, and they'll still be
>> available in TSAN builds.
>>
>> *From: *Man Cao <manc at google.com>
>> *Date: *Mon, May 6, 2019 at 9:48 AM
>> *To: * <tsan-dev at openjdk.java.net>
>>
>> Hi all,
>> >
>> > Can I have reviews for this bug fix and cleanup?
>> > http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/
>> >
>> > Copying commit message below:
>> > Use TSAN_RUNTIME_ONLY for most instrumentation.
>> > This fixes build error using --with-jvm-features=-tsan,
>> > and reduces boilerplate code for instrumentation.
>> > Also simplified oopDesc:is_oop() uses and updated a comment.
>> >
>> > -Man
>> >
>>
>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Mon May  6 20:24:59 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 6 May 2019 13:24:59 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
In-Reply-To: <CA+w6HxZf2b=EWq=Tka3L9m1Kx+Uq6yMNRKa6GnT3+WUZRb1aow@mail.gmail.com>
References: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
 <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>
 <CAF9BGByzNmwNGYCgaqAjY99eq8jkO2NR5NV73R3tN3DztEnseA@mail.gmail.com>
 <CA+w6HxZf2b=EWq=Tka3L9m1Kx+Uq6yMNRKa6GnT3+WUZRb1aow@mail.gmail.com>
Message-ID: <CAPW48srT0kNG0p-sDgmF5g02UU+F0QhkTXHGu1UB6U=4XusLaQ@mail.gmail.com>

LGTM.

*From: *Man Cao <manc at google.com>
*Date: *Mon, May 6, 2019 at 11:56 AM
*To: *Jean Christophe Beyler
*Cc: *Arthur Eubanks, <tsan-dev at openjdk.java.net>

Done. New webrev and incremental webrev:
>
> https://cr.openjdk.java.net/~manc/tsan20190506/webrev.01/
> https://cr.openjdk.java.net/~manc/tsan20190506/webrev.diff.00-01/
>
> -Man
>
>
> *From: *Jean Christophe Beyler <jcbeyler at google.com>
> *Date: *Mon, May 6, 2019 at 10:24 AM
> *To: *Arthur Eubanks
> *Cc: *Man Cao, <tsan-dev at openjdk.java.net>
>
> I agree with Arthur, though nested macros makes me sad, it would be better
>> than having to do LINE/FILE every time...
>> Jc
>>
>> On Mon, May 6, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> Having to type "__FILE__, __LINE__" to every TsanRawLock*() call is
>>> annoying. Is it possible to still keep the TSAN_RAW_LOCK_* macros in
>>> tsan.hpp and call them in a TSAN_RUNTIME_ONLY block? That way they're
>>> stripped out by the preprocessor in non-TSAN builds, and they'll still be
>>> available in TSAN builds.
>>>
>>> *From: *Man Cao <manc at google.com>
>>> *Date: *Mon, May 6, 2019 at 9:48 AM
>>> *To: * <tsan-dev at openjdk.java.net>
>>>
>>> Hi all,
>>> >
>>> > Can I have reviews for this bug fix and cleanup?
>>> > http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/
>>> >
>>> > Copying commit message below:
>>> > Use TSAN_RUNTIME_ONLY for most instrumentation.
>>> > This fixes build error using --with-jvm-features=-tsan,
>>> > and reduces boilerplate code for instrumentation.
>>> > Also simplified oopDesc:is_oop() uses and updated a comment.
>>> >
>>> > -Man
>>> >
>>>
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From jcbeyler at google.com  Mon May  6 22:19:07 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 6 May 2019 15:19:07 -0700
Subject: RFR (M): Use TSAN_RUNTIME_ONLY for most instrumentation
In-Reply-To: <CAPW48srT0kNG0p-sDgmF5g02UU+F0QhkTXHGu1UB6U=4XusLaQ@mail.gmail.com>
References: <CA+w6HxYxTH_UDgWaFmRK-tkdRt0fP1dD3FahqAsAjO2AOv=gOw@mail.gmail.com>
 <CAPW48sqFf3a-kWFmQCxS2WcarqDw0V4bXBnRMKC8eWqi241pEw@mail.gmail.com>
 <CAF9BGByzNmwNGYCgaqAjY99eq8jkO2NR5NV73R3tN3DztEnseA@mail.gmail.com>
 <CA+w6HxZf2b=EWq=Tka3L9m1Kx+Uq6yMNRKa6GnT3+WUZRb1aow@mail.gmail.com>
 <CAPW48srT0kNG0p-sDgmF5g02UU+F0QhkTXHGu1UB6U=4XusLaQ@mail.gmail.com>
Message-ID: <CAF9BGByjkY4=ZzK775RHRyp8=NNka8FeWQB-CgAJ0=ticF5PrQ@mail.gmail.com>

LGTM :)

On Mon, May 6, 2019 at 1:25 PM Arthur Eubanks <aeubanks at google.com> wrote:

> LGTM.
>
> *From: *Man Cao <manc at google.com>
> *Date: *Mon, May 6, 2019 at 11:56 AM
> *To: *Jean Christophe Beyler
> *Cc: *Arthur Eubanks, <tsan-dev at openjdk.java.net>
>
> Done. New webrev and incremental webrev:
>>
>> https://cr.openjdk.java.net/~manc/tsan20190506/webrev.01/
>> https://cr.openjdk.java.net/~manc/tsan20190506/webrev.diff.00-01/
>>
>> -Man
>>
>>
>> *From: *Jean Christophe Beyler <jcbeyler at google.com>
>> *Date: *Mon, May 6, 2019 at 10:24 AM
>> *To: *Arthur Eubanks
>> *Cc: *Man Cao, <tsan-dev at openjdk.java.net>
>>
>> I agree with Arthur, though nested macros makes me sad, it would be
>>> better than having to do LINE/FILE every time...
>>> Jc
>>>
>>> On Mon, May 6, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>>> Having to type "__FILE__, __LINE__" to every TsanRawLock*() call is
>>>> annoying. Is it possible to still keep the TSAN_RAW_LOCK_* macros in
>>>> tsan.hpp and call them in a TSAN_RUNTIME_ONLY block? That way they're
>>>> stripped out by the preprocessor in non-TSAN builds, and they'll still
>>>> be
>>>> available in TSAN builds.
>>>>
>>>> *From: *Man Cao <manc at google.com>
>>>> *Date: *Mon, May 6, 2019 at 9:48 AM
>>>> *To: * <tsan-dev at openjdk.java.net>
>>>>
>>>> Hi all,
>>>> >
>>>> > Can I have reviews for this bug fix and cleanup?
>>>> > http://cr.openjdk.java.net/~manc/tsan/20190506/webrev.00/
>>>> >
>>>> > Copying commit message below:
>>>> > Use TSAN_RUNTIME_ONLY for most instrumentation.
>>>> > This fixes build error using --with-jvm-features=-tsan,
>>>> > and reduces boilerplate code for instrumentation.
>>>> > Also simplified oopDesc:is_oop() uses and updated a comment.
>>>> >
>>>> > -Man
>>>> >
>>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>

-- 

Thanks,
Jc

From manc at google.com  Mon May  6 22:34:59 2019
From: manc at google.com (manc at google.com)
Date: Mon, 06 May 2019 22:34:59 +0000
Subject: hg: tsan/dev: Use TSAN_RUNTIME_ONLY for most instrumentation.
Message-ID: <201905062235.x46MZ04Q027030@aojmv0008.oracle.com>

Changeset: f94cc4cf247f
Author:    manc
Date:      2019-05-05 19:21 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/f94cc4cf247f

Use TSAN_RUNTIME_ONLY for most instrumentation.

This fixes build error using --with-jvm-features=-tsan,
and reduces boilerplate code for instrumentation.

Also simplified oopDesc:is_oop() uses and updated a comment.

Reviewed-by: jcbeyler, aeubanks

! src/hotspot/cpu/x86/interp_masm_x86.cpp
! src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
! src/hotspot/share/classfile/systemDictionary.cpp
! src/hotspot/share/prims/jvm.cpp
! src/hotspot/share/prims/jvmtiEnv.cpp
! src/hotspot/share/prims/jvmtiRawMonitor.cpp
! src/hotspot/share/prims/jvmtiTagMap.cpp
! src/hotspot/share/runtime/arguments.cpp
! src/hotspot/share/runtime/init.cpp
! src/hotspot/share/runtime/sharedRuntime.cpp
! src/hotspot/share/runtime/synchronizer.cpp
! src/hotspot/share/tsan/tsan.hpp
! src/hotspot/share/tsan/tsanExternalDecls.hpp
! src/hotspot/share/utilities/macros.hpp


From manc at google.com  Tue May 14 05:08:29 2019
From: manc at google.com (Man Cao)
Date: Mon, 13 May 2019 22:08:29 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the launcher
 with Clang
Message-ID: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>

Hi all,

Can I have reviews for this patch that hopefully will unblock us from the
libtsan.so's version issue?
https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/

This is a workaround for the version issue with libtsan.so in GCC. If we
build the JDK with Clang, we will have a newer version of the TSAN runtime
statically linked into the launcher:
$ nm images/jdk/bin/java | grep tsan_symbolize
000000000048e640 W __tsan_symbolize_external
000000000048e650 W __tsan_symbolize_external_ex

In the long term, Java TSAN will still work with GCC, but stack trace
symbolization will be broken until OpenJDK can be built with GCC9, which
contains the new version of libtsan.so that we need.
Also need to write distinct values to the global variable in
libAbstractNativeLoop.cpp, otherwise some optimization in Clang could treat
the variable as constant, and TSAN reports "Write of size 1" instead of
"size 4".

I have tested this with both LLVM 8.0.0 and GCC.

-Man

From manc at google.com  Tue May 14 05:14:42 2019
From: manc at google.com (Man Cao)
Date: Mon, 13 May 2019 22:14:42 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
Message-ID: <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>

FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
http://releases.llvm.org/download.html#8.0.0

I will also update the wiki page if everyone likes this workaround.

-Man


*From: *Man Cao <manc at google.com>
*Date: *Mon, May 13, 2019 at 10:08 PM
*To: * <tsan-dev at openjdk.java.net>

Hi all,
>
> Can I have reviews for this patch that hopefully will unblock us from the
> libtsan.so's version issue?
> https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
>
> This is a workaround for the version issue with libtsan.so in GCC. If we
> build the JDK with Clang, we will have a newer version of the TSAN runtime
> statically linked into the launcher:
> $ nm images/jdk/bin/java | grep tsan_symbolize
> 000000000048e640 W __tsan_symbolize_external
> 000000000048e650 W __tsan_symbolize_external_ex
>
> In the long term, Java TSAN will still work with GCC, but stack trace
> symbolization will be broken until OpenJDK can be built with GCC9, which
> contains the new version of libtsan.so that we need.
> Also need to write distinct values to the global variable in
> libAbstractNativeLoop.cpp, otherwise some optimization in Clang could treat
> the variable as constant, and TSAN reports "Write of size 1" instead of
> "size 4".
>
> I have tested this with both LLVM 8.0.0 and GCC.
>
> -Man
>

From jcbeyler at google.com  Tue May 14 17:30:51 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 14 May 2019 10:30:51 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
 <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
Message-ID: <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>

LGTM, I am curious about the clang write of size 1 issue but I suggest we
push that to post this submission; that way we all have the info to build
and test,
Jc

*From: *Man Cao <manc at google.com>
*Date: *Mon, May 13, 2019 at 10:15 PM
*To: * <tsan-dev at openjdk.java.net>

FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
> http://releases.llvm.org/download.html#8.0.0
>
> I will also update the wiki page if everyone likes this workaround.
>
> -Man
>
>
> *From: *Man Cao <manc at google.com>
> *Date: *Mon, May 13, 2019 at 10:08 PM
> *To: * <tsan-dev at openjdk.java.net>
>
> Hi all,
> >
> > Can I have reviews for this patch that hopefully will unblock us from the
> > libtsan.so's version issue?
> > https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
> >
> > This is a workaround for the version issue with libtsan.so in GCC. If we
> > build the JDK with Clang, we will have a newer version of the TSAN
> runtime
> > statically linked into the launcher:
> > $ nm images/jdk/bin/java | grep tsan_symbolize
> > 000000000048e640 W __tsan_symbolize_external
> > 000000000048e650 W __tsan_symbolize_external_ex
> >
> > In the long term, Java TSAN will still work with GCC, but stack trace
> > symbolization will be broken until OpenJDK can be built with GCC9, which
> > contains the new version of libtsan.so that we need.
> > Also need to write distinct values to the global variable in
> > libAbstractNativeLoop.cpp, otherwise some optimization in Clang could
> treat
> > the variable as constant, and TSAN reports "Write of size 1" instead of
> > "size 4".
> >
> > I have tested this with both LLVM 8.0.0 and GCC.
> >
> > -Man
> >
>


-- 

Thanks,
Jc

From aeubanks at google.com  Tue May 14 17:50:05 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 14 May 2019 10:50:05 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
 <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
 <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>
Message-ID: <CAPW48sruZNrKFV2B=51+3cDN3Kn3igO1qcdOptauifenFWoHDw@mail.gmail.com>

This looks promising, let me do some testing.

Regarding the size of the write in native code, we should probably not
depend on the exact size, since that's not what we care about.

*From: *Jean Christophe Beyler <jcbeyler at google.com>
*Date: *Tue, May 14, 2019 at 10:31 AM
*To: *Man Cao
*Cc: * <tsan-dev at openjdk.java.net>

LGTM, I am curious about the clang write of size 1 issue but I suggest we
> push that to post this submission; that way we all have the info to build
> and test,
> Jc
>
> *From: *Man Cao <manc at google.com>
> *Date: *Mon, May 13, 2019 at 10:15 PM
> *To: * <tsan-dev at openjdk.java.net>
>
> FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
> > http://releases.llvm.org/download.html#8.0.0
> >
> > I will also update the wiki page if everyone likes this workaround.
> >
> > -Man
> >
> >
> > *From: *Man Cao <manc at google.com>
> > *Date: *Mon, May 13, 2019 at 10:08 PM
> > *To: * <tsan-dev at openjdk.java.net>
> >
> > Hi all,
> > >
> > > Can I have reviews for this patch that hopefully will unblock us from
> the
> > > libtsan.so's version issue?
> > > https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
> > >
> > > This is a workaround for the version issue with libtsan.so in GCC. If
> we
> > > build the JDK with Clang, we will have a newer version of the TSAN
> > runtime
> > > statically linked into the launcher:
> > > $ nm images/jdk/bin/java | grep tsan_symbolize
> > > 000000000048e640 W __tsan_symbolize_external
> > > 000000000048e650 W __tsan_symbolize_external_ex
> > >
> > > In the long term, Java TSAN will still work with GCC, but stack trace
> > > symbolization will be broken until OpenJDK can be built with GCC9,
> which
> > > contains the new version of libtsan.so that we need.
> > > Also need to write distinct values to the global variable in
> > > libAbstractNativeLoop.cpp, otherwise some optimization in Clang could
> > treat
> > > the variable as constant, and TSAN reports "Write of size 1" instead of
> > > "size 4".
> > >
> > > I have tested this with both LLVM 8.0.0 and GCC.
> > >
> > > -Man
> > >
> >
>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue May 14 18:05:01 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 14 May 2019 11:05:01 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CAPW48sruZNrKFV2B=51+3cDN3Kn3igO1qcdOptauifenFWoHDw@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
 <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
 <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>
 <CAPW48sruZNrKFV2B=51+3cDN3Kn3igO1qcdOptauifenFWoHDw@mail.gmail.com>
Message-ID: <CAF9BGByHm6MiyevdQYH_P_31_uty4AnT9CT9XmhaeZudqzADvQ@mail.gmail.com>

Well we do care about correctness and there was no reason for TSAN to
provide a "wrong" size. In this case, it is an optimization of clang that
provokes this 1 vs 4 byte difference.

Basically clang is able to ascertain a single value is being written into
the global variable and transform it into a boolean, "if you wrote it in,
I'll return the value you wanted to write in it"
Jc

*From: *Arthur Eubanks <aeubanks at google.com>
*Date: *Tue, May 14, 2019 at 10:50 AM
*To: *Jean Christophe Beyler
*Cc: *Man Cao, <tsan-dev at openjdk.java.net>

This looks promising, let me do some testing.
>
> Regarding the size of the write in native code, we should probably not
> depend on the exact size, since that's not what we care about.
>
> *From: *Jean Christophe Beyler <jcbeyler at google.com>
> *Date: *Tue, May 14, 2019 at 10:31 AM
> *To: *Man Cao
> *Cc: * <tsan-dev at openjdk.java.net>
>
> LGTM, I am curious about the clang write of size 1 issue but I suggest we
>> push that to post this submission; that way we all have the info to build
>> and test,
>> Jc
>>
>> *From: *Man Cao <manc at google.com>
>> *Date: *Mon, May 13, 2019 at 10:15 PM
>> *To: * <tsan-dev at openjdk.java.net>
>>
>> FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
>> > http://releases.llvm.org/download.html#8.0.0
>> >
>> > I will also update the wiki page if everyone likes this workaround.
>> >
>> > -Man
>> >
>> >
>> > *From: *Man Cao <manc at google.com>
>> > *Date: *Mon, May 13, 2019 at 10:08 PM
>> > *To: * <tsan-dev at openjdk.java.net>
>> >
>> > Hi all,
>> > >
>> > > Can I have reviews for this patch that hopefully will unblock us from
>> the
>> > > libtsan.so's version issue?
>> > > https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
>> > >
>> > > This is a workaround for the version issue with libtsan.so in GCC. If
>> we
>> > > build the JDK with Clang, we will have a newer version of the TSAN
>> > runtime
>> > > statically linked into the launcher:
>> > > $ nm images/jdk/bin/java | grep tsan_symbolize
>> > > 000000000048e640 W __tsan_symbolize_external
>> > > 000000000048e650 W __tsan_symbolize_external_ex
>> > >
>> > > In the long term, Java TSAN will still work with GCC, but stack trace
>> > > symbolization will be broken until OpenJDK can be built with GCC9,
>> which
>> > > contains the new version of libtsan.so that we need.
>> > > Also need to write distinct values to the global variable in
>> > > libAbstractNativeLoop.cpp, otherwise some optimization in Clang could
>> > treat
>> > > the variable as constant, and TSAN reports "Write of size 1" instead
>> of
>> > > "size 4".
>> > >
>> > > I have tested this with both LLVM 8.0.0 and GCC.
>> > >
>> > > -Man
>> > >
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Tue May 14 19:05:50 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 14 May 2019 12:05:50 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CAF9BGByHm6MiyevdQYH_P_31_uty4AnT9CT9XmhaeZudqzADvQ@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
 <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
 <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>
 <CAPW48sruZNrKFV2B=51+3cDN3Kn3igO1qcdOptauifenFWoHDw@mail.gmail.com>
 <CAF9BGByHm6MiyevdQYH_P_31_uty4AnT9CT9XmhaeZudqzADvQ@mail.gmail.com>
Message-ID: <CAPW48sogy-r+2V9SUy3kvEmA3a4xN4QEXk2fHD7FWQw-Fqo56g@mail.gmail.com>

I tested it with the patch that adds Java memory instrumentation and the
stack traces look good (the package name is missing, but that's a different
problem).

One worry I have is that people who want to do this will have to build
their native code with -Wl,--unresolved-symbols=ignore-in-object-files if
they're using gcc, and also make sure not to link with -fsanitize=thread
regardless of the compiler. Maybe for now that doesn't matter, but it will
be a usability problem later on.

LGTM.

*From: *Jean Christophe Beyler <jcbeyler at google.com>
*Date: *Tue, May 14, 2019 at 11:05 AM
*To: *Arthur Eubanks
*Cc: *Man Cao, <tsan-dev at openjdk.java.net>

Well we do care about correctness and there was no reason for TSAN to
> provide a "wrong" size. In this case, it is an optimization of clang that
> provokes this 1 vs 4 byte difference.
>
> Basically clang is able to ascertain a single value is being written into
> the global variable and transform it into a boolean, "if you wrote it in,
> I'll return the value you wanted to write in it"
> Jc
>
> *From: *Arthur Eubanks <aeubanks at google.com>
> *Date: *Tue, May 14, 2019 at 10:50 AM
> *To: *Jean Christophe Beyler
> *Cc: *Man Cao, <tsan-dev at openjdk.java.net>
>
> This looks promising, let me do some testing.
>>
>> Regarding the size of the write in native code, we should probably not
>> depend on the exact size, since that's not what we care about.
>>
>> *From: *Jean Christophe Beyler <jcbeyler at google.com>
>> *Date: *Tue, May 14, 2019 at 10:31 AM
>> *To: *Man Cao
>> *Cc: * <tsan-dev at openjdk.java.net>
>>
>> LGTM, I am curious about the clang write of size 1 issue but I suggest we
>>> push that to post this submission; that way we all have the info to build
>>> and test,
>>> Jc
>>>
>>> *From: *Man Cao <manc at google.com>
>>> *Date: *Mon, May 13, 2019 at 10:15 PM
>>> *To: * <tsan-dev at openjdk.java.net>
>>>
>>> FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
>>> > http://releases.llvm.org/download.html#8.0.0
>>> >
>>> > I will also update the wiki page if everyone likes this workaround.
>>> >
>>> > -Man
>>> >
>>> >
>>> > *From: *Man Cao <manc at google.com>
>>> > *Date: *Mon, May 13, 2019 at 10:08 PM
>>> > *To: * <tsan-dev at openjdk.java.net>
>>> >
>>> > Hi all,
>>> > >
>>> > > Can I have reviews for this patch that hopefully will unblock us
>>> from the
>>> > > libtsan.so's version issue?
>>> > > https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
>>> > >
>>> > > This is a workaround for the version issue with libtsan.so in GCC.
>>> If we
>>> > > build the JDK with Clang, we will have a newer version of the TSAN
>>> > runtime
>>> > > statically linked into the launcher:
>>> > > $ nm images/jdk/bin/java | grep tsan_symbolize
>>> > > 000000000048e640 W __tsan_symbolize_external
>>> > > 000000000048e650 W __tsan_symbolize_external_ex
>>> > >
>>> > > In the long term, Java TSAN will still work with GCC, but stack trace
>>> > > symbolization will be broken until OpenJDK can be built with GCC9,
>>> which
>>> > > contains the new version of libtsan.so that we need.
>>> > > Also need to write distinct values to the global variable in
>>> > > libAbstractNativeLoop.cpp, otherwise some optimization in Clang could
>>> > treat
>>> > > the variable as constant, and TSAN reports "Write of size 1" instead
>>> of
>>> > > "size 4".
>>> > >
>>> > > I have tested this with both LLVM 8.0.0 and GCC.
>>> > >
>>> > > -Man
>>> > >
>>> >
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue May 14 19:11:11 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 14 May 2019 12:11:11 -0700
Subject: RFR (S): Supporting VM operations
Message-ID: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>

Hi all,

Here is a webrev that adds a bit of logic to get the VM operation supported
and not provoke false positive races:
http://cr.openjdk.java.net/~jcbeyler/vm_operation/

The backstory to this is:
   - When a thread is going to do certain JVMTI operations (or other btw),
it can do a VM operation. When this happens, there might be a callback into
user code which could change the state of the user memory.
     - If this happens, TSAN did not see that the thread actually has
passed owner ship to the VM thread via a vm event queue and wait
   - To solve this, the best solution I have found is to tell TSAN of the
thread memory via an acquire/release on the thread object itself

This thus becomes (note all mentions of locking below are fictitious and
just calls to TSAN to make it believe all is well in the world):
   - A thread running has a "lock" on its on JavaThread object
   - When it is going to wait for a VMThread to do something (including do
a callback) it relinquishes its "lock" and then waits (if the operation is
concurrent)
   - Then the VM thread gets that "lock" and does its operation; including
doing callbacks
    - When the VM thread is done, it relinquishes the "lock" again and the
JavaThread can acquire it back

This in essence does seem to work, the JvmtiTaggerTest no longer requires a
user lock on its own memory. The down-side is that now we have to track
this thread "lock" and I had to instrument the start of a thread.

Thanks,
Jc

From manc at google.com  Tue May 14 19:17:42 2019
From: manc at google.com (manc at google.com)
Date: Tue, 14 May 2019 19:17:42 +0000
Subject: hg: tsan/dev: Supports statically linking TSAN runtime into the
 launcher with Clang.
Message-ID: <201905141917.x4EJHgCp014120@aojmv0008.oracle.com>

Changeset: f761a445f8ed
Author:    manc
Date:      2019-05-13 19:22 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/f761a445f8ed

Supports statically linking TSAN runtime into the launcher with Clang.

This is a workaround for the version issue with libtsan.so in GCC.
If we build the JDK with Clang, we will have a newer version of the
TSAN runtime statically linked into the launcher:

$ nm images/jdk/bin/java | grep tsan_symbolize
000000000048e640 W __tsan_symbolize_external
000000000048e650 W __tsan_symbolize_external_ex

In the long term, Java TSAN will still work with GCC, but stack trace
symbolization will be broken until OpenJDK can be built with GCC9,
which contains the new version of libtsan.so that we need.

Also need to write distinct values to the global variable in
libAbstractNativeLoop.cpp, otherwise some optimization in Clang
could treat the variable as constant, and TSAN reports "Write of size 1"
instead of "size 4".

Reviewed-by: jcbeyler, aeubanks

! make/launcher/LauncherCommon.gmk
! make/test/JtregNativeHotspot.gmk
! test/hotspot/jtreg/tsan/libAbstractNativeLoop.cpp


From manc at google.com  Tue May 14 19:18:19 2019
From: manc at google.com (Man Cao)
Date: Tue, 14 May 2019 12:18:19 -0700
Subject: RFR (S): Supports statically linking TSAN runtime into the
 launcher with Clang
In-Reply-To: <CAPW48sogy-r+2V9SUy3kvEmA3a4xN4QEXk2fHD7FWQw-Fqo56g@mail.gmail.com>
References: <CA+w6Hxb1R3doHidAPm5BmcMrDvs_5kqForKYA6hTFWKWmKcr=Q@mail.gmail.com>
 <CA+w6HxbHWknYtGSjFM-VCuL7tsiDni11+Z0OeyygB_NVp4U_jQ@mail.gmail.com>
 <CAF9BGByk5eOd2hgSwsnwTtOCtBEFiGF1MVK2GmPiGJ4CKiWQzw@mail.gmail.com>
 <CAPW48sruZNrKFV2B=51+3cDN3Kn3igO1qcdOptauifenFWoHDw@mail.gmail.com>
 <CAF9BGByHm6MiyevdQYH_P_31_uty4AnT9CT9XmhaeZudqzADvQ@mail.gmail.com>
 <CAPW48sogy-r+2V9SUy3kvEmA3a4xN4QEXk2fHD7FWQw-Fqo56g@mail.gmail.com>
Message-ID: <CA+w6HxaK0oJNui_YFwdO0YJKjE=KrFHbbT44kiYn-kCLC_oViw@mail.gmail.com>

Good to hear that it works. Thanks for the reviews!

-Man


*From: *Arthur Eubanks <aeubanks at google.com>
*Date: *Tue, May 14, 2019 at 12:06 PM
*To: *Jean Christophe Beyler
*Cc: *Man Cao, <tsan-dev at openjdk.java.net>

I tested it with the patch that adds Java memory instrumentation and the
> stack traces look good (the package name is missing, but that's a different
> problem).
>
> One worry I have is that people who want to do this will have to build
> their native code with -Wl,--unresolved-symbols=ignore-in-object-files if
> they're using gcc, and also make sure not to link with -fsanitize=thread
> regardless of the compiler. Maybe for now that doesn't matter, but it will
> be a usability problem later on.
>
> LGTM.
>
> *From: *Jean Christophe Beyler <jcbeyler at google.com>
> *Date: *Tue, May 14, 2019 at 11:05 AM
> *To: *Arthur Eubanks
> *Cc: *Man Cao, <tsan-dev at openjdk.java.net>
>
> Well we do care about correctness and there was no reason for TSAN to
>> provide a "wrong" size. In this case, it is an optimization of clang that
>> provokes this 1 vs 4 byte difference.
>>
>> Basically clang is able to ascertain a single value is being written into
>> the global variable and transform it into a boolean, "if you wrote it in,
>> I'll return the value you wanted to write in it"
>> Jc
>>
>> *From: *Arthur Eubanks <aeubanks at google.com>
>> *Date: *Tue, May 14, 2019 at 10:50 AM
>> *To: *Jean Christophe Beyler
>> *Cc: *Man Cao, <tsan-dev at openjdk.java.net>
>>
>> This looks promising, let me do some testing.
>>>
>>> Regarding the size of the write in native code, we should probably not
>>> depend on the exact size, since that's not what we care about.
>>>
>>> *From: *Jean Christophe Beyler <jcbeyler at google.com>
>>> *Date: *Tue, May 14, 2019 at 10:31 AM
>>> *To: *Man Cao
>>> *Cc: * <tsan-dev at openjdk.java.net>
>>>
>>> LGTM, I am curious about the clang write of size 1 issue but I suggest we
>>>> push that to post this submission; that way we all have the info to
>>>> build
>>>> and test,
>>>> Jc
>>>>
>>>> *From: *Man Cao <manc at google.com>
>>>> *Date: *Mon, May 13, 2019 at 10:15 PM
>>>> *To: * <tsan-dev at openjdk.java.net>
>>>>
>>>> FYI, I used the pre-built LLVM 8.0.0 for Ubuntu 18.04 from here:
>>>> > http://releases.llvm.org/download.html#8.0.0
>>>> >
>>>> > I will also update the wiki page if everyone likes this workaround.
>>>> >
>>>> > -Man
>>>> >
>>>> >
>>>> > *From: *Man Cao <manc at google.com>
>>>> > *Date: *Mon, May 13, 2019 at 10:08 PM
>>>> > *To: * <tsan-dev at openjdk.java.net>
>>>> >
>>>> > Hi all,
>>>> > >
>>>> > > Can I have reviews for this patch that hopefully will unblock us
>>>> from the
>>>> > > libtsan.so's version issue?
>>>> > > https://cr.openjdk.java.net/~manc/tsan20190513/webrev.00/
>>>> > >
>>>> > > This is a workaround for the version issue with libtsan.so in GCC.
>>>> If we
>>>> > > build the JDK with Clang, we will have a newer version of the TSAN
>>>> > runtime
>>>> > > statically linked into the launcher:
>>>> > > $ nm images/jdk/bin/java | grep tsan_symbolize
>>>> > > 000000000048e640 W __tsan_symbolize_external
>>>> > > 000000000048e650 W __tsan_symbolize_external_ex
>>>> > >
>>>> > > In the long term, Java TSAN will still work with GCC, but stack
>>>> trace
>>>> > > symbolization will be broken until OpenJDK can be built with GCC9,
>>>> which
>>>> > > contains the new version of libtsan.so that we need.
>>>> > > Also need to write distinct values to the global variable in
>>>> > > libAbstractNativeLoop.cpp, otherwise some optimization in Clang
>>>> could
>>>> > treat
>>>> > > the variable as constant, and TSAN reports "Write of size 1"
>>>> instead of
>>>> > > "size 4".
>>>> > >
>>>> > > I have tested this with both LLVM 8.0.0 and GCC.
>>>> > >
>>>> > > -Man
>>>> > >
>>>> >
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From aeubanks at google.com  Tue May 14 22:49:40 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 14 May 2019 15:49:40 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
Message-ID: <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>

src/hotspot/share/runtime/vmThread.cpp:
In VMThread::execute(), would having an object that called
TSAN_RAW_LOCK_RELEASED() on construct and TSAN_RAW_LOCK_ACQUIRED() on
destruct be cleaner? That way you don't need to have the
TSAN_RAW_LOCK_ACQUIRED() in every exit path. If you did that, it could also
work with VMThread::evaluate_operation()

*From: *Jean Christophe Beyler <jcbeyler at google.com>
*Date: *Tue, May 14, 2019 at 12:12 PM
*To: * <tsan-dev at openjdk.java.net>

Hi all,
>
> Here is a webrev that adds a bit of logic to get the VM operation supported
> and not provoke false positive races:
> http://cr.openjdk.java.net/~jcbeyler/vm_operation/
>
> The backstory to this is:
>    - When a thread is going to do certain JVMTI operations (or other btw),
> it can do a VM operation. When this happens, there might be a callback into
> user code which could change the state of the user memory.
>      - If this happens, TSAN did not see that the thread actually has
> passed owner ship to the VM thread via a vm event queue and wait
>    - To solve this, the best solution I have found is to tell TSAN of the
> thread memory via an acquire/release on the thread object itself
>
> This thus becomes (note all mentions of locking below are fictitious and
> just calls to TSAN to make it believe all is well in the world):
>    - A thread running has a "lock" on its on JavaThread object
>    - When it is going to wait for a VMThread to do something (including do
> a callback) it relinquishes its "lock" and then waits (if the operation is
> concurrent)
>    - Then the VM thread gets that "lock" and does its operation; including
> doing callbacks
>     - When the VM thread is done, it relinquishes the "lock" again and the
> JavaThread can acquire it back
>
> This in essence does seem to work, the JvmtiTaggerTest no longer requires a
> user lock on its own memory. The down-side is that now we have to track
> this thread "lock" and I had to instrument the start of a thread.
>
> Thanks,
> Jc
>

From manc at google.com  Wed May 15 17:28:43 2019
From: manc at google.com (Man Cao)
Date: Wed, 15 May 2019 10:28:43 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
 <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
Message-ID: <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>

I don't think this approach fixes all the issues. We want to capture the
happens-before edge at a safepoint between all Java threads and the VM
thread, but this approach only captures the edge between the Java thread
that calls jvmti->IterateThroughHeap() with the VM thread.

Consider the following example:
T1, T2 are Java threads executing JNI code.

static int sum = 0;

T1:
sum = 10;

T2:
jvmti->IterateThroughHeap(&callback, &sum);

VMThread:
TSAN_RAW_LOCK_ACQUIRED(T2)
*sum_ptr += 1;
TSAN_RAW_LOCK_RELEASED(T2)

Then it will still report race between T1 and the VM thread.

Another question, should we call TSAN_RAW_LOCK_CREATE and
TSAN_RAW_LOCK_DESTROY for the fake per-thread lock also?

-Man

On Tue, May 14, 2019 at 3:50 PM Arthur Eubanks <aeubanks at google.com> wrote:

> src/hotspot/share/runtime/vmThread.cpp:
> In VMThread::execute(), would having an object that called
> TSAN_RAW_LOCK_RELEASED() on construct and TSAN_RAW_LOCK_ACQUIRED() on
> destruct be cleaner? That way you don't need to have the
> TSAN_RAW_LOCK_ACQUIRED() in every exit path. If you did that, it could also
> work with VMThread::evaluate_operation()
>
> *From: *Jean Christophe Beyler <jcbeyler at google.com>
> *Date: *Tue, May 14, 2019 at 12:12 PM
> *To: * <tsan-dev at openjdk.java.net>
>
> Hi all,
> >
> > Here is a webrev that adds a bit of logic to get the VM operation
> supported
> > and not provoke false positive races:
> > http://cr.openjdk.java.net/~jcbeyler/vm_operation/
> >
> > The backstory to this is:
> >    - When a thread is going to do certain JVMTI operations (or other
> btw),
> > it can do a VM operation. When this happens, there might be a callback
> into
> > user code which could change the state of the user memory.
> >      - If this happens, TSAN did not see that the thread actually has
> > passed owner ship to the VM thread via a vm event queue and wait
> >    - To solve this, the best solution I have found is to tell TSAN of the
> > thread memory via an acquire/release on the thread object itself
> >
> > This thus becomes (note all mentions of locking below are fictitious and
> > just calls to TSAN to make it believe all is well in the world):
> >    - A thread running has a "lock" on its on JavaThread object
> >    - When it is going to wait for a VMThread to do something (including
> do
> > a callback) it relinquishes its "lock" and then waits (if the operation
> is
> > concurrent)
> >    - Then the VM thread gets that "lock" and does its operation;
> including
> > doing callbacks
> >     - When the VM thread is done, it relinquishes the "lock" again and
> the
> > JavaThread can acquire it back
> >
> > This in essence does seem to work, the JvmtiTaggerTest no longer
> requires a
> > user lock on its own memory. The down-side is that now we have to track
> > this thread "lock" and I had to instrument the start of a thread.
> >
> > Thanks,
> > Jc
> >
>

From aeubanks at google.com  Wed May 15 17:34:18 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 15 May 2019 10:34:18 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
 <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
 <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>
Message-ID: <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>

>
> I don't think this approach fixes all the issues. We want to capture the
> happens-before edge at a safepoint between all Java threads and the VM
> thread, but this approach only captures the edge between the Java thread
> that calls jvmti->IterateThroughHeap() with the VM thread.
>
> Consider the following example:
> T1, T2 are Java threads executing JNI code.
>
> static int sum = 0;
>
> T1:
> sum = 10;
>
> T2:
> jvmti->IterateThroughHeap(&callback, &sum);
>
> VMThread:
> TSAN_RAW_LOCK_ACQUIRED(T2)
> *sum_ptr += 1;
> TSAN_RAW_LOCK_RELEASED(T2)
>
> Then it will still report race between T1 and the VM thread.
>
Is that not a real race? What's the synchronization between T1 and
T2/VMThread?

>
> Another question, should we call TSAN_RAW_LOCK_CREATE and
> TSAN_RAW_LOCK_DESTROY for the fake per-thread lock also?
>
Yes that's probably necessary for correctness since if a thread is
destroyed and another one is created in its place, TSAN will think it's the
same lock (I think?).

>
> -Man
>

From jcbeyler at google.com  Wed May 15 18:11:48 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 15 May 2019 11:11:48 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
 <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
 <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>
 <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>
Message-ID: <CAF9BGBwPMkTxZ3-yGzqYgA1Dd9Wzoi_vtd7yHHUhbL4c0YdcAg@mail.gmail.com>

Hi Man,

I believe as Arthur that your example is a real race. There's nothing we
can do about that case and we should actually expose it as a race in my
opinion.

For the create/destroy, I'll add that to the webrev and do testing and I'll
do some clean up to have a class do the acquire/release as Arthur pointed
out. I hesitated when I did it but you've convinced me :)
Jc

*From: *Arthur Eubanks <aeubanks at google.com>
*Date: *Wed, May 15, 2019 at 10:34 AM
*To: *Man Cao
*Cc: * <tsan-dev at openjdk.java.net>, Jean Christophe Beyler

I don't think this approach fixes all the issues. We want to capture the
>> happens-before edge at a safepoint between all Java threads and the VM
>> thread, but this approach only captures the edge between the Java thread
>> that calls jvmti->IterateThroughHeap() with the VM thread.
>>
>> Consider the following example:
>> T1, T2 are Java threads executing JNI code.
>>
>> static int sum = 0;
>>
>> T1:
>> sum = 10;
>>
>> T2:
>> jvmti->IterateThroughHeap(&callback, &sum);
>>
>> VMThread:
>> TSAN_RAW_LOCK_ACQUIRED(T2)
>> *sum_ptr += 1;
>> TSAN_RAW_LOCK_RELEASED(T2)
>>
>> Then it will still report race between T1 and the VM thread.
>>
> Is that not a real race? What's the synchronization between T1 and
> T2/VMThread?
>
>>
>> Another question, should we call TSAN_RAW_LOCK_CREATE and
>> TSAN_RAW_LOCK_DESTROY for the fake per-thread lock also?
>>
> Yes that's probably necessary for correctness since if a thread is
> destroyed and another one is created in its place, TSAN will think it's the
> same lock (I think?).
>
>>
>> -Man
>>
>

-- 

Thanks,
Jc

From manc at google.com  Wed May 15 19:03:06 2019
From: manc at google.com (Man Cao)
Date: Wed, 15 May 2019 12:03:06 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
 <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
 <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>
 <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>
Message-ID: <CA+w6HxZEgDU1KuSDQDm9YA_0bW=qiGEj7mjdm9haonRmTA8wjQ@mail.gmail.com>

I don't think there's a race. T2 does not have any synchronization with T1,
but T2 does not read or write to sum.

T1 and the VM thread are synchronized by the stop-the-world safe point,
which behaves as a global barrier, so there is no race either.

-Man


On Wed, May 15, 2019, 10:34 Arthur Eubanks <aeubanks at google.com> wrote:

> I don't think this approach fixes all the issues. We want to capture the
>> happens-before edge at a safepoint between all Java threads and the VM
>> thread, but this approach only captures the edge between the Java thread
>> that calls jvmti->IterateThroughHeap() with the VM thread.
>>
>> Consider the following example:
>> T1, T2 are Java threads executing JNI code.
>>
>> static int sum = 0;
>>
>> T1:
>> sum = 10;
>>
>> T2:
>> jvmti->IterateThroughHeap(&callback, &sum);
>>
>> VMThread:
>> TSAN_RAW_LOCK_ACQUIRED(T2)
>> *sum_ptr += 1;
>> TSAN_RAW_LOCK_RELEASED(T2)
>>
>> Then it will still report race between T1 and the VM thread.
>>
> Is that not a real race? What's the synchronization between T1 and
> T2/VMThread?
>
>>
>> Another question, should we call TSAN_RAW_LOCK_CREATE and
>> TSAN_RAW_LOCK_DESTROY for the fake per-thread lock also?
>>
> Yes that's probably necessary for correctness since if a thread is
> destroyed and another one is created in its place, TSAN will think it's the
> same lock (I think?).
>
>>
>> -Man
>>
>

From jcbeyler at google.com  Thu May 16 01:04:42 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 15 May 2019 18:04:42 -0700
Subject: RFR (S): Supporting VM operations
In-Reply-To: <CA+w6HxZEgDU1KuSDQDm9YA_0bW=qiGEj7mjdm9haonRmTA8wjQ@mail.gmail.com>
References: <CAF9BGBzTUDbxbxTDwUx0ykoZCaKND5MJCox9G-enPwSCJqdUjA@mail.gmail.com>
 <CAPW48soj27a3byhSHDuD7TCcL1b+_M=tVwtZq6FVDCoU9Echsg@mail.gmail.com>
 <CA+w6HxZPGG4vQT4b6U-nf36A-9b0Vazj-KKo0w=8NNLLzx=C3Q@mail.gmail.com>
 <CAPW48sq+cKQRM6yCYjo3fSVNTJQCPf-k0iugcEQz4XXnzYVyHw@mail.gmail.com>
 <CA+w6HxZEgDU1KuSDQDm9YA_0bW=qiGEj7mjdm9haonRmTA8wjQ@mail.gmail.com>
Message-ID: <CAF9BGByYU4-cHgzh2_=t6o4caem2EQ7Vhxe3LbRw=+ZDfFvZOA@mail.gmail.com>

Fine... I think it is a very weird case and probably is a user mistake BUT
you are right, it's not a race since there are the right locks in place.

I'll dive into safepoint code handling to see if I can move the
lock/release model from the handling of the VM_Operation queue to a more
generic safepoint/VM checked all threads are in a safepoint mechanism.

*From: *Man Cao <manc at google.com>
*Date: *Wed, May 15, 2019 at 12:03 PM
*To: *Arthur Eubanks
*Cc: * <tsan-dev at openjdk.java.net>, Jean Christophe Beyler

I don't think there's a race. T2 does not have any synchronization with T1,
> but T2 does not read or write to sum.
>
> T1 and the VM thread are synchronized by the stop-the-world safe point,
> which behaves as a global barrier, so there is no race either.
>
> -Man
>
>
> On Wed, May 15, 2019, 10:34 Arthur Eubanks <aeubanks at google.com> wrote:
>
>> I don't think this approach fixes all the issues. We want to capture the
>>> happens-before edge at a safepoint between all Java threads and the VM
>>> thread, but this approach only captures the edge between the Java thread
>>> that calls jvmti->IterateThroughHeap() with the VM thread.
>>>
>>> Consider the following example:
>>> T1, T2 are Java threads executing JNI code.
>>>
>>> static int sum = 0;
>>>
>>> T1:
>>> sum = 10;
>>>
>>> T2:
>>> jvmti->IterateThroughHeap(&callback, &sum);
>>>
>>> VMThread:
>>> TSAN_RAW_LOCK_ACQUIRED(T2)
>>> *sum_ptr += 1;
>>> TSAN_RAW_LOCK_RELEASED(T2)
>>>
>>> Then it will still report race between T1 and the VM thread.
>>>
>> Is that not a real race? What's the synchronization between T1 and
>> T2/VMThread?
>>
>>>
>>> Another question, should we call TSAN_RAW_LOCK_CREATE and
>>> TSAN_RAW_LOCK_DESTROY for the fake per-thread lock also?
>>>
>> Yes that's probably necessary for correctness since if a thread is
>> destroyed and another one is created in its place, TSAN will think it's the
>> same lock (I think?).
>>
>>>
>>> -Man
>>>
>>

-- 

Thanks,
Jc

From aeubanks at google.com  Mon May 20 20:27:46 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 20 May 2019 13:27:46 -0700
Subject: [RFR] Get symbolization properly working
Message-ID: <CAPW48sq0gA4UG507M=m1kUwvh1RQu=L1Poo404rni_C1JiSX7w@mail.gmail.com>

Webrev:
http://cr.openjdk.java.net/~aeubanks/tsansymbolizelauncher/webrev.00/index.html

Move TSAN symbolizer function override to launcher

Symbolization doesn't work because __tsan_symbolize_external_ex() is not in
the launcher but in libjvm.so.

Rename the existing __tsan_symbolize_external_ex() to TsanSymbolize() and
export from libjvm.so. Get a pointer to it in libjli. The launcher can call
TsanSymbolize() through libjli in its override of
__tsan_symbolize_external_ex().


Tested by applying the Java memory instrumentation and running the
RacyIntMemberLoopTest test. Symbolization works (although the package name
is missing from the class name).

From jcbeyler at google.com  Tue May 21 15:28:47 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 21 May 2019 08:28:47 -0700
Subject: [RFR] Get symbolization properly working
In-Reply-To: <CAPW48sq0gA4UG507M=m1kUwvh1RQu=L1Poo404rni_C1JiSX7w@mail.gmail.com>
References: <CAPW48sq0gA4UG507M=m1kUwvh1RQu=L1Poo404rni_C1JiSX7w@mail.gmail.com>
Message-ID: <CAF9BGBzEDSyohGsSHWnWhj0zcOcVnxipsbmjyhDQRr5UQCMRmA@mail.gmail.com>

Looks good to me :),
Jc


On Mon, May 20, 2019 at 1:28 PM Arthur Eubanks <aeubanks at google.com> wrote:

> Webrev:
>
> http://cr.openjdk.java.net/~aeubanks/tsansymbolizelauncher/webrev.00/index.html
>
> Move TSAN symbolizer function override to launcher
>
> Symbolization doesn't work because __tsan_symbolize_external_ex() is not in
> the launcher but in libjvm.so.
>
> Rename the existing __tsan_symbolize_external_ex() to TsanSymbolize() and
> export from libjvm.so. Get a pointer to it in libjli. The launcher can call
> TsanSymbolize() through libjli in its override of
> __tsan_symbolize_external_ex().
>
>
> Tested by applying the Java memory instrumentation and running the
> RacyIntMemberLoopTest test. Symbolization works (although the package name
> is missing from the class name).
>


-- 

Thanks,
Jc

From aeubanks at google.com  Tue May 21 15:42:20 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 21 May 2019 08:42:20 -0700
Subject: [RFR] Get symbolization properly working
In-Reply-To: <CAF9BGBzEDSyohGsSHWnWhj0zcOcVnxipsbmjyhDQRr5UQCMRmA@mail.gmail.com>
References: <CAPW48sq0gA4UG507M=m1kUwvh1RQu=L1Poo404rni_C1JiSX7w@mail.gmail.com>
 <CAF9BGBzEDSyohGsSHWnWhj0zcOcVnxipsbmjyhDQRr5UQCMRmA@mail.gmail.com>
Message-ID: <CAPW48spaBCsViweLRtZ_KsE4s-vHD2=R16qgBu08XpFy-hiAow@mail.gmail.com>

Fixed the "thse" typo in main.c

On Tue, May 21, 2019 at 8:29 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Looks good to me :),
> Jc
>
>
> On Mon, May 20, 2019 at 1:28 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> Webrev:
>>
>> http://cr.openjdk.java.net/~aeubanks/tsansymbolizelauncher/webrev.00/index.html
>>
>> Move TSAN symbolizer function override to launcher
>>
>> Symbolization doesn't work because __tsan_symbolize_external_ex() is not
>> in
>> the launcher but in libjvm.so.
>>
>> Rename the existing __tsan_symbolize_external_ex() to TsanSymbolize() and
>> export from libjvm.so. Get a pointer to it in libjli. The launcher can
>> call
>> TsanSymbolize() through libjli in its override of
>> __tsan_symbolize_external_ex().
>>
>>
>> Tested by applying the Java memory instrumentation and running the
>> RacyIntMemberLoopTest test. Symbolization works (although the package name
>> is missing from the class name).
>>
>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Tue May 21 19:58:12 2019
From: manc at google.com (Man Cao)
Date: Tue, 21 May 2019 12:58:12 -0700
Subject: [RFR] Get symbolization properly working
In-Reply-To: <CAPW48spaBCsViweLRtZ_KsE4s-vHD2=R16qgBu08XpFy-hiAow@mail.gmail.com>
References: <CAPW48sq0gA4UG507M=m1kUwvh1RQu=L1Poo404rni_C1JiSX7w@mail.gmail.com>
 <CAF9BGBzEDSyohGsSHWnWhj0zcOcVnxipsbmjyhDQRr5UQCMRmA@mail.gmail.com>
 <CAPW48spaBCsViweLRtZ_KsE4s-vHD2=R16qgBu08XpFy-hiAow@mail.gmail.com>
Message-ID: <CA+w6HxaaB-DtVVs-q0wCny5xnzG-LUE=xmRBxNxbJYLtC5N08w@mail.gmail.com>

Looks good.

-Man


On Tue, May 21, 2019 at 8:43 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Fixed the "thse" typo in main.c
>
> On Tue, May 21, 2019 at 8:29 AM Jean Christophe Beyler <
> jcbeyler at google.com>
> wrote:
>
> > Looks good to me :),
> > Jc
> >
> >
> > On Mon, May 20, 2019 at 1:28 PM Arthur Eubanks <aeubanks at google.com>
> > wrote:
> >
> >> Webrev:
> >>
> >>
> http://cr.openjdk.java.net/~aeubanks/tsansymbolizelauncher/webrev.00/index.html
> >>
> >> Move TSAN symbolizer function override to launcher
> >>
> >> Symbolization doesn't work because __tsan_symbolize_external_ex() is not
> >> in
> >> the launcher but in libjvm.so.
> >>
> >> Rename the existing __tsan_symbolize_external_ex() to TsanSymbolize()
> and
> >> export from libjvm.so. Get a pointer to it in libjli. The launcher can
> >> call
> >> TsanSymbolize() through libjli in its override of
> >> __tsan_symbolize_external_ex().
> >>
> >>
> >> Tested by applying the Java memory instrumentation and running the
> >> RacyIntMemberLoopTest test. Symbolization works (although the package
> name
> >> is missing from the class name).
> >>
> >
> >
> > --
> >
> > Thanks,
> > Jc
> >
>

From aeubanks at google.com  Tue May 21 20:08:43 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Tue, 21 May 2019 20:08:43 +0000
Subject: hg: tsan/dev: Move TSAN symbolizer function override to launcher
Message-ID: <201905212008.x4LK8hmi000506@aojmv0008.oracle.com>

Changeset: 4451b6504248
Author:    aeubanks
Date:      2019-04-23 09:13 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/4451b6504248

Move TSAN symbolizer function override to launcher

Symbolization doesn't work because __tsan_symbolize_external_ex() is not in the launcher but in libjvm.so.

Rename the existing __tsan_symbolize_external_ex() to TsanSymbolize() and export from libjvm.so. Get a pointer to it in libjli. The launcher can call TsanSymbolize() through libjli in its override of __tsan_symbolize_external_ex().

! make/hotspot/symbols/symbols-shared
! make/launcher/LauncherCommon.gmk
! make/lib/CoreLibraries.gmk
! src/hotspot/share/tsan/tsan.cpp
! src/java.base/share/native/launcher/main.c
! src/java.base/share/native/libjli/java.c
! src/java.base/share/native/libjli/java.h
! src/java.base/unix/native/libjli/java_md_solinux.c


From aeubanks at google.com  Tue May 21 22:36:31 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 21 May 2019 15:36:31 -0700
Subject: [RFR] Java memory instrumentation for TSAN
Message-ID: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>

webrev:
http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.00/index.html

This change adds instrumentation for Java memory accesses.

From manc at google.com  Wed May 22 18:12:59 2019
From: manc at google.com (Man Cao)
Date: Wed, 22 May 2019 11:12:59 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
Message-ID: <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>

Much of the code is not guarded by the INCLUDE_TSAN or TSAN_RUNTIME_ONLY
macro.
Can we update it accordingly?

-Man


On Tue, May 21, 2019 at 3:37 PM Arthur Eubanks <aeubanks at google.com> wrote:

> webrev:
> http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.00/index.html
>
> This change adds instrumentation for Java memory accesses.
>

From aeubanks at google.com  Wed May 22 23:11:30 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 22 May 2019 16:11:30 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
 <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
Message-ID: <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>

Sorry about that, this change was done before all this was in place and I
forgot to update it. Updated in
http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.01/index.html.

On Wed, May 22, 2019 at 11:13 AM Man Cao <manc at google.com> wrote:

> Much of the code is not guarded by the INCLUDE_TSAN or TSAN_RUNTIME_ONLY
> macro.
> Can we update it accordingly?
>
> -Man
>
>
> On Tue, May 21, 2019 at 3:37 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> webrev:
>> http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.00/index.html
>>
>> This change adds instrumentation for Java memory accesses.
>>
>

From manc at google.com  Fri May 24 21:20:15 2019
From: manc at google.com (Man Cao)
Date: Fri, 24 May 2019 14:20:15 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
 <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
 <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>
Message-ID: <CA+w6HxY8mp-2VFMDFtc5E3YqSk63re-F__6J4Zo9jSy1KhM7=g@mail.gmail.com>

We can add the "ThreadSanitizerJavaMem" flag, so it is consistent with our
internal implementation.
We can set ThreadSanitizerJavaMem's default value to true here, because all
the uses of this flag should be guarded by "if (ThreadSanitizer)".

Only the following places need to be changed to use ThreadSanitizerJavaMem:
In sharedRuntime.cpp:
#define TSAN_MEMORY_ACCESS(name)
...
assert(ThreadSanitizer, "Need -XX:+ThreadSanitizer");

In templateTable_x86.cpp:
TemplateTable::tsan_observe_get_or_put()
....
 if (ThreadSanitizer) {
  __ jmp(Done);

TemplateTable::tsan_observe_load_or_store()
It can add an assert for ThreadSanitizer, and check
for ThreadSanitizerJavaMem instead.

Other minor issues:
In arguments.cpp, can we change it to:
TSAN_RUNTIME_ONLY(
  set_mode_flags..
  FLAG_SET_ERGO..
  FLAG_SET_ERGO..
);

In instanceKlass.cpp:
Can we move "#if INCLUDE_TSAN \n #include "runtime/sharedRuntime.hpp""to
the end of the include statements, so to consistent with other conditional
includes?

In templateTable_x86.cpp:
In TemplateTable::tsan_release_acquire_method()
 785   ShouldNotReachHere();
 786   return NULL;

Can we remove "return NULL"? Or would it trigger a compiler warning?

In TemplateTable::load_field_cp_cache_entry():
2883 #ifdef INCLUDE_TSAN
2884 if (ThreadSanitizer) {
2885  // Draw a happens-before edge from the class's static initializer to
2886  // this lookup.

This should use TSAN_RUNTIME_ONLY instead.

-Man

On Wed, May 22, 2019 at 4:11 PM Arthur Eubanks <aeubanks at google.com> wrote:

> Sorry about that, this change was done before all this was in place and I
> forgot to update it. Updated in
> http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.01/index.html.
>
> On Wed, May 22, 2019 at 11:13 AM Man Cao <manc at google.com> wrote:
>
>> Much of the code is not guarded by the INCLUDE_TSAN or TSAN_RUNTIME_ONLY
>> macro.
>> Can we update it accordingly?
>>
>> -Man
>>
>>
>> On Tue, May 21, 2019 at 3:37 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> webrev:
>>> http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.00/index.html
>>>
>>> This change adds instrumentation for Java memory accesses.
>>>
>>

From aeubanks at google.com  Tue May 28 20:02:51 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 28 May 2019 13:02:51 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CA+w6HxY8mp-2VFMDFtc5E3YqSk63re-F__6J4Zo9jSy1KhM7=g@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
 <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
 <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>
 <CA+w6HxY8mp-2VFMDFtc5E3YqSk63re-F__6J4Zo9jSy1KhM7=g@mail.gmail.com>
Message-ID: <CAPW48sohpDZNj31-EyS-QqRq+hGQh9xifPrgbmPvwjvZDLH0zg@mail.gmail.com>

On Fri, May 24, 2019 at 2:20 PM Man Cao <manc at google.com> wrote:

> We can add the "ThreadSanitizerJavaMem" flag, so it is consistent with our
> internal implementation.
>
Added a new flag named "ThreadSanitizerJavaMemory" (no point in shortening
the name by 3 chars).

> We can set ThreadSanitizerJavaMem's default value to true here, because
> all the uses of this flag should be guarded by "if (ThreadSanitizer)".
>
> Only the following places need to be changed to use ThreadSanitizerJavaMem:
> In sharedRuntime.cpp:
> #define TSAN_MEMORY_ACCESS(name)
> ...
> assert(ThreadSanitizer, "Need -XX:+ThreadSanitizer");
>
Done (asserted both ThreadSanitizer and ThreadSanitizerJavaMem).

>
> In templateTable_x86.cpp:
> TemplateTable::tsan_observe_get_or_put()
> ....
>  if (ThreadSanitizer) {
>   __ jmp(Done);
>
Done.

>
> TemplateTable::tsan_observe_load_or_store()
> It can add an assert for ThreadSanitizer, and check
> for ThreadSanitizerJavaMem instead.
>
Done.

>
> Other minor issues:
> In arguments.cpp, can we change it to:
> TSAN_RUNTIME_ONLY(
>   set_mode_flags..
>   FLAG_SET_ERGO..
>   FLAG_SET_ERGO..
> );
>
Done. I didn't want to do this initially because I think it's ugly to have
a semicolon in the statements and at the end of the macro, but oh well.

>
> In instanceKlass.cpp:
> Can we move "#if INCLUDE_TSAN \n #include "runtime/sharedRuntime.hpp""to
> the end of the include statements, so to consistent with other conditional
> includes?
>
Done.

>
> In templateTable_x86.cpp:
> In TemplateTable::tsan_release_acquire_method()
>  785   ShouldNotReachHere();
>  786   return NULL;
>
> Can we remove "return NULL"? Or would it trigger a compiler warning?
>
Yeah it triggers a compiler warning.

>
> In TemplateTable::load_field_cp_cache_entry():
> 2883 #ifdef INCLUDE_TSAN
> 2884 if (ThreadSanitizer) {
> 2885  // Draw a happens-before edge from the class's static initializer to
> 2886  // this lookup.
>
> This should use TSAN_RUNTIME_ONLY instead.
>
> -Man
>

also added a test to check that there are no races when
ThreadSanitizerJavaMemory is turned off.

new webrev: http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/

From manc at google.com  Wed May 29 15:55:10 2019
From: manc at google.com (Man Cao)
Date: Wed, 29 May 2019 08:55:10 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CAPW48sohpDZNj31-EyS-QqRq+hGQh9xifPrgbmPvwjvZDLH0zg@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
 <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
 <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>
 <CA+w6HxY8mp-2VFMDFtc5E3YqSk63re-F__6J4Zo9jSy1KhM7=g@mail.gmail.com>
 <CAPW48sohpDZNj31-EyS-QqRq+hGQh9xifPrgbmPvwjvZDLH0zg@mail.gmail.com>
Message-ID: <CA+w6HxbJyEhCC0e+2OHDVeYiH8fQ8rFmxmiuabTgZGhiMfQnJw@mail.gmail.com>

Looks good.

-Man


On Tue, May 28, 2019 at 1:03 PM Arthur Eubanks <aeubanks at google.com> wrote:

> On Fri, May 24, 2019 at 2:20 PM Man Cao <manc at google.com> wrote:
>
>> We can add the "ThreadSanitizerJavaMem" flag, so it is consistent with
>> our internal implementation.
>>
> Added a new flag named "ThreadSanitizerJavaMemory" (no point in shortening
> the name by 3 chars).
>
>> We can set ThreadSanitizerJavaMem's default value to true here, because
>> all the uses of this flag should be guarded by "if (ThreadSanitizer)".
>>
>> Only the following places need to be changed to
>> use ThreadSanitizerJavaMem:
>> In sharedRuntime.cpp:
>> #define TSAN_MEMORY_ACCESS(name)
>> ...
>> assert(ThreadSanitizer, "Need -XX:+ThreadSanitizer");
>>
> Done (asserted both ThreadSanitizer and ThreadSanitizerJavaMem).
>
>>
>> In templateTable_x86.cpp:
>> TemplateTable::tsan_observe_get_or_put()
>> ....
>>  if (ThreadSanitizer) {
>>   __ jmp(Done);
>>
> Done.
>
>>
>> TemplateTable::tsan_observe_load_or_store()
>> It can add an assert for ThreadSanitizer, and check
>> for ThreadSanitizerJavaMem instead.
>>
> Done.
>
>>
>> Other minor issues:
>> In arguments.cpp, can we change it to:
>> TSAN_RUNTIME_ONLY(
>>   set_mode_flags..
>>   FLAG_SET_ERGO..
>>   FLAG_SET_ERGO..
>> );
>>
> Done. I didn't want to do this initially because I think it's ugly to have
> a semicolon in the statements and at the end of the macro, but oh well.
>
>>
>> In instanceKlass.cpp:
>> Can we move "#if INCLUDE_TSAN \n #include "runtime/sharedRuntime.hpp""to
>> the end of the include statements, so to consistent with other conditional
>> includes?
>>
> Done.
>
>>
>> In templateTable_x86.cpp:
>> In TemplateTable::tsan_release_acquire_method()
>>  785   ShouldNotReachHere();
>>  786   return NULL;
>>
>> Can we remove "return NULL"? Or would it trigger a compiler warning?
>>
> Yeah it triggers a compiler warning.
>
>>
>> In TemplateTable::load_field_cp_cache_entry():
>> 2883 #ifdef INCLUDE_TSAN
>> 2884 if (ThreadSanitizer) {
>> 2885  // Draw a happens-before edge from the class's static initializer to
>> 2886  // this lookup.
>>
>> This should use TSAN_RUNTIME_ONLY instead.
>>
>> -Man
>>
>
> also added a test to check that there are no races when
> ThreadSanitizerJavaMemory is turned off.
>
> new webrev: http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/
>

From jcbeyler at google.com  Wed May 29 18:15:39 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 29 May 2019 11:15:39 -0700
Subject: [RFR] Java memory instrumentation for TSAN
In-Reply-To: <CA+w6HxbJyEhCC0e+2OHDVeYiH8fQ8rFmxmiuabTgZGhiMfQnJw@mail.gmail.com>
References: <CAPW48srwV85P6hRYo5X4TL6k1fX-q7mN=+g0F9-yaYrMbaLkqQ@mail.gmail.com>
 <CA+w6Hxax6UZpyFRrA7tJCC7YZYJTnpMBvDAca7QUCGZnN4DLeQ@mail.gmail.com>
 <CAPW48sqM1he+60wW9t2QG0qm6fQJMOJUotSxtbZSOt+jiMLkRQ@mail.gmail.com>
 <CA+w6HxY8mp-2VFMDFtc5E3YqSk63re-F__6J4Zo9jSy1KhM7=g@mail.gmail.com>
 <CAPW48sohpDZNj31-EyS-QqRq+hGQh9xifPrgbmPvwjvZDLH0zg@mail.gmail.com>
 <CA+w6HxbJyEhCC0e+2OHDVeYiH8fQ8rFmxmiuabTgZGhiMfQnJw@mail.gmail.com>
Message-ID: <CAF9BGByhW7Q3JH2vhMhBrg6LDiBN6ryawz4MAPEC5v4vm5QjqA@mail.gmail.com>

Hi all,

Sorry I'm late at the party:

http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/src/hotspot/cpu/x86/templateTable_x86.hpp.udiff.html
that an array has ben
  -> been

http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/src/hotspot/share/runtime/arguments.cpp.udiff.html
  -> down the pplication
    -> down the application

http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/src/hotspot/cpu/x86/templateTable_x86.cpp.udiff.html
  -> Label Done
    - Label done

Apart from that, it looks good to me, no need of a new webrev :)
Jc

On Wed, May 29, 2019 at 8:56 AM Man Cao <manc at google.com> wrote:

> Looks good.
>
> -Man
>
>
> On Tue, May 28, 2019 at 1:03 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > On Fri, May 24, 2019 at 2:20 PM Man Cao <manc at google.com> wrote:
> >
> >> We can add the "ThreadSanitizerJavaMem" flag, so it is consistent with
> >> our internal implementation.
> >>
> > Added a new flag named "ThreadSanitizerJavaMemory" (no point in
> shortening
> > the name by 3 chars).
> >
> >> We can set ThreadSanitizerJavaMem's default value to true here, because
> >> all the uses of this flag should be guarded by "if (ThreadSanitizer)".
> >>
> >> Only the following places need to be changed to
> >> use ThreadSanitizerJavaMem:
> >> In sharedRuntime.cpp:
> >> #define TSAN_MEMORY_ACCESS(name)
> >> ...
> >> assert(ThreadSanitizer, "Need -XX:+ThreadSanitizer");
> >>
> > Done (asserted both ThreadSanitizer and ThreadSanitizerJavaMem).
> >
> >>
> >> In templateTable_x86.cpp:
> >> TemplateTable::tsan_observe_get_or_put()
> >> ....
> >>  if (ThreadSanitizer) {
> >>   __ jmp(Done);
> >>
> > Done.
> >
> >>
> >> TemplateTable::tsan_observe_load_or_store()
> >> It can add an assert for ThreadSanitizer, and check
> >> for ThreadSanitizerJavaMem instead.
> >>
> > Done.
> >
> >>
> >> Other minor issues:
> >> In arguments.cpp, can we change it to:
> >> TSAN_RUNTIME_ONLY(
> >>   set_mode_flags..
> >>   FLAG_SET_ERGO..
> >>   FLAG_SET_ERGO..
> >> );
> >>
> > Done. I didn't want to do this initially because I think it's ugly to
> have
> > a semicolon in the statements and at the end of the macro, but oh well.
> >
> >>
> >> In instanceKlass.cpp:
> >> Can we move "#if INCLUDE_TSAN \n #include "runtime/sharedRuntime.hpp""to
> >> the end of the include statements, so to consistent with other
> conditional
> >> includes?
> >>
> > Done.
> >
> >>
> >> In templateTable_x86.cpp:
> >> In TemplateTable::tsan_release_acquire_method()
> >>  785   ShouldNotReachHere();
> >>  786   return NULL;
> >>
> >> Can we remove "return NULL"? Or would it trigger a compiler warning?
> >>
> > Yeah it triggers a compiler warning.
> >
> >>
> >> In TemplateTable::load_field_cp_cache_entry():
> >> 2883 #ifdef INCLUDE_TSAN
> >> 2884 if (ThreadSanitizer) {
> >> 2885  // Draw a happens-before edge from the class's static initializer
> to
> >> 2886  // this lookup.
> >>
> >> This should use TSAN_RUNTIME_ONLY instead.
> >>
> >> -Man
> >>
> >
> > also added a test to check that there are no races when
> > ThreadSanitizerJavaMemory is turned off.
> >
> > new webrev: http://cr.openjdk.java.net/~aeubanks/tsanjavamem/webrev.02/
> >
>


-- 

Thanks,
Jc

From aeubanks at google.com  Wed May 29 18:21:30 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Wed, 29 May 2019 18:21:30 +0000
Subject: hg: tsan/dev: Add Java memory instrumentation
Message-ID: <201905291821.x4TILULe010798@aojmv0008.oracle.com>

Changeset: bfdc93046b52
Author:    aeubanks
Date:      2019-04-23 09:13 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/bfdc93046b52

Add Java memory instrumentation

Now we can detect Java races.

! src/hotspot/cpu/x86/templateTable_x86.cpp
! src/hotspot/cpu/x86/templateTable_x86.hpp
! src/hotspot/share/classfile/javaClasses.cpp
! src/hotspot/share/classfile/javaClasses.hpp
! src/hotspot/share/oops/instanceKlass.cpp
! src/hotspot/share/runtime/arguments.cpp
! src/hotspot/share/runtime/globals.hpp
! src/hotspot/share/runtime/sharedRuntime.cpp
! src/hotspot/share/runtime/sharedRuntime.hpp
! src/hotspot/share/tsan/tsanExternalDecls.hpp
! test/hotspot/jtreg/tsan/JvmtiTaggerTest.java
! test/hotspot/jtreg/tsan/NonRawRacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/RacyIntMemberLoopTest.java
+ test/hotspot/jtreg/tsan/RacyIntMemberNoJavaMemLoopTest.java
! test/hotspot/jtreg/tsan/RacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/TsanRunner.java