From aeubanks at google.com  Mon Apr  1 15:34:50 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 1 Apr 2019 08:34:50 -0700
Subject: RFR (M): Make TSAN a build-time JVM feature
In-Reply-To: <CAF9BGByzo40fiVbA2fYpTG31DHyGTqBsGqBYtHghGhh_RnsGhA@mail.gmail.com>
References: <CA+w6Hxaa4wLd3W+hhHYP9BkNKEjo9LjhJSpCqankgLdKsQ8e7Q@mail.gmail.com>
 <CA+w6HxadR8+WJw8hBOy6DQQa28PfdKCjGP583DsHFfy60FJ2fw@mail.gmail.com>
 <CAF9BGByzo40fiVbA2fYpTG31DHyGTqBsGqBYtHghGhh_RnsGhA@mail.gmail.com>
Message-ID: <CAPW48sokYkaw+bBixtZX6VyBLha57e4aRXhDQnUryUjqhYG2gg@mail.gmail.com>

nit: in make/hotspot/lib/JvmFeatures.gmk don't add a trailing slash, the
others JVM_EXCLUDE_PATTERNS mostly don't have a trailing slash. No need for
new webrev.

otherwise LGTM

On Fri, Mar 29, 2019 at 7:51 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Agreed Man on all points,
>
> LGTM,
> Jc
>
> On Fri, Mar 29, 2019 at 7:14 PM Man Cao <manc at google.com> wrote:
>
> > Note that I moved the two new files for TSAN into the directory
> > "share/tsan".
> > Looking at our internal TSAN code, there will be more new files for TSAN
> to
> > add in the future. A separate directory seems cleaner.
> >
> > -Man
> >
> >
> > On Fri, Mar 29, 2019 at 7:10 PM Man Cao <manc at google.com> wrote:
> >
> > > Hi,
> > >
> > > Could I get review for this:
> > > http://cr.openjdk.java.net/~manc/tsan/20190329/webrev.00/
> > >
> > > It makes it possible to disable TSAN at build time, using
> > > "--with-jvm-features=-tsan". My initial purpose was to add a "#ifdef"
> > macro
> > > for TSAN, but it seems making TSAN a JVM feature is the right way to go
> > for
> > > the long term.
> > >
> > > Tested by building with and without TSAN, and running LD_PRELOAD.
> > >
> > > -Man
> > >
> >
>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Mon Apr  1 17:48:53 2019
From: manc at google.com (Man Cao)
Date: Mon, 1 Apr 2019 10:48:53 -0700
Subject: RFR (M): Make TSAN a build-time JVM feature
In-Reply-To: <CAPW48sokYkaw+bBixtZX6VyBLha57e4aRXhDQnUryUjqhYG2gg@mail.gmail.com>
References: <CA+w6Hxaa4wLd3W+hhHYP9BkNKEjo9LjhJSpCqankgLdKsQ8e7Q@mail.gmail.com>
 <CA+w6HxadR8+WJw8hBOy6DQQa28PfdKCjGP583DsHFfy60FJ2fw@mail.gmail.com>
 <CAF9BGByzo40fiVbA2fYpTG31DHyGTqBsGqBYtHghGhh_RnsGhA@mail.gmail.com>
 <CAPW48sokYkaw+bBixtZX6VyBLha57e4aRXhDQnUryUjqhYG2gg@mail.gmail.com>
Message-ID: <CA+w6HxabREugmjz2FKKAB+E=UKPNsLzcn51znRt3B=db6CYMaA@mail.gmail.com>

Done.

-Man


On Mon, Apr 1, 2019 at 8:35 AM Arthur Eubanks <aeubanks at google.com> wrote:

> nit: in make/hotspot/lib/JvmFeatures.gmk don't add a trailing slash, the
> others JVM_EXCLUDE_PATTERNS mostly don't have a trailing slash. No need for
> new webrev.
>
> otherwise LGTM
>
> On Fri, Mar 29, 2019 at 7:51 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Agreed Man on all points,
>>
>> LGTM,
>> Jc
>>
>> On Fri, Mar 29, 2019 at 7:14 PM Man Cao <manc at google.com> wrote:
>>
>> > Note that I moved the two new files for TSAN into the directory
>> > "share/tsan".
>> > Looking at our internal TSAN code, there will be more new files for
>> TSAN to
>> > add in the future. A separate directory seems cleaner.
>> >
>> > -Man
>> >
>> >
>> > On Fri, Mar 29, 2019 at 7:10 PM Man Cao <manc at google.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > Could I get review for this:
>> > > http://cr.openjdk.java.net/~manc/tsan/20190329/webrev.00/
>> > >
>> > > It makes it possible to disable TSAN at build time, using
>> > > "--with-jvm-features=-tsan". My initial purpose was to add a "#ifdef"
>> > macro
>> > > for TSAN, but it seems making TSAN a JVM feature is the right way to
>> go
>> > for
>> > > the long term.
>> > >
>> > > Tested by building with and without TSAN, and running LD_PRELOAD.
>> > >
>> > > -Man
>> > >
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From manc at google.com  Mon Apr  1 17:49:48 2019
From: manc at google.com (manc at google.com)
Date: Mon, 01 Apr 2019 17:49:48 +0000
Subject: hg: tsan/dev: Make tsan a build-time JVM feature
Message-ID: <201904011749.x31HnnFo020197@aojmv0008.oracle.com>

Changeset: 6e020afc1a13
Author:    manc
Date:      2019-03-29 18:27 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/6e020afc1a13

Make tsan a build-time JVM feature

! make/autoconf/hotspot.m4
! make/hotspot/lib/JvmFeatures.gmk
! src/hotspot/cpu/x86/interp_masm_x86.cpp
! src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
! src/hotspot/share/runtime/arguments.cpp
! src/hotspot/share/runtime/globals.hpp
! src/hotspot/share/runtime/init.cpp
! src/hotspot/share/runtime/sharedRuntime.cpp
! src/hotspot/share/runtime/sharedRuntime.hpp
- src/hotspot/share/runtime/tsan.cpp
+ src/hotspot/share/tsan/tsan.cpp
! src/hotspot/share/tsan/tsanExternalDecls.hpp < src/hotspot/share/runtime/tsanExternalDecls.hpp
! src/hotspot/share/utilities/macros.hpp


From aeubanks at google.com  Tue Apr  2 00:03:51 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 1 Apr 2019 17:03:51 -0700
Subject: [RFR]: Stack frame symbolization
Message-ID: <CAPW48spwLAQ60RKde9+V55JrQXwZ__nyDqTWtF4QtvwyTJX92g@mail.gmail.com>

Here's a first attempt at symbolizing stack frames. We'll have to wait
until TSAN works before we can verify that this actually works...

http://cr.openjdk.java.net/~aeubanks/tsansymbolize/webrev.00

From jcbeyler at google.com  Tue Apr  2 02:03:58 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 1 Apr 2019 19:03:58 -0700
Subject: [RFR]: Stack frame symbolization
In-Reply-To: <CAPW48spwLAQ60RKde9+V55JrQXwZ__nyDqTWtF4QtvwyTJX92g@mail.gmail.com>
References: <CAPW48spwLAQ60RKde9+V55JrQXwZ__nyDqTWtF4QtvwyTJX92g@mail.gmail.com>
Message-ID: <CAF9BGBxXXupuwF9-fZZggPLT3=bYOi2JEYdKx9LRKyCHRMTswg@mail.gmail.com>

Hi Arthur,

Looks good  to me, as you said, we will only know when it gets used... :)

However, I wonder why we are doing our own bit manipulations and not using
a bitfield? Any reason for that?

Thanks,
Jc

On Mon, Apr 1, 2019 at 5:05 PM Arthur Eubanks <aeubanks at google.com> wrote:

> Here's a first attempt at symbolizing stack frames. We'll have to wait
> until TSAN works before we can verify that this actually works...
>
> http://cr.openjdk.java.net/~aeubanks/tsansymbolize/webrev.00
>


-- 

Thanks,
Jc

From manc at google.com  Tue Apr  2 04:13:52 2019
From: manc at google.com (Man Cao)
Date: Mon, 1 Apr 2019 21:13:52 -0700
Subject: [RFR]: Stack frame symbolization
In-Reply-To: <CAF9BGBxXXupuwF9-fZZggPLT3=bYOi2JEYdKx9LRKyCHRMTswg@mail.gmail.com>
References: <CAPW48spwLAQ60RKde9+V55JrQXwZ__nyDqTWtF4QtvwyTJX92g@mail.gmail.com>
 <CAF9BGBxXXupuwF9-fZZggPLT3=bYOi2JEYdKx9LRKyCHRMTswg@mail.gmail.com>
Message-ID: <CA+w6HxbAo2XMrSVF90iHOKyDzROm75CeO7Ox3fe=1u+OnAJ_gw@mail.gmail.com>

Looks good as well.

> However, I wonder why we are doing our own bit manipulations and not using
> a bitfield?  Any reason for that?
I guess it might be due to the fact that bit field has some
implementation-defined behaviors, and we want TSAN's data structures to
have rock-solid exact number of bits.

-Man


On Mon, Apr 1, 2019 at 7:05 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Arthur,
>
> Looks good  to me, as you said, we will only know when it gets used... :)
>
> However, I wonder why we are doing our own bit manipulations and not using
> a bitfield? Any reason for that?
>
> Thanks,
> Jc
>
> On Mon, Apr 1, 2019 at 5:05 PM Arthur Eubanks <aeubanks at google.com> wrote:
>
> > Here's a first attempt at symbolizing stack frames. We'll have to wait
> > until TSAN works before we can verify that this actually works...
> >
> > http://cr.openjdk.java.net/~aeubanks/tsansymbolize/webrev.00
> >
>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr  2 14:37:43 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 2 Apr 2019 07:37:43 -0700
Subject: [RFR]: Stack frame symbolization
In-Reply-To: <CA+w6HxbAo2XMrSVF90iHOKyDzROm75CeO7Ox3fe=1u+OnAJ_gw@mail.gmail.com>
References: <CAPW48spwLAQ60RKde9+V55JrQXwZ__nyDqTWtF4QtvwyTJX92g@mail.gmail.com>
 <CAF9BGBxXXupuwF9-fZZggPLT3=bYOi2JEYdKx9LRKyCHRMTswg@mail.gmail.com>
 <CA+w6HxbAo2XMrSVF90iHOKyDzROm75CeO7Ox3fe=1u+OnAJ_gw@mail.gmail.com>
Message-ID: <CAF9BGBxvtrA_Fz3GQ5zosqxsAyCrU8YKT9ZkmShrXUe2A9QqtA@mail.gmail.com>

On Mon, Apr 1, 2019 at 9:14 PM Man Cao <manc at google.com> wrote:

> Looks good as well.
>
> > However, I wonder why we are doing our own bit manipulations and not
> using
> > a bitfield?  Any reason for that?
> I guess it might be due to the fact that bit field has some
> implementation-defined behaviors, and we want TSAN's data structures to
> have rock-solid exact number of bits.
>
>
How we learn nitty gritty details sometimes. This is a difference between
the C-norm and the C++ norm, which behave differently regarding bitfields.

LGTM,
Jc



> -Man
>
>
> On Mon, Apr 1, 2019 at 7:05 PM Jean Christophe Beyler <jcbeyler at google.com>
> wrote:
>
>> Hi Arthur,
>>
>> Looks good  to me, as you said, we will only know when it gets used... :)
>>
>> However, I wonder why we are doing our own bit manipulations and not using
>> a bitfield? Any reason for that?
>>
>> Thanks,
>> Jc
>>
>> On Mon, Apr 1, 2019 at 5:05 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> > Here's a first attempt at symbolizing stack frames. We'll have to wait
>> > until TSAN works before we can verify that this actually works...
>> >
>> > http://cr.openjdk.java.net/~aeubanks/tsansymbolize/webrev.00
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Tue Apr  2 15:05:55 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Tue, 02 Apr 2019 15:05:55 +0000
Subject: hg: tsan/dev: Add code to symbolize stack traces
Message-ID: <201904021505.x32F5tXU025700@aojmv0008.oracle.com>

Changeset: 2cb59b7088e4
Author:    aeubanks
Date:      2019-04-01 16:57 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/2cb59b7088e4

Add code to symbolize stack traces

! src/hotspot/share/runtime/sharedRuntime.hpp
! src/hotspot/share/tsan/tsan.cpp


From aeubanks at google.com  Tue Apr  2 22:39:52 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 2 Apr 2019 15:39:52 -0700
Subject: [RFR]: Update/add some tests
Message-ID: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>

http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html

I'll let the commit message do the talking:

Add native code that can be called by TSAN tests.
Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
to repeat the success/failure checks.
Add two native tests, one racy, one not. (right now some JVM races are
reported so the non-racy one fails)

"make test" doesn't work since it doesn't work with LD_PRELOAD. Instead, run:
$ make jdk-image test-image
$ JTREG=path/to/jtreg
$ JDK_ROOT=path/to/tsan_jdk
$ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
-jdk:$JDK_ROOT/build/images/jdk/
-nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
$JDK_ROOT/test/hotspot/jtreg/tsan/
where $JDK_ROOT must be an absolute path because -nativepath only
works with an absolute path.

From jcbeyler at google.com  Wed Apr  3 16:20:12 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 3 Apr 2019 09:20:12 -0700
Subject: [RFR]: Update/add some tests
In-Reply-To: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
References: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
Message-ID: <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>

Hi Arthur,

Looks good to me, slight nit:
http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java.html

The documentation of the class is wrong, it should say it is for a non racy
test, no?

Thanks,
Jc

On Tue, Apr 2, 2019 at 3:40 PM Arthur Eubanks <aeubanks at google.com> wrote:

> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html
>
> I'll let the commit message do the talking:
>
> Add native code that can be called by TSAN tests.
> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
> to repeat the success/failure checks.
> Add two native tests, one racy, one not. (right now some JVM races are
> reported so the non-racy one fails)
>
> "make test" doesn't work since it doesn't work with LD_PRELOAD. Instead,
> run:
> $ make jdk-image test-image
> $ JTREG=path/to/jtreg
> $ JDK_ROOT=path/to/tsan_jdk
> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
> -jdk:$JDK_ROOT/build/images/jdk/
> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
> $JDK_ROOT/test/hotspot/jtreg/tsan/
> where $JDK_ROOT must be an absolute path because -nativepath only
> works with an absolute path.
>


-- 

Thanks,
Jc

From aeubanks at google.com  Wed Apr  3 16:44:08 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 3 Apr 2019 09:44:08 -0700
Subject: [RFR]: Update/add some tests
In-Reply-To: <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>
References: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
 <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>
Message-ID: <CAPW48sqCSEQGOQxEHK0d7kRaYW1W_c1E1jUeGZ4biV5LYcPAPQ@mail.gmail.com>

Done.
http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.01

On Wed, Apr 3, 2019 at 9:20 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Arthur,
>
> Looks good to me, slight nit:
>
> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java.html
>
> The documentation of the class is wrong, it should say it is for a non
> racy test, no?
>
> Thanks,
> Jc
>
> On Tue, Apr 2, 2019 at 3:40 PM Arthur Eubanks <aeubanks at google.com> wrote:
>
>> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html
>>
>> I'll let the commit message do the talking:
>>
>> Add native code that can be called by TSAN tests.
>> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
>> to repeat the success/failure checks.
>> Add two native tests, one racy, one not. (right now some JVM races are
>> reported so the non-racy one fails)
>>
>> "make test" doesn't work since it doesn't work with LD_PRELOAD. Instead,
>> run:
>> $ make jdk-image test-image
>> $ JTREG=path/to/jtreg
>> $ JDK_ROOT=path/to/tsan_jdk
>> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
>> -jdk:$JDK_ROOT/build/images/jdk/
>> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
>> $JDK_ROOT/test/hotspot/jtreg/tsan/
>> where $JDK_ROOT must be an absolute path because -nativepath only
>> works with an absolute path.
>>
>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Wed Apr  3 18:29:49 2019
From: manc at google.com (Man Cao)
Date: Wed, 3 Apr 2019 11:29:49 -0700
Subject: [RFR]: Update/add some tests
In-Reply-To: <CAPW48sqCSEQGOQxEHK0d7kRaYW1W_c1E1jUeGZ4biV5LYcPAPQ@mail.gmail.com>
References: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
 <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>
 <CAPW48sqCSEQGOQxEHK0d7kRaYW1W_c1E1jUeGZ4biV5LYcPAPQ@mail.gmail.com>
Message-ID: <CA+w6HxZmOvzb_uEbxQ4BWExBhm6UKhD29bZexaKF_h=sXrWKRQ@mail.gmail.com>

I wonder if it is cleaner to put the native method declaration and
System.loadLibrary() call in a subclass of AbstractLoop, say "abstract
class AbstractNativeLoop extends AbstractLoop".

This will avoid having "@run main/othervm/native" for the TSAN tests
that do not test native code.



In TsanRunner.java:

35 public class TsanRunner {

36   protected native void writeNative();

Is this writeNative() necessary? Seems unused.


-Man


On Wed, Apr 3, 2019 at 9:45 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Done.
> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.01
>
> On Wed, Apr 3, 2019 at 9:20 AM Jean Christophe Beyler <jcbeyler at google.com
> >
> wrote:
>
> > Hi Arthur,
> >
> > Looks good to me, slight nit:
> >
> >
> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java.html
> >
> > The documentation of the class is wrong, it should say it is for a non
> > racy test, no?
> >
> > Thanks,
> > Jc
> >
> > On Tue, Apr 2, 2019 at 3:40 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
> >
> >> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html
> >>
> >> I'll let the commit message do the talking:
> >>
> >> Add native code that can be called by TSAN tests.
> >> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
> >> to repeat the success/failure checks.
> >> Add two native tests, one racy, one not. (right now some JVM races are
> >> reported so the non-racy one fails)
> >>
> >> "make test" doesn't work since it doesn't work with LD_PRELOAD. Instead,
> >> run:
> >> $ make jdk-image test-image
> >> $ JTREG=path/to/jtreg
> >> $ JDK_ROOT=path/to/tsan_jdk
> >> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
> >> -jdk:$JDK_ROOT/build/images/jdk/
> >> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
> >> $JDK_ROOT/test/hotspot/jtreg/tsan/
> >> where $JDK_ROOT must be an absolute path because -nativepath only
> >> works with an absolute path.
> >>
> >
> >
> > --
> >
> > Thanks,
> > Jc
> >
>

From aeubanks at google.com  Wed Apr  3 18:59:14 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 3 Apr 2019 11:59:14 -0700
Subject: [RFR]: Update/add some tests
In-Reply-To: <CA+w6HxZmOvzb_uEbxQ4BWExBhm6UKhD29bZexaKF_h=sXrWKRQ@mail.gmail.com>
References: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
 <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>
 <CAPW48sqCSEQGOQxEHK0d7kRaYW1W_c1E1jUeGZ4biV5LYcPAPQ@mail.gmail.com>
 <CA+w6HxZmOvzb_uEbxQ4BWExBhm6UKhD29bZexaKF_h=sXrWKRQ@mail.gmail.com>
Message-ID: <CAPW48sqZTgDVnwB=6nWQRgA1vaU5wm-PmNOnjq2aVLsaV6fg9w@mail.gmail.com>

New webrev: http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.02

On Wed, Apr 3, 2019 at 11:30 AM Man Cao <manc at google.com> wrote:

> I wonder if it is cleaner to put the native method declaration and System.loadLibrary() call in a subclass of AbstractLoop, say "abstract class AbstractNativeLoop extends AbstractLoop".
>
> This will avoid having "@run main/othervm/native" for the TSAN tests that do not test native code.
>
> I had considered that, but was too lazy to do it. But now that you've
mentioned it I've gone ahead and done that.

>
>
> In TsanRunner.java:
>
> 35 public class TsanRunner {
>
> 36   protected native void writeNative();
>
> Is this writeNative() necessary? Seems unused.
>
Oops, removed.

>
>
> -Man
>
>
> On Wed, Apr 3, 2019 at 9:45 AM Arthur Eubanks <aeubanks at google.com> wrote:
>
>> Done.
>> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.01
>>
>> On Wed, Apr 3, 2019 at 9:20 AM Jean Christophe Beyler <
>> jcbeyler at google.com>
>> wrote:
>>
>> > Hi Arthur,
>> >
>> > Looks good to me, slight nit:
>> >
>> >
>> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java.html
>> >
>> > The documentation of the class is wrong, it should say it is for a non
>> > racy test, no?
>> >
>> > Thanks,
>> > Jc
>> >
>> > On Tue, Apr 2, 2019 at 3:40 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>> >
>> >> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html
>> >>
>> >> I'll let the commit message do the talking:
>> >>
>> >> Add native code that can be called by TSAN tests.
>> >> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
>> >> to repeat the success/failure checks.
>> >> Add two native tests, one racy, one not. (right now some JVM races are
>> >> reported so the non-racy one fails)
>> >>
>> >> "make test" doesn't work since it doesn't work with LD_PRELOAD.
>> Instead,
>> >> run:
>> >> $ make jdk-image test-image
>> >> $ JTREG=path/to/jtreg
>> >> $ JDK_ROOT=path/to/tsan_jdk
>> >> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
>> >> -jdk:$JDK_ROOT/build/images/jdk/
>> >> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
>> >> $JDK_ROOT/test/hotspot/jtreg/tsan/
>> >> where $JDK_ROOT must be an absolute path because -nativepath only
>> >> works with an absolute path.
>> >>
>> >
>> >
>> > --
>> >
>> > Thanks,
>> > Jc
>> >
>>
>

From manc at google.com  Wed Apr  3 19:35:22 2019
From: manc at google.com (Man Cao)
Date: Wed, 3 Apr 2019 12:35:22 -0700
Subject: [RFR]: Update/add some tests
In-Reply-To: <CAPW48sqZTgDVnwB=6nWQRgA1vaU5wm-PmNOnjq2aVLsaV6fg9w@mail.gmail.com>
References: <CAPW48spw2otfcFacUQoAJdM+i27jQV4Mykpw0f-1-CisjHjNEw@mail.gmail.com>
 <CAF9BGBxKth=dq7r1pqpAsGV_aSpiegV73qj0dw9JSdVJbcjxbQ@mail.gmail.com>
 <CAPW48sqCSEQGOQxEHK0d7kRaYW1W_c1E1jUeGZ4biV5LYcPAPQ@mail.gmail.com>
 <CA+w6HxZmOvzb_uEbxQ4BWExBhm6UKhD29bZexaKF_h=sXrWKRQ@mail.gmail.com>
 <CAPW48sqZTgDVnwB=6nWQRgA1vaU5wm-PmNOnjq2aVLsaV6fg9w@mail.gmail.com>
Message-ID: <CA+w6Hxax54LOnj3CDadWrTp95UwYLRnh3fbZkVUSWhYPz4ZtaQ@mail.gmail.com>

Looks good, except bin/tsan.supp seems coming from a different commit.
No need for new webrev, just make sure not to push tsan.supp.

-Man


On Wed, Apr 3, 2019 at 11:59 AM Arthur Eubanks <aeubanks at google.com> wrote:

> New webrev: http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.02
>
> On Wed, Apr 3, 2019 at 11:30 AM Man Cao <manc at google.com> wrote:
>
>> I wonder if it is cleaner to put the native method declaration and System.loadLibrary() call in a subclass of AbstractLoop, say "abstract class AbstractNativeLoop extends AbstractLoop".
>>
>> This will avoid having "@run main/othervm/native" for the TSAN tests that do not test native code.
>>
>> I had considered that, but was too lazy to do it. But now that you've
> mentioned it I've gone ahead and done that.
>
>>
>>
>> In TsanRunner.java:
>>
>> 35 public class TsanRunner {
>>
>> 36   protected native void writeNative();
>>
>> Is this writeNative() necessary? Seems unused.
>>
> Oops, removed.
>
>>
>>
>> -Man
>>
>>
>> On Wed, Apr 3, 2019 at 9:45 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> Done.
>>> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.01
>>>
>>> On Wed, Apr 3, 2019 at 9:20 AM Jean Christophe Beyler <
>>> jcbeyler at google.com>
>>> wrote:
>>>
>>> > Hi Arthur,
>>> >
>>> > Looks good to me, slight nit:
>>> >
>>> >
>>> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java.html
>>> >
>>> > The documentation of the class is wrong, it should say it is for a non
>>> > racy test, no?
>>> >
>>> > Thanks,
>>> > Jc
>>> >
>>> > On Tue, Apr 2, 2019 at 3:40 PM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>> >
>>> >> http://cr.openjdk.java.net/~aeubanks/tsantests/webrev.00/index.html
>>> >>
>>> >> I'll let the commit message do the talking:
>>> >>
>>> >> Add native code that can be called by TSAN tests.
>>> >> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have
>>> >> to repeat the success/failure checks.
>>> >> Add two native tests, one racy, one not. (right now some JVM races are
>>> >> reported so the non-racy one fails)
>>> >>
>>> >> "make test" doesn't work since it doesn't work with LD_PRELOAD.
>>> Instead,
>>> >> run:
>>> >> $ make jdk-image test-image
>>> >> $ JTREG=path/to/jtreg
>>> >> $ JDK_ROOT=path/to/tsan_jdk
>>> >> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
>>> >> -jdk:$JDK_ROOT/build/images/jdk/
>>> >> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
>>> >> $JDK_ROOT/test/hotspot/jtreg/tsan/
>>> >> where $JDK_ROOT must be an absolute path because -nativepath only
>>> >> works with an absolute path.
>>> >>
>>> >
>>> >
>>> > --
>>> >
>>> > Thanks,
>>> > Jc
>>> >
>>>
>>

From aeubanks at google.com  Wed Apr  3 19:58:20 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Wed, 03 Apr 2019 19:58:20 +0000
Subject: hg: tsan/dev: Update/clean/add TSAN tests
Message-ID: <201904031958.x33JwKH7005614@aojmv0008.oracle.com>

Changeset: b09ef0dcf54c
Author:    aeubanks
Date:      2019-04-02 15:28 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/b09ef0dcf54c

Update/clean/add TSAN tests

Add native code that can be called by TSAN tests.
Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have to repeat the success/failure checks.
Add two native tests, one racy, one not. (right now unrelated JVM races are reported so the non-racy one fails)

"make test" doesn't work since it doesn't work with LD_PRELOAD. Instead, run:
$ make jdk-image test-image
$ JTREG=path/to/jtreg
$ JDK_ROOT=path/to/tsan_jdk
$ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so -jdk:$JDK_ROOT/build/images/jdk/ -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/ $JDK_ROOT/test/hotspot/jtreg/tsan/
where $JDK_ROOT must be an absolute path because -nativepath only works with an absolute path.

! make/test/JtregNativeHotspot.gmk
+ test/hotspot/jtreg/tsan/AbstractNativeLoop.java
+ test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/RacyIntMemberLoopTest.java
+ test/hotspot/jtreg/tsan/RacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/TsanRunner.java
+ test/hotspot/jtreg/tsan/libAbstractNativeLoop.c


From aeubanks at google.com  Wed Apr  3 20:20:16 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 3 Apr 2019 13:20:16 -0700
Subject: [RFR]: Suppress list
Message-ID: <CAPW48sr+WxbOqW1b77mhvYRAsUO844ErZ0y_xQR6ReSUa6oP+Q@mail.gmail.com>

http://cr.openjdk.java.net/~aeubanks/tsansupp/webrev.00/

Add a suppress list, currently consisting of libjvm.so and libjimage.so.
This can be used with TSAN_OPTIONS=suppressions=path/to/jdk/bin/tsan.supp.

Eventually we want to use __tsan_default_suppressions() so we don't have an
extra file. However, when I add __tsan_default_suppressions(), it doesn't
get called by TSAN. I think it's because __tsan_init() (which calls
__tsan_default_suppressions()) is called too early (LD_PRELOAD time?) and
our overridden __tsan_default_suppressions() is not available yet. Might be
solved by statically linking TSAN.

From jcbeyler at google.com  Wed Apr  3 20:44:44 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 3 Apr 2019 13:44:44 -0700
Subject: hg: tsan/dev: Update/clean/add TSAN tests
In-Reply-To: <201904031958.x33JwKH7005614@aojmv0008.oracle.com>
References: <201904031958.x33JwKH7005614@aojmv0008.oracle.com>
Message-ID: <CAF9BGBy_xyGDGEr2Wn7FiR32FR4+Qj7FuqNKPRN0-2Q1QFi46g@mail.gmail.com>

Hi Arthur,

Looks good; are you updating the documentation in the wiki at the same time
as your patches are going through?

Thanks,
Jc

On Wed, Apr 3, 2019 at 1:02 PM <aeubanks at google.com> wrote:

> Changeset: b09ef0dcf54c
> Author:    aeubanks
> Date:      2019-04-02 15:28 -0700
> URL:       http://hg.openjdk.java.net/tsan/dev/rev/b09ef0dcf54c
>
> Update/clean/add TSAN tests
>
> Add native code that can be called by TSAN tests.
> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have to
> repeat the success/failure checks.
> Add two native tests, one racy, one not. (right now unrelated JVM races
> are reported so the non-racy one fails)
>
> "make test" doesn't work since it doesn't work with LD_PRELOAD. Instead,
> run:
> $ make jdk-image test-image
> $ JTREG=path/to/jtreg
> $ JDK_ROOT=path/to/tsan_jdk
> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
> -jdk:$JDK_ROOT/build/images/jdk/
> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
> $JDK_ROOT/test/hotspot/jtreg/tsan/
> where $JDK_ROOT must be an absolute path because -nativepath only works
> with an absolute path.
>
> ! make/test/JtregNativeHotspot.gmk
> + test/hotspot/jtreg/tsan/AbstractNativeLoop.java
> + test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java
> ! test/hotspot/jtreg/tsan/RacyIntMemberLoopTest.java
> + test/hotspot/jtreg/tsan/RacyNativeLoopTest.java
> ! test/hotspot/jtreg/tsan/TsanRunner.java
> + test/hotspot/jtreg/tsan/libAbstractNativeLoop.c
>
>

-- 

Thanks,
Jc

From manc at google.com  Wed Apr  3 20:47:07 2019
From: manc at google.com (Man Cao)
Date: Wed, 3 Apr 2019 13:47:07 -0700
Subject: [RFR]: Suppress list
In-Reply-To: <CAPW48sr+WxbOqW1b77mhvYRAsUO844ErZ0y_xQR6ReSUa6oP+Q@mail.gmail.com>
References: <CAPW48sr+WxbOqW1b77mhvYRAsUO844ErZ0y_xQR6ReSUa6oP+Q@mail.gmail.com>
Message-ID: <CA+w6Hxb8riFSb9uJg93D+uwxJF9J-Ux0S_UaTjx-ns0oYfy_aA@mail.gmail.com>

Could "!/bin/bash" be removed from header? And it should have Oracle in
copyright notice.
Looks good otherwise, no need for new webrev.

-Man


On Wed, Apr 3, 2019 at 1:21 PM Arthur Eubanks <aeubanks at google.com> wrote:

> http://cr.openjdk.java.net/~aeubanks/tsansupp/webrev.00/
>
> Add a suppress list, currently consisting of libjvm.so and libjimage.so.
> This can be used with TSAN_OPTIONS=suppressions=path/to/jdk/bin/tsan.supp.
>
> Eventually we want to use __tsan_default_suppressions() so we don't have an
> extra file. However, when I add __tsan_default_suppressions(), it doesn't
> get called by TSAN. I think it's because __tsan_init() (which calls
> __tsan_default_suppressions()) is called too early (LD_PRELOAD time?) and
> our overridden __tsan_default_suppressions() is not available yet. Might be
> solved by statically linking TSAN.
>

From aeubanks at google.com  Wed Apr  3 20:46:23 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 3 Apr 2019 13:46:23 -0700
Subject: hg: tsan/dev: Update/clean/add TSAN tests
In-Reply-To: <CAF9BGBy_xyGDGEr2Wn7FiR32FR4+Qj7FuqNKPRN0-2Q1QFi46g@mail.gmail.com>
References: <201904031958.x33JwKH7005614@aojmv0008.oracle.com>
 <CAF9BGBy_xyGDGEr2Wn7FiR32FR4+Qj7FuqNKPRN0-2Q1QFi46g@mail.gmail.com>
Message-ID: <CAPW48soLaDGWOjGx09i=9y-7A3EHicjKbv18ekKBod0rgnzzFQ@mail.gmail.com>

Yes, I was planning on updating the wiki after this change.

On Wed, Apr 3, 2019 at 1:44 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Arthur,
>
> Looks good; are you updating the documentation in the wiki at the same
> time as your patches are going through?
>
> Thanks,
> Jc
>
> On Wed, Apr 3, 2019 at 1:02 PM <aeubanks at google.com> wrote:
>
>> Changeset: b09ef0dcf54c
>> Author:    aeubanks
>> Date:      2019-04-02 15:28 -0700
>> URL:       http://hg.openjdk.java.net/tsan/dev/rev/b09ef0dcf54c
>>
>> Update/clean/add TSAN tests
>>
>> Add native code that can be called by TSAN tests.
>> Add runTsanTestExpectFailure/Success to TsanRunner so tests don't have to
>> repeat the success/failure checks.
>> Add two native tests, one racy, one not. (right now unrelated JVM races
>> are reported so the non-racy one fails)
>>
>> "make test" doesn't work since it doesn't work with LD_PRELOAD. Instead,
>> run:
>> $ make jdk-image test-image
>> $ JTREG=path/to/jtreg
>> $ JDK_ROOT=path/to/tsan_jdk
>> $ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so
>> -jdk:$JDK_ROOT/build/images/jdk/
>> -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/
>> $JDK_ROOT/test/hotspot/jtreg/tsan/
>> where $JDK_ROOT must be an absolute path because -nativepath only works
>> with an absolute path.
>>
>> ! make/test/JtregNativeHotspot.gmk
>> + test/hotspot/jtreg/tsan/AbstractNativeLoop.java
>> + test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java
>> ! test/hotspot/jtreg/tsan/RacyIntMemberLoopTest.java
>> + test/hotspot/jtreg/tsan/RacyNativeLoopTest.java
>> ! test/hotspot/jtreg/tsan/TsanRunner.java
>> + test/hotspot/jtreg/tsan/libAbstractNativeLoop.c
>>
>>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Wed Apr  3 20:51:17 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 3 Apr 2019 13:51:17 -0700
Subject: [RFR]: Suppress list
In-Reply-To: <CA+w6Hxb8riFSb9uJg93D+uwxJF9J-Ux0S_UaTjx-ns0oYfy_aA@mail.gmail.com>
References: <CAPW48sr+WxbOqW1b77mhvYRAsUO844ErZ0y_xQR6ReSUa6oP+Q@mail.gmail.com>
 <CA+w6Hxb8riFSb9uJg93D+uwxJF9J-Ux0S_UaTjx-ns0oYfy_aA@mail.gmail.com>
Message-ID: <CAPW48spvGnoe0P8ensveOmK5hBpLC7-uPq2E6mBhqLS52NiDnw@mail.gmail.com>

Fixed.

On Wed, Apr 3, 2019 at 1:47 PM Man Cao <manc at google.com> wrote:

> Could "!/bin/bash" be removed from header? And it should have Oracle in
> copyright notice.
> Looks good otherwise, no need for new webrev.
>
> -Man
>
>
> On Wed, Apr 3, 2019 at 1:21 PM Arthur Eubanks <aeubanks at google.com> wrote:
>
>> http://cr.openjdk.java.net/~aeubanks/tsansupp/webrev.00/
>>
>> Add a suppress list, currently consisting of libjvm.so and libjimage.so.
>> This can be used with TSAN_OPTIONS=suppressions=path/to/jdk/bin/tsan.supp.
>>
>> Eventually we want to use __tsan_default_suppressions() so we don't have
>> an
>> extra file. However, when I add __tsan_default_suppressions(), it doesn't
>> get called by TSAN. I think it's because __tsan_init() (which calls
>> __tsan_default_suppressions()) is called too early (LD_PRELOAD time?) and
>> our overridden __tsan_default_suppressions() is not available yet. Might
>> be
>> solved by statically linking TSAN.
>>
>

From aeubanks at google.com  Wed Apr  3 20:51:39 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Wed, 03 Apr 2019 20:51:39 +0000
Subject: hg: tsan/dev: Add a TSAN suppression list
Message-ID: <201904032051.x33KpesP002376@aojmv0008.oracle.com>

Changeset: 49c7e40a0222
Author:    aeubanks
Date:      2019-04-02 17:17 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/49c7e40a0222

Add a TSAN suppression list

use with TSAN_OPTIONS=suppressions=path/to/jdk/bin/tsan.supp

This will tell TSAN to not report any races from libjvm.so and libjimage.so in tsan.supp.

Now there's an extra argument to pass to jtreg: -e:TSAN_OPTIONS
$ $JTREG -e:LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so -jdk:$JDK_ROOT/build/images/jdk/ -nativepath:$JDK_ROOT/build/images/test/hotspot/jtreg/native/ -e:TSAN_OPTIONS=suppressions=$JDK_ROOT/bin/tsan.supp $JDK_ROOT/test/hotspot/jtreg/tsan/

+ bin/tsan.supp


From manc at google.com  Fri Apr 12 00:44:01 2019
From: manc at google.com (Man Cao)
Date: Thu, 11 Apr 2019 17:44:01 -0700
Subject: RFR(M): Support running TSAN without LD_PRELOAD.
Message-ID: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>

Can I have reviews for this:
http://cr.openjdk.java.net/~manc/tsan/20190411/

Commit message:

This dynamically links TSAN runtime with the launcher bin/java, and moves
the default suppression list into hard-coded string in the launcher's
source. With this change, we can just use the following command to run tsan
tests:
$ make test TEST="hotspot/jtreg/tsan"

Currently only NonRacyNativeLoopTest fails due to missing lock
instrumentation.
Also refactored hotspot.m4 to consolidate TSAN-related code into one block.

-Man

From aeubanks at google.com  Fri Apr 12 01:58:04 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Thu, 11 Apr 2019 18:58:04 -0700
Subject: RFR(M): Support running TSAN without LD_PRELOAD.
In-Reply-To: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>
References: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>
Message-ID: <CAPW48spJx+m07yfoeC1PsNutaMqcVoDdAdgOxWKuAvzqiCvQ-A@mail.gmail.com>

TSAN was enabled by default right? I think when this gets upstreamed it
should be off by default, but for our branch this is good.
Awesome that "make test" just works now!

Is it possible to put the TSAN stuff in the gmk higher up so you can use
LIBS instead of LDFLAGS?

On Thu, Apr 11, 2019 at 5:44 PM Man Cao <manc at google.com> wrote:

> Can I have reviews for this:
> http://cr.openjdk.java.net/~manc/tsan/20190411/
>
> Commit message:
>
> This dynamically links TSAN runtime with the launcher bin/java, and moves
> the default suppression list into hard-coded string in the launcher's
> source. With this change, we can just use the following command to run tsan
> tests:
> $ make test TEST="hotspot/jtreg/tsan"
>
> Currently only NonRacyNativeLoopTest fails due to missing lock
> instrumentation.
> Also refactored hotspot.m4 to consolidate TSAN-related code into one block.
>
> -Man
>

From jcbeyler at google.com  Fri Apr 12 02:23:21 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Thu, 11 Apr 2019 19:23:21 -0700
Subject: RFR(M): Support running TSAN without LD_PRELOAD.
In-Reply-To: <CAPW48spJx+m07yfoeC1PsNutaMqcVoDdAdgOxWKuAvzqiCvQ-A@mail.gmail.com>
References: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>
 <CAPW48spJx+m07yfoeC1PsNutaMqcVoDdAdgOxWKuAvzqiCvQ-A@mail.gmail.com>
Message-ID: <CAF9BGBw_+BvSOQ+xCvrmBAY2nWmG=z1XDq0XEjutFyLvGLnh5A@mail.gmail.com>

LGTM, I would like to try to figure out how to have the suppress list in a
file at some point but we can figure that out later,
Jc

Ps: Please update installation and testing info in the wiki once merged :)

On Thu, Apr 11, 2019 at 6:58 PM Arthur Eubanks <aeubanks at google.com> wrote:

> TSAN was enabled by default right? I think when this gets upstreamed it
> should be off by default, but for our branch this is good.
> Awesome that "make test" just works now!
>
> Is it possible to put the TSAN stuff in the gmk higher up so you can use
> LIBS instead of LDFLAGS?
>
> On Thu, Apr 11, 2019 at 5:44 PM Man Cao <manc at google.com> wrote:
>
> > Can I have reviews for this:
> > http://cr.openjdk.java.net/~manc/tsan/20190411/
> >
> > Commit message:
> >
> > This dynamically links TSAN runtime with the launcher bin/java, and moves
> > the default suppression list into hard-coded string in the launcher's
> > source. With this change, we can just use the following command to run
> tsan
> > tests:
> > $ make test TEST="hotspot/jtreg/tsan"
> >
> > Currently only NonRacyNativeLoopTest fails due to missing lock
> > instrumentation.
> > Also refactored hotspot.m4 to consolidate TSAN-related code into one
> block.
> >
> > -Man
> >
>


-- 

Thanks,
Jc

From manc at google.com  Fri Apr 12 03:03:18 2019
From: manc at google.com (Man Cao)
Date: Thu, 11 Apr 2019 20:03:18 -0700
Subject: RFR(M): Support running TSAN without LD_PRELOAD.
In-Reply-To: <CAF9BGBw_+BvSOQ+xCvrmBAY2nWmG=z1XDq0XEjutFyLvGLnh5A@mail.gmail.com>
References: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>
 <CAPW48spJx+m07yfoeC1PsNutaMqcVoDdAdgOxWKuAvzqiCvQ-A@mail.gmail.com>
 <CAF9BGBw_+BvSOQ+xCvrmBAY2nWmG=z1XDq0XEjutFyLvGLnh5A@mail.gmail.com>
Message-ID: <CA+w6HxZRgtNvHagxzHwp6uw7qjhMKo0OFMrkeWQM2smJts2Y0A@mail.gmail.com>

> TSAN was enabled by default right? I think when this gets upstreamed it
should be off by default, but for our branch this is good.
Yes. It is enabled by default, and can be disabled via
--with-jvm-features=-tsan.
It is easy to change to disable it by default by modifying hotspot.m4.

> Is it possible to put the TSAN stuff in the gmk higher up so you can use
LIBS instead of LDFLAGS?
I think that would require propagating the INCLUDE_TSAN parameter all the
way through SetupJdkExecutable -> SetupNativeCompilation.
Then we end up at "make/common/NativeCompilation.gmk:1047", which has
another GLOBAL_LIBS that could prepend other -l flags before -ltsan.
It seems too much work and could affect compilations outside the launcher
directory

> Ps: Please update installation and testing info in the wiki once merged
:)
Will do. The wiki site seems down at the moment.

-Man


On Thu, Apr 11, 2019 at 7:23 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> LGTM, I would like to try to figure out how to have the suppress list in a
> file at some point but we can figure that out later,
> Jc
>
> Ps: Please update installation and testing info in the wiki once merged :)
>
> On Thu, Apr 11, 2019 at 6:58 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> TSAN was enabled by default right? I think when this gets upstreamed it
>> should be off by default, but for our branch this is good.
>> Awesome that "make test" just works now!
>>
>> Is it possible to put the TSAN stuff in the gmk higher up so you can use
>> LIBS instead of LDFLAGS?
>>
>> On Thu, Apr 11, 2019 at 5:44 PM Man Cao <manc at google.com> wrote:
>>
>> > Can I have reviews for this:
>> > http://cr.openjdk.java.net/~manc/tsan/20190411/
>> >
>> > Commit message:
>> >
>> > This dynamically links TSAN runtime with the launcher bin/java, and
>> moves
>> > the default suppression list into hard-coded string in the launcher's
>> > source. With this change, we can just use the following command to run
>> tsan
>> > tests:
>> > $ make test TEST="hotspot/jtreg/tsan"
>> >
>> > Currently only NonRacyNativeLoopTest fails due to missing lock
>> > instrumentation.
>> > Also refactored hotspot.m4 to consolidate TSAN-related code into one
>> block.
>> >
>> > -Man
>> >
>>
>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Fri Apr 12 03:43:36 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Thu, 11 Apr 2019 20:43:36 -0700
Subject: RFR(M): Support running TSAN without LD_PRELOAD.
In-Reply-To: <CA+w6HxZRgtNvHagxzHwp6uw7qjhMKo0OFMrkeWQM2smJts2Y0A@mail.gmail.com>
References: <CA+w6HxbTVV_w8GjK5pkFGL7nz1nxdN+0fsaPj-Vq+d8Gg1OEAw@mail.gmail.com>
 <CAPW48spJx+m07yfoeC1PsNutaMqcVoDdAdgOxWKuAvzqiCvQ-A@mail.gmail.com>
 <CAF9BGBw_+BvSOQ+xCvrmBAY2nWmG=z1XDq0XEjutFyLvGLnh5A@mail.gmail.com>
 <CA+w6HxZRgtNvHagxzHwp6uw7qjhMKo0OFMrkeWQM2smJts2Y0A@mail.gmail.com>
Message-ID: <CAPW48sroKuxmc1312xDwfiBLyKpW0tTfmer87gCoUGXXt587VA@mail.gmail.com>

LGTM

On Thu, Apr 11, 2019 at 8:03 PM Man Cao <manc at google.com> wrote:

> > TSAN was enabled by default right? I think when this gets upstreamed it
> should be off by default, but for our branch this is good.
> Yes. It is enabled by default, and can be disabled via
> --with-jvm-features=-tsan.
> It is easy to change to disable it by default by modifying hotspot.m4.
>
> > Is it possible to put the TSAN stuff in the gmk higher up so you can use
> LIBS instead of LDFLAGS?
> I think that would require propagating the INCLUDE_TSAN parameter all the
> way through SetupJdkExecutable -> SetupNativeCompilation.
> Then we end up at "make/common/NativeCompilation.gmk:1047", which has
> another GLOBAL_LIBS that could prepend other -l flags before -ltsan.
> It seems too much work and could affect compilations outside the launcher
> directory
>
> > Ps: Please update installation and testing info in the wiki once merged
> :)
> Will do. The wiki site seems down at the moment.
>
> -Man
>
>
> On Thu, Apr 11, 2019 at 7:23 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> LGTM, I would like to try to figure out how to have the suppress list in
>> a file at some point but we can figure that out later,
>> Jc
>>
>> Ps: Please update installation and testing info in the wiki once merged :)
>>
>> On Thu, Apr 11, 2019 at 6:58 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> TSAN was enabled by default right? I think when this gets upstreamed it
>>> should be off by default, but for our branch this is good.
>>> Awesome that "make test" just works now!
>>>
>>> Is it possible to put the TSAN stuff in the gmk higher up so you can use
>>> LIBS instead of LDFLAGS?
>>>
>>> On Thu, Apr 11, 2019 at 5:44 PM Man Cao <manc at google.com> wrote:
>>>
>>> > Can I have reviews for this:
>>> > http://cr.openjdk.java.net/~manc/tsan/20190411/
>>> >
>>> > Commit message:
>>> >
>>> > This dynamically links TSAN runtime with the launcher bin/java, and
>>> moves
>>> > the default suppression list into hard-coded string in the launcher's
>>> > source. With this change, we can just use the following command to run
>>> tsan
>>> > tests:
>>> > $ make test TEST="hotspot/jtreg/tsan"
>>> >
>>> > Currently only NonRacyNativeLoopTest fails due to missing lock
>>> > instrumentation.
>>> > Also refactored hotspot.m4 to consolidate TSAN-related code into one
>>> block.
>>> >
>>> > -Man
>>> >
>>>
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From manc at google.com  Fri Apr 12 17:56:54 2019
From: manc at google.com (manc at google.com)
Date: Fri, 12 Apr 2019 17:56:54 +0000
Subject: hg: tsan/dev: Support running TSAN without LD_PRELOAD.
Message-ID: <201904121756.x3CHuskg024686@aojmv0008.oracle.com>

Changeset: bf4236bed41c
Author:    manc
Date:      2019-04-11 17:30 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/bf4236bed41c

Support running TSAN without LD_PRELOAD.

This dynamically links TSAN runtime with the launcher bin/java, and moves
the default suppression list into hard-coded string in the launcher's
source. With this change, we can just use the following command to run
tsan tests:

$ make test TEST="hotspot/jtreg/tsan"

Currently only NonRacyNativeLoopTest fails due to missing lock instrumentation.

Also refactored hotspot.m4 to consolidate TSAN-related code into one block.

- bin/tsan.supp
! make/autoconf/hotspot.m4
! make/autoconf/spec.gmk.in
! make/launcher/Launcher-java.base.gmk
! make/launcher/LauncherCommon.gmk
! src/java.base/share/native/launcher/main.c


From aeubanks at google.com  Fri Apr 19 17:50:27 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Fri, 19 Apr 2019 10:50:27 -0700
Subject: [RFR]: Don't manually pass the thread to
 google_tsan_interp_method_entry
Message-ID: <CAPW48sq4JhoTj0JVF3PzuhHq5TTikZNv37hs4YC1=dkzFdhmfA@mail.gmail.com>

Don't manually pass the thread to google_tsan_interp_method_entry
The JavaThread is handled automatically by JRT_ENTRY.

http://cr.openjdk.java.net/~aeubanks/tsanjavathread/webrev.00/index.html

From manc at google.com  Fri Apr 19 17:54:03 2019
From: manc at google.com (Man Cao)
Date: Fri, 19 Apr 2019 10:54:03 -0700
Subject: [RFR]: Don't manually pass the thread to
 google_tsan_interp_method_entry
In-Reply-To: <CAPW48sq4JhoTj0JVF3PzuhHq5TTikZNv37hs4YC1=dkzFdhmfA@mail.gmail.com>
References: <CAPW48sq4JhoTj0JVF3PzuhHq5TTikZNv37hs4YC1=dkzFdhmfA@mail.gmail.com>
Message-ID: <CA+w6HxaGZscH0OMaCt7HmdtCqjUe7-BaXZ=9A6kg4hfgDbz_Eg@mail.gmail.com>

Looks good.

-Man


On Fri, Apr 19, 2019 at 10:51 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Don't manually pass the thread to google_tsan_interp_method_entry
> The JavaThread is handled automatically by JRT_ENTRY.
>
> http://cr.openjdk.java.net/~aeubanks/tsanjavathread/webrev.00/index.html
>

From jcbeyler at google.com  Fri Apr 19 18:16:24 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 19 Apr 2019 13:16:24 -0500
Subject: [RFR]: Don't manually pass the thread to
 google_tsan_interp_method_entry
In-Reply-To: <CA+w6HxaGZscH0OMaCt7HmdtCqjUe7-BaXZ=9A6kg4hfgDbz_Eg@mail.gmail.com>
References: <CAPW48sq4JhoTj0JVF3PzuhHq5TTikZNv37hs4YC1=dkzFdhmfA@mail.gmail.com>
 <CA+w6HxaGZscH0OMaCt7HmdtCqjUe7-BaXZ=9A6kg4hfgDbz_Eg@mail.gmail.com>
Message-ID: <CAF9BGBz=UsvtzwES=q+oG2X4eJ20MzsSN9zoPKCxUjb+dbPjcA@mail.gmail.com>

Looks good too :)
Jc

On Fri, Apr 19, 2019 at 12:54 PM Man Cao <manc at google.com> wrote:

> Looks good.
>
> -Man
>
>
> On Fri, Apr 19, 2019 at 10:51 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > Don't manually pass the thread to google_tsan_interp_method_entry
> > The JavaThread is handled automatically by JRT_ENTRY.
> >
> > http://cr.openjdk.java.net/~aeubanks/tsanjavathread/webrev.00/index.html
> >
>


-- 

Thanks,
Jc

From aeubanks at google.com  Fri Apr 19 18:18:22 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Fri, 19 Apr 2019 18:18:22 +0000
Subject: hg: tsan/dev: Don't manually pass the thread to
 google_tsan_interp_method_entry
Message-ID: <201904191818.x3JIINt4026179@aojmv0008.oracle.com>

Changeset: b15ed66eb136
Author:    aeubanks
Date:      2019-04-19 10:45 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/b15ed66eb136

Don't manually pass the thread to google_tsan_interp_method_entry

The JavaThread is handled automatically by JRT_ENTRY.

! src/hotspot/cpu/x86/interp_masm_x86.cpp


From aeubanks at google.com  Fri Apr 19 18:21:42 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Fri, 19 Apr 2019 11:21:42 -0700
Subject: [RFR]: Force jmethodIDs to be populated under TSAN
Message-ID: <CAPW48sp=DLSb6KmBqB=bSzEVAfwRy7dYX9MCStFWiyktnQ2_Og@mail.gmail.com>

Force jmethodIDs to be populated under TSAN
Otherwise they might be null when we need them.

http://cr.openjdk.java.net/~aeubanks/tsanjmethodid/webrev.00/index.html

From jcbeyler at google.com  Fri Apr 19 18:24:00 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 19 Apr 2019 13:24:00 -0500
Subject: [RFR]: Force jmethodIDs to be populated under TSAN
In-Reply-To: <CAPW48sp=DLSb6KmBqB=bSzEVAfwRy7dYX9MCStFWiyktnQ2_Og@mail.gmail.com>
References: <CAPW48sp=DLSb6KmBqB=bSzEVAfwRy7dYX9MCStFWiyktnQ2_Og@mail.gmail.com>
Message-ID: <CAF9BGBy3qs7iR5qTaKFt=B0PBH_Ac5Rf1vzxPzngF_n2hgYGfQ@mail.gmail.com>

Looks good :)
Jc

On Fri, Apr 19, 2019 at 1:22 PM Arthur Eubanks <aeubanks at google.com> wrote:

> Force jmethodIDs to be populated under TSAN
> Otherwise they might be null when we need them.
>
> http://cr.openjdk.java.net/~aeubanks/tsanjmethodid/webrev.00/index.html
>


-- 

Thanks,
Jc

From manc at google.com  Fri Apr 19 18:28:56 2019
From: manc at google.com (Man Cao)
Date: Fri, 19 Apr 2019 11:28:56 -0700
Subject: [RFR]: Force jmethodIDs to be populated under TSAN
In-Reply-To: <CAF9BGBy3qs7iR5qTaKFt=B0PBH_Ac5Rf1vzxPzngF_n2hgYGfQ@mail.gmail.com>
References: <CAPW48sp=DLSb6KmBqB=bSzEVAfwRy7dYX9MCStFWiyktnQ2_Og@mail.gmail.com>
 <CAF9BGBy3qs7iR5qTaKFt=B0PBH_Ac5Rf1vzxPzngF_n2hgYGfQ@mail.gmail.com>
Message-ID: <CA+w6HxZoowxCQWvfNEwNJXhTd+efq42b4zqNeLwSQd3XDPYFBA@mail.gmail.com>

Use "#if INCLUDE_TSAN" instead of "#ifdef INCLUDE_TSAN". Otherwise looks
good.
No need for new webrev.

-Man


On Fri, Apr 19, 2019 at 11:24 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Looks good :)
> Jc
>
> On Fri, Apr 19, 2019 at 1:22 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > Force jmethodIDs to be populated under TSAN
> > Otherwise they might be null when we need them.
> >
> > http://cr.openjdk.java.net/~aeubanks/tsanjmethodid/webrev.00/index.html
> >
>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Fri Apr 19 18:31:41 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Fri, 19 Apr 2019 11:31:41 -0700
Subject: [RFR]: Force jmethodIDs to be populated under TSAN
In-Reply-To: <CA+w6HxZoowxCQWvfNEwNJXhTd+efq42b4zqNeLwSQd3XDPYFBA@mail.gmail.com>
References: <CAPW48sp=DLSb6KmBqB=bSzEVAfwRy7dYX9MCStFWiyktnQ2_Og@mail.gmail.com>
 <CAF9BGBy3qs7iR5qTaKFt=B0PBH_Ac5Rf1vzxPzngF_n2hgYGfQ@mail.gmail.com>
 <CA+w6HxZoowxCQWvfNEwNJXhTd+efq42b4zqNeLwSQd3XDPYFBA@mail.gmail.com>
Message-ID: <CAPW48spUfcR-s6UPZJMSCX4X2vRd8bvmc_55hcJkRoKqM3ZMZw@mail.gmail.com>

Done.

On Fri, Apr 19, 2019 at 11:29 AM Man Cao <manc at google.com> wrote:

> Use "#if INCLUDE_TSAN" instead of "#ifdef INCLUDE_TSAN". Otherwise looks
> good.
> No need for new webrev.
>
> -Man
>
>
> On Fri, Apr 19, 2019 at 11:24 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Looks good :)
>> Jc
>>
>> On Fri, Apr 19, 2019 at 1:22 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> > Force jmethodIDs to be populated under TSAN
>> > Otherwise they might be null when we need them.
>> >
>> > http://cr.openjdk.java.net/~aeubanks/tsanjmethodid/webrev.00/index.html
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From aeubanks at google.com  Fri Apr 19 18:33:12 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Fri, 19 Apr 2019 18:33:12 +0000
Subject: hg: tsan/dev: Force jmethodIDs to be populated under TSAN
Message-ID: <201904191833.x3JIXDWI004411@aojmv0008.oracle.com>

Changeset: b9d5f42a7ac9
Author:    aeubanks
Date:      2019-04-19 10:59 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/b9d5f42a7ac9

Force jmethodIDs to be populated under TSAN

Otherwise they might be null when we need them.

! src/hotspot/share/classfile/classFileParser.cpp


From aeubanks at google.com  Fri Apr 19 18:42:56 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Fri, 19 Apr 2019 11:42:56 -0700
Subject: INCLUDE_TSAN inconsistent between launcher and hotspot
Message-ID: <CAPW48sqdqG8Xv1r8pL8T0dvwdp5gmqzEFo2O4-Q4Kd-6Z-Vf2Q@mail.gmail.com>

In a previous change it was mentioned that we should use "#if
INCLUDE_TSAN", not "#ifdef INCLUDE_TSAN".

I looked at all INCLUDE_TSAN, and the one
in src/java.base/share/native/launcher/main.c is different from the ones in
hotspot. I tried the following change:
http://cr.openjdk.java.net/~aeubanks/tsanmainincludetsan/webrev.00/src/java.base/share/native/launcher/main.c.udiff.html
but I get
error: "INCLUDE_TSAN" is not defined, evaluates to 0 [-Werror=undef]

In src/hotspot/share/utilities/macros.hpp in hotspot, there's
#ifndef INCLUDE_TSAN
#define INCLUDE_TSAN 1
#endif

Do we need something like this in the launcher? Or is the difference ok?

From manc at google.com  Fri Apr 19 19:06:29 2019
From: manc at google.com (Man Cao)
Date: Fri, 19 Apr 2019 12:06:29 -0700
Subject: INCLUDE_TSAN inconsistent between launcher and hotspot
In-Reply-To: <CAPW48sqdqG8Xv1r8pL8T0dvwdp5gmqzEFo2O4-Q4Kd-6Z-Vf2Q@mail.gmail.com>
References: <CAPW48sqdqG8Xv1r8pL8T0dvwdp5gmqzEFo2O4-Q4Kd-6Z-Vf2Q@mail.gmail.com>
Message-ID: <CA+w6Hxap51_wUaFt0q1s0w9yey+AapB1UL6AZ0z4JGhLuz_czw@mail.gmail.com>

The launcher's source code is completely separate from HotSpot. The
launcher cannot use the macros defined in macros.hpp.
There are two macros both called INCLUDE_TSAN, but for HotSpot it is
defined to 0 or 1, for launcher it is just a symbol without any value.

I think the difference is OK, but a bit confusing for readers of the code.
Do we want to rename one of them?

-Man


On Fri, Apr 19, 2019 at 11:43 AM Arthur Eubanks <aeubanks at google.com> wrote:

> In a previous change it was mentioned that we should use "#if
> INCLUDE_TSAN", not "#ifdef INCLUDE_TSAN".
>
> I looked at all INCLUDE_TSAN, and the one
> in src/java.base/share/native/launcher/main.c is different from the ones in
> hotspot. I tried the following change:
>
> http://cr.openjdk.java.net/~aeubanks/tsanmainincludetsan/webrev.00/src/java.base/share/native/launcher/main.c.udiff.html
> but I get
> error: "INCLUDE_TSAN" is not defined, evaluates to 0 [-Werror=undef]
>
> In src/hotspot/share/utilities/macros.hpp in hotspot, there's
> #ifndef INCLUDE_TSAN
> #define INCLUDE_TSAN 1
> #endif
>
> Do we need something like this in the launcher? Or is the difference ok?
>

From jcbeyler at google.com  Fri Apr 19 19:42:02 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 19 Apr 2019 14:42:02 -0500
Subject: INCLUDE_TSAN inconsistent between launcher and hotspot
In-Reply-To: <CA+w6Hxap51_wUaFt0q1s0w9yey+AapB1UL6AZ0z4JGhLuz_czw@mail.gmail.com>
References: <CAPW48sqdqG8Xv1r8pL8T0dvwdp5gmqzEFo2O4-Q4Kd-6Z-Vf2Q@mail.gmail.com>
 <CA+w6Hxap51_wUaFt0q1s0w9yey+AapB1UL6AZ0z4JGhLuz_czw@mail.gmail.com>
Message-ID: <CAF9BGBy7SvQ5BWmaor6ZWsza+p+DnGe03x-B1aVgfR+yYur+aw@mail.gmail.com>

Renaming seems to remove any problems, perhaps LAUNCHER_INCLUDE_TSAN or
something?
Jc

On Fri, Apr 19, 2019 at 2:07 PM Man Cao <manc at google.com> wrote:

> The launcher's source code is completely separate from HotSpot. The
> launcher cannot use the macros defined in macros.hpp.
> There are two macros both called INCLUDE_TSAN, but for HotSpot it is
> defined to 0 or 1, for launcher it is just a symbol without any value.
>
> I think the difference is OK, but a bit confusing for readers of the code.
> Do we want to rename one of them?
>
> -Man
>
>
> On Fri, Apr 19, 2019 at 11:43 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > In a previous change it was mentioned that we should use "#if
> > INCLUDE_TSAN", not "#ifdef INCLUDE_TSAN".
> >
> > I looked at all INCLUDE_TSAN, and the one
> > in src/java.base/share/native/launcher/main.c is different from the ones
> in
> > hotspot. I tried the following change:
> >
> >
> http://cr.openjdk.java.net/~aeubanks/tsanmainincludetsan/webrev.00/src/java.base/share/native/launcher/main.c.udiff.html
> > but I get
> > error: "INCLUDE_TSAN" is not defined, evaluates to 0 [-Werror=undef]
> >
> > In src/hotspot/share/utilities/macros.hpp in hotspot, there's
> > #ifndef INCLUDE_TSAN
> > #define INCLUDE_TSAN 1
> > #endif
> >
> > Do we need something like this in the launcher? Or is the difference ok?
> >
>


-- 

Thanks,
Jc

From aeubanks at google.com  Fri Apr 19 20:20:37 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Fri, 19 Apr 2019 13:20:37 -0700
Subject: [RFR]: Change TSAN_INCLUDE define to LAUNCHER_TSAN_INCLUDE in launcher
Message-ID: <CAPW48srvDpjCF9cqKtuQbEsGoixzrTusEOBra66MOnNy_4uWAQ@mail.gmail.com>

http://cr.openjdk.java.net/~aeubanks/tsandeflauncher/webrev.00/index.html

Rename INCLUDE_TSAN to LAUNCHER_INCLUDE_TSAN in launcher
It's confusing that it's the same name as the hotspot macro but uses a
different style (#ifdef vs #if).

From manc at google.com  Fri Apr 19 20:26:19 2019
From: manc at google.com (Man Cao)
Date: Fri, 19 Apr 2019 13:26:19 -0700
Subject: [RFR]: Change TSAN_INCLUDE define to LAUNCHER_TSAN_INCLUDE in
 launcher
In-Reply-To: <CAPW48srvDpjCF9cqKtuQbEsGoixzrTusEOBra66MOnNy_4uWAQ@mail.gmail.com>
References: <CAPW48srvDpjCF9cqKtuQbEsGoixzrTusEOBra66MOnNy_4uWAQ@mail.gmail.com>
Message-ID: <CA+w6HxZOFaHGeyPVLQs7Qar6ncN540RhsYYVR1zSpFBRYnKPuA@mail.gmail.com>

Looks good.

-Man


On Fri, Apr 19, 2019 at 1:21 PM Arthur Eubanks <aeubanks at google.com> wrote:

> http://cr.openjdk.java.net/~aeubanks/tsandeflauncher/webrev.00/index.html
>
> Rename INCLUDE_TSAN to LAUNCHER_INCLUDE_TSAN in launcher
> It's confusing that it's the same name as the hotspot macro but uses a
> different style (#ifdef vs #if).
>

From jcbeyler at google.com  Fri Apr 19 20:27:36 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 19 Apr 2019 15:27:36 -0500
Subject: [RFR]: Change TSAN_INCLUDE define to LAUNCHER_TSAN_INCLUDE in
 launcher
In-Reply-To: <CA+w6HxZOFaHGeyPVLQs7Qar6ncN540RhsYYVR1zSpFBRYnKPuA@mail.gmail.com>
References: <CAPW48srvDpjCF9cqKtuQbEsGoixzrTusEOBra66MOnNy_4uWAQ@mail.gmail.com>
 <CA+w6HxZOFaHGeyPVLQs7Qar6ncN540RhsYYVR1zSpFBRYnKPuA@mail.gmail.com>
Message-ID: <CAF9BGBxFnCKQUuF35MTwwQVkYV8hTnZy1dgp=AxKqXySdHPQvg@mail.gmail.com>

LGTM

On Fri, Apr 19, 2019 at 3:27 PM Man Cao <manc at google.com> wrote:

> Looks good.
>
> -Man
>
>
> On Fri, Apr 19, 2019 at 1:21 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> >
> http://cr.openjdk.java.net/~aeubanks/tsandeflauncher/webrev.00/index.html
> >
> > Rename INCLUDE_TSAN to LAUNCHER_INCLUDE_TSAN in launcher
> > It's confusing that it's the same name as the hotspot macro but uses a
> > different style (#ifdef vs #if).
> >
>


-- 

Thanks,
Jc

From aeubanks at google.com  Fri Apr 19 20:28:20 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Fri, 19 Apr 2019 20:28:20 +0000
Subject: hg: tsan/dev: Rename INCLUDE_TSAN to LAUNCHER_INCLUDE_TSAN in launcher
Message-ID: <201904192028.x3JKSLXM005719@aojmv0008.oracle.com>

Changeset: d44d0b28af5d
Author:    aeubanks
Date:      2019-04-19 11:35 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/d44d0b28af5d

Rename INCLUDE_TSAN to LAUNCHER_INCLUDE_TSAN in launcher

It's confusing that it's the same name as the hotspot macro but uses a different style (#ifdef vs #if).

! make/launcher/LauncherCommon.gmk
! src/java.base/share/native/launcher/main.c


From aeubanks at google.com  Tue Apr 23 16:28:40 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 09:28:40 -0700
Subject: [RFR]: Lock instrumentation
Message-ID: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>

This changes adds instrumentation for various locks in the JVM. (not
including JVMTI)
http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00

Note the NonRacy tests still don't pass for some reason. I believe it's due
to dynamic linking of libtsan.so.

From jcbeyler at google.com  Tue Apr 23 16:49:02 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 09:49:02 -0700
Subject: [RFR]: Lock instrumentation
In-Reply-To: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
References: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
Message-ID: <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>

Hi Arthur,

Looks good to me, I was looking at if we could factorize a bit the
assertions and what-not but for now it's fine.

(I did notice that we let creep in the #endif // INCLUDE_TSAN which does
not seem to be that much the style of the general code base so we might
want to remove them at some point)

But that's a detail, LGTM,
Jc


On Tue, Apr 23, 2019 at 9:29 AM Arthur Eubanks <aeubanks at google.com> wrote:

> This changes adds instrumentation for various locks in the JVM. (not
> including JVMTI)
> http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00
>
> Note the NonRacy tests still don't pass for some reason. I believe it's due
> to dynamic linking of libtsan.so.
>


-- 

Thanks,
Jc

From aeubanks at google.com  Tue Apr 23 16:58:24 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 09:58:24 -0700
Subject: [RFR]: Add test which does synchronization in native land
Message-ID: <CAPW48sr4E5v0qzOdCCB8hR99yXLDoZeVapcGj3gQLeEEfgW9=g@mail.gmail.com>

http://cr.openjdk.java.net/~aeubanks/tsannativesync/webrev.00/index.html

From jcbeyler at google.com  Tue Apr 23 18:45:48 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 11:45:48 -0700
Subject: [RFR]: Add test which does synchronization in native land
In-Reply-To: <CAPW48sr4E5v0qzOdCCB8hR99yXLDoZeVapcGj3gQLeEEfgW9=g@mail.gmail.com>
References: <CAPW48sr4E5v0qzOdCCB8hR99yXLDoZeVapcGj3gQLeEEfgW9=g@mail.gmail.com>
Message-ID: <CAF9BGBy485mH2wJrjf9Wzg9rqPiRxiEFbgC=oRpWLdaOYnC+cQ@mail.gmail.com>

LGTM

On Tue, Apr 23, 2019 at 9:59 AM Arthur Eubanks <aeubanks at google.com> wrote:

> http://cr.openjdk.java.net/~aeubanks/tsannativesync/webrev.00/index.html
>


-- 

Thanks,
Jc

From jcbeyler at google.com  Tue Apr 23 19:56:21 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 12:56:21 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
Message-ID: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>

Hi all,

Please review this webrev that moves tsan being enabled in the subprocess
and not the process being tested.

Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/

Thanks,
Jc

From aeubanks at google.com  Tue Apr 23 19:58:54 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 12:58:54 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
Message-ID: <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>

Now the tests don't need to be in "othervm" right?

On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi all,
>
> Please review this webrev that moves tsan being enabled in the subprocess
> and not the process being tested.
>
> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr 23 20:03:19 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 13:03:19 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
Message-ID: <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>

As far as I can tell, othervm just is an isolation mode for the various
actions; it should change nothing to have it or not and I think it is the
"default" to use as things will be more isolated.

I'd vote to keep it to be honest if what I am reading/understanding is
correct,
Jc

On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com> wrote:

> Now the tests don't need to be in "othervm" right?
>
> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi all,
>>
>> Please review this webrev that moves tsan being enabled in the subprocess
>> and not the process being tested.
>>
>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Tue Apr 23 20:06:47 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 13:06:47 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
Message-ID: <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>

I don't think it's the "default", it's only necessary when you need to spin
up a new VM to run the test (which we're doing ourselves manually). It's
slower and unnecessary in our case.
I think it's almost always only used when you need new a new VM with
different flags.

On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> As far as I can tell, othervm just is an isolation mode for the various
> actions; it should change nothing to have it or not and I think it is the
> "default" to use as things will be more isolated.
>
> I'd vote to keep it to be honest if what I am reading/understanding is
> correct,
> Jc
>
> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> Now the tests don't need to be in "othervm" right?
>>
>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Hi all,
>>>
>>> Please review this webrev that moves tsan being enabled in the subprocess
>>> and not the process being tested.
>>>
>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Tue Apr 23 20:23:22 2019
From: manc at google.com (Man Cao)
Date: Tue, 23 Apr 2019 13:23:22 -0700
Subject: [RFR]: Lock instrumentation
In-Reply-To: <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>
References: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
 <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>
Message-ID: <CA+w6HxZ=WEfUnM3gvbTLKRPx8doeJJymk_yXS672dVsrK7noFw@mail.gmail.com>

In synchronizer.cpp, in ReleaseJavaMonitorsClosure::do_monitor(), the TSAN
instrumentation should be inside the code block of the conditional "if
(mid->owner() == THREAD) {".
Otherwise looks good. No need for a new webrev after fixing this.

Regarding the "#if INCLUDE_TSAN" macro, I think we could define a macro in
macros.hpp like the following:
#if INCLUDE_TSAN
#define TSAN_RUNTIME_ONLY(code) if (ThreadSanitizer) { code }
#else
#define TSAN_RUNTIME_ONLY(code)
#endif

Then it could simplify some (but not all) of the instrumentation.

-Man


On Tue, Apr 23, 2019 at 9:49 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Arthur,
>
> Looks good to me, I was looking at if we could factorize a bit the
> assertions and what-not but for now it's fine.
>
> (I did notice that we let creep in the #endif // INCLUDE_TSAN which does
> not seem to be that much the style of the general code base so we might
> want to remove them at some point)
>
> But that's a detail, LGTM,
> Jc
>
>
> On Tue, Apr 23, 2019 at 9:29 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > This changes adds instrumentation for various locks in the JVM. (not
> > including JVMTI)
> > http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00
> >
> > Note the NonRacy tests still don't pass for some reason. I believe it's
> due
> > to dynamic linking of libtsan.so.
> >
>
>
> --
>
> Thanks,
> Jc
>

From alexey.menkov at oracle.com  Tue Apr 23 20:54:21 2019
From: alexey.menkov at oracle.com (Alex Menkov)
Date: Tue, 23 Apr 2019 13:54:21 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
 <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
Message-ID: <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>

I thought agentvm is the default, but looks like it's true only for 
"make run-test":

https://openjdk.java.net/jtreg/faq.html#what-are-the-agentvm-and-othervm-modes

--alex

On 04/23/2019 13:06, Arthur Eubanks wrote:
> I don't think it's the "default", it's only necessary when you need to spin
> up a new VM to run the test (which we're doing ourselves manually). It's
> slower and unnecessary in our case.
> I think it's almost always only used when you need new a new VM with
> different flags.
> 
> On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <jcbeyler at google.com>
> wrote:
> 
>> As far as I can tell, othervm just is an isolation mode for the various
>> actions; it should change nothing to have it or not and I think it is the
>> "default" to use as things will be more isolated.
>>
>> I'd vote to keep it to be honest if what I am reading/understanding is
>> correct,
>> Jc
>>
>> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> Now the tests don't need to be in "othervm" right?
>>>
>>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Please review this webrev that moves tsan being enabled in the subprocess
>>>> and not the process being tested.
>>>>
>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>

From jcbeyler at google.com  Tue Apr 23 20:57:03 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 13:57:03 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
 <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
 <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>
Message-ID: <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>

Yeah I noticed that too.

It also seems that I need othervm if native is defined so for the native
racy tests, I need them. I removed it the non native test though.

Let me know what you think, I can generate a new webrev if you like,
Jc

On Tue, Apr 23, 2019 at 1:54 PM Alex Menkov <alexey.menkov at oracle.com>
wrote:

> I thought agentvm is the default, but looks like it's true only for
> "make run-test":
>
>
> https://openjdk.java.net/jtreg/faq.html#what-are-the-agentvm-and-othervm-modes
>
> --alex
>
> On 04/23/2019 13:06, Arthur Eubanks wrote:
> > I don't think it's the "default", it's only necessary when you need to
> spin
> > up a new VM to run the test (which we're doing ourselves manually). It's
> > slower and unnecessary in our case.
> > I think it's almost always only used when you need new a new VM with
> > different flags.
> >
> > On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <
> jcbeyler at google.com>
> > wrote:
> >
> >> As far as I can tell, othervm just is an isolation mode for the various
> >> actions; it should change nothing to have it or not and I think it is
> the
> >> "default" to use as things will be more isolated.
> >>
> >> I'd vote to keep it to be honest if what I am reading/understanding is
> >> correct,
> >> Jc
> >>
> >> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com>
> >> wrote:
> >>
> >>> Now the tests don't need to be in "othervm" right?
> >>>
> >>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
> >>> jcbeyler at google.com> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> Please review this webrev that moves tsan being enabled in the
> subprocess
> >>>> and not the process being tested.
> >>>>
> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
> >>>>
> >>>> Thanks,
> >>>> Jc
> >>>>
> >>>
> >>
> >> --
> >>
> >> Thanks,
> >> Jc
> >>
>


-- 

Thanks,
Jc

From manc at google.com  Tue Apr 23 21:06:08 2019
From: manc at google.com (Man Cao)
Date: Tue, 23 Apr 2019 14:06:08 -0700
Subject: [RFR]: Add test which does synchronization in native land
In-Reply-To: <CAF9BGBy485mH2wJrjf9Wzg9rqPiRxiEFbgC=oRpWLdaOYnC+cQ@mail.gmail.com>
References: <CAPW48sr4E5v0qzOdCCB8hR99yXLDoZeVapcGj3gQLeEEfgW9=g@mail.gmail.com>
 <CAF9BGBy485mH2wJrjf9Wzg9rqPiRxiEFbgC=oRpWLdaOYnC+cQ@mail.gmail.com>
Message-ID: <CA+w6HxYGG0AS1MwPxt2Zy_PHWcNcdKzdWsYtT2dCMwdYM4PcGw@mail.gmail.com>

Is it better to add "static" to "pthread_mutex_t mutex"? It will make the
variable local to the module.
Also it seems all uses of PTHREAD_MUTEX_INITIALIZER in OpenJDK are for
static global variables.

Otherwise looks good. No need for another webrev for adding "static".

-Man


On Tue, Apr 23, 2019 at 11:46 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> LGTM
>
> On Tue, Apr 23, 2019 at 9:59 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > http://cr.openjdk.java.net/~aeubanks/tsannativesync/webrev.00/index.html
> >
>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Tue Apr 23 21:11:15 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 14:11:15 -0700
Subject: [RFR]: Add test which does synchronization in native land
In-Reply-To: <CA+w6HxYGG0AS1MwPxt2Zy_PHWcNcdKzdWsYtT2dCMwdYM4PcGw@mail.gmail.com>
References: <CAPW48sr4E5v0qzOdCCB8hR99yXLDoZeVapcGj3gQLeEEfgW9=g@mail.gmail.com>
 <CAF9BGBy485mH2wJrjf9Wzg9rqPiRxiEFbgC=oRpWLdaOYnC+cQ@mail.gmail.com>
 <CA+w6HxYGG0AS1MwPxt2Zy_PHWcNcdKzdWsYtT2dCMwdYM4PcGw@mail.gmail.com>
Message-ID: <CAPW48spFBXcix1HjkBaU2aD305=UfWVwEU7VBzU+-A=2H+jqNA@mail.gmail.com>

Done.

On Tue, Apr 23, 2019 at 2:06 PM Man Cao <manc at google.com> wrote:

> Is it better to add "static" to "pthread_mutex_t mutex"? It will make the
> variable local to the module.
> Also it seems all uses of PTHREAD_MUTEX_INITIALIZER in OpenJDK are for
> static global variables.
>
> Otherwise looks good. No need for another webrev for adding "static".
>
> -Man
>
>
> On Tue, Apr 23, 2019 at 11:46 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> LGTM
>>
>> On Tue, Apr 23, 2019 at 9:59 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> >
>> http://cr.openjdk.java.net/~aeubanks/tsannativesync/webrev.00/index.html
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From aeubanks at google.com  Tue Apr 23 21:12:51 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Tue, 23 Apr 2019 21:12:51 +0000
Subject: hg: tsan/dev: Add test which does synchronization in native land
Message-ID: <201904232112.x3NLCpnr021336@aojmv0008.oracle.com>

Changeset: bc4100503ba4
Author:    aeubanks
Date:      2019-04-22 15:03 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/bc4100503ba4

Add test which does synchronization in native land

! test/hotspot/jtreg/tsan/AbstractNativeLoop.java
+ test/hotspot/jtreg/tsan/NonRacyNativeLoopNativeSyncTest.java
! test/hotspot/jtreg/tsan/libAbstractNativeLoop.c


From aeubanks at google.com  Tue Apr 23 21:17:55 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 23 Apr 2019 14:17:55 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
 <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
 <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>
 <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>
Message-ID: <CAPW48sr8Fe5S-7jSQsfNMCJy2pe+6g3R4VYFtEkKOhNmqeWhRw@mail.gmail.com>

Hmm I'm not sure why native would need othervm as well. I tried main/native
and it complained about not finding libAbstractLoop.so in
System.loadLibrary(). I would have thought that only the VM that actually
runs with TSAN would need to be in its own VM.

Anyway, LGTM, but there's another test that I just pushed that also needs
to be cleaned up :)

On Tue, Apr 23, 2019 at 1:57 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Yeah I noticed that too.
>
> It also seems that I need othervm if native is defined so for the native
> racy tests, I need them. I removed it the non native test though.
>
> Let me know what you think, I can generate a new webrev if you like,
> Jc
>
> On Tue, Apr 23, 2019 at 1:54 PM Alex Menkov <alexey.menkov at oracle.com>
> wrote:
>
>> I thought agentvm is the default, but looks like it's true only for
>> "make run-test":
>>
>>
>> https://openjdk.java.net/jtreg/faq.html#what-are-the-agentvm-and-othervm-modes
>>
>> --alex
>>
>> On 04/23/2019 13:06, Arthur Eubanks wrote:
>> > I don't think it's the "default", it's only necessary when you need to
>> spin
>> > up a new VM to run the test (which we're doing ourselves manually). It's
>> > slower and unnecessary in our case.
>> > I think it's almost always only used when you need new a new VM with
>> > different flags.
>> >
>> > On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <
>> jcbeyler at google.com>
>> > wrote:
>> >
>> >> As far as I can tell, othervm just is an isolation mode for the various
>> >> actions; it should change nothing to have it or not and I think it is
>> the
>> >> "default" to use as things will be more isolated.
>> >>
>> >> I'd vote to keep it to be honest if what I am reading/understanding is
>> >> correct,
>> >> Jc
>> >>
>> >> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com>
>> >> wrote:
>> >>
>> >>> Now the tests don't need to be in "othervm" right?
>> >>>
>> >>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
>> >>> jcbeyler at google.com> wrote:
>> >>>
>> >>>> Hi all,
>> >>>>
>> >>>> Please review this webrev that moves tsan being enabled in the
>> subprocess
>> >>>> and not the process being tested.
>> >>>>
>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>> >>>>
>> >>>> Thanks,
>> >>>> Jc
>> >>>>
>> >>>
>> >>
>> >> --
>> >>
>> >> Thanks,
>> >> Jc
>> >>
>>
>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Tue Apr 23 21:18:09 2019
From: manc at google.com (Man Cao)
Date: Tue, 23 Apr 2019 14:18:09 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
 <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
 <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>
 <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>
Message-ID: <CA+w6HxaTUV6ufV4DiZz9YJbPdYUK1HWKyuRzKznPB++uiHQkXQ@mail.gmail.com>

Looks and sounds good.
Just make sure to also remove "-XX:+ThreadSanitizer" from this commit:
http://hg.openjdk.java.net/tsan/dev/rev/bc4100503ba4

-Man


On Tue, Apr 23, 2019 at 1:57 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Yeah I noticed that too.
>
> It also seems that I need othervm if native is defined so for the native
> racy tests, I need them. I removed it the non native test though.
>
> Let me know what you think, I can generate a new webrev if you like,
> Jc
>
> On Tue, Apr 23, 2019 at 1:54 PM Alex Menkov <alexey.menkov at oracle.com>
> wrote:
>
> > I thought agentvm is the default, but looks like it's true only for
> > "make run-test":
> >
> >
> >
> https://openjdk.java.net/jtreg/faq.html#what-are-the-agentvm-and-othervm-modes
> >
> > --alex
> >
> > On 04/23/2019 13:06, Arthur Eubanks wrote:
> > > I don't think it's the "default", it's only necessary when you need to
> > spin
> > > up a new VM to run the test (which we're doing ourselves manually).
> It's
> > > slower and unnecessary in our case.
> > > I think it's almost always only used when you need new a new VM with
> > > different flags.
> > >
> > > On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <
> > jcbeyler at google.com>
> > > wrote:
> > >
> > >> As far as I can tell, othervm just is an isolation mode for the
> various
> > >> actions; it should change nothing to have it or not and I think it is
> > the
> > >> "default" to use as things will be more isolated.
> > >>
> > >> I'd vote to keep it to be honest if what I am reading/understanding is
> > >> correct,
> > >> Jc
> > >>
> > >> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com>
> > >> wrote:
> > >>
> > >>> Now the tests don't need to be in "othervm" right?
> > >>>
> > >>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
> > >>> jcbeyler at google.com> wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>> Please review this webrev that moves tsan being enabled in the
> > subprocess
> > >>>> and not the process being tested.
> > >>>>
> > >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
> > >>>>
> > >>>> Thanks,
> > >>>> Jc
> > >>>>
> > >>>
> > >>
> > >> --
> > >>
> > >> Thanks,
> > >> Jc
> > >>
> >
>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr 23 21:21:17 2019
From: jcbeyler at google.com (jcbeyler at google.com)
Date: Tue, 23 Apr 2019 21:21:17 +0000
Subject: hg: tsan/dev: Move -XX:+ThreadSanitizer to the testing process
Message-ID: <201904232121.x3NLLIFo024725@aojmv0008.oracle.com>

Changeset: 3501b80b3480
Author:    jcbeyler
Date:      2019-04-23 14:20 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/3501b80b3480

Move -XX:+ThreadSanitizer to the testing process

Otherwise we are never testing TSAN :)

This passes the non racy ones but fails the NonRacyNativeLoop one,
which is expected

! test/hotspot/jtreg/tsan/NonRacyNativeLoopNativeSyncTest.java
! test/hotspot/jtreg/tsan/NonRacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/RacyIntMemberLoopTest.java
! test/hotspot/jtreg/tsan/RacyNativeLoopTest.java
! test/hotspot/jtreg/tsan/TsanRunner.java


From jcbeyler at google.com  Tue Apr 23 21:21:34 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 14:21:34 -0700
Subject: RFR: Move -XX:+ThreadSanitizer to the testing process
In-Reply-To: <CA+w6HxaTUV6ufV4DiZz9YJbPdYUK1HWKyuRzKznPB++uiHQkXQ@mail.gmail.com>
References: <CAF9BGBzCaO9OBYhYXLzruL0TGQwz32i6prnqtA4JgRFX1-GV3w@mail.gmail.com>
 <CAPW48srP7LU2ZUjFmBKCXu9z=NfD4xKUGszCEGeFxXgB_u2f0w@mail.gmail.com>
 <CAF9BGBxBxOfxNHUkKJV9mXK3F8sm7Hidr8StkUUbnd41w2ze8g@mail.gmail.com>
 <CAPW48sq-K=mkQq6abhgz5PKutiL5weqpatmMf2H3DDZrwBCkxw@mail.gmail.com>
 <c5a20603-543f-b48c-4f8e-c275781a64ed@oracle.com>
 <CAF9BGBxgO1noM2ht46H2z13kranQizkgmLkOKxoiwh1TO+t64Q@mail.gmail.com>
 <CA+w6HxaTUV6ufV4DiZz9YJbPdYUK1HWKyuRzKznPB++uiHQkXQ@mail.gmail.com>
Message-ID: <CAF9BGBx9HnTf=sSJHfJs=qQoXAmHx0wfK0-ni_B_Q7k-eFKh2g@mail.gmail.com>

Done, thanks for the reviews :)
Jc

On Tue, Apr 23, 2019 at 2:18 PM Man Cao <manc at google.com> wrote:

> Looks and sounds good.
> Just make sure to also remove "-XX:+ThreadSanitizer" from this commit:
> http://hg.openjdk.java.net/tsan/dev/rev/bc4100503ba4
>
> -Man
>
>
> On Tue, Apr 23, 2019 at 1:57 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Yeah I noticed that too.
>>
>> It also seems that I need othervm if native is defined so for the native
>> racy tests, I need them. I removed it the non native test though.
>>
>> Let me know what you think, I can generate a new webrev if you like,
>> Jc
>>
>> On Tue, Apr 23, 2019 at 1:54 PM Alex Menkov <alexey.menkov at oracle.com>
>> wrote:
>>
>> > I thought agentvm is the default, but looks like it's true only for
>> > "make run-test":
>> >
>> >
>> >
>> https://openjdk.java.net/jtreg/faq.html#what-are-the-agentvm-and-othervm-modes
>> >
>> > --alex
>> >
>> > On 04/23/2019 13:06, Arthur Eubanks wrote:
>> > > I don't think it's the "default", it's only necessary when you need to
>> > spin
>> > > up a new VM to run the test (which we're doing ourselves manually).
>> It's
>> > > slower and unnecessary in our case.
>> > > I think it's almost always only used when you need new a new VM with
>> > > different flags.
>> > >
>> > > On Tue, Apr 23, 2019 at 1:03 PM Jean Christophe Beyler <
>> > jcbeyler at google.com>
>> > > wrote:
>> > >
>> > >> As far as I can tell, othervm just is an isolation mode for the
>> various
>> > >> actions; it should change nothing to have it or not and I think it is
>> > the
>> > >> "default" to use as things will be more isolated.
>> > >>
>> > >> I'd vote to keep it to be honest if what I am reading/understanding
>> is
>> > >> correct,
>> > >> Jc
>> > >>
>> > >> On Tue, Apr 23, 2019 at 12:59 PM Arthur Eubanks <aeubanks at google.com
>> >
>> > >> wrote:
>> > >>
>> > >>> Now the tests don't need to be in "othervm" right?
>> > >>>
>> > >>> On Tue, Apr 23, 2019 at 12:57 PM Jean Christophe Beyler <
>> > >>> jcbeyler at google.com> wrote:
>> > >>>
>> > >>>> Hi all,
>> > >>>>
>> > >>>> Please review this webrev that moves tsan being enabled in the
>> > subprocess
>> > >>>> and not the process being tested.
>> > >>>>
>> > >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/tsan_flag/
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Jc
>> > >>>>
>> > >>>
>> > >>
>> > >> --
>> > >>
>> > >> Thanks,
>> > >> Jc
>> > >>
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From jcbeyler at google.com  Tue Apr 23 21:29:05 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 23 Apr 2019 14:29:05 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
Message-ID: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>

Hi all,

I added the locks for the JVMTI raw monitors and added two tests, it is
failing the non racy ones due to the JVM locks not yet included but I think
that will resolve itself soon :)

I also moved the agent library to C++.

FWIW, we also need to clean up our tests that check for failing, right now
they pass because they generically just are checking for a TSAN race but
not specifically which one, which is a bit wrong / imprecise.

Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/

Let me know what you think,
Jc

From aeubanks at google.com  Wed Apr 24 16:58:22 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 24 Apr 2019 09:58:22 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
Message-ID: <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>

Overall:
Since AnnotateRWLock* are direct TSAN calls, I think we should have a
wrapper around them. Then we can have things like checking if the lock is
NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
seems wrong.
"#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add the
comment for the #endifs.

test/hotspot/jtreg/tsan/TsanRunner.java
Could you rename "args" to "vmArgs"?

src/hotspot/share/prims/jvmtiEnv.cpp.udiff
Indentation around 3258 is wrong.


On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi all,
>
> I added the locks for the JVMTI raw monitors and added two tests, it is
> failing the non racy ones due to the JVM locks not yet included but I think
> that will resolve itself soon :)
>
> I also moved the agent library to C++.
>
> FWIW, we also need to clean up our tests that check for failing, right now
> they pass because they generically just are checking for a TSAN race but
> not specifically which one, which is a bit wrong / imprecise.
>
> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>
> Let me know what you think,
> Jc
>

From jcbeyler at google.com  Wed Apr 24 17:13:55 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 24 Apr 2019 10:13:55 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
Message-ID: <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>

Hi Arthur,

What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems like
it is what we did internally and what you were doing for the lock webrev as
well, ie have that as our include header. Are you suggesting we make
another tsan header for these methods.

Ok for all other points (except the indentation, just the way webrev does
things by default, the indentation is actually correct in my code),
Jc

On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Overall:
> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
> wrapper around them. Then we can have things like checking if the lock is
> NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
> seems wrong.
> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add the
> comment for the #endifs.
>
> test/hotspot/jtreg/tsan/TsanRunner.java
> Could you rename "args" to "vmArgs"?
>
> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
> Indentation around 3258 is wrong.
>
>
> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi all,
>>
>> I added the locks for the JVMTI raw monitors and added two tests, it is
>> failing the non racy ones due to the JVM locks not yet included but I
>> think
>> that will resolve itself soon :)
>>
>> I also moved the agent library to C++.
>>
>> FWIW, we also need to clean up our tests that check for failing, right now
>> they pass because they generically just are checking for a TSAN race but
>> not specifically which one, which is a bit wrong / imprecise.
>>
>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>
>> Let me know what you think,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Wed Apr 24 17:18:00 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 24 Apr 2019 10:18:00 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
Message-ID: <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>

Yeah we should have another tsan header. Kind of like how our interface for
the other TSAN functions is through SharedRuntime. Only that file would
include tsanExternalDecls.hpp. The other files would include the new header
file.

On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Arthur,
>
> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems like
> it is what we did internally and what you were doing for the lock webrev as
> well, ie have that as our include header. Are you suggesting we make
> another tsan header for these methods.
>
> Ok for all other points (except the indentation, just the way webrev does
> things by default, the indentation is actually correct in my code),
> Jc
>
> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> Overall:
>> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
>> wrapper around them. Then we can have things like checking if the lock is
>> NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
>> seems wrong.
>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add the
>> comment for the #endifs.
>>
>> test/hotspot/jtreg/tsan/TsanRunner.java
>> Could you rename "args" to "vmArgs"?
>>
>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>> Indentation around 3258 is wrong.
>>
>>
>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Hi all,
>>>
>>> I added the locks for the JVMTI raw monitors and added two tests, it is
>>> failing the non racy ones due to the JVM locks not yet included but I
>>> think
>>> that will resolve itself soon :)
>>>
>>> I also moved the agent library to C++.
>>>
>>> FWIW, we also need to clean up our tests that check for failing, right
>>> now
>>> they pass because they generically just are checking for a TSAN race but
>>> not specifically which one, which is a bit wrong / imprecise.
>>>
>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>
>>> Let me know what you think,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Wed Apr 24 17:37:34 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Wed, 24 Apr 2019 10:37:34 -0700
Subject: [RFR]: Lock instrumentation
In-Reply-To: <CA+w6HxZ=WEfUnM3gvbTLKRPx8doeJJymk_yXS672dVsrK7noFw@mail.gmail.com>
References: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
 <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>
 <CA+w6HxZ=WEfUnM3gvbTLKRPx8doeJJymk_yXS672dVsrK7noFw@mail.gmail.com>
Message-ID: <CAPW48sovshkEypUcV4=zQv9cDwZ-4HPYvyfokDtfYaFUuWrUWg@mail.gmail.com>

With JC's fix to the tests and the ReleaseJavaMonitorsClosure::do_monitor()
fix, now all the tests pass!

I'm mixed on the TSAN_RUNTIME_ONLY() suggestion, but either way it can come
in a later change.

On Tue, Apr 23, 2019 at 1:23 PM Man Cao <manc at google.com> wrote:

> In synchronizer.cpp, in ReleaseJavaMonitorsClosure::do_monitor(), the TSAN
> instrumentation should be inside the code block of the conditional "if
> (mid->owner() == THREAD) {".
> Otherwise looks good. No need for a new webrev after fixing this.
>
> Regarding the "#if INCLUDE_TSAN" macro, I think we could define a macro in
> macros.hpp like the following:
> #if INCLUDE_TSAN
> #define TSAN_RUNTIME_ONLY(code) if (ThreadSanitizer) { code }
> #else
> #define TSAN_RUNTIME_ONLY(code)
> #endif
>
> Then it could simplify some (but not all) of the instrumentation.
>
> -Man
>
>
> On Tue, Apr 23, 2019 at 9:49 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi Arthur,
>>
>> Looks good to me, I was looking at if we could factorize a bit the
>> assertions and what-not but for now it's fine.
>>
>> (I did notice that we let creep in the #endif // INCLUDE_TSAN which does
>> not seem to be that much the style of the general code base so we might
>> want to remove them at some point)
>>
>> But that's a detail, LGTM,
>> Jc
>>
>>
>> On Tue, Apr 23, 2019 at 9:29 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> > This changes adds instrumentation for various locks in the JVM. (not
>> > including JVMTI)
>> > http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00
>> >
>> > Note the NonRacy tests still don't pass for some reason. I believe it's
>> due
>> > to dynamic linking of libtsan.so.
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From aeubanks at google.com  Wed Apr 24 17:39:19 2019
From: aeubanks at google.com (aeubanks at google.com)
Date: Wed, 24 Apr 2019 17:39:19 +0000
Subject: hg: tsan/dev: Provide (un)lock info from Java to TSAN
Message-ID: <201904241739.x3OHdKGT027379@aojmv0008.oracle.com>

Changeset: e8047ff3da99
Author:    aeubanks
Date:      2019-04-02 08:50 -0700
URL:       http://hg.openjdk.java.net/tsan/dev/rev/e8047ff3da99

Provide (un)lock info from Java to TSAN

! src/hotspot/cpu/x86/interp_masm_x86.cpp
! src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
! src/hotspot/share/classfile/systemDictionary.cpp
! src/hotspot/share/runtime/sharedRuntime.cpp
! src/hotspot/share/runtime/sharedRuntime.hpp
! src/hotspot/share/runtime/synchronizer.cpp
! src/hotspot/share/tsan/tsanExternalDecls.hpp


From jcbeyler at google.com  Wed Apr 24 17:45:17 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 24 Apr 2019 10:45:17 -0700
Subject: [RFR]: Lock instrumentation
In-Reply-To: <CAPW48sovshkEypUcV4=zQv9cDwZ-4HPYvyfokDtfYaFUuWrUWg@mail.gmail.com>
References: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
 <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>
 <CA+w6HxZ=WEfUnM3gvbTLKRPx8doeJJymk_yXS672dVsrK7noFw@mail.gmail.com>
 <CAPW48sovshkEypUcV4=zQv9cDwZ-4HPYvyfokDtfYaFUuWrUWg@mail.gmail.com>
Message-ID: <CAF9BGBxawJcS58sc+de3+pDqBPaPvfWk6qYN3MTCDSeAOvoQeg@mail.gmail.com>

Yeah for the test, I'm actually adding that macro now in the JVMTI patch so
we can see what it looks and decide :), then I can refactor the other
changes to the same style in a later webrev,
Jc

On Wed, Apr 24, 2019 at 10:37 AM Arthur Eubanks <aeubanks at google.com> wrote:

> With JC's fix to the tests and
> the ReleaseJavaMonitorsClosure::do_monitor() fix, now all the tests pass!
>
> I'm mixed on the TSAN_RUNTIME_ONLY() suggestion, but either way it can
> come in a later change.
>
> On Tue, Apr 23, 2019 at 1:23 PM Man Cao <manc at google.com> wrote:
>
>> In synchronizer.cpp, in ReleaseJavaMonitorsClosure::do_monitor(), the
>> TSAN instrumentation should be inside the code block of the conditional "if
>> (mid->owner() == THREAD) {".
>> Otherwise looks good. No need for a new webrev after fixing this.
>>
>> Regarding the "#if INCLUDE_TSAN" macro, I think we could define a macro
>> in macros.hpp like the following:
>> #if INCLUDE_TSAN
>> #define TSAN_RUNTIME_ONLY(code) if (ThreadSanitizer) { code }
>> #else
>> #define TSAN_RUNTIME_ONLY(code)
>> #endif
>>
>> Then it could simplify some (but not all) of the instrumentation.
>>
>> -Man
>>
>>
>> On Tue, Apr 23, 2019 at 9:49 AM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Hi Arthur,
>>>
>>> Looks good to me, I was looking at if we could factorize a bit the
>>> assertions and what-not but for now it's fine.
>>>
>>> (I did notice that we let creep in the #endif // INCLUDE_TSAN which does
>>> not seem to be that much the style of the general code base so we might
>>> want to remove them at some point)
>>>
>>> But that's a detail, LGTM,
>>> Jc
>>>
>>>
>>> On Tue, Apr 23, 2019 at 9:29 AM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>> > This changes adds instrumentation for various locks in the JVM. (not
>>> > including JVMTI)
>>> > http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00
>>> >
>>> > Note the NonRacy tests still don't pass for some reason. I believe
>>> it's due
>>> > to dynamic linking of libtsan.so.
>>> >
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>

-- 

Thanks,
Jc

From manc at google.com  Wed Apr 24 17:54:53 2019
From: manc at google.com (Man Cao)
Date: Wed, 24 Apr 2019 10:54:53 -0700
Subject: [RFR]: Lock instrumentation
In-Reply-To: <CAF9BGBxawJcS58sc+de3+pDqBPaPvfWk6qYN3MTCDSeAOvoQeg@mail.gmail.com>
References: <CAPW48sp8fKzB4XTi_CLakApyF2gE0FfRVOSQV0_4MaQe+Ery6Q@mail.gmail.com>
 <CAF9BGByz9zWGWzJHeX=gGR2EdZu5rfm-f7h_ms7dRRn6vh57pw@mail.gmail.com>
 <CA+w6HxZ=WEfUnM3gvbTLKRPx8doeJJymk_yXS672dVsrK7noFw@mail.gmail.com>
 <CAPW48sovshkEypUcV4=zQv9cDwZ-4HPYvyfokDtfYaFUuWrUWg@mail.gmail.com>
 <CAF9BGBxawJcS58sc+de3+pDqBPaPvfWk6qYN3MTCDSeAOvoQeg@mail.gmail.com>
Message-ID: <CA+w6Hxbqf1Quw5UoGpWUED=iYiC5A=OEZVCFUYbkJYFKJGYQZw@mail.gmail.com>

Great to hear that the test works now.

For the macro, I think as long as the use of the macro looks like a scope
(like JRT_LEAF / JRT_END), or a function call (like JFR_ONLY), the style
should be OK.

-Man


On Wed, Apr 24, 2019 at 10:45 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Yeah for the test, I'm actually adding that macro now in the JVMTI patch
> so we can see what it looks and decide :), then I can refactor the other
> changes to the same style in a later webrev,
> Jc
>
> On Wed, Apr 24, 2019 at 10:37 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> With JC's fix to the tests and
>> the ReleaseJavaMonitorsClosure::do_monitor() fix, now all the tests pass!
>>
>> I'm mixed on the TSAN_RUNTIME_ONLY() suggestion, but either way it can
>> come in a later change.
>>
>> On Tue, Apr 23, 2019 at 1:23 PM Man Cao <manc at google.com> wrote:
>>
>>> In synchronizer.cpp, in ReleaseJavaMonitorsClosure::do_monitor(), the
>>> TSAN instrumentation should be inside the code block of the conditional "if
>>> (mid->owner() == THREAD) {".
>>> Otherwise looks good. No need for a new webrev after fixing this.
>>>
>>> Regarding the "#if INCLUDE_TSAN" macro, I think we could define a macro
>>> in macros.hpp like the following:
>>> #if INCLUDE_TSAN
>>> #define TSAN_RUNTIME_ONLY(code) if (ThreadSanitizer) { code }
>>> #else
>>> #define TSAN_RUNTIME_ONLY(code)
>>> #endif
>>>
>>> Then it could simplify some (but not all) of the instrumentation.
>>>
>>> -Man
>>>
>>>
>>> On Tue, Apr 23, 2019 at 9:49 AM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Hi Arthur,
>>>>
>>>> Looks good to me, I was looking at if we could factorize a bit the
>>>> assertions and what-not but for now it's fine.
>>>>
>>>> (I did notice that we let creep in the #endif // INCLUDE_TSAN which does
>>>> not seem to be that much the style of the general code base so we might
>>>> want to remove them at some point)
>>>>
>>>> But that's a detail, LGTM,
>>>> Jc
>>>>
>>>>
>>>> On Tue, Apr 23, 2019 at 9:29 AM Arthur Eubanks <aeubanks at google.com>
>>>> wrote:
>>>>
>>>> > This changes adds instrumentation for various locks in the JVM. (not
>>>> > including JVMTI)
>>>> > http://cr.openjdk.java.net/~aeubanks/tsanlock/webrev.00
>>>> >
>>>> > Note the NonRacy tests still don't pass for some reason. I believe
>>>> it's due
>>>> > to dynamic linking of libtsan.so.
>>>> >
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Wed Apr 24 18:00:43 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Wed, 24 Apr 2019 11:00:43 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
Message-ID: <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>

Sounds good to me.

Here it is:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/

I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works well
for one liners for code readability to be honest. If we want this, I'll do
another pass to propagate it around the rest of the changes in a separate
webrev.
cf:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html

And now with Arthur's change, all tests now pass (still would like to fix
the racing tests to be more restrictive),
Jc


On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com> wrote:

> Yeah we should have another tsan header. Kind of like how our interface
> for the other TSAN functions is through SharedRuntime. Only that file would
> include tsanExternalDecls.hpp. The other files would include the new header
> file.
>
> On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi Arthur,
>>
>> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems like
>> it is what we did internally and what you were doing for the lock webrev as
>> well, ie have that as our include header. Are you suggesting we make
>> another tsan header for these methods.
>>
>> Ok for all other points (except the indentation, just the way webrev does
>> things by default, the indentation is actually correct in my code),
>> Jc
>>
>> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> Overall:
>>> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
>>> wrapper around them. Then we can have things like checking if the lock is
>>> NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
>>> seems wrong.
>>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add the
>>> comment for the #endifs.
>>>
>>> test/hotspot/jtreg/tsan/TsanRunner.java
>>> Could you rename "args" to "vmArgs"?
>>>
>>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>> Indentation around 3258 is wrong.
>>>
>>>
>>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I added the locks for the JVMTI raw monitors and added two tests, it is
>>>> failing the non racy ones due to the JVM locks not yet included but I
>>>> think
>>>> that will resolve itself soon :)
>>>>
>>>> I also moved the agent library to C++.
>>>>
>>>> FWIW, we also need to clean up our tests that check for failing, right
>>>> now
>>>> they pass because they generically just are checking for a TSAN race but
>>>> not specifically which one, which is a bit wrong / imprecise.
>>>>
>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>
>>>> Let me know what you think,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From manc at google.com  Wed Apr 24 23:58:42 2019
From: manc at google.com (Man Cao)
Date: Wed, 24 Apr 2019 16:58:42 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
Message-ID: <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>

Code in HotSpot looks good. Except that we have additional instrumentation
in jvmtiTagMap.cpp that could be added in a following change.

A few comments on tests:
In AbstractNativeLoop.java:
Would it make sense to make all native methods "static"? They don't access
any instance field, right?

In NonRawRacyNativeLoopTest.java:
There are two calls to createRawLock() in NonRawRacyNativeLoopRunner. Could
the one in main() be deleted?

In RawRacyNativeLoopTest.java:
Similarly, could the createRawLock() in main() be deleted?
I think calling createRawLock() in run() would result in each thread
creating 50000 raw monitors, due to the way in which AbstractLoop is
written. Can we avoid doing this so the test can be a bit faster?

-Man


On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Sounds good to me.
>
> Here it is:
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>
> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works well
> for one liners for code readability to be honest. If we want this, I'll do
> another pass to propagate it around the rest of the changes in a separate
> webrev.
> cf:
>
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>
> And now with Arthur's change, all tests now pass (still would like to fix
> the racing tests to be more restrictive),
> Jc
>
>
> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > Yeah we should have another tsan header. Kind of like how our interface
> > for the other TSAN functions is through SharedRuntime. Only that file
> would
> > include tsanExternalDecls.hpp. The other files would include the new
> header
> > file.
> >
> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
> > jcbeyler at google.com> wrote:
> >
> >> Hi Arthur,
> >>
> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems
> like
> >> it is what we did internally and what you were doing for the lock
> webrev as
> >> well, ie have that as our include header. Are you suggesting we make
> >> another tsan header for these methods.
> >>
> >> Ok for all other points (except the indentation, just the way webrev
> does
> >> things by default, the indentation is actually correct in my code),
> >> Jc
> >>
> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
> >> wrote:
> >>
> >>> Overall:
> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
> >>> wrapper around them. Then we can have things like checking if the lock
> is
> >>> NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
> >>> seems wrong.
> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add
> the
> >>> comment for the #endifs.
> >>>
> >>> test/hotspot/jtreg/tsan/TsanRunner.java
> >>> Could you rename "args" to "vmArgs"?
> >>>
> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
> >>> Indentation around 3258 is wrong.
> >>>
> >>>
> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
> >>> jcbeyler at google.com> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I added the locks for the JVMTI raw monitors and added two tests, it
> is
> >>>> failing the non racy ones due to the JVM locks not yet included but I
> >>>> think
> >>>> that will resolve itself soon :)
> >>>>
> >>>> I also moved the agent library to C++.
> >>>>
> >>>> FWIW, we also need to clean up our tests that check for failing, right
> >>>> now
> >>>> they pass because they generically just are checking for a TSAN race
> but
> >>>> not specifically which one, which is a bit wrong / imprecise.
> >>>>
> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
> >>>>
> >>>> Let me know what you think,
> >>>> Jc
> >>>>
> >>>
> >>
> >> --
> >>
> >> Thanks,
> >> Jc
> >>
> >
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Thu Apr 25 22:01:06 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Thu, 25 Apr 2019 15:01:06 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
Message-ID: <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>

Done on all points I believe Man.

I added the JvmtiTagMap change and added a new test for that as well. The
test was a bit annoying due to the fact that it was difficult to explain
that the "sum" parameter was safe to be updated before/after the heap
iteration. Currently, I've had to put a mutex around it all, which seems
like overkill but it was the only way Tsan would not complain.

I'm unclear if that is right, it seems that it is due to the fact that it
does not see the VM operation as a fork/join operation perhaps? I believe
this is systemic to any JVMTI operation that has a VM operation style
callback so we might have more work there.

Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/

Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems half
the #endif do it and half don't so I'm not sure what is the right approach.

Let me know what you think,
Jc

On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:

> Code in HotSpot looks good. Except that we have additional instrumentation
> in jvmtiTagMap.cpp that could be added in a following change.
>
> A few comments on tests:
> In AbstractNativeLoop.java:
> Would it make sense to make all native methods "static"? They don't access
> any instance field, right?
>
> In NonRawRacyNativeLoopTest.java:
> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
> Could the one in main() be deleted?
>
> In RawRacyNativeLoopTest.java:
> Similarly, could the createRawLock() in main() be deleted?
> I think calling createRawLock() in run() would result in each thread
> creating 50000 raw monitors, due to the way in which AbstractLoop is
> written. Can we avoid doing this so the test can be a bit faster?
>
> -Man
>
>
> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Sounds good to me.
>>
>> Here it is:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>
>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works well
>> for one liners for code readability to be honest. If we want this, I'll do
>> another pass to propagate it around the rest of the changes in a separate
>> webrev.
>> cf:
>>
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>
>> And now with Arthur's change, all tests now pass (still would like to fix
>> the racing tests to be more restrictive),
>> Jc
>>
>>
>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> > Yeah we should have another tsan header. Kind of like how our interface
>> > for the other TSAN functions is through SharedRuntime. Only that file
>> would
>> > include tsanExternalDecls.hpp. The other files would include the new
>> header
>> > file.
>> >
>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>> > jcbeyler at google.com> wrote:
>> >
>> >> Hi Arthur,
>> >>
>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems
>> like
>> >> it is what we did internally and what you were doing for the lock
>> webrev as
>> >> well, ie have that as our include header. Are you suggesting we make
>> >> another tsan header for these methods.
>> >>
>> >> Ok for all other points (except the indentation, just the way webrev
>> does
>> >> things by default, the indentation is actually correct in my code),
>> >> Jc
>> >>
>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>> >> wrote:
>> >>
>> >>> Overall:
>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
>> >>> wrapper around them. Then we can have things like checking if the
>> lock is
>> >>> NULL, etc, in a central location. And pulling in tsanExternalDecls.hpp
>> >>> seems wrong.
>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add
>> the
>> >>> comment for the #endifs.
>> >>>
>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>> >>> Could you rename "args" to "vmArgs"?
>> >>>
>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>> >>> Indentation around 3258 is wrong.
>> >>>
>> >>>
>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>> >>> jcbeyler at google.com> wrote:
>> >>>
>> >>>> Hi all,
>> >>>>
>> >>>> I added the locks for the JVMTI raw monitors and added two tests, it
>> is
>> >>>> failing the non racy ones due to the JVM locks not yet included but I
>> >>>> think
>> >>>> that will resolve itself soon :)
>> >>>>
>> >>>> I also moved the agent library to C++.
>> >>>>
>> >>>> FWIW, we also need to clean up our tests that check for failing,
>> right
>> >>>> now
>> >>>> they pass because they generically just are checking for a TSAN race
>> but
>> >>>> not specifically which one, which is a bit wrong / imprecise.
>> >>>>
>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>> >>>>
>> >>>> Let me know what you think,
>> >>>> Jc
>> >>>>
>> >>>
>> >>
>> >> --
>> >>
>> >> Thanks,
>> >> Jc
>> >>
>> >
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From manc at google.com  Fri Apr 26 05:41:32 2019
From: manc at google.com (Man Cao)
Date: Thu, 25 Apr 2019 22:41:32 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
Message-ID: <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>

It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
2. TsanVisibleMutexLocker should replace MutexLocker in
JvmtiTagMap::get_objects_with_tags().
Will adding these fix the issue with "sum" in the test?

For the new tests, there is "-agentlib:AbstractNativeLoop" on the command
line, and they will do System.loadLibrary("AbstractNativeLoop") again in
class initializer. I'm not sure if this could cause any unexpected
behavior, but it is probably better to avoid doing this. Could we put
JVMTI-related code in a new file?

In RawRacyNativeLoopTest, I think there is a race on the "private long
lock" field itself. Two threads will write different values to the same
field, so TSAN will likely report a race on this field regardless of what
happens in JVMTI code.

Minor comments:
I think a better name for TsanSimulatedMutexLocker could be TsanMutexScope,
because TsanSimulatedMutexLocker has nothing to do with the MutexLocker
class. Also, this class could be a subclase of StackObj.
I'd keep our internal comment about "Simulates barrier synchronization on
safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are quite
useful to explain why we need the TSAN instrumentation.
In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see any
string functions?

-Man


On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Done on all points I believe Man.
>
> I added the JvmtiTagMap change and added a new test for that as well. The
> test was a bit annoying due to the fact that it was difficult to explain
> that the "sum" parameter was safe to be updated before/after the heap
> iteration. Currently, I've had to put a mutex around it all, which seems
> like overkill but it was the only way Tsan would not complain.
>
> I'm unclear if that is right, it seems that it is due to the fact that it
> does not see the VM operation as a fork/join operation perhaps? I believe
> this is systemic to any JVMTI operation that has a VM operation style
> callback so we might have more work there.
>
> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>
> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
> half the #endif do it and half don't so I'm not sure what is the right
> approach.
>
> Let me know what you think,
> Jc
>
> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>
>> Code in HotSpot looks good. Except that we have additional
>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>> change.
>>
>> A few comments on tests:
>> In AbstractNativeLoop.java:
>> Would it make sense to make all native methods "static"? They don't
>> access any instance field, right?
>>
>> In NonRawRacyNativeLoopTest.java:
>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>> Could the one in main() be deleted?
>>
>> In RawRacyNativeLoopTest.java:
>> Similarly, could the createRawLock() in main() be deleted?
>> I think calling createRawLock() in run() would result in each thread
>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>> written. Can we avoid doing this so the test can be a bit faster?
>>
>> -Man
>>
>>
>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Sounds good to me.
>>>
>>> Here it is:
>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>
>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works
>>> well
>>> for one liners for code readability to be honest. If we want this, I'll
>>> do
>>> another pass to propagate it around the rest of the changes in a separate
>>> webrev.
>>> cf:
>>>
>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>
>>> And now with Arthur's change, all tests now pass (still would like to fix
>>> the racing tests to be more restrictive),
>>> Jc
>>>
>>>
>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>> > Yeah we should have another tsan header. Kind of like how our interface
>>> > for the other TSAN functions is through SharedRuntime. Only that file
>>> would
>>> > include tsanExternalDecls.hpp. The other files would include the new
>>> header
>>> > file.
>>> >
>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>> > jcbeyler at google.com> wrote:
>>> >
>>> >> Hi Arthur,
>>> >>
>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems
>>> like
>>> >> it is what we did internally and what you were doing for the lock
>>> webrev as
>>> >> well, ie have that as our include header. Are you suggesting we make
>>> >> another tsan header for these methods.
>>> >>
>>> >> Ok for all other points (except the indentation, just the way webrev
>>> does
>>> >> things by default, the indentation is actually correct in my code),
>>> >> Jc
>>> >>
>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>>> >> wrote:
>>> >>
>>> >>> Overall:
>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should have a
>>> >>> wrapper around them. Then we can have things like checking if the
>>> lock is
>>> >>> NULL, etc, in a central location. And pulling in
>>> tsanExternalDecls.hpp
>>> >>> seems wrong.
>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you add
>>> the
>>> >>> comment for the #endifs.
>>> >>>
>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>> >>> Could you rename "args" to "vmArgs"?
>>> >>>
>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>> >>> Indentation around 3258 is wrong.
>>> >>>
>>> >>>
>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>> >>> jcbeyler at google.com> wrote:
>>> >>>
>>> >>>> Hi all,
>>> >>>>
>>> >>>> I added the locks for the JVMTI raw monitors and added two tests,
>>> it is
>>> >>>> failing the non racy ones due to the JVM locks not yet included but
>>> I
>>> >>>> think
>>> >>>> that will resolve itself soon :)
>>> >>>>
>>> >>>> I also moved the agent library to C++.
>>> >>>>
>>> >>>> FWIW, we also need to clean up our tests that check for failing,
>>> right
>>> >>>> now
>>> >>>> they pass because they generically just are checking for a TSAN
>>> race but
>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>> >>>>
>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>> >>>>
>>> >>>> Let me know what you think,
>>> >>>> Jc
>>> >>>>
>>> >>>
>>> >>
>>> >> --
>>> >>
>>> >> Thanks,
>>> >> Jc
>>> >>
>>> >
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Fri Apr 26 16:50:08 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 26 Apr 2019 09:50:08 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
Message-ID: <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>

Hi Man,

New webrev is here:

On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:

> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>

So funny thing I can't do that actually. When I do, TSAN complains about a
race between the create and the lock itself. I don't know why it didn't
show up in our internal testing but I've added a note to it. I'm going to
run some tests on our version to see if I'm right that it just didn't show
up due to how we were using it (enough time between the create and the
iterate to not show up). But my study of the code this morning seems to
lead to : yes TSAN sees it as a race but it is not really one.


> 2. TsanVisibleMutexLocker should replace MutexLocker in
> JvmtiTagMap::get_objects_with_tags().
>

Done, there was another one as well that now I did the same to. Same as
above, I think we just did the minimal that we saw, but not the full set.

(Note: I think I'm missing other tag callbacks that should be checked for
races, I'll check the code and see in a second phase; probably with a more
extensive study of the create/lock tsan race)


> Will adding these fix the issue with "sum" in the test?
>

No they don't, I still have the same issue, the problem is that sum is
changed without a lock in user code, and I fail to see how to do a "fence"
for tsan from user to JVM and back. Except if we have a solution right now,
I'd rather figure this out in a subsequent webrev as this webrev is already
growing quite a bit, if that's ok :-)


>
> For the new tests, there is "-agentlib:AbstractNativeLoop" on the command
> line, and they will do System.loadLibrary("AbstractNativeLoop") again in
> class initializer. I'm not sure if this could cause any unexpected
> behavior, but it is probably better to avoid doing this. Could we put
> JVMTI-related code in a new file?
>
> In RawRacyNativeLoopTest, I think there is a race on the "private long
> lock" field itself. Two threads will write different values to the same
> field, so TSAN will likely report a race on this field regardless of what
> happens in JVMTI code.
>

Done, added a threadlocal instead to make it really localized. This is yet
another reason I'd like our race detection in our tests to check the
signatures but we need symbolization first (hint hint).
Also tested and it now fails with the right race.


>
>
> Minor comments:
> I think a better name for TsanSimulatedMutexLocker could be
> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
> MutexLocker class. Also, this class could be a subclase of StackObj.
>

Done, I think the StackObj would have been caught when running this in
fastdebug, which we have not yet set up to be done by default in our
testing...


> I'd keep our internal comment about "Simulates barrier synchronization on
> safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are quite
> useful to explain why we need the TSAN instrumentation.
>

Done.


> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see any
> string functions?
>

memset is defined there :)



>
> -Man
>
>
> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Done on all points I believe Man.
>>
>> I added the JvmtiTagMap change and added a new test for that as well. The
>> test was a bit annoying due to the fact that it was difficult to explain
>> that the "sum" parameter was safe to be updated before/after the heap
>> iteration. Currently, I've had to put a mutex around it all, which seems
>> like overkill but it was the only way Tsan would not complain.
>>
>> I'm unclear if that is right, it seems that it is due to the fact that it
>> does not see the VM operation as a fork/join operation perhaps? I believe
>> this is systemic to any JVMTI operation that has a VM operation style
>> callback so we might have more work there.
>>
>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>
>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
>> half the #endif do it and half don't so I'm not sure what is the right
>> approach.
>>
>> Let me know what you think,
>> Jc
>>
>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>
>>> Code in HotSpot looks good. Except that we have additional
>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>> change.
>>>
>>> A few comments on tests:
>>> In AbstractNativeLoop.java:
>>> Would it make sense to make all native methods "static"? They don't
>>> access any instance field, right?
>>>
>>> In NonRawRacyNativeLoopTest.java:
>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>> Could the one in main() be deleted?
>>>
>>> In RawRacyNativeLoopTest.java:
>>> Similarly, could the createRawLock() in main() be deleted?
>>> I think calling createRawLock() in run() would result in each thread
>>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>>> written. Can we avoid doing this so the test can be a bit faster?
>>>
>>> -Man
>>>
>>>
>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Sounds good to me.
>>>>
>>>> Here it is:
>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>
>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works
>>>> well
>>>> for one liners for code readability to be honest. If we want this, I'll
>>>> do
>>>> another pass to propagate it around the rest of the changes in a
>>>> separate
>>>> webrev.
>>>> cf:
>>>>
>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>
>>>> And now with Arthur's change, all tests now pass (still would like to
>>>> fix
>>>> the racing tests to be more restrictive),
>>>> Jc
>>>>
>>>>
>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>>> wrote:
>>>>
>>>> > Yeah we should have another tsan header. Kind of like how our
>>>> interface
>>>> > for the other TSAN functions is through SharedRuntime. Only that file
>>>> would
>>>> > include tsanExternalDecls.hpp. The other files would include the new
>>>> header
>>>> > file.
>>>> >
>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>> > jcbeyler at google.com> wrote:
>>>> >
>>>> >> Hi Arthur,
>>>> >>
>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong? Seems
>>>> like
>>>> >> it is what we did internally and what you were doing for the lock
>>>> webrev as
>>>> >> well, ie have that as our include header. Are you suggesting we make
>>>> >> another tsan header for these methods.
>>>> >>
>>>> >> Ok for all other points (except the indentation, just the way webrev
>>>> does
>>>> >> things by default, the indentation is actually correct in my code),
>>>> >> Jc
>>>> >>
>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com>
>>>> >> wrote:
>>>> >>
>>>> >>> Overall:
>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should have
>>>> a
>>>> >>> wrapper around them. Then we can have things like checking if the
>>>> lock is
>>>> >>> NULL, etc, in a central location. And pulling in
>>>> tsanExternalDecls.hpp
>>>> >>> seems wrong.
>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you
>>>> add the
>>>> >>> comment for the #endifs.
>>>> >>>
>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>> >>> Could you rename "args" to "vmArgs"?
>>>> >>>
>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>> >>> Indentation around 3258 is wrong.
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>> >>> jcbeyler at google.com> wrote:
>>>> >>>
>>>> >>>> Hi all,
>>>> >>>>
>>>> >>>> I added the locks for the JVMTI raw monitors and added two tests,
>>>> it is
>>>> >>>> failing the non racy ones due to the JVM locks not yet included
>>>> but I
>>>> >>>> think
>>>> >>>> that will resolve itself soon :)
>>>> >>>>
>>>> >>>> I also moved the agent library to C++.
>>>> >>>>
>>>> >>>> FWIW, we also need to clean up our tests that check for failing,
>>>> right
>>>> >>>> now
>>>> >>>> they pass because they generically just are checking for a TSAN
>>>> race but
>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>> >>>>
>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>> >>>>
>>>> >>>> Let me know what you think,
>>>> >>>> Jc
>>>> >>>>
>>>> >>>
>>>> >>
>>>> >> --
>>>> >>
>>>> >> Thanks,
>>>> >> Jc
>>>> >>
>>>> >
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From jcbeyler at google.com  Fri Apr 26 21:43:35 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Fri, 26 Apr 2019 14:43:35 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
Message-ID: <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>

Seems like I forgot to say two things:

1) First here is the link to the newest webrev:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
2) JTREG does not really allow two different native files, so as far as I
can tell all other examples do what I've done here for -agentlib and
System.loadlibrary

Thanks,
Jc

On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Man,
>
> New webrev is here:
>
> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>
>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>
>
> So funny thing I can't do that actually. When I do, TSAN complains about a
> race between the create and the lock itself. I don't know why it didn't
> show up in our internal testing but I've added a note to it. I'm going to
> run some tests on our version to see if I'm right that it just didn't show
> up due to how we were using it (enough time between the create and the
> iterate to not show up). But my study of the code this morning seems to
> lead to : yes TSAN sees it as a race but it is not really one.
>
>
>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>> JvmtiTagMap::get_objects_with_tags().
>>
>
> Done, there was another one as well that now I did the same to. Same as
> above, I think we just did the minimal that we saw, but not the full set.
>
> (Note: I think I'm missing other tag callbacks that should be checked for
> races, I'll check the code and see in a second phase; probably with a more
> extensive study of the create/lock tsan race)
>
>
>> Will adding these fix the issue with "sum" in the test?
>>
>
> No they don't, I still have the same issue, the problem is that sum is
> changed without a lock in user code, and I fail to see how to do a "fence"
> for tsan from user to JVM and back. Except if we have a solution right now,
> I'd rather figure this out in a subsequent webrev as this webrev is already
> growing quite a bit, if that's ok :-)
>
>
>>
>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>> again in class initializer. I'm not sure if this could cause any unexpected
>> behavior, but it is probably better to avoid doing this. Could we put
>> JVMTI-related code in a new file?
>>
>> In RawRacyNativeLoopTest, I think there is a race on the "private long
>> lock" field itself. Two threads will write different values to the same
>> field, so TSAN will likely report a race on this field regardless of what
>> happens in JVMTI code.
>>
>
> Done, added a threadlocal instead to make it really localized. This is yet
> another reason I'd like our race detection in our tests to check the
> signatures but we need symbolization first (hint hint).
> Also tested and it now fails with the right race.
>
>
>>
>>
>> Minor comments:
>> I think a better name for TsanSimulatedMutexLocker could be
>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>
>
> Done, I think the StackObj would have been caught when running this in
> fastdebug, which we have not yet set up to be done by default in our
> testing...
>
>
>> I'd keep our internal comment about "Simulates barrier synchronization on
>> safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are quite
>> useful to explain why we need the TSAN instrumentation.
>>
>
> Done.
>
>
>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see
>> any string functions?
>>
>
> memset is defined there :)
>
>
>
>>
>> -Man
>>
>>
>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Done on all points I believe Man.
>>>
>>> I added the JvmtiTagMap change and added a new test for that as well.
>>> The test was a bit annoying due to the fact that it was difficult to
>>> explain that the "sum" parameter was safe to be updated before/after the
>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>> seems like overkill but it was the only way Tsan would not complain.
>>>
>>> I'm unclear if that is right, it seems that it is due to the fact that
>>> it does not see the VM operation as a fork/join operation perhaps? I
>>> believe this is systemic to any JVMTI operation that has a VM operation
>>> style callback so we might have more work there.
>>>
>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>
>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
>>> half the #endif do it and half don't so I'm not sure what is the right
>>> approach.
>>>
>>> Let me know what you think,
>>> Jc
>>>
>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>
>>>> Code in HotSpot looks good. Except that we have additional
>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>> change.
>>>>
>>>> A few comments on tests:
>>>> In AbstractNativeLoop.java:
>>>> Would it make sense to make all native methods "static"? They don't
>>>> access any instance field, right?
>>>>
>>>> In NonRawRacyNativeLoopTest.java:
>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>> Could the one in main() be deleted?
>>>>
>>>> In RawRacyNativeLoopTest.java:
>>>> Similarly, could the createRawLock() in main() be deleted?
>>>> I think calling createRawLock() in run() would result in each thread
>>>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>>>> written. Can we avoid doing this so the test can be a bit faster?
>>>>
>>>> -Man
>>>>
>>>>
>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Sounds good to me.
>>>>>
>>>>> Here it is:
>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>
>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works
>>>>> well
>>>>> for one liners for code readability to be honest. If we want this,
>>>>> I'll do
>>>>> another pass to propagate it around the rest of the changes in a
>>>>> separate
>>>>> webrev.
>>>>> cf:
>>>>>
>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>
>>>>> And now with Arthur's change, all tests now pass (still would like to
>>>>> fix
>>>>> the racing tests to be more restrictive),
>>>>> Jc
>>>>>
>>>>>
>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>>>> wrote:
>>>>>
>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>> interface
>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>> file would
>>>>> > include tsanExternalDecls.hpp. The other files would include the new
>>>>> header
>>>>> > file.
>>>>> >
>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>> > jcbeyler at google.com> wrote:
>>>>> >
>>>>> >> Hi Arthur,
>>>>> >>
>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>> Seems like
>>>>> >> it is what we did internally and what you were doing for the lock
>>>>> webrev as
>>>>> >> well, ie have that as our include header. Are you suggesting we make
>>>>> >> another tsan header for these methods.
>>>>> >>
>>>>> >> Ok for all other points (except the indentation, just the way
>>>>> webrev does
>>>>> >> things by default, the indentation is actually correct in my code),
>>>>> >> Jc
>>>>> >>
>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <aeubanks at google.com
>>>>> >
>>>>> >> wrote:
>>>>> >>
>>>>> >>> Overall:
>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>> have a
>>>>> >>> wrapper around them. Then we can have things like checking if the
>>>>> lock is
>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>> tsanExternalDecls.hpp
>>>>> >>> seems wrong.
>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you
>>>>> add the
>>>>> >>> comment for the #endifs.
>>>>> >>>
>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>> >>>
>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>> >>> Indentation around 3258 is wrong.
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>> >>> jcbeyler at google.com> wrote:
>>>>> >>>
>>>>> >>>> Hi all,
>>>>> >>>>
>>>>> >>>> I added the locks for the JVMTI raw monitors and added two tests,
>>>>> it is
>>>>> >>>> failing the non racy ones due to the JVM locks not yet included
>>>>> but I
>>>>> >>>> think
>>>>> >>>> that will resolve itself soon :)
>>>>> >>>>
>>>>> >>>> I also moved the agent library to C++.
>>>>> >>>>
>>>>> >>>> FWIW, we also need to clean up our tests that check for failing,
>>>>> right
>>>>> >>>> now
>>>>> >>>> they pass because they generically just are checking for a TSAN
>>>>> race but
>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>> >>>>
>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>> >>>>
>>>>> >>>> Let me know what you think,
>>>>> >>>> Jc
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >> --
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Jc
>>>>> >>
>>>>> >
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>


-- 

Thanks,
Jc

From aeubanks at google.com  Mon Apr 29 20:10:31 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 29 Apr 2019 13:10:31 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
Message-ID: <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>

src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
-  return new Mutex(Mutex::native, "JVM_RawMonitorCreate");
+  void *mon = (void*) new Mutex(Mutex::native, "JVM_RawMonitorCreate");
+
+  TSAN_RAW_LOCK_CREATE(mon);
+  return mon;

src/hotspot/share/prims/jvmtiEnv.cpp: the comment shouldn't be there, or it
should be updated
+  // A wait is modeled in Tsan as a simple release-acquire pair.
+  // The matching acquire annotation is below.
+  TSAN_RAW_LOCK_ACQUIRED(rmonitor);
+
   switch (r) {
   case ObjectMonitor::OM_INTERRUPTED:
     return JVMTI_ERROR_INTERRUPT;
   case ObjectMonitor::OM_ILLEGAL_MONITOR_STATE:
     return JVMTI_ERROR_NOT_MONITOR_OWNER;

src/hotspot/share/prims/jvmtiTagMap.cpp:
It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
makes the separation of Tsan-specific features and the rest of the JVM less
clear. I would prefer something like
#if INCLUDE_TSAN
TsanMutexScope tsanMutexScope(lock());
#endif
as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
TsanVisibleMutexLocker uses multiple inheritance? ew. What about having the
two superclasses as two member variables instead?

I don't understand "good thing" in the comments below:
+  // This means that this lock is created under a mutex (good thing) but
then,
+  // subsequent uses do have a lock to protect it (good thing because not
+  // needed in this case), however TSAN sees it as being needed because:
+  //  - Another thread can come and get the newly created JvmtiTagMap
without a
+  //  lock and acquire the lock.
+  //  - This provokes a race for TSAN on the lock itself, though there is
no
+  //  real issue.

test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
Does "lock" have to be static?

On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Seems like I forgot to say two things:
>
> 1) First here is the link to the newest webrev:
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
> 2) JTREG does not really allow two different native files, so as far as I
> can tell all other examples do what I've done here for -agentlib and
> System.loadlibrary
>
> Thanks,
> Jc
>
> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi Man,
>>
>> New webrev is here:
>>
>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>
>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>
>>
>> So funny thing I can't do that actually. When I do, TSAN complains about
>> a race between the create and the lock itself. I don't know why it didn't
>> show up in our internal testing but I've added a note to it. I'm going to
>> run some tests on our version to see if I'm right that it just didn't show
>> up due to how we were using it (enough time between the create and the
>> iterate to not show up). But my study of the code this morning seems to
>> lead to : yes TSAN sees it as a race but it is not really one.
>>
>>
>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>> JvmtiTagMap::get_objects_with_tags().
>>>
>>
>> Done, there was another one as well that now I did the same to. Same as
>> above, I think we just did the minimal that we saw, but not the full set.
>>
>> (Note: I think I'm missing other tag callbacks that should be checked for
>> races, I'll check the code and see in a second phase; probably with a more
>> extensive study of the create/lock tsan race)
>>
>>
>>> Will adding these fix the issue with "sum" in the test?
>>>
>>
>> No they don't, I still have the same issue, the problem is that sum is
>> changed without a lock in user code, and I fail to see how to do a "fence"
>> for tsan from user to JVM and back. Except if we have a solution right now,
>> I'd rather figure this out in a subsequent webrev as this webrev is already
>> growing quite a bit, if that's ok :-)
>>
>>
>>>
>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>> again in class initializer. I'm not sure if this could cause any unexpected
>>> behavior, but it is probably better to avoid doing this. Could we put
>>> JVMTI-related code in a new file?
>>>
>>> In RawRacyNativeLoopTest, I think there is a race on the "private long
>>> lock" field itself. Two threads will write different values to the same
>>> field, so TSAN will likely report a race on this field regardless of what
>>> happens in JVMTI code.
>>>
>>
>> Done, added a threadlocal instead to make it really localized. This is
>> yet another reason I'd like our race detection in our tests to check the
>> signatures but we need symbolization first (hint hint).
>> Also tested and it now fails with the right race.
>>
>>
>>>
>>>
>>> Minor comments:
>>> I think a better name for TsanSimulatedMutexLocker could be
>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>
>>
>> Done, I think the StackObj would have been caught when running this in
>> fastdebug, which we have not yet set up to be done by default in our
>> testing...
>>
>>
>>> I'd keep our internal comment about "Simulates barrier synchronization
>>> on safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are
>>> quite useful to explain why we need the TSAN instrumentation.
>>>
>>
>> Done.
>>
>>
>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see
>>> any string functions?
>>>
>>
>> memset is defined there :)
>>
>>
>>
>>>
>>> -Man
>>>
>>>
>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Done on all points I believe Man.
>>>>
>>>> I added the JvmtiTagMap change and added a new test for that as well.
>>>> The test was a bit annoying due to the fact that it was difficult to
>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>
>>>> I'm unclear if that is right, it seems that it is due to the fact that
>>>> it does not see the VM operation as a fork/join operation perhaps? I
>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>> style callback so we might have more work there.
>>>>
>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>
>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
>>>> half the #endif do it and half don't so I'm not sure what is the right
>>>> approach.
>>>>
>>>> Let me know what you think,
>>>> Jc
>>>>
>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>
>>>>> Code in HotSpot looks good. Except that we have additional
>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>> change.
>>>>>
>>>>> A few comments on tests:
>>>>> In AbstractNativeLoop.java:
>>>>> Would it make sense to make all native methods "static"? They don't
>>>>> access any instance field, right?
>>>>>
>>>>> In NonRawRacyNativeLoopTest.java:
>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>> Could the one in main() be deleted?
>>>>>
>>>>> In RawRacyNativeLoopTest.java:
>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>> I think calling createRawLock() in run() would result in each thread
>>>>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>>>>> written. Can we avoid doing this so the test can be a bit faster?
>>>>>
>>>>> -Man
>>>>>
>>>>>
>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>> jcbeyler at google.com> wrote:
>>>>>
>>>>>> Sounds good to me.
>>>>>>
>>>>>> Here it is:
>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>
>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it works
>>>>>> well
>>>>>> for one liners for code readability to be honest. If we want this,
>>>>>> I'll do
>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>> separate
>>>>>> webrev.
>>>>>> cf:
>>>>>>
>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>
>>>>>> And now with Arthur's change, all tests now pass (still would like to
>>>>>> fix
>>>>>> the racing tests to be more restrictive),
>>>>>> Jc
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>> interface
>>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>>> file would
>>>>>> > include tsanExternalDecls.hpp. The other files would include the
>>>>>> new header
>>>>>> > file.
>>>>>> >
>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>> > jcbeyler at google.com> wrote:
>>>>>> >
>>>>>> >> Hi Arthur,
>>>>>> >>
>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>> Seems like
>>>>>> >> it is what we did internally and what you were doing for the lock
>>>>>> webrev as
>>>>>> >> well, ie have that as our include header. Are you suggesting we
>>>>>> make
>>>>>> >> another tsan header for these methods.
>>>>>> >>
>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>> webrev does
>>>>>> >> things by default, the indentation is actually correct in my code),
>>>>>> >> Jc
>>>>>> >>
>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>> aeubanks at google.com>
>>>>>> >> wrote:
>>>>>> >>
>>>>>> >>> Overall:
>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>>> have a
>>>>>> >>> wrapper around them. Then we can have things like checking if the
>>>>>> lock is
>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>> tsanExternalDecls.hpp
>>>>>> >>> seems wrong.
>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you
>>>>>> add the
>>>>>> >>> comment for the #endifs.
>>>>>> >>>
>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>> >>>
>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>> >>> Indentation around 3258 is wrong.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>> >>>
>>>>>> >>>> Hi all,
>>>>>> >>>>
>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>> tests, it is
>>>>>> >>>> failing the non racy ones due to the JVM locks not yet included
>>>>>> but I
>>>>>> >>>> think
>>>>>> >>>> that will resolve itself soon :)
>>>>>> >>>>
>>>>>> >>>> I also moved the agent library to C++.
>>>>>> >>>>
>>>>>> >>>> FWIW, we also need to clean up our tests that check for failing,
>>>>>> right
>>>>>> >>>> now
>>>>>> >>>> they pass because they generically just are checking for a TSAN
>>>>>> race but
>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>> >>>>
>>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>> >>>>
>>>>>> >>>> Let me know what you think,
>>>>>> >>>> Jc
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>> >> --
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> Jc
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Jc
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Mon Apr 29 21:00:11 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 29 Apr 2019 14:00:11 -0700
Subject: Symbolization
Message-ID: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>

It looks like the libtsan.so from gcc 8 is out of date. It doesn't
contain __tsan_symbolize_external_ex(), only the
legacy __tsan_symbolize_external(). This messes with testing the Java Tsan
patches.

$ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym

I looked at the latest gcc sources and it does have a version of Tsan that
includes __tsan_symbolize_external_ex(). We could tell people to compile
their own gcc and use that libtsan.so.
$ apt search gcc
only shows up to gcc 8 on my machine.

Thoughts?

From jcbeyler at google.com  Mon Apr 29 21:17:19 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 29 Apr 2019 14:17:19 -0700
Subject: Symbolization
In-Reply-To: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
References: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
Message-ID: <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>

Compiling gcc to get libtsan... Is there no way to build libtsan then from
scratch instead?

On Mon, Apr 29, 2019 at 2:00 PM Arthur Eubanks <aeubanks at google.com> wrote:

> It looks like the libtsan.so from gcc 8 is out of date. It doesn't
> contain __tsan_symbolize_external_ex(), only the
> legacy __tsan_symbolize_external(). This messes with testing the Java Tsan
> patches.
>
> $ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym
>
> I looked at the latest gcc sources and it does have a version of Tsan that
> includes __tsan_symbolize_external_ex(). We could tell people to compile
> their own gcc and use that libtsan.so.
> $ apt search gcc
> only shows up to gcc 8 on my machine.
>
> Thoughts?
>


-- 

Thanks,
Jc

From manc at google.com  Mon Apr 29 21:18:08 2019
From: manc at google.com (Man Cao)
Date: Mon, 29 Apr 2019 14:18:08 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
Message-ID: <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>

webrev.04 looks good to me.
Minor comment is that I'd add TODO or comment about
pthread_mutex_lock/unlock should be removed from PerObjectCallback()
and Java_JvmtiTaggerLoopRunner_iterateOverTags()
in libAbstractNativeLoop.cpp, once we have handled the lock creation race.
No need for a webrev for this.

For the lock creation race in JvmtiTagMap::JvmtiTagMap(), I think there is
a valid C++ data race in double-checked locking
in JvmtiTagMap::tag_map_for().
Ideally accesses to JvmtiEnvBase::_tag_map should use
OrderAccess::load_acquire() and OrderAccess::release_store(), and we should
make TSAN aware of these fence semantics, then TSAN would not complain
about the lock creation race.
I can try sending serviceability-dev@ a change for OrderAccess.

Lastly for Arthur's comment:
> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
> Does "lock" have to be static?
ThreadLocals are typically private static fields by convention:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ThreadLocal.html
.

-Man


On Mon, Apr 29, 2019 at 1:10 PM Arthur Eubanks <aeubanks at google.com> wrote:

> src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
> -  return new Mutex(Mutex::native, "JVM_RawMonitorCreate");
> +  void *mon = (void*) new Mutex(Mutex::native, "JVM_RawMonitorCreate");
> +
> +  TSAN_RAW_LOCK_CREATE(mon);
> +  return mon;
>
> src/hotspot/share/prims/jvmtiEnv.cpp: the comment shouldn't be there, or
> it should be updated
> +  // A wait is modeled in Tsan as a simple release-acquire pair.
> +  // The matching acquire annotation is below.
> +  TSAN_RAW_LOCK_ACQUIRED(rmonitor);
> +
>    switch (r) {
>    case ObjectMonitor::OM_INTERRUPTED:
>      return JVMTI_ERROR_INTERRUPT;
>    case ObjectMonitor::OM_ILLEGAL_MONITOR_STATE:
>      return JVMTI_ERROR_NOT_MONITOR_OWNER;
>
> src/hotspot/share/prims/jvmtiTagMap.cpp:
> It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
> makes the separation of Tsan-specific features and the rest of the JVM less
> clear. I would prefer something like
> #if INCLUDE_TSAN
> TsanMutexScope tsanMutexScope(lock());
> #endif
> as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
> TsanVisibleMutexLocker uses multiple inheritance? ew. What about having
> the two superclasses as two member variables instead?
>
> I don't understand "good thing" in the comments below:
> +  // This means that this lock is created under a mutex (good thing) but
> then,
> +  // subsequent uses do have a lock to protect it (good thing because not
> +  // needed in this case), however TSAN sees it as being needed because:
> +  //  - Another thread can come and get the newly created JvmtiTagMap
> without a
> +  //  lock and acquire the lock.
> +  //  - This provokes a race for TSAN on the lock itself, though there is
> no
> +  //  real issue.
>
> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
> Does "lock" have to be static?
>
> On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Seems like I forgot to say two things:
>>
>> 1) First here is the link to the newest webrev:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
>> 2) JTREG does not really allow two different native files, so as far as I
>> can tell all other examples do what I've done here for -agentlib and
>> System.loadlibrary
>>
>> Thanks,
>> Jc
>>
>> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Hi Man,
>>>
>>> New webrev is here:
>>>
>>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>>
>>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>>
>>>
>>> So funny thing I can't do that actually. When I do, TSAN complains about
>>> a race between the create and the lock itself. I don't know why it didn't
>>> show up in our internal testing but I've added a note to it. I'm going to
>>> run some tests on our version to see if I'm right that it just didn't show
>>> up due to how we were using it (enough time between the create and the
>>> iterate to not show up). But my study of the code this morning seems to
>>> lead to : yes TSAN sees it as a race but it is not really one.
>>>
>>>
>>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>>> JvmtiTagMap::get_objects_with_tags().
>>>>
>>>
>>> Done, there was another one as well that now I did the same to. Same as
>>> above, I think we just did the minimal that we saw, but not the full set.
>>>
>>> (Note: I think I'm missing other tag callbacks that should be checked
>>> for races, I'll check the code and see in a second phase; probably with a
>>> more extensive study of the create/lock tsan race)
>>>
>>>
>>>> Will adding these fix the issue with "sum" in the test?
>>>>
>>>
>>> No they don't, I still have the same issue, the problem is that sum is
>>> changed without a lock in user code, and I fail to see how to do a "fence"
>>> for tsan from user to JVM and back. Except if we have a solution right now,
>>> I'd rather figure this out in a subsequent webrev as this webrev is already
>>> growing quite a bit, if that's ok :-)
>>>
>>>
>>>>
>>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>>> again in class initializer. I'm not sure if this could cause any unexpected
>>>> behavior, but it is probably better to avoid doing this. Could we put
>>>> JVMTI-related code in a new file?
>>>>
>>>> In RawRacyNativeLoopTest, I think there is a race on the "private long
>>>> lock" field itself. Two threads will write different values to the same
>>>> field, so TSAN will likely report a race on this field regardless of what
>>>> happens in JVMTI code.
>>>>
>>>
>>> Done, added a threadlocal instead to make it really localized. This is
>>> yet another reason I'd like our race detection in our tests to check the
>>> signatures but we need symbolization first (hint hint).
>>> Also tested and it now fails with the right race.
>>>
>>>
>>>>
>>>>
>>>> Minor comments:
>>>> I think a better name for TsanSimulatedMutexLocker could be
>>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>>
>>>
>>> Done, I think the StackObj would have been caught when running this in
>>> fastdebug, which we have not yet set up to be done by default in our
>>> testing...
>>>
>>>
>>>> I'd keep our internal comment about "Simulates barrier synchronization
>>>> on safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are
>>>> quite useful to explain why we need the TSAN instrumentation.
>>>>
>>>
>>> Done.
>>>
>>>
>>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see
>>>> any string functions?
>>>>
>>>
>>> memset is defined there :)
>>>
>>>
>>>
>>>>
>>>> -Man
>>>>
>>>>
>>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Done on all points I believe Man.
>>>>>
>>>>> I added the JvmtiTagMap change and added a new test for that as well.
>>>>> The test was a bit annoying due to the fact that it was difficult to
>>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>>
>>>>> I'm unclear if that is right, it seems that it is due to the fact that
>>>>> it does not see the VM operation as a fork/join operation perhaps? I
>>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>>> style callback so we might have more work there.
>>>>>
>>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>>
>>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
>>>>> half the #endif do it and half don't so I'm not sure what is the right
>>>>> approach.
>>>>>
>>>>> Let me know what you think,
>>>>> Jc
>>>>>
>>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>>
>>>>>> Code in HotSpot looks good. Except that we have additional
>>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>>> change.
>>>>>>
>>>>>> A few comments on tests:
>>>>>> In AbstractNativeLoop.java:
>>>>>> Would it make sense to make all native methods "static"? They don't
>>>>>> access any instance field, right?
>>>>>>
>>>>>> In NonRawRacyNativeLoopTest.java:
>>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>>> Could the one in main() be deleted?
>>>>>>
>>>>>> In RawRacyNativeLoopTest.java:
>>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>>> I think calling createRawLock() in run() would result in each thread
>>>>>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>>>>>> written. Can we avoid doing this so the test can be a bit faster?
>>>>>>
>>>>>> -Man
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>>> jcbeyler at google.com> wrote:
>>>>>>
>>>>>>> Sounds good to me.
>>>>>>>
>>>>>>> Here it is:
>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>>
>>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it
>>>>>>> works well
>>>>>>> for one liners for code readability to be honest. If we want this,
>>>>>>> I'll do
>>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>>> separate
>>>>>>> webrev.
>>>>>>> cf:
>>>>>>>
>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>>
>>>>>>> And now with Arthur's change, all tests now pass (still would like
>>>>>>> to fix
>>>>>>> the racing tests to be more restrictive),
>>>>>>> Jc
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>>> interface
>>>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>>>> file would
>>>>>>> > include tsanExternalDecls.hpp. The other files would include the
>>>>>>> new header
>>>>>>> > file.
>>>>>>> >
>>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>>> > jcbeyler at google.com> wrote:
>>>>>>> >
>>>>>>> >> Hi Arthur,
>>>>>>> >>
>>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>>> Seems like
>>>>>>> >> it is what we did internally and what you were doing for the lock
>>>>>>> webrev as
>>>>>>> >> well, ie have that as our include header. Are you suggesting we
>>>>>>> make
>>>>>>> >> another tsan header for these methods.
>>>>>>> >>
>>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>>> webrev does
>>>>>>> >> things by default, the indentation is actually correct in my
>>>>>>> code),
>>>>>>> >> Jc
>>>>>>> >>
>>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>>> aeubanks at google.com>
>>>>>>> >> wrote:
>>>>>>> >>
>>>>>>> >>> Overall:
>>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>>>> have a
>>>>>>> >>> wrapper around them. Then we can have things like checking if
>>>>>>> the lock is
>>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>>> tsanExternalDecls.hpp
>>>>>>> >>> seems wrong.
>>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you
>>>>>>> add the
>>>>>>> >>> comment for the #endifs.
>>>>>>> >>>
>>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>>> >>>
>>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>>> >>> Indentation around 3258 is wrong.
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>>> >>>
>>>>>>> >>>> Hi all,
>>>>>>> >>>>
>>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>>> tests, it is
>>>>>>> >>>> failing the non racy ones due to the JVM locks not yet included
>>>>>>> but I
>>>>>>> >>>> think
>>>>>>> >>>> that will resolve itself soon :)
>>>>>>> >>>>
>>>>>>> >>>> I also moved the agent library to C++.
>>>>>>> >>>>
>>>>>>> >>>> FWIW, we also need to clean up our tests that check for
>>>>>>> failing, right
>>>>>>> >>>> now
>>>>>>> >>>> they pass because they generically just are checking for a TSAN
>>>>>>> race but
>>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>>> >>>>
>>>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>>> >>>>
>>>>>>> >>>> Let me know what you think,
>>>>>>> >>>> Jc
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >>
>>>>>>> >> Thanks,
>>>>>>> >> Jc
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jc
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From manc at google.com  Mon Apr 29 21:35:54 2019
From: manc at google.com (Man Cao)
Date: Mon, 29 Apr 2019 14:35:54 -0700
Subject: Symbolization
In-Reply-To: <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>
References: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
 <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>
Message-ID: <CA+w6HxY7YHbvBYnGC2Ort7V1hFBEpVDSTMhcwuAH9a7d9n2Z6g@mail.gmail.com>

I think the best way is still to find a way to build libtsan.so from latest
source, with the GCC version supported by OpenJDK:
https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms.
Even if we can get libtsan.so from a newer GCC, I'm not sure if it is safe
to build OpenJDK with the newer GCC, or to use a newer libtsan.so with an
old GCC.

-Man


On Mon, Apr 29, 2019 at 2:18 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Compiling gcc to get libtsan... Is there no way to build libtsan then from
> scratch instead?
>
> On Mon, Apr 29, 2019 at 2:00 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
> > It looks like the libtsan.so from gcc 8 is out of date. It doesn't
> > contain __tsan_symbolize_external_ex(), only the
> > legacy __tsan_symbolize_external(). This messes with testing the Java
> Tsan
> > patches.
> >
> > $ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym
> >
> > I looked at the latest gcc sources and it does have a version of Tsan
> that
> > includes __tsan_symbolize_external_ex(). We could tell people to compile
> > their own gcc and use that libtsan.so.
> > $ apt search gcc
> > only shows up to gcc 8 on my machine.
> >
> > Thoughts?
> >
>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Mon Apr 29 21:41:19 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 29 Apr 2019 14:41:19 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
Message-ID: <CAF9BGBxwUDo+2KUzRpLc8v6S9ymaZ0+LccfqXiSJe=4TTeKUTg@mail.gmail.com>

On Mon, Apr 29, 2019 at 1:10 PM Arthur Eubanks <aeubanks at google.com> wrote:

> src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
>
>
Done, removed :)



> src/hotspot/share/prims/jvmtiTagMap.cpp:
> It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
> makes the separation of Tsan-specific features and the rest of the JVM less
> clear. I would prefer something like
> #if INCLUDE_TSAN
> TsanMutexScope tsanMutexScope(lock());
> #endif
> as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
> TsanVisibleMutexLocker uses multiple inheritance? ew. What about having
> the two superclasses as two member variables instead?
>

Done, I used the TSAN_RUNTIME_ONLY (which we can still negotiate later the
name, etc. ;-))


>
> I don't understand "good thing" in the comments below:
> +  // This means that this lock is created under a mutex (good thing) but
> then,
> +  // subsequent uses do have a lock to protect it (good thing because not
> +  // needed in this case), however TSAN sees it as being needed because:
> +  //  - Another thread can come and get the newly created JvmtiTagMap
> without a
> +  //  lock and acquire the lock.
> +  //  - This provokes a race for TSAN on the lock itself, though there is
> no
> +  //  real issue.
>
>
Removed them :)


> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
> Does "lock" have to be static?
>

Man explains this, so I'll assume it's answered :)

Thanks for the review :)
Jc


> On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Seems like I forgot to say two things:
>>
>> 1) First here is the link to the newest webrev:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
>> 2) JTREG does not really allow two different native files, so as far as I
>> can tell all other examples do what I've done here for -agentlib and
>> System.loadlibrary
>>
>> Thanks,
>> Jc
>>
>> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Hi Man,
>>>
>>> New webrev is here:
>>>
>>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>>
>>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>>
>>>
>>> So funny thing I can't do that actually. When I do, TSAN complains about
>>> a race between the create and the lock itself. I don't know why it didn't
>>> show up in our internal testing but I've added a note to it. I'm going to
>>> run some tests on our version to see if I'm right that it just didn't show
>>> up due to how we were using it (enough time between the create and the
>>> iterate to not show up). But my study of the code this morning seems to
>>> lead to : yes TSAN sees it as a race but it is not really one.
>>>
>>>
>>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>>> JvmtiTagMap::get_objects_with_tags().
>>>>
>>>
>>> Done, there was another one as well that now I did the same to. Same as
>>> above, I think we just did the minimal that we saw, but not the full set.
>>>
>>> (Note: I think I'm missing other tag callbacks that should be checked
>>> for races, I'll check the code and see in a second phase; probably with a
>>> more extensive study of the create/lock tsan race)
>>>
>>>
>>>> Will adding these fix the issue with "sum" in the test?
>>>>
>>>
>>> No they don't, I still have the same issue, the problem is that sum is
>>> changed without a lock in user code, and I fail to see how to do a "fence"
>>> for tsan from user to JVM and back. Except if we have a solution right now,
>>> I'd rather figure this out in a subsequent webrev as this webrev is already
>>> growing quite a bit, if that's ok :-)
>>>
>>>
>>>>
>>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>>> again in class initializer. I'm not sure if this could cause any unexpected
>>>> behavior, but it is probably better to avoid doing this. Could we put
>>>> JVMTI-related code in a new file?
>>>>
>>>> In RawRacyNativeLoopTest, I think there is a race on the "private long
>>>> lock" field itself. Two threads will write different values to the same
>>>> field, so TSAN will likely report a race on this field regardless of what
>>>> happens in JVMTI code.
>>>>
>>>
>>> Done, added a threadlocal instead to make it really localized. This is
>>> yet another reason I'd like our race detection in our tests to check the
>>> signatures but we need symbolization first (hint hint).
>>> Also tested and it now fails with the right race.
>>>
>>>
>>>>
>>>>
>>>> Minor comments:
>>>> I think a better name for TsanSimulatedMutexLocker could be
>>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>>
>>>
>>> Done, I think the StackObj would have been caught when running this in
>>> fastdebug, which we have not yet set up to be done by default in our
>>> testing...
>>>
>>>
>>>> I'd keep our internal comment about "Simulates barrier synchronization
>>>> on safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are
>>>> quite useful to explain why we need the TSAN instrumentation.
>>>>
>>>
>>> Done.
>>>
>>>
>>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see
>>>> any string functions?
>>>>
>>>
>>> memset is defined there :)
>>>
>>>
>>>
>>>>
>>>> -Man
>>>>
>>>>
>>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Done on all points I believe Man.
>>>>>
>>>>> I added the JvmtiTagMap change and added a new test for that as well.
>>>>> The test was a bit annoying due to the fact that it was difficult to
>>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>>
>>>>> I'm unclear if that is right, it seems that it is due to the fact that
>>>>> it does not see the VM operation as a fork/join operation perhaps? I
>>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>>> style callback so we might have more work there.
>>>>>
>>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>>
>>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it seems
>>>>> half the #endif do it and half don't so I'm not sure what is the right
>>>>> approach.
>>>>>
>>>>> Let me know what you think,
>>>>> Jc
>>>>>
>>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>>
>>>>>> Code in HotSpot looks good. Except that we have additional
>>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>>> change.
>>>>>>
>>>>>> A few comments on tests:
>>>>>> In AbstractNativeLoop.java:
>>>>>> Would it make sense to make all native methods "static"? They don't
>>>>>> access any instance field, right?
>>>>>>
>>>>>> In NonRawRacyNativeLoopTest.java:
>>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>>> Could the one in main() be deleted?
>>>>>>
>>>>>> In RawRacyNativeLoopTest.java:
>>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>>> I think calling createRawLock() in run() would result in each thread
>>>>>> creating 50000 raw monitors, due to the way in which AbstractLoop is
>>>>>> written. Can we avoid doing this so the test can be a bit faster?
>>>>>>
>>>>>> -Man
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>>> jcbeyler at google.com> wrote:
>>>>>>
>>>>>>> Sounds good to me.
>>>>>>>
>>>>>>> Here it is:
>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>>
>>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it
>>>>>>> works well
>>>>>>> for one liners for code readability to be honest. If we want this,
>>>>>>> I'll do
>>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>>> separate
>>>>>>> webrev.
>>>>>>> cf:
>>>>>>>
>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>>
>>>>>>> And now with Arthur's change, all tests now pass (still would like
>>>>>>> to fix
>>>>>>> the racing tests to be more restrictive),
>>>>>>> Jc
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <aeubanks at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>>> interface
>>>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>>>> file would
>>>>>>> > include tsanExternalDecls.hpp. The other files would include the
>>>>>>> new header
>>>>>>> > file.
>>>>>>> >
>>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>>> > jcbeyler at google.com> wrote:
>>>>>>> >
>>>>>>> >> Hi Arthur,
>>>>>>> >>
>>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>>> Seems like
>>>>>>> >> it is what we did internally and what you were doing for the lock
>>>>>>> webrev as
>>>>>>> >> well, ie have that as our include header. Are you suggesting we
>>>>>>> make
>>>>>>> >> another tsan header for these methods.
>>>>>>> >>
>>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>>> webrev does
>>>>>>> >> things by default, the indentation is actually correct in my
>>>>>>> code),
>>>>>>> >> Jc
>>>>>>> >>
>>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>>> aeubanks at google.com>
>>>>>>> >> wrote:
>>>>>>> >>
>>>>>>> >>> Overall:
>>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>>>> have a
>>>>>>> >>> wrapper around them. Then we can have things like checking if
>>>>>>> the lock is
>>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>>> tsanExternalDecls.hpp
>>>>>>> >>> seems wrong.
>>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could you
>>>>>>> add the
>>>>>>> >>> comment for the #endifs.
>>>>>>> >>>
>>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>>> >>>
>>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>>> >>> Indentation around 3258 is wrong.
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>>> >>>
>>>>>>> >>>> Hi all,
>>>>>>> >>>>
>>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>>> tests, it is
>>>>>>> >>>> failing the non racy ones due to the JVM locks not yet included
>>>>>>> but I
>>>>>>> >>>> think
>>>>>>> >>>> that will resolve itself soon :)
>>>>>>> >>>>
>>>>>>> >>>> I also moved the agent library to C++.
>>>>>>> >>>>
>>>>>>> >>>> FWIW, we also need to clean up our tests that check for
>>>>>>> failing, right
>>>>>>> >>>> now
>>>>>>> >>>> they pass because they generically just are checking for a TSAN
>>>>>>> race but
>>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>>> >>>>
>>>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>>> >>>>
>>>>>>> >>>> Let me know what you think,
>>>>>>> >>>> Jc
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >>
>>>>>>> >> Thanks,
>>>>>>> >> Jc
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jc
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From jcbeyler at google.com  Mon Apr 29 21:47:08 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 29 Apr 2019 14:47:08 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
Message-ID: <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>

Hi Man & Arthur,

New webrev is here:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.05/

Let me know what you both think :)
Jc


-------------------------------------

On Mon, Apr 29, 2019 at 2:18 PM Man Cao <manc at google.com> wrote:

> webrev.04 looks good to me.
> Minor comment is that I'd add TODO or comment about
> pthread_mutex_lock/unlock should be removed from PerObjectCallback()
> and Java_JvmtiTaggerLoopRunner_iterateOverTags()
> in libAbstractNativeLoop.cpp, once we have handled the lock creation race.
> No need for a webrev for this.
>
>
Done :)


> For the lock creation race in JvmtiTagMap::JvmtiTagMap(), I think there is
> a valid C++ data race in double-checked locking
> in JvmtiTagMap::tag_map_for().
> Ideally accesses to JvmtiEnvBase::_tag_map should use
> OrderAccess::load_acquire() and OrderAccess::release_store(), and we should
> make TSAN aware of these fence semantics, then TSAN would not complain
> about the lock creation race.
> I can try sending serviceability-dev@ a change for OrderAccess.
>

For what it's worth, even if you fix that, you still will get a race from
TSAN since tsan does not care about this particular race; it cares about
the lock creation in the instance itself. You would need to fix that and
not the setting of this tag map.
Good news is that you can use the jvmtiTaggertest to see if you are right
and succeed :)


>
> Lastly for Arthur's comment:
> > test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
> > Does "lock" have to be static?
> ThreadLocals are typically private static fields by convention:
> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ThreadLocal.html
> .
>
> -Man
>
>
> On Mon, Apr 29, 2019 at 1:10 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
>> -  return new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>> +  void *mon = (void*) new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>> +
>> +  TSAN_RAW_LOCK_CREATE(mon);
>> +  return mon;
>>
>> src/hotspot/share/prims/jvmtiEnv.cpp: the comment shouldn't be there, or
>> it should be updated
>> +  // A wait is modeled in Tsan as a simple release-acquire pair.
>> +  // The matching acquire annotation is below.
>> +  TSAN_RAW_LOCK_ACQUIRED(rmonitor);
>> +
>>    switch (r) {
>>    case ObjectMonitor::OM_INTERRUPTED:
>>      return JVMTI_ERROR_INTERRUPT;
>>    case ObjectMonitor::OM_ILLEGAL_MONITOR_STATE:
>>      return JVMTI_ERROR_NOT_MONITOR_OWNER;
>>
>> src/hotspot/share/prims/jvmtiTagMap.cpp:
>> It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
>> makes the separation of Tsan-specific features and the rest of the JVM less
>> clear. I would prefer something like
>> #if INCLUDE_TSAN
>> TsanMutexScope tsanMutexScope(lock());
>> #endif
>> as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
>> TsanVisibleMutexLocker uses multiple inheritance? ew. What about having
>> the two superclasses as two member variables instead?
>>
>> I don't understand "good thing" in the comments below:
>> +  // This means that this lock is created under a mutex (good thing) but
>> then,
>> +  // subsequent uses do have a lock to protect it (good thing because not
>> +  // needed in this case), however TSAN sees it as being needed because:
>> +  //  - Another thread can come and get the newly created JvmtiTagMap
>> without a
>> +  //  lock and acquire the lock.
>> +  //  - This provokes a race for TSAN on the lock itself, though there
>> is no
>> +  //  real issue.
>>
>> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
>> Does "lock" have to be static?
>>
>> On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Seems like I forgot to say two things:
>>>
>>> 1) First here is the link to the newest webrev:
>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
>>> 2) JTREG does not really allow two different native files, so as far as
>>> I can tell all other examples do what I've done here for -agentlib and
>>> System.loadlibrary
>>>
>>> Thanks,
>>> Jc
>>>
>>> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Hi Man,
>>>>
>>>> New webrev is here:
>>>>
>>>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>>>
>>>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>>>
>>>>
>>>> So funny thing I can't do that actually. When I do, TSAN complains
>>>> about a race between the create and the lock itself. I don't know why it
>>>> didn't show up in our internal testing but I've added a note to it. I'm
>>>> going to run some tests on our version to see if I'm right that it just
>>>> didn't show up due to how we were using it (enough time between the create
>>>> and the iterate to not show up). But my study of the code this morning
>>>> seems to lead to : yes TSAN sees it as a race but it is not really one.
>>>>
>>>>
>>>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>>>> JvmtiTagMap::get_objects_with_tags().
>>>>>
>>>>
>>>> Done, there was another one as well that now I did the same to. Same as
>>>> above, I think we just did the minimal that we saw, but not the full set.
>>>>
>>>> (Note: I think I'm missing other tag callbacks that should be checked
>>>> for races, I'll check the code and see in a second phase; probably with a
>>>> more extensive study of the create/lock tsan race)
>>>>
>>>>
>>>>> Will adding these fix the issue with "sum" in the test?
>>>>>
>>>>
>>>> No they don't, I still have the same issue, the problem is that sum is
>>>> changed without a lock in user code, and I fail to see how to do a "fence"
>>>> for tsan from user to JVM and back. Except if we have a solution right now,
>>>> I'd rather figure this out in a subsequent webrev as this webrev is already
>>>> growing quite a bit, if that's ok :-)
>>>>
>>>>
>>>>>
>>>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>>>> again in class initializer. I'm not sure if this could cause any unexpected
>>>>> behavior, but it is probably better to avoid doing this. Could we put
>>>>> JVMTI-related code in a new file?
>>>>>
>>>>> In RawRacyNativeLoopTest, I think there is a race on the "private
>>>>> long lock" field itself. Two threads will write different values to the
>>>>> same field, so TSAN will likely report a race on this field regardless of
>>>>> what happens in JVMTI code.
>>>>>
>>>>
>>>> Done, added a threadlocal instead to make it really localized. This is
>>>> yet another reason I'd like our race detection in our tests to check the
>>>> signatures but we need symbolization first (hint hint).
>>>> Also tested and it now fails with the right race.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Minor comments:
>>>>> I think a better name for TsanSimulatedMutexLocker could be
>>>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>>>
>>>>
>>>> Done, I think the StackObj would have been caught when running this in
>>>> fastdebug, which we have not yet set up to be done by default in our
>>>> testing...
>>>>
>>>>
>>>>> I'd keep our internal comment about "Simulates barrier synchronization
>>>>> on safepoint" in doit() functions in jvmtiTagMap.cpp. I think they are
>>>>> quite useful to explain why we need the TSAN instrumentation.
>>>>>
>>>>
>>>> Done.
>>>>
>>>>
>>>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't see
>>>>> any string functions?
>>>>>
>>>>
>>>> memset is defined there :)
>>>>
>>>>
>>>>
>>>>>
>>>>> -Man
>>>>>
>>>>>
>>>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>>>> jcbeyler at google.com> wrote:
>>>>>
>>>>>> Done on all points I believe Man.
>>>>>>
>>>>>> I added the JvmtiTagMap change and added a new test for that as well.
>>>>>> The test was a bit annoying due to the fact that it was difficult to
>>>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>>>
>>>>>> I'm unclear if that is right, it seems that it is due to the fact
>>>>>> that it does not see the VM operation as a fork/join operation perhaps? I
>>>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>>>> style callback so we might have more work there.
>>>>>>
>>>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>>>
>>>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it
>>>>>> seems half the #endif do it and half don't so I'm not sure what is the
>>>>>> right approach.
>>>>>>
>>>>>> Let me know what you think,
>>>>>> Jc
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>>>
>>>>>>> Code in HotSpot looks good. Except that we have additional
>>>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>>>> change.
>>>>>>>
>>>>>>> A few comments on tests:
>>>>>>> In AbstractNativeLoop.java:
>>>>>>> Would it make sense to make all native methods "static"? They don't
>>>>>>> access any instance field, right?
>>>>>>>
>>>>>>> In NonRawRacyNativeLoopTest.java:
>>>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>>>> Could the one in main() be deleted?
>>>>>>>
>>>>>>> In RawRacyNativeLoopTest.java:
>>>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>>>> I think calling createRawLock() in run() would result in each
>>>>>>> thread creating 50000 raw monitors, due to the way in which
>>>>>>> AbstractLoop is written. Can we avoid doing this so the test can be a bit
>>>>>>> faster?
>>>>>>>
>>>>>>> -Man
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>>>> jcbeyler at google.com> wrote:
>>>>>>>
>>>>>>>> Sounds good to me.
>>>>>>>>
>>>>>>>> Here it is:
>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>>>
>>>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it
>>>>>>>> works well
>>>>>>>> for one liners for code readability to be honest. If we want this,
>>>>>>>> I'll do
>>>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>>>> separate
>>>>>>>> webrev.
>>>>>>>> cf:
>>>>>>>>
>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>>>
>>>>>>>> And now with Arthur's change, all tests now pass (still would like
>>>>>>>> to fix
>>>>>>>> the racing tests to be more restrictive),
>>>>>>>> Jc
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <
>>>>>>>> aeubanks at google.com> wrote:
>>>>>>>>
>>>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>>>> interface
>>>>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>>>>> file would
>>>>>>>> > include tsanExternalDecls.hpp. The other files would include the
>>>>>>>> new header
>>>>>>>> > file.
>>>>>>>> >
>>>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>>>> > jcbeyler at google.com> wrote:
>>>>>>>> >
>>>>>>>> >> Hi Arthur,
>>>>>>>> >>
>>>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>>>> Seems like
>>>>>>>> >> it is what we did internally and what you were doing for the
>>>>>>>> lock webrev as
>>>>>>>> >> well, ie have that as our include header. Are you suggesting we
>>>>>>>> make
>>>>>>>> >> another tsan header for these methods.
>>>>>>>> >>
>>>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>>>> webrev does
>>>>>>>> >> things by default, the indentation is actually correct in my
>>>>>>>> code),
>>>>>>>> >> Jc
>>>>>>>> >>
>>>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>>>> aeubanks at google.com>
>>>>>>>> >> wrote:
>>>>>>>> >>
>>>>>>>> >>> Overall:
>>>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>>>>> have a
>>>>>>>> >>> wrapper around them. Then we can have things like checking if
>>>>>>>> the lock is
>>>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>>>> tsanExternalDecls.hpp
>>>>>>>> >>> seems wrong.
>>>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could
>>>>>>>> you add the
>>>>>>>> >>> comment for the #endifs.
>>>>>>>> >>>
>>>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>>>> >>>
>>>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>>>> >>> Indentation around 3258 is wrong.
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>>>> >>>
>>>>>>>> >>>> Hi all,
>>>>>>>> >>>>
>>>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>>>> tests, it is
>>>>>>>> >>>> failing the non racy ones due to the JVM locks not yet
>>>>>>>> included but I
>>>>>>>> >>>> think
>>>>>>>> >>>> that will resolve itself soon :)
>>>>>>>> >>>>
>>>>>>>> >>>> I also moved the agent library to C++.
>>>>>>>> >>>>
>>>>>>>> >>>> FWIW, we also need to clean up our tests that check for
>>>>>>>> failing, right
>>>>>>>> >>>> now
>>>>>>>> >>>> they pass because they generically just are checking for a
>>>>>>>> TSAN race but
>>>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>>>> >>>>
>>>>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>>>> >>>>
>>>>>>>> >>>> Let me know what you think,
>>>>>>>> >>>> Jc
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >>
>>>>>>>> >> Thanks,
>>>>>>>> >> Jc
>>>>>>>> >>
>>>>>>>> >
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jc
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Jc
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>

-- 

Thanks,
Jc

From jcbeyler at google.com  Mon Apr 29 21:48:22 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 29 Apr 2019 14:48:22 -0700
Subject: Symbolization
In-Reply-To: <CA+w6HxY7YHbvBYnGC2Ort7V1hFBEpVDSTMhcwuAH9a7d9n2Z6g@mail.gmail.com>
References: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
 <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>
 <CA+w6HxY7YHbvBYnGC2Ort7V1hFBEpVDSTMhcwuAH9a7d9n2Z6g@mail.gmail.com>
Message-ID: <CAF9BGBwr+Khm0=a2uTvHQ3wEvoUpkJpteTF9sthbK-o7OswUjQ@mail.gmail.com>

FWIW, it does not really matter to me at this point what we say to do as
long as it works for us. At some point, when we get a bit further along, we
would want to figure out a "sane" way of doing things; requiring to compile
GCC by hand is not sane in my opinion but is ok for now :)
Jc

On Mon, Apr 29, 2019 at 2:36 PM Man Cao <manc at google.com> wrote:

> I think the best way is still to find a way to build libtsan.so from
> latest source, with the GCC version supported by OpenJDK:
> https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms.
> Even if we can get libtsan.so from a newer GCC, I'm not sure if it is safe
> to build OpenJDK with the newer GCC, or to use a newer libtsan.so with an
> old GCC.
>
> -Man
>
>
> On Mon, Apr 29, 2019 at 2:18 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Compiling gcc to get libtsan... Is there no way to build libtsan then from
>> scratch instead?
>>
>> On Mon, Apr 29, 2019 at 2:00 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>> > It looks like the libtsan.so from gcc 8 is out of date. It doesn't
>> > contain __tsan_symbolize_external_ex(), only the
>> > legacy __tsan_symbolize_external(). This messes with testing the Java
>> Tsan
>> > patches.
>> >
>> > $ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym
>> >
>> > I looked at the latest gcc sources and it does have a version of Tsan
>> that
>> > includes __tsan_symbolize_external_ex(). We could tell people to compile
>> > their own gcc and use that libtsan.so.
>> > $ apt search gcc
>> > only shows up to gcc 8 on my machine.
>> >
>> > Thoughts?
>> >
>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Mon Apr 29 21:58:00 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 29 Apr 2019 14:58:00 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
Message-ID: <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>

I don't see the comment change in src/hotspot/share/prims/jvmtiEnv.cpp.
Other than that, LGTM. (no need for another webrev)

On Mon, Apr 29, 2019 at 2:47 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Hi Man & Arthur,
>
> New webrev is here:
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.05/
>
> Let me know what you both think :)
> Jc
>
>
> -------------------------------------
>
> On Mon, Apr 29, 2019 at 2:18 PM Man Cao <manc at google.com> wrote:
>
>> webrev.04 looks good to me.
>> Minor comment is that I'd add TODO or comment about
>> pthread_mutex_lock/unlock should be removed from PerObjectCallback()
>> and Java_JvmtiTaggerLoopRunner_iterateOverTags()
>> in libAbstractNativeLoop.cpp, once we have handled the lock creation race.
>> No need for a webrev for this.
>>
>>
> Done :)
>
>
>> For the lock creation race in JvmtiTagMap::JvmtiTagMap(), I think there
>> is a valid C++ data race in double-checked locking
>> in JvmtiTagMap::tag_map_for().
>> Ideally accesses to JvmtiEnvBase::_tag_map should use
>> OrderAccess::load_acquire() and OrderAccess::release_store(), and we should
>> make TSAN aware of these fence semantics, then TSAN would not complain
>> about the lock creation race.
>> I can try sending serviceability-dev@ a change for OrderAccess.
>>
>
> For what it's worth, even if you fix that, you still will get a race from
> TSAN since tsan does not care about this particular race; it cares about
> the lock creation in the instance itself. You would need to fix that and
> not the setting of this tag map.
> Good news is that you can use the jvmtiTaggertest to see if you are right
> and succeed :)
>
>
>>
>> Lastly for Arthur's comment:
>> > test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
>> > Does "lock" have to be static?
>> ThreadLocals are typically private static fields by convention:
>> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ThreadLocal.html
>> .
>>
>> -Man
>>
>>
>> On Mon, Apr 29, 2019 at 1:10 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
>>> -  return new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>>> +  void *mon = (void*) new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>>> +
>>> +  TSAN_RAW_LOCK_CREATE(mon);
>>> +  return mon;
>>>
>>> src/hotspot/share/prims/jvmtiEnv.cpp: the comment shouldn't be there, or
>>> it should be updated
>>> +  // A wait is modeled in Tsan as a simple release-acquire pair.
>>> +  // The matching acquire annotation is below.
>>> +  TSAN_RAW_LOCK_ACQUIRED(rmonitor);
>>> +
>>>    switch (r) {
>>>    case ObjectMonitor::OM_INTERRUPTED:
>>>      return JVMTI_ERROR_INTERRUPT;
>>>    case ObjectMonitor::OM_ILLEGAL_MONITOR_STATE:
>>>      return JVMTI_ERROR_NOT_MONITOR_OWNER;
>>>
>>> src/hotspot/share/prims/jvmtiTagMap.cpp:
>>> It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
>>> makes the separation of Tsan-specific features and the rest of the JVM less
>>> clear. I would prefer something like
>>> #if INCLUDE_TSAN
>>> TsanMutexScope tsanMutexScope(lock());
>>> #endif
>>> as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
>>> TsanVisibleMutexLocker uses multiple inheritance? ew. What about having
>>> the two superclasses as two member variables instead?
>>>
>>> I don't understand "good thing" in the comments below:
>>> +  // This means that this lock is created under a mutex (good thing)
>>> but then,
>>> +  // subsequent uses do have a lock to protect it (good thing because
>>> not
>>> +  // needed in this case), however TSAN sees it as being needed because:
>>> +  //  - Another thread can come and get the newly created JvmtiTagMap
>>> without a
>>> +  //  lock and acquire the lock.
>>> +  //  - This provokes a race for TSAN on the lock itself, though there
>>> is no
>>> +  //  real issue.
>>>
>>> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
>>> Does "lock" have to be static?
>>>
>>> On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Seems like I forgot to say two things:
>>>>
>>>> 1) First here is the link to the newest webrev:
>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
>>>> 2) JTREG does not really allow two different native files, so as far as
>>>> I can tell all other examples do what I've done here for -agentlib and
>>>> System.loadlibrary
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Hi Man,
>>>>>
>>>>> New webrev is here:
>>>>>
>>>>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>>>>
>>>>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>>>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>>>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>>>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>>>>
>>>>>
>>>>> So funny thing I can't do that actually. When I do, TSAN complains
>>>>> about a race between the create and the lock itself. I don't know why it
>>>>> didn't show up in our internal testing but I've added a note to it. I'm
>>>>> going to run some tests on our version to see if I'm right that it just
>>>>> didn't show up due to how we were using it (enough time between the create
>>>>> and the iterate to not show up). But my study of the code this morning
>>>>> seems to lead to : yes TSAN sees it as a race but it is not really one.
>>>>>
>>>>>
>>>>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>>>>> JvmtiTagMap::get_objects_with_tags().
>>>>>>
>>>>>
>>>>> Done, there was another one as well that now I did the same to. Same
>>>>> as above, I think we just did the minimal that we saw, but not the full set.
>>>>>
>>>>> (Note: I think I'm missing other tag callbacks that should be checked
>>>>> for races, I'll check the code and see in a second phase; probably with a
>>>>> more extensive study of the create/lock tsan race)
>>>>>
>>>>>
>>>>>> Will adding these fix the issue with "sum" in the test?
>>>>>>
>>>>>
>>>>> No they don't, I still have the same issue, the problem is that sum is
>>>>> changed without a lock in user code, and I fail to see how to do a "fence"
>>>>> for tsan from user to JVM and back. Except if we have a solution right now,
>>>>> I'd rather figure this out in a subsequent webrev as this webrev is already
>>>>> growing quite a bit, if that's ok :-)
>>>>>
>>>>>
>>>>>>
>>>>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>>>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>>>>> again in class initializer. I'm not sure if this could cause any unexpected
>>>>>> behavior, but it is probably better to avoid doing this. Could we put
>>>>>> JVMTI-related code in a new file?
>>>>>>
>>>>>> In RawRacyNativeLoopTest, I think there is a race on the "private
>>>>>> long lock" field itself. Two threads will write different values to the
>>>>>> same field, so TSAN will likely report a race on this field regardless of
>>>>>> what happens in JVMTI code.
>>>>>>
>>>>>
>>>>> Done, added a threadlocal instead to make it really localized. This is
>>>>> yet another reason I'd like our race detection in our tests to check the
>>>>> signatures but we need symbolization first (hint hint).
>>>>> Also tested and it now fails with the right race.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Minor comments:
>>>>>> I think a better name for TsanSimulatedMutexLocker could be
>>>>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>>>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>>>>
>>>>>
>>>>> Done, I think the StackObj would have been caught when running this in
>>>>> fastdebug, which we have not yet set up to be done by default in our
>>>>> testing...
>>>>>
>>>>>
>>>>>> I'd keep our internal comment about "Simulates barrier
>>>>>> synchronization on safepoint" in doit() functions in jvmtiTagMap.cpp. I
>>>>>> think they are quite useful to explain why we need the TSAN instrumentation.
>>>>>>
>>>>>
>>>>> Done.
>>>>>
>>>>>
>>>>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't
>>>>>> see any string functions?
>>>>>>
>>>>>
>>>>> memset is defined there :)
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> -Man
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>>>>> jcbeyler at google.com> wrote:
>>>>>>
>>>>>>> Done on all points I believe Man.
>>>>>>>
>>>>>>> I added the JvmtiTagMap change and added a new test for that as
>>>>>>> well. The test was a bit annoying due to the fact that it was difficult to
>>>>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>>>>
>>>>>>> I'm unclear if that is right, it seems that it is due to the fact
>>>>>>> that it does not see the VM operation as a fork/join operation perhaps? I
>>>>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>>>>> style callback so we might have more work there.
>>>>>>>
>>>>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>>>>
>>>>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it
>>>>>>> seems half the #endif do it and half don't so I'm not sure what is the
>>>>>>> right approach.
>>>>>>>
>>>>>>> Let me know what you think,
>>>>>>> Jc
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>>>>
>>>>>>>> Code in HotSpot looks good. Except that we have additional
>>>>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>>>>> change.
>>>>>>>>
>>>>>>>> A few comments on tests:
>>>>>>>> In AbstractNativeLoop.java:
>>>>>>>> Would it make sense to make all native methods "static"? They don't
>>>>>>>> access any instance field, right?
>>>>>>>>
>>>>>>>> In NonRawRacyNativeLoopTest.java:
>>>>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>>>>> Could the one in main() be deleted?
>>>>>>>>
>>>>>>>> In RawRacyNativeLoopTest.java:
>>>>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>>>>> I think calling createRawLock() in run() would result in each
>>>>>>>> thread creating 50000 raw monitors, due to the way in which
>>>>>>>> AbstractLoop is written. Can we avoid doing this so the test can be a bit
>>>>>>>> faster?
>>>>>>>>
>>>>>>>> -Man
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>>>>> jcbeyler at google.com> wrote:
>>>>>>>>
>>>>>>>>> Sounds good to me.
>>>>>>>>>
>>>>>>>>> Here it is:
>>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>>>>
>>>>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it
>>>>>>>>> works well
>>>>>>>>> for one liners for code readability to be honest. If we want this,
>>>>>>>>> I'll do
>>>>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>>>>> separate
>>>>>>>>> webrev.
>>>>>>>>> cf:
>>>>>>>>>
>>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>>>>
>>>>>>>>> And now with Arthur's change, all tests now pass (still would like
>>>>>>>>> to fix
>>>>>>>>> the racing tests to be more restrictive),
>>>>>>>>> Jc
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <
>>>>>>>>> aeubanks at google.com> wrote:
>>>>>>>>>
>>>>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>>>>> interface
>>>>>>>>> > for the other TSAN functions is through SharedRuntime. Only that
>>>>>>>>> file would
>>>>>>>>> > include tsanExternalDecls.hpp. The other files would include the
>>>>>>>>> new header
>>>>>>>>> > file.
>>>>>>>>> >
>>>>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>>>>> > jcbeyler at google.com> wrote:
>>>>>>>>> >
>>>>>>>>> >> Hi Arthur,
>>>>>>>>> >>
>>>>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>>>>> Seems like
>>>>>>>>> >> it is what we did internally and what you were doing for the
>>>>>>>>> lock webrev as
>>>>>>>>> >> well, ie have that as our include header. Are you suggesting we
>>>>>>>>> make
>>>>>>>>> >> another tsan header for these methods.
>>>>>>>>> >>
>>>>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>>>>> webrev does
>>>>>>>>> >> things by default, the indentation is actually correct in my
>>>>>>>>> code),
>>>>>>>>> >> Jc
>>>>>>>>> >>
>>>>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>>>>> aeubanks at google.com>
>>>>>>>>> >> wrote:
>>>>>>>>> >>
>>>>>>>>> >>> Overall:
>>>>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we should
>>>>>>>>> have a
>>>>>>>>> >>> wrapper around them. Then we can have things like checking if
>>>>>>>>> the lock is
>>>>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>>>>> tsanExternalDecls.hpp
>>>>>>>>> >>> seems wrong.
>>>>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could
>>>>>>>>> you add the
>>>>>>>>> >>> comment for the #endifs.
>>>>>>>>> >>>
>>>>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>>>>> >>>
>>>>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>>>>> >>> Indentation around 3258 is wrong.
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>>> Hi all,
>>>>>>>>> >>>>
>>>>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>>>>> tests, it is
>>>>>>>>> >>>> failing the non racy ones due to the JVM locks not yet
>>>>>>>>> included but I
>>>>>>>>> >>>> think
>>>>>>>>> >>>> that will resolve itself soon :)
>>>>>>>>> >>>>
>>>>>>>>> >>>> I also moved the agent library to C++.
>>>>>>>>> >>>>
>>>>>>>>> >>>> FWIW, we also need to clean up our tests that check for
>>>>>>>>> failing, right
>>>>>>>>> >>>> now
>>>>>>>>> >>>> they pass because they generically just are checking for a
>>>>>>>>> TSAN race but
>>>>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>>>>> >>>>
>>>>>>>>> >>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>>>>> >>>>
>>>>>>>>> >>>> Let me know what you think,
>>>>>>>>> >>>> Jc
>>>>>>>>> >>>>
>>>>>>>>> >>>
>>>>>>>>> >>
>>>>>>>>> >> --
>>>>>>>>> >>
>>>>>>>>> >> Thanks,
>>>>>>>>> >> Jc
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jc
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jc
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>
> --
>
> Thanks,
> Jc
>

From manc at google.com  Mon Apr 29 22:03:28 2019
From: manc at google.com (Man Cao)
Date: Mon, 29 Apr 2019 15:03:28 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
Message-ID: <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>

Just discussed with JC that the following will not work,
because TSAN_RUNTIME_ONLY defines its own scope, and it will immediately
call the destructor:
TSAN_RUNTIME_ONLY(TsanMutexScope tms(lock()));


-Man


On Mon, Apr 29, 2019 at 2:58 PM Arthur Eubanks <aeubanks at google.com> wrote:

> I don't see the comment change in src/hotspot/share/prims/jvmtiEnv.cpp.
> Other than that, LGTM. (no need for another webrev)
>
> On Mon, Apr 29, 2019 at 2:47 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Hi Man & Arthur,
>>
>> New webrev is here:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.05/
>>
>> Let me know what you both think :)
>> Jc
>>
>>
>> -------------------------------------
>>
>> On Mon, Apr 29, 2019 at 2:18 PM Man Cao <manc at google.com> wrote:
>>
>>> webrev.04 looks good to me.
>>> Minor comment is that I'd add TODO or comment about
>>> pthread_mutex_lock/unlock should be removed from PerObjectCallback()
>>> and Java_JvmtiTaggerLoopRunner_iterateOverTags()
>>> in libAbstractNativeLoop.cpp, once we have handled the lock creation race.
>>> No need for a webrev for this.
>>>
>>>
>> Done :)
>>
>>
>>> For the lock creation race in JvmtiTagMap::JvmtiTagMap(), I think there
>>> is a valid C++ data race in double-checked locking
>>> in JvmtiTagMap::tag_map_for().
>>> Ideally accesses to JvmtiEnvBase::_tag_map should use
>>> OrderAccess::load_acquire() and OrderAccess::release_store(), and we should
>>> make TSAN aware of these fence semantics, then TSAN would not complain
>>> about the lock creation race.
>>> I can try sending serviceability-dev@ a change for OrderAccess.
>>>
>>
>> For what it's worth, even if you fix that, you still will get a race from
>> TSAN since tsan does not care about this particular race; it cares about
>> the lock creation in the instance itself. You would need to fix that and
>> not the setting of this tag map.
>> Good news is that you can use the jvmtiTaggertest to see if you are right
>> and succeed :)
>>
>>
>>>
>>> Lastly for Arthur's comment:
>>> > test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
>>> > Does "lock" have to be static?
>>> ThreadLocals are typically private static fields by convention:
>>> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ThreadLocal.html
>>> .
>>>
>>> -Man
>>>
>>>
>>> On Mon, Apr 29, 2019 at 1:10 PM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>>> src/hotspot/share/prims/jvm.cpp: is the cast to (void *) necessary?
>>>> -  return new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>>>> +  void *mon = (void*) new Mutex(Mutex::native, "JVM_RawMonitorCreate");
>>>> +
>>>> +  TSAN_RAW_LOCK_CREATE(mon);
>>>> +  return mon;
>>>>
>>>> src/hotspot/share/prims/jvmtiEnv.cpp: the comment shouldn't be there,
>>>> or it should be updated
>>>> +  // A wait is modeled in Tsan as a simple release-acquire pair.
>>>> +  // The matching acquire annotation is below.
>>>> +  TSAN_RAW_LOCK_ACQUIRED(rmonitor);
>>>> +
>>>>    switch (r) {
>>>>    case ObjectMonitor::OM_INTERRUPTED:
>>>>      return JVMTI_ERROR_INTERRUPT;
>>>>    case ObjectMonitor::OM_ILLEGAL_MONITOR_STATE:
>>>>      return JVMTI_ERROR_NOT_MONITOR_OWNER;
>>>>
>>>> src/hotspot/share/prims/jvmtiTagMap.cpp:
>>>> It seems wrong to be mixing in Tsan specific stuff with MutexLocker. It
>>>> makes the separation of Tsan-specific features and the rest of the JVM less
>>>> clear. I would prefer something like
>>>> #if INCLUDE_TSAN
>>>> TsanMutexScope tsanMutexScope(lock());
>>>> #endif
>>>> as opposed to replacing MutexLocker with TsanVisibleMutexLocker.
>>>> TsanVisibleMutexLocker uses multiple inheritance? ew. What about having
>>>> the two superclasses as two member variables instead?
>>>>
>>>> I don't understand "good thing" in the comments below:
>>>> +  // This means that this lock is created under a mutex (good thing)
>>>> but then,
>>>> +  // subsequent uses do have a lock to protect it (good thing because
>>>> not
>>>> +  // needed in this case), however TSAN sees it as being needed
>>>> because:
>>>> +  //  - Another thread can come and get the newly created JvmtiTagMap
>>>> without a
>>>> +  //  lock and acquire the lock.
>>>> +  //  - This provokes a race for TSAN on the lock itself, though there
>>>> is no
>>>> +  //  real issue.
>>>>
>>>> test/hotspot/jtreg/tsan/RawRacyNativeLoopTest.java:
>>>> Does "lock" have to be static?
>>>>
>>>> On Fri, Apr 26, 2019 at 2:43 PM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Seems like I forgot to say two things:
>>>>>
>>>>> 1) First here is the link to the newest webrev:
>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.04/
>>>>> 2) JTREG does not really allow two different native files, so as far
>>>>> as I can tell all other examples do what I've done here for -agentlib and
>>>>> System.loadlibrary
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>> On Fri, Apr 26, 2019 at 9:50 AM Jean Christophe Beyler <
>>>>> jcbeyler at google.com> wrote:
>>>>>
>>>>>> Hi Man,
>>>>>>
>>>>>> New webrev is here:
>>>>>>
>>>>>> On Thu, Apr 25, 2019 at 10:41 PM Man Cao <manc at google.com> wrote:
>>>>>>
>>>>>>> It seems there are a few missing instrumentation in jvmtiTagMap.cpp:
>>>>>>> 1. There should be a call to TSAN_RAW_LOCK_CREATE(&_lock) in
>>>>>>> JvmtiTagMap::JvmtiTagMap(JvmtiEnv* env), and a call to
>>>>>>> TSAN_RAW_LOCK_DESTROY(&_lock) in JvmtiTagMap::~JvmtiTagMap().
>>>>>>>
>>>>>>
>>>>>> So funny thing I can't do that actually. When I do, TSAN complains
>>>>>> about a race between the create and the lock itself. I don't know why it
>>>>>> didn't show up in our internal testing but I've added a note to it. I'm
>>>>>> going to run some tests on our version to see if I'm right that it just
>>>>>> didn't show up due to how we were using it (enough time between the create
>>>>>> and the iterate to not show up). But my study of the code this morning
>>>>>> seems to lead to : yes TSAN sees it as a race but it is not really one.
>>>>>>
>>>>>>
>>>>>>> 2. TsanVisibleMutexLocker should replace MutexLocker in
>>>>>>> JvmtiTagMap::get_objects_with_tags().
>>>>>>>
>>>>>>
>>>>>> Done, there was another one as well that now I did the same to. Same
>>>>>> as above, I think we just did the minimal that we saw, but not the full set.
>>>>>>
>>>>>> (Note: I think I'm missing other tag callbacks that should be checked
>>>>>> for races, I'll check the code and see in a second phase; probably with a
>>>>>> more extensive study of the create/lock tsan race)
>>>>>>
>>>>>>
>>>>>>> Will adding these fix the issue with "sum" in the test?
>>>>>>>
>>>>>>
>>>>>> No they don't, I still have the same issue, the problem is that sum
>>>>>> is changed without a lock in user code, and I fail to see how to do a
>>>>>> "fence" for tsan from user to JVM and back. Except if we have a solution
>>>>>> right now, I'd rather figure this out in a subsequent webrev as this webrev
>>>>>> is already growing quite a bit, if that's ok :-)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> For the new tests, there is "-agentlib:AbstractNativeLoop" on the
>>>>>>> command line, and they will do System.loadLibrary("AbstractNativeLoop")
>>>>>>> again in class initializer. I'm not sure if this could cause any unexpected
>>>>>>> behavior, but it is probably better to avoid doing this. Could we put
>>>>>>> JVMTI-related code in a new file?
>>>>>>>
>>>>>>> In RawRacyNativeLoopTest, I think there is a race on the "private
>>>>>>> long lock" field itself. Two threads will write different values to the
>>>>>>> same field, so TSAN will likely report a race on this field regardless of
>>>>>>> what happens in JVMTI code.
>>>>>>>
>>>>>>
>>>>>> Done, added a threadlocal instead to make it really localized. This
>>>>>> is yet another reason I'd like our race detection in our tests to check the
>>>>>> signatures but we need symbolization first (hint hint).
>>>>>> Also tested and it now fails with the right race.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Minor comments:
>>>>>>> I think a better name for TsanSimulatedMutexLocker could be
>>>>>>> TsanMutexScope, because TsanSimulatedMutexLocker has nothing to do with the
>>>>>>> MutexLocker class. Also, this class could be a subclase of StackObj.
>>>>>>>
>>>>>>
>>>>>> Done, I think the StackObj would have been caught when running this
>>>>>> in fastdebug, which we have not yet set up to be done by default in our
>>>>>> testing...
>>>>>>
>>>>>>
>>>>>>> I'd keep our internal comment about "Simulates barrier
>>>>>>> synchronization on safepoint" in doit() functions in jvmtiTagMap.cpp. I
>>>>>>> think they are quite useful to explain why we need the TSAN instrumentation.
>>>>>>>
>>>>>>
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>>> In libAbstractNativeLoop.cpp, why "include <string.h>" but I don't
>>>>>>> see any string functions?
>>>>>>>
>>>>>>
>>>>>> memset is defined there :)
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> -Man
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 25, 2019 at 3:01 PM Jean Christophe Beyler <
>>>>>>> jcbeyler at google.com> wrote:
>>>>>>>
>>>>>>>> Done on all points I believe Man.
>>>>>>>>
>>>>>>>> I added the JvmtiTagMap change and added a new test for that as
>>>>>>>> well. The test was a bit annoying due to the fact that it was difficult to
>>>>>>>> explain that the "sum" parameter was safe to be updated before/after the
>>>>>>>> heap iteration. Currently, I've had to put a mutex around it all, which
>>>>>>>> seems like overkill but it was the only way Tsan would not complain.
>>>>>>>>
>>>>>>>> I'm unclear if that is right, it seems that it is due to the fact
>>>>>>>> that it does not see the VM operation as a fork/join operation perhaps? I
>>>>>>>> believe this is systemic to any JVMTI operation that has a VM operation
>>>>>>>> style callback so we might have more work there.
>>>>>>>>
>>>>>>>> Webrev: http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.02/
>>>>>>>>
>>>>>>>> Btw, @Arthur: the // INCLUDE_TSAN is actually questionable... it
>>>>>>>> seems half the #endif do it and half don't so I'm not sure what is the
>>>>>>>> right approach.
>>>>>>>>
>>>>>>>> Let me know what you think,
>>>>>>>> Jc
>>>>>>>>
>>>>>>>> On Wed, Apr 24, 2019 at 4:58 PM Man Cao <manc at google.com> wrote:
>>>>>>>>
>>>>>>>>> Code in HotSpot looks good. Except that we have additional
>>>>>>>>> instrumentation in jvmtiTagMap.cpp that could be added in a following
>>>>>>>>> change.
>>>>>>>>>
>>>>>>>>> A few comments on tests:
>>>>>>>>> In AbstractNativeLoop.java:
>>>>>>>>> Would it make sense to make all native methods "static"? They
>>>>>>>>> don't access any instance field, right?
>>>>>>>>>
>>>>>>>>> In NonRawRacyNativeLoopTest.java:
>>>>>>>>> There are two calls to createRawLock() in NonRawRacyNativeLoopRunner.
>>>>>>>>> Could the one in main() be deleted?
>>>>>>>>>
>>>>>>>>> In RawRacyNativeLoopTest.java:
>>>>>>>>> Similarly, could the createRawLock() in main() be deleted?
>>>>>>>>> I think calling createRawLock() in run() would result in each
>>>>>>>>> thread creating 50000 raw monitors, due to the way in which
>>>>>>>>> AbstractLoop is written. Can we avoid doing this so the test can be a bit
>>>>>>>>> faster?
>>>>>>>>>
>>>>>>>>> -Man
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 24, 2019 at 11:01 AM Jean Christophe Beyler <
>>>>>>>>> jcbeyler at google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sounds good to me.
>>>>>>>>>>
>>>>>>>>>> Here it is:
>>>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/
>>>>>>>>>>
>>>>>>>>>> I also took the idea of adding a  TSAN_RUNTIME_ONLY, I think it
>>>>>>>>>> works well
>>>>>>>>>> for one liners for code readability to be honest. If we want
>>>>>>>>>> this, I'll do
>>>>>>>>>> another pass to propagate it around the rest of the changes in a
>>>>>>>>>> separate
>>>>>>>>>> webrev.
>>>>>>>>>> cf:
>>>>>>>>>>
>>>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.01/src/hotspot/share/tsan/tsan.hpp.html
>>>>>>>>>>
>>>>>>>>>> And now with Arthur's change, all tests now pass (still would
>>>>>>>>>> like to fix
>>>>>>>>>> the racing tests to be more restrictive),
>>>>>>>>>> Jc
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 24, 2019 at 10:18 AM Arthur Eubanks <
>>>>>>>>>> aeubanks at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>> > Yeah we should have another tsan header. Kind of like how our
>>>>>>>>>> interface
>>>>>>>>>> > for the other TSAN functions is through SharedRuntime. Only
>>>>>>>>>> that file would
>>>>>>>>>> > include tsanExternalDecls.hpp. The other files would include
>>>>>>>>>> the new header
>>>>>>>>>> > file.
>>>>>>>>>> >
>>>>>>>>>> > On Wed, Apr 24, 2019 at 10:14 AM Jean Christophe Beyler <
>>>>>>>>>> > jcbeyler at google.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> >> Hi Arthur,
>>>>>>>>>> >>
>>>>>>>>>> >> What do you mean: "pulling in tsanExternalDecls.hpp" is wrong?
>>>>>>>>>> Seems like
>>>>>>>>>> >> it is what we did internally and what you were doing for the
>>>>>>>>>> lock webrev as
>>>>>>>>>> >> well, ie have that as our include header. Are you suggesting
>>>>>>>>>> we make
>>>>>>>>>> >> another tsan header for these methods.
>>>>>>>>>> >>
>>>>>>>>>> >> Ok for all other points (except the indentation, just the way
>>>>>>>>>> webrev does
>>>>>>>>>> >> things by default, the indentation is actually correct in my
>>>>>>>>>> code),
>>>>>>>>>> >> Jc
>>>>>>>>>> >>
>>>>>>>>>> >> On Wed, Apr 24, 2019 at 9:58 AM Arthur Eubanks <
>>>>>>>>>> aeubanks at google.com>
>>>>>>>>>> >> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >>> Overall:
>>>>>>>>>> >>> Since AnnotateRWLock* are direct TSAN calls, I think we
>>>>>>>>>> should have a
>>>>>>>>>> >>> wrapper around them. Then we can have things like checking if
>>>>>>>>>> the lock is
>>>>>>>>>> >>> NULL, etc, in a central location. And pulling in
>>>>>>>>>> tsanExternalDecls.hpp
>>>>>>>>>> >>> seems wrong.
>>>>>>>>>> >>> "#if INCLUDE_TSAN" ends with "#endif // INCLUDE_TSAN", could
>>>>>>>>>> you add the
>>>>>>>>>> >>> comment for the #endifs.
>>>>>>>>>> >>>
>>>>>>>>>> >>> test/hotspot/jtreg/tsan/TsanRunner.java
>>>>>>>>>> >>> Could you rename "args" to "vmArgs"?
>>>>>>>>>> >>>
>>>>>>>>>> >>> src/hotspot/share/prims/jvmtiEnv.cpp.udiff
>>>>>>>>>> >>> Indentation around 3258 is wrong.
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Tue, Apr 23, 2019 at 2:29 PM Jean Christophe Beyler <
>>>>>>>>>> >>> jcbeyler at google.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>>> Hi all,
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> I added the locks for the JVMTI raw monitors and added two
>>>>>>>>>> tests, it is
>>>>>>>>>> >>>> failing the non racy ones due to the JVM locks not yet
>>>>>>>>>> included but I
>>>>>>>>>> >>>> think
>>>>>>>>>> >>>> that will resolve itself soon :)
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> I also moved the agent library to C++.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> FWIW, we also need to clean up our tests that check for
>>>>>>>>>> failing, right
>>>>>>>>>> >>>> now
>>>>>>>>>> >>>> they pass because they generically just are checking for a
>>>>>>>>>> TSAN race but
>>>>>>>>>> >>>> not specifically which one, which is a bit wrong / imprecise.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Webrev:
>>>>>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.00/
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Let me know what you think,
>>>>>>>>>> >>>> Jc
>>>>>>>>>> >>>>
>>>>>>>>>> >>>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >>
>>>>>>>>>> >> Thanks,
>>>>>>>>>> >> Jc
>>>>>>>>>> >>
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Jc
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jc
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Jc
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

From aeubanks at google.com  Mon Apr 29 22:16:49 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Mon, 29 Apr 2019 15:16:49 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
Message-ID: <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>

There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
helpful?

>

From jcbeyler at google.com  Tue Apr 30 01:58:19 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Mon, 29 Apr 2019 18:58:19 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
Message-ID: <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>

Fair enough, I forgot you had added it there while I was adding it in mine.

I merged it to use yours, and because you don't do the fancy do/while in
the macro, it works as intended then and we can leave it at that then for
now :)

New webrev is here:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/

I'll add in a future webrev the tests for the other VM operations and try
to see how we can fix the VM operation also in a future webrev; as soon as
symbolization is up, we should be able to get testing to be a bit more
reliable :)

Thanks for the reviews!
Jc

On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com> wrote:

> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
> helpful?
>
>>

-- 

Thanks,
Jc

From manc at google.com  Tue Apr 30 18:04:48 2019
From: manc at google.com (Man Cao)
Date: Tue, 30 Apr 2019 11:04:48 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
Message-ID: <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>

The issue now is the code is not guarded by the -XX:+ThreadSanitizer flag
any more.

Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in tsan.hpp? E.g.:
#define TSAN_RAW_LOCK_CREATE(lock) \
   TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__, __LINE__,
lock); })

Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the following?
#define TSAN_RAW_LOCK_CREATE(lock) \
  TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))

In any case, it seems that there is no clean way to guard the creation of
TsanMutexScope objects with the +ThreadSanitizer flag, but it is probably
OK as it is no-op with -ThreadSanitizer.

-Man


On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Fair enough, I forgot you had added it there while I was adding it in mine.
>
> I merged it to use yours, and because you don't do the fancy do/while in
> the macro, it works as intended then and we can leave it at that then for
> now :)
>
> New webrev is here:
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>
> I'll add in a future webrev the tests for the other VM operations and try
> to see how we can fix the VM operation also in a future webrev; as soon as
> symbolization is up, we should be able to get testing to be a bit more
> reliable :)
>
> Thanks for the reviews!
> Jc
>
> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
> wrote:
>
>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
>> helpful?
>>
>>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr 30 20:28:31 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 30 Apr 2019 13:28:31 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
Message-ID: <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>

Good point, what do you prefer here? Just leave like this for now since it
is a noop? I do :)

But I'm biased at preferring forward progress on this webrev :)
Jc

On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:

> The issue now is the code is not guarded by the -XX:+ThreadSanitizer flag
> any more.
>
> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in tsan.hpp?
> E.g.:
> #define TSAN_RAW_LOCK_CREATE(lock) \
>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__, __LINE__,
> lock); })
>
> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the following?
> #define TSAN_RAW_LOCK_CREATE(lock) \
>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>
> In any case, it seems that there is no clean way to guard the creation of
> TsanMutexScope objects with the +ThreadSanitizer flag, but it is probably
> OK as it is no-op with -ThreadSanitizer.
>
> -Man
>
>
> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Fair enough, I forgot you had added it there while I was adding it in
>> mine.
>>
>> I merged it to use yours, and because you don't do the fancy do/while in
>> the macro, it works as intended then and we can leave it at that then for
>> now :)
>>
>> New webrev is here:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>
>> I'll add in a future webrev the tests for the other VM operations and try
>> to see how we can fix the VM operation also in a future webrev; as soon as
>> symbolization is up, we should be able to get testing to be a bit more
>> reliable :)
>>
>> Thanks for the reviews!
>> Jc
>>
>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>> wrote:
>>
>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
>>> helpful?
>>>
>>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From manc at google.com  Tue Apr 30 21:34:32 2019
From: manc at google.com (Man Cao)
Date: Tue, 30 Apr 2019 14:34:32 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
Message-ID: <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>

I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
simplify other code.
The current webrev if left unchanged could cause issues with
-XX:-ThreadSanitizer.

-Man


On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Good point, what do you prefer here? Just leave like this for now since it
> is a noop? I do :)
>
> But I'm biased at preferring forward progress on this webrev :)
> Jc
>
> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>
>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer flag
>> any more.
>>
>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in tsan.hpp?
>> E.g.:
>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__, __LINE__,
>> lock); })
>>
>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the following?
>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>
>> In any case, it seems that there is no clean way to guard the creation of
>> TsanMutexScope objects with the +ThreadSanitizer flag, but it is probably
>> OK as it is no-op with -ThreadSanitizer.
>>
>> -Man
>>
>>
>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Fair enough, I forgot you had added it there while I was adding it in
>>> mine.
>>>
>>> I merged it to use yours, and because you don't do the fancy do/while in
>>> the macro, it works as intended then and we can leave it at that then for
>>> now :)
>>>
>>> New webrev is here:
>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>
>>> I'll add in a future webrev the tests for the other VM operations and
>>> try to see how we can fix the VM operation also in a future webrev; as soon
>>> as symbolization is up, we should be able to get testing to be a bit more
>>> reliable :)
>>>
>>> Thanks for the reviews!
>>> Jc
>>>
>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
>>>> helpful?
>>>>
>>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From aeubanks at google.com  Tue Apr 30 22:18:44 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 30 Apr 2019 15:18:44 -0700
Subject: Symbolization
In-Reply-To: <CAF9BGBwr+Khm0=a2uTvHQ3wEvoUpkJpteTF9sthbK-o7OswUjQ@mail.gmail.com>
References: <CAPW48sp5MkURwbkf3sHwWjoFY7owmedyuCe0thoQDxUsTdJqCw@mail.gmail.com>
 <CAF9BGBwQV_s1vqFdfcD0Gm=ZA4_6DHoir3tsabK_yjujbWRLCQ@mail.gmail.com>
 <CA+w6HxY7YHbvBYnGC2Ort7V1hFBEpVDSTMhcwuAH9a7d9n2Z6g@mail.gmail.com>
 <CAF9BGBwr+Khm0=a2uTvHQ3wEvoUpkJpteTF9sthbK-o7OswUjQ@mail.gmail.com>
Message-ID: <CAPW48sp2H7hMU-2DiJXLA_aL1R3w8zMecQgSZB=68xHu9R=Vvg@mail.gmail.com>

The earliest GCC that uses __tsan_symbolize_external_ex() is GCC 9. So
using the GCC version supported by OpenJDK (GCC 7) is not feasible.

I tried compiling GCC 9 and using it as the toolchain (install GCC 9 with
prefix $HOME/bin/gcc9, then configure and build OpenJDK
--with-toolchain-path=$HOME/bin/gcc9).

Running the TSAN tests, I get
result: Failed. Unexpected exit from test [exit code: 139]

Doing some searching, it looks like exit code 139 is something to do with
memory fragmentation, or SIGSEGV, or something like that. So my guess is
that mixing .so's between the Linux distro and the compiled GCC is not okay.

On Mon, Apr 29, 2019 at 2:48 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> FWIW, it does not really matter to me at this point what we say to do as
> long as it works for us. At some point, when we get a bit further along, we
> would want to figure out a "sane" way of doing things; requiring to compile
> GCC by hand is not sane in my opinion but is ok for now :)
> Jc
>
> On Mon, Apr 29, 2019 at 2:36 PM Man Cao <manc at google.com> wrote:
>
>> I think the best way is still to find a way to build libtsan.so from
>> latest source, with the GCC version supported by OpenJDK:
>> https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms.
>> Even if we can get libtsan.so from a newer GCC, I'm not sure if it is
>> safe to build OpenJDK with the newer GCC, or to use a newer libtsan.so with
>> an old GCC.
>>
>> -Man
>>
>>
>> On Mon, Apr 29, 2019 at 2:18 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Compiling gcc to get libtsan... Is there no way to build libtsan then
>>> from
>>> scratch instead?
>>>
>>> On Mon, Apr 29, 2019 at 2:00 PM Arthur Eubanks <aeubanks at google.com>
>>> wrote:
>>>
>>> > It looks like the libtsan.so from gcc 8 is out of date. It doesn't
>>> > contain __tsan_symbolize_external_ex(), only the
>>> > legacy __tsan_symbolize_external(). This messes with testing the Java
>>> Tsan
>>> > patches.
>>> >
>>> > $ strings /usr/lib/gcc/x86_64-linux-gnu/8/libtsan.so | grep __tsan_sym
>>> >
>>> > I looked at the latest gcc sources and it does have a version of Tsan
>>> that
>>> > includes __tsan_symbolize_external_ex(). We could tell people to
>>> compile
>>> > their own gcc and use that libtsan.so.
>>> > $ apt search gcc
>>> > only shows up to gcc 8 on my machine.
>>> >
>>> > Thoughts?
>>> >
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr 30 22:19:19 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 30 Apr 2019 15:19:19 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
 <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
Message-ID: <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>

Fair enough :)

Done here:
http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.07/

(I left the jvmtiTagMap ones using TSAN_ONLY, as if TSAN is defined, they
call the macros which then use TSAN_RUNTIME_ONLY so that works too)

Tested that the test still passes :), let me know what you think,
Jc


On Tue, Apr 30, 2019 at 2:34 PM Man Cao <manc at google.com> wrote:

> I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
> tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
> simplify other code.
> The current webrev if left unchanged could cause issues with
> -XX:-ThreadSanitizer.
>
> -Man
>
>
> On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Good point, what do you prefer here? Just leave like this for now since
>> it is a noop? I do :)
>>
>> But I'm biased at preferring forward progress on this webrev :)
>> Jc
>>
>> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>>
>>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer
>>> flag any more.
>>>
>>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in tsan.hpp?
>>> E.g.:
>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__,
>>> __LINE__, lock); })
>>>
>>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the
>>> following?
>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>>
>>> In any case, it seems that there is no clean way to guard the creation
>>> of TsanMutexScope objects with the +ThreadSanitizer flag, but it is
>>> probably OK as it is no-op with -ThreadSanitizer.
>>>
>>> -Man
>>>
>>>
>>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Fair enough, I forgot you had added it there while I was adding it in
>>>> mine.
>>>>
>>>> I merged it to use yours, and because you don't do the fancy do/while
>>>> in the macro, it works as intended then and we can leave it at that then
>>>> for now :)
>>>>
>>>> New webrev is here:
>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>>
>>>> I'll add in a future webrev the tests for the other VM operations and
>>>> try to see how we can fix the VM operation also in a future webrev; as soon
>>>> as symbolization is up, we should be able to get testing to be a bit more
>>>> reliable :)
>>>>
>>>> Thanks for the reviews!
>>>> Jc
>>>>
>>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>>> wrote:
>>>>
>>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe that's
>>>>> helpful?
>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc

From aeubanks at google.com  Tue Apr 30 22:23:31 2019
From: aeubanks at google.com (Arthur Eubanks)
Date: Tue, 30 Apr 2019 15:23:31 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
 <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
 <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>
Message-ID: <CAPW48srbr7_ZXCpaWXkxeHm9bN6kwQU8zuVk=VyU64cOoXV1sA@mail.gmail.com>

The comment in src/hotspot/share/prims/jvmtiEnv.cpp still says "below".

Otherwise LGTM.

On Tue, Apr 30, 2019 at 3:19 PM Jean Christophe Beyler <jcbeyler at google.com>
wrote:

> Fair enough :)
>
> Done here:
> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.07/
>
> (I left the jvmtiTagMap ones using TSAN_ONLY, as if TSAN is defined, they
> call the macros which then use TSAN_RUNTIME_ONLY so that works too)
>
> Tested that the test still passes :), let me know what you think,
> Jc
>
>
> On Tue, Apr 30, 2019 at 2:34 PM Man Cao <manc at google.com> wrote:
>
>> I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
>> tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
>> simplify other code.
>> The current webrev if left unchanged could cause issues with
>> -XX:-ThreadSanitizer.
>>
>> -Man
>>
>>
>> On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <
>> jcbeyler at google.com> wrote:
>>
>>> Good point, what do you prefer here? Just leave like this for now since
>>> it is a noop? I do :)
>>>
>>> But I'm biased at preferring forward progress on this webrev :)
>>> Jc
>>>
>>> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>>>
>>>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer
>>>> flag any more.
>>>>
>>>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in tsan.hpp?
>>>> E.g.:
>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__,
>>>> __LINE__, lock); })
>>>>
>>>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the
>>>> following?
>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>>>
>>>> In any case, it seems that there is no clean way to guard the creation
>>>> of TsanMutexScope objects with the +ThreadSanitizer flag, but it is
>>>> probably OK as it is no-op with -ThreadSanitizer.
>>>>
>>>> -Man
>>>>
>>>>
>>>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>>>> jcbeyler at google.com> wrote:
>>>>
>>>>> Fair enough, I forgot you had added it there while I was adding it in
>>>>> mine.
>>>>>
>>>>> I merged it to use yours, and because you don't do the fancy do/while
>>>>> in the macro, it works as intended then and we can leave it at that then
>>>>> for now :)
>>>>>
>>>>> New webrev is here:
>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>>>
>>>>> I'll add in a future webrev the tests for the other VM operations and
>>>>> try to see how we can fix the VM operation also in a future webrev; as soon
>>>>> as symbolization is up, we should be able to get testing to be a bit more
>>>>> reliable :)
>>>>>
>>>>> Thanks for the reviews!
>>>>> Jc
>>>>>
>>>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>>>> wrote:
>>>>>
>>>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe
>>>>>> that's helpful?
>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Jc
>>>>>
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Jc
>>>
>>
>
> --
>
> Thanks,
> Jc
>

From jcbeyler at google.com  Tue Apr 30 22:25:34 2019
From: jcbeyler at google.com (Jean Christophe Beyler)
Date: Tue, 30 Apr 2019 15:25:34 -0700
Subject: RFR: Add JVMTI raw monitor lock/unlock mechanisms
In-Reply-To: <CAPW48srbr7_ZXCpaWXkxeHm9bN6kwQU8zuVk=VyU64cOoXV1sA@mail.gmail.com>
References: <CAF9BGBxkcZHVt8gTdy5RLhpVtCP0vhOhUPRoGoXfApUWy_sFbw@mail.gmail.com>
 <CAPW48spCPesDv1z2=YBfE_xUxQB-uH_9STDmF2Np2EfiQ8+V7A@mail.gmail.com>
 <CAF9BGBzvnivdBsfF74W7jDFSX=8ayobfpVAzph0iaTDOiPe82A@mail.gmail.com>
 <CAPW48sqwUvnKRjObBX9WU1imxqwVkpXSA92+yHt=KU7+UCLjkw@mail.gmail.com>
 <CAF9BGBw7Q6iczDBC8fYEMe=8__Z_-KWKt1QVZe_Ec+8Du+Wx4g@mail.gmail.com>
 <CA+w6HxZNjTFOv-7bqvnnbbqzkAr88RPVEUWNzds_YHp2Asevxw@mail.gmail.com>
 <CAF9BGByoUPi2JGrpNx4-kOoERYOQg6G6mFHGG5OP=vrgcURzEA@mail.gmail.com>
 <CA+w6HxY80OFp+aw32wt=hBFTgWz1HqVRi_HoxDSuj8RgOkNz1Q@mail.gmail.com>
 <CAF9BGByRKrO3uPos1iQLzOU6NqbBQ3Mi_ojO+ROeWrcwQjFK8Q@mail.gmail.com>
 <CAF9BGBx9v8LDf39hLBBMZbZ6HDS6jDUoA7+EqHovG_h=U5jmMA@mail.gmail.com>
 <CAPW48spXkf0AjOYhjespn+xNa=7igAw_Nj8rZHsFY-EPDjXSYA@mail.gmail.com>
 <CA+w6HxYEg5+dTur48DuFjFyMt-RL69Cu99iMyNA7TVG=CPVzfw@mail.gmail.com>
 <CAF9BGBwz3UHQUT+V70meqRAXhD+x8ug8=pSGNH+fuCGPzhCWOQ@mail.gmail.com>
 <CAPW48sqD__o4F5dJXrzBykLMZB2BwDaXCkuR8ttyH8yVOOJi_g@mail.gmail.com>
 <CA+w6HxYOarJ1acatuObZOjNMG_NBXLCje1jF8bEtMrFrKCYJWw@mail.gmail.com>
 <CAPW48sq5ttCd7Ao1kcZrYw3Wzn5Jt0cHWXp8uucTm-tgdpZe3Q@mail.gmail.com>
 <CAF9BGBw=n311bDUG7fkwY=AvLhHs+gM1E7qev=6rradn6DUMCQ@mail.gmail.com>
 <CA+w6HxbkjyGx++qpKkhvJX2sRbBZhP3+6GgppZ971DNnUN-hTQ@mail.gmail.com>
 <CAF9BGBwcWhJzfgXHejwg49NFr11tJH8qkbsZ+RP3D_Oo_YkVoA@mail.gmail.com>
 <CA+w6HxZ6jZXY54YFsadKFS9Zd-Ytxy_J5JK5TKwep0Z55LKCqw@mail.gmail.com>
 <CAF9BGBxDJ_o4=qaam6T4Ys3m8W+AZ_PoZxc5ruf=dNpEysSFHQ@mail.gmail.com>
 <CAPW48srbr7_ZXCpaWXkxeHm9bN6kwQU8zuVk=VyU64cOoXV1sA@mail.gmail.com>
Message-ID: <CAF9BGBwe3K1oSwPWq4_Ee325e67VCVrqVxPGzO9xcLChVeBB+Q@mail.gmail.com>

Fixed on my local branch:)
Jc

On Tue, Apr 30, 2019 at 3:23 PM Arthur Eubanks <aeubanks at google.com> wrote:

> The comment in src/hotspot/share/prims/jvmtiEnv.cpp still says "below".
>
> Otherwise LGTM.
>
> On Tue, Apr 30, 2019 at 3:19 PM Jean Christophe Beyler <
> jcbeyler at google.com> wrote:
>
>> Fair enough :)
>>
>> Done here:
>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.07/
>>
>> (I left the jvmtiTagMap ones using TSAN_ONLY, as if TSAN is defined, they
>> call the macros which then use TSAN_RUNTIME_ONLY so that works too)
>>
>> Tested that the test still passes :), let me know what you think,
>> Jc
>>
>>
>> On Tue, Apr 30, 2019 at 2:34 PM Man Cao <manc at google.com> wrote:
>>
>>> I'd prefer to revive the TSAN_RUNTIME_ONLY, and change the 4 macros in
>>> tsan.hpp to use it. I like your TSAN_RUNTIME_ONLY and it could be used to
>>> simplify other code.
>>> The current webrev if left unchanged could cause issues with
>>> -XX:-ThreadSanitizer.
>>>
>>> -Man
>>>
>>>
>>> On Tue, Apr 30, 2019 at 1:28 PM Jean Christophe Beyler <
>>> jcbeyler at google.com> wrote:
>>>
>>>> Good point, what do you prefer here? Just leave like this for now since
>>>> it is a noop? I do :)
>>>>
>>>> But I'm biased at preferring forward progress on this webrev :)
>>>> Jc
>>>>
>>>> On Tue, Apr 30, 2019 at 11:05 AM Man Cao <manc at google.com> wrote:
>>>>
>>>>> The issue now is the code is not guarded by the -XX:+ThreadSanitizer
>>>>> flag any more.
>>>>>
>>>>> Perhaps we can add "if (ThreadSanitizer)" to the 4 macros in
>>>>> tsan.hpp? E.g.:
>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>    TSAN_ONLY(if (ThreadSanitizer) { TsanRawLockCreate(__FILE__,
>>>>> __LINE__, lock); })
>>>>>
>>>>> Or maybe we can revive the TSAN_RUNTIME_ONLY macro, and use the
>>>>> following?
>>>>> #define TSAN_RAW_LOCK_CREATE(lock) \
>>>>>   TSAN_RUNTIME_ONLY(TsanRawLockCreate(__FILE__, __LINE__, lock))
>>>>>
>>>>> In any case, it seems that there is no clean way to guard the creation
>>>>> of TsanMutexScope objects with the +ThreadSanitizer flag, but it is
>>>>> probably OK as it is no-op with -ThreadSanitizer.
>>>>>
>>>>> -Man
>>>>>
>>>>>
>>>>> On Mon, Apr 29, 2019 at 6:58 PM Jean Christophe Beyler <
>>>>> jcbeyler at google.com> wrote:
>>>>>
>>>>>> Fair enough, I forgot you had added it there while I was adding it in
>>>>>> mine.
>>>>>>
>>>>>> I merged it to use yours, and because you don't do the fancy do/while
>>>>>> in the macro, it works as intended then and we can leave it at that then
>>>>>> for now :)
>>>>>>
>>>>>> New webrev is here:
>>>>>> http://cr.openjdk.java.net/~jcbeyler/jvmti/webrev.06/
>>>>>>
>>>>>> I'll add in a future webrev the tests for the other VM operations and
>>>>>> try to see how we can fix the VM operation also in a future webrev; as soon
>>>>>> as symbolization is up, we should be able to get testing to be a bit more
>>>>>> reliable :)
>>>>>>
>>>>>> Thanks for the reviews!
>>>>>> Jc
>>>>>>
>>>>>> On Mon, Apr 29, 2019 at 3:17 PM Arthur Eubanks <aeubanks at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> There is TSAN_ONLY in hotspot/share/utilities/macros.hpp, maybe
>>>>>>> that's helpful?
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Jc
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Jc
>>>>
>>>
>>
>> --
>>
>> Thanks,
>> Jc
>>
>

-- 

Thanks,
Jc