From jiangli at openjdk.org  Thu Aug  1 22:25:59 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Thu, 1 Aug 2024 22:25:59 GMT
Subject: RFR: Merge jdk21u
In-Reply-To: <a6sYpIaZHEQP6neMJfY8qZMxgbrSlei_fwiTZVe89uQ=.3f76f7d1-e98a-4de7-89cd-33948a087e36@github.com>
References: <qXzzRCrFt_IyKl8CeqFqcRtz_xZb0pzqXEANyv-7EaQ=.7187be2c-1ead-4915-8a5d-27e23d86de26@github.com>
 <x8AxYIldQ6ilT2OW4dGsPofuW6X0TIoqYGHM9-dthSw=.2b49806b-d1ac-4bee-94f7-93cf8c480423@github.com>
 <7nsFb2fBntwF7Mi7l9KWHlvySDIG-AF4e9rrLTXPD5w=.394e5da4-a88e-425a-bab6-a68d2d840979@github.com>
 <zPEhd5lT5bAPZ19j7apAlHyfMLayKnCOr2l-xipw_aE=.4c1396ae-20ab-45a9-9d71-b8a91a17d827@github.com>
 <a6sYpIaZHEQP6neMJfY8qZMxgbrSlei_fwiTZVe89uQ=.3f76f7d1-e98a-4de7-89cd-33948a087e36@github.com>
Message-ID: <EhHk1EbG2jGym5deRSsPIOOUvCpX_dXQp7p5aoyS7J8=.602eee7d-d031-4d66-92d9-ce4867b64a1b@github.com>

On Wed, 31 Jul 2024 00:00:08 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

> > > Could we create a new branch (e.g. "tsan-21u") in the TSAN project to track TSAN based on JDK 21u? The current "tsan" branch in intended to track OpenJDK's tip.
> > 
> > 
> > SG for creating tsan-21u branch.
> 
> An alternative is to only track the JDK mainline and not track any of the JDK update repo. That can avoid complications with handling branches. Let me try integrate the current tsan repo internally.
> 
> For JDK 21, there is a bit delay. For future JDK LTS/STS integrations, we can integrate them to the tsan repo sooner after a `ga` release.

I tried integrating the current tsan changes (based on `jdk-21-ga`) with a JDK code base mirroring JDK 21.0.3. The integration went smoothly. `TsanRunner` needed the `ProcessTools.createLimitedTestJavaProcessBuilder()` change, otherwise there were no significant integration issues. So tracking with the JDK mainline is quite practical. It avoids the complications with managing branches related to tracking JDK update repo.

I'll close the PR based on the above.

-------------

PR Comment: https://git.openjdk.org/tsan/pull/21#issuecomment-2264115331

From jiangli at openjdk.org  Thu Aug  1 22:25:59 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Thu, 1 Aug 2024 22:25:59 GMT
Subject: Withdrawn: Merge jdk21u
In-Reply-To: <qXzzRCrFt_IyKl8CeqFqcRtz_xZb0pzqXEANyv-7EaQ=.7187be2c-1ead-4915-8a5d-27e23d86de26@github.com>
References: <qXzzRCrFt_IyKl8CeqFqcRtz_xZb0pzqXEANyv-7EaQ=.7187be2c-1ead-4915-8a5d-27e23d86de26@github.com>
Message-ID: <dEKtmAVhPKhm0vBTEKYm03Oa1TVSM5gZVsU3qE7MuHM=.0cde17f3-1b3a-453a-a74f-09a4986201fd@github.com>

On Tue, 30 Jul 2024 19:04:57 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

> This PR merges with JDK 21.0.3 (https://github.com/openjdk/jdk/commit/05f18b10620834883180013bb8873a6b4bdc6df2).
> 
> Merge was done most automatically. See manual edit and change details below.
> 
> **Manual changes:**
> - Edited `.jcheck/conf` and `src/hotspot/share/logging/logTag.hpp` to resolve trivial merge conflicts.
> - Changed `TsanRunner `to use `ProcessTools.createLimitedTestJavaProcessBuilder()` to resolve tsan jtreg test failures. `ProcessTools.createJavaProcessBuilder()` was renamed to `ProcessTools.createLimitedTestJavaProcessBuilder` by https://github.com/openjdk/jdk/pull/15452.
> 
> **Merge commands:**
> 
> $ git remote add 21u https://github.com/openjdk/jdk21u.git
> $ git remote update
> $ git merge 05f18b1
> #resolve merge conflicts and commit
> $ git remote remove 21u
> 
> 
> **Testing:**
> 
> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang --disable-warnings-as-errors
> 
> 
> 
> $ make test TEST=hotspot/jtreg/tsan
> 
> All tests pass with `release` binary
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hotspot/jtreg/tsan                        79    79     0     0   
> ==============================
> TEST SUCCESS

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/tsan/pull/21

From manc at openjdk.org  Tue Aug 13 20:45:21 2024
From: manc at openjdk.org (Man Cao)
Date: Tue, 13 Aug 2024 20:45:21 GMT
Subject: RFR: Allow creating branches with /branch commit command
Message-ID: <wNcwZ1-R_pwBYiPTP0joGX2OONqbE1p4SccIO_QSoW4=.f12ace2a-1675-4fdb-bb92-dd09de092d9b@github.com>

Hello,

Could anyone approve this change to allow creating branches in the TSAN project? See https://bugs.openjdk.org/browse/SKARA-2345 and https://bugs.openjdk.org/browse/JDK-8333743.

-Man

-------------

Commit messages:
 - Allow creating branches with /branch commit command

Changes: https://git.openjdk.org/tsan/pull/22/files
  Webrev: https://webrevs.openjdk.org/?repo=tsan&pr=22&range=00
  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/tsan/pull/22.diff
  Fetch: git fetch https://git.openjdk.org/tsan.git pull/22/head:pull/22

PR: https://git.openjdk.org/tsan/pull/22

From jiangli at openjdk.org  Thu Aug 15 00:11:17 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Thu, 15 Aug 2024 00:11:17 GMT
Subject: RFR: Tsan oop map table hash fix
Message-ID: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>

Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:

1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`

2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`

With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.

The fix is to compute `primitive_hash` using the oop address instead of `identity_hash`. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.

Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could compute incorrect hash value for the current entry. So it should use the updated oop address instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.

**Testing**
Tsan jtreg tests on linux-x64:
`$ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang`
==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/hotspot/jtreg/tsan                        79    79     0     0   
==============================
TEST SUCCESS

-------------

Commit messages:
 - - Remove obj->fast_no_hash_check() case from TsanOopMapTable::add_entry. We don't use oop
 - Add TsanOopMapImpl::MovedEntry. Also save the oop size for a moved oop and avoid calling oop size() when adding a moved
 - Update comments.
 - Remove moved_entry_sizes array.
 - Change to compute primitive_hash using the oop address instead of oop identity_hash for an entry's hash.

Changes: https://git.openjdk.org/tsan/pull/23/files
  Webrev: https://webrevs.openjdk.org/?repo=tsan&pr=23&range=00
  Stats: 65 lines in 3 files changed: 53 ins; 4 del; 8 mod
  Patch: https://git.openjdk.org/tsan/pull/23.diff
  Fetch: git fetch https://git.openjdk.org/tsan.git pull/23/head:pull/23

PR: https://git.openjdk.org/tsan/pull/23

From jiangli at openjdk.org  Thu Aug 15 01:00:04 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Thu, 15 Aug 2024 01:00:04 GMT
Subject: RFR: Allow creating branches with /branch commit command
In-Reply-To: <wNcwZ1-R_pwBYiPTP0joGX2OONqbE1p4SccIO_QSoW4=.f12ace2a-1675-4fdb-bb92-dd09de092d9b@github.com>
References: <wNcwZ1-R_pwBYiPTP0joGX2OONqbE1p4SccIO_QSoW4=.f12ace2a-1675-4fdb-bb92-dd09de092d9b@github.com>
Message-ID: <F9ReWkwlxvjIHjYOMKL1ems5xHsJdtw0z0ns6Vusz6Y=.9d7c92a1-fee2-4c12-8cfe-d5636453cf10@github.com>

On Tue, 13 Aug 2024 20:40:40 GMT, Man Cao <manc at openjdk.org> wrote:

> Hello,
> 
> Could anyone approve this change to allow creating branches in the TSAN project? See https://bugs.openjdk.org/browse/SKARA-2345 and https://bugs.openjdk.org/browse/JDK-8333743.
> 
> -Man

Seems fine.

-------------

Marked as reviewed by jiangli (Committer).

PR Review: https://git.openjdk.org/tsan/pull/22#pullrequestreview-2239403089

From manc at google.com  Thu Aug 15 01:47:59 2024
From: manc at google.com (Man Cao)
Date: Wed, 14 Aug 2024 18:47:59 -0700
Subject: CFV: New Tsan Reviewer: Jiangli Zhou
Message-ID: <CA+w6Hxbf=mRTVGnGsz93aDaapy5_hXwGuF1Kx1CHzF1bbC0PRg@mail.gmail.com>

I hereby nominate Jiangli Zhou (OpenJDK id: jiangli) to Reviewer role in
Project Tsan.

Jiangli has contributed several large, significant changes into the Tsan
[1], so that Tsan can run on JDK 21.
Jiangli is an experienced OpenJDK contributor and has been a Reviewer for
the jdk and jdk-updates projects.
Tsan project is currently in maintenance mode to support future JDK
releases. Jiangli and I are the main active maintainers.
Tsan project does not need to follow the guideline of 32 significant PRs
for Reviewer nomination, since there is not much development of new
features.

Votes are due by August 21, 2024.

Only current Tsan Reviewers [2] are eligible to vote on this nomination.
Votes must be cast in the open by replying to this mailing list.

For Lazy Consensus voting instructions, see [3].

Best,
Man Cao

[1]
https://github.com/openjdk/tsan/pulls?q=is%3Apr+is%3Aclosed+author%3Ajianglizhou
[2] https://openjdk.org/census#tsan
[3] https://openjdk.org/projects/#committer-vote
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/tsan-dev/attachments/20240814/11db43fa/attachment.htm>

From manc at openjdk.org  Thu Aug 15 02:36:08 2024
From: manc at openjdk.org (Man Cao)
Date: Thu, 15 Aug 2024 02:36:08 GMT
Subject: RFR: Tsan oop map table hash fix
In-Reply-To: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
Message-ID: <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>

On Thu, 15 Aug 2024 00:07:22 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
> 
> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
> 
> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
> 
> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
> 
> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
> 
> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
> 
> Testing
> 
> Tsan jtreg tests on linux-x64:
> 
> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
> 
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hotspot/jtreg/tsan                        79    79     0     0   
> ==============================
> TEST SUCCESS

Changes requested by manc (Reviewer).

src/hotspot/share/tsan/tsanOopMap.cpp line 235:

> 233:       const TsanOopMapImpl::MovedEntry &e = moved_entries.at(i);
> 234:       _oop_map->add_entry(e.key(), e.value());
> 235:     }

Is there a memory leak on the `MovedEntry::TsanOopMapTableKey*` after here?
`TsanOopMapTable::add_entry()` makes a copy `*(e.key())`, but `e.key()` appears never freed.

src/hotspot/share/tsan/tsanOopMapTable.cpp line 76:

> 74:   bool added;
> 75:   size_t* v = _table.put_if_absent(*entry, size, &added);
> 76:   assert(added, "must be");

> Since primitive_hash is now used, I also changed TsanOopMapTableKey::get_hash and TsanOopMapTableKey::equals to use entry.object_no_keepalive() instead of entry._obj. If entry._obj is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. 

I think the concurrent processing of `TsanOopMapTable` and another worker thread processing `OopStorageSet` to update TSAN's `WeakHandles` is a scary situation, when we start using `entry.object_no_keepalive()` or `WeakHandle::peek()`. `peek()` may return an updated or not-yet-updated value, which will likely cause hidden bugs.

I came up with the buggy scenario below:
Suppose object A was at 0x1200, object B was at 0x2300. During GC, A becomes dead, and GC moves B to A's location, so B is now at 0x1200.
Worker thread T1 processed part of the `OopStorageSet` and updated B's weak handle (wh_B).  The weak handle for A (wh_A) is not yet processed.
T1 starts to process `TsanOopMapTable`, and it will try adding the new B to the table, with `wh_B.peek()` returing 0x1200. It will fail to add this entry, because the new B looks identical to the entry for A, with `wh_A.peek()` still returning 0x1200.
Then object B will be lost from the `TsanOopMapTable`.

I think a better approach is ensure the `TsanOopMapTable` is only processed once after all of the `OopStorageSet` is updated.
One option is the alternative approach of inserting  `TsanOopMap::notify_tsan_for_freed_and_moved_objects()` to individual GCs as mentioned in https://github.com/openjdk/tsan/pull/19.
Another possibility is to develop a mechanism to check whether `OopStorageSet` has been fully processed, before doing actual work to update `TsanOopMapTable`.

src/hotspot/share/tsan/tsanOopMapTable.cpp line 161:

> 159:         }
> 160: 
> 161:         entry.update_obj();

It is probably better to remove the `TsanOopMapTableKey::update_obj()` function altogether, so `TsanOopMapTableKey` becomes an immutable object. There is no point to update the old object when it is going to be removed immediately afterwards.

The code can use `wh_obj` to create a new `TsanOopMapTableKey`. `TsanOopMapTableKey` may need a new constructor though:

TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj) {
  _wh = src._wh;
  _obj = obj;
}

-------------

PR Review: https://git.openjdk.org/tsan/pull/23#pullrequestreview-2239376358
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1717670966
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1717793720
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1717677419

From jiangli at openjdk.org  Thu Aug 15 23:06:09 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Thu, 15 Aug 2024 23:06:09 GMT
Subject: RFR: Tsan oop map table hash fix
In-Reply-To: <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>
Message-ID: <IlHqt2X-AgyzhE863qVkiHGs-LI0WE7ZMQVlrhLgPEs=.2e82c084-61ef-4d2b-bf24-89ff7f71e7df@github.com>

On Thu, 15 Aug 2024 02:17:36 GMT, Man Cao <manc at openjdk.org> wrote:

>> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
>> 
>> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
>> 
>> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
>> 
>> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
>> 
>> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
>> 
>> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
>> 
>> Testing
>> 
>> Tsan jtreg tests on linux-x64:
>> 
>> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
>> 
>> 
>> ==============================
>> Test summary
>> ==============================
>>    TEST                                              TOTAL  PASS  FAIL ERROR   
>>    jtreg:test/hotspot/jtreg/tsan                        79    79     0     0   
>> ==============================
>> TEST SUCCESS
>
> src/hotspot/share/tsan/tsanOopMapTable.cpp line 76:
> 
>> 74:   bool added;
>> 75:   size_t* v = _table.put_if_absent(*entry, size, &added);
>> 76:   assert(added, "must be");
> 
>> Since primitive_hash is now used, I also changed TsanOopMapTableKey::get_hash and TsanOopMapTableKey::equals to use entry.object_no_keepalive() instead of entry._obj. If entry._obj is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. 
> 
> I think the concurrent processing of `TsanOopMapTable` and another worker thread processing `OopStorageSet` to update TSAN's `WeakHandles` is a scary situation, when we start using `entry.object_no_keepalive()` or `WeakHandle::peek()`. `peek()` may return an updated or not-yet-updated value, which will likely cause hidden bugs.
> 
> I came up with the buggy scenario below:
> Suppose object A was at 0x1200, object B was at 0x2300. During GC, A becomes dead, and GC moves B to A's location, so B is now at 0x1200.
> Worker thread T1 processed part of the `OopStorageSet` and updated B's weak handle (wh_B).  The weak handle for A (wh_A) is not yet processed.
> T1 starts to process `TsanOopMapTable`, and it will try adding the new B to the table, with `wh_B.peek()` returing 0x1200. It will fail to add this entry, because the new B looks identical to the entry for A, with `wh_A.peek()` still returning 0x1200.
> Then object B will be lost from the `TsanOopMapTable`.
> 
> I think a better approach is ensure the `TsanOopMapTable` is only processed once after all of the `OopStorageSet` is updated.
> One option is the alternative approach of inserting  `TsanOopMap::notify_tsan_for_freed_and_moved_objects()` to individual GCs as mentioned in https://github.com/openjdk/tsan/pull/19.
> Another possibility is to develop a mechanism to check whether `OopStorageSet` has been fully processed, before doing actual work to update `TsanOopMapTable`.

Thanks for the analysis. The scenario describes a potential rare case where an `oop` is moved to an address location containing a freed `oop` and tsan is notified about the moved `oop` before the freed `oop`. I agree that's a potential bug.

Doing `TsanOopMap::notify_tsan_for_freed_and_moved_objects()` to individual GCs has higher risks of not covering all different cases if GC changes. I like you suggestion of implementing "a mechanism to check whether OopStorageSet has been fully processed". Implementing within `OopStorageSet` could be complicated. I think it might be doable in `WeakProcessor::Task::work` to check if all tasks are done. I'm trying that approach.

-------------

PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1719096635

From jiangli at openjdk.org  Sat Aug 17 01:51:22 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Sat, 17 Aug 2024 01:51:22 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
Message-ID: <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>

> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
> 
> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
> 
> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
> 
> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
> 
> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
> 
> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
> 
> Testing
> 
> Tsan jtreg tests on linux-x64:
> 
> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
> 
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hotspot/jtreg/tsan                        79    79     0     0   
> ==============================
> TEST SUCCESS

Jiangli Zhou has updated the pull request incrementally with two additional commits since the last revision:

 - - Release memory for the malloced new TsanOopMapTableKey after adding back to the TsanOopMap. Thanks manc@ for
     noticing the memory leak.
   - Add TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj).
 - Change to call TsanOopMap::notify_tsan_for_freed_and_moved_objects from the last worker thread that completes
   the parallel WeakProcessor::Task work:
   - Add WeakProcessor::Task::_nworkers_completed to track the completed workers. This is only used if
     INCLUDE_TSAN is true.
   - Increment WeakProcessor::Task::_nworkers_completed in WeakProcessor::Task::work when each parallel worker
     finishes work.
   
   With the change, TsanOopMap::notify_tsan_for_freed_and_moved_objects is only called from one thread. This is
   to address the issue described in https://github.com/openjdk/tsan/pull/23#discussion_r1717793720.

-------------

Changes:
  - all: https://git.openjdk.org/tsan/pull/23/files
  - new: https://git.openjdk.org/tsan/pull/23/files/e1f56f32..e4f46136

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=tsan&pr=23&range=01
 - incr: https://webrevs.openjdk.org/?repo=tsan&pr=23&range=00-01

  Stats: 44 lines in 6 files changed: 22 ins; 12 del; 10 mod
  Patch: https://git.openjdk.org/tsan/pull/23.diff
  Fetch: git fetch https://git.openjdk.org/tsan.git pull/23/head:pull/23

PR: https://git.openjdk.org/tsan/pull/23

From jiangli at openjdk.org  Sat Aug 17 01:57:06 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Sat, 17 Aug 2024 01:57:06 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>
Message-ID: <2t6p3tJlolE6uIfruQ_PUCPbNSzHXIJeQQUOCqOGMy0=.220a16c7-ddc7-4a09-b250-e88e5507977d@github.com>

On Thu, 15 Aug 2024 00:33:31 GMT, Man Cao <manc at openjdk.org> wrote:

>> Jiangli Zhou has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - - Release memory for the malloced new TsanOopMapTableKey after adding back to the TsanOopMap. Thanks manc@ for
>>      noticing the memory leak.
>>    - Add TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj).
>>  - Change to call TsanOopMap::notify_tsan_for_freed_and_moved_objects from the last worker thread that completes
>>    the parallel WeakProcessor::Task work:
>>    - Add WeakProcessor::Task::_nworkers_completed to track the completed workers. This is only used if
>>      INCLUDE_TSAN is true.
>>    - Increment WeakProcessor::Task::_nworkers_completed in WeakProcessor::Task::work when each parallel worker
>>      finishes work.
>>    
>>    With the change, TsanOopMap::notify_tsan_for_freed_and_moved_objects is only called from one thread. This is
>>    to address the issue described in https://github.com/openjdk/tsan/pull/23#discussion_r1717793720.
>
> src/hotspot/share/tsan/tsanOopMap.cpp line 235:
> 
>> 233:       const TsanOopMapImpl::MovedEntry &e = moved_entries.at(i);
>> 234:       _oop_map->add_entry(e.key(), e.value());
>> 235:     }
> 
> Is there a memory leak on the `MovedEntry::TsanOopMapTableKey*` after here?
> `TsanOopMapTable::add_entry()` makes a copy `*(e.key())`, but `e.key()` appears never freed.

Fixed. Thanks for noticing it.

> src/hotspot/share/tsan/tsanOopMapTable.cpp line 161:
> 
>> 159:         }
>> 160: 
>> 161:         entry.update_obj();
> 
> It is probably better to remove the `TsanOopMapTableKey::update_obj()` function altogether, so `TsanOopMapTableKey` becomes an immutable object. There is no point to update the old object when it is going to be removed immediately afterwards.
> 
> The code can use `wh_obj` to create a new `TsanOopMapTableKey`. `TsanOopMapTableKey` may need a new constructor though:
> 
> TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj) {
>   _wh = src._wh;
>   _obj = obj;
> }

Adding TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj) and removing TsanOopMapTableKey::update_obj() sounds ok to me. Done.

-------------

PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1720531124
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1720532042

From jiangli at openjdk.org  Sat Aug 17 01:57:06 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Sat, 17 Aug 2024 01:57:06 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <IlHqt2X-AgyzhE863qVkiHGs-LI0WE7ZMQVlrhLgPEs=.2e82c084-61ef-4d2b-bf24-89ff7f71e7df@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <gaa8Wea7k3qRuQdoQw8hxWWrCe-CHno3aFD3yThljwA=.c5f5052f-565a-4ecf-a239-32f7de30b803@github.com>
 <IlHqt2X-AgyzhE863qVkiHGs-LI0WE7ZMQVlrhLgPEs=.2e82c084-61ef-4d2b-bf24-89ff7f71e7df@github.com>
Message-ID: <B-E1f0YJk5dFenq7VDn5Ul0eljArVRSqKLMwxtjA-tM=.20bf8263-b821-40b3-b99f-a76243eeb75a@github.com>

On Thu, 15 Aug 2024 23:03:09 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> src/hotspot/share/tsan/tsanOopMapTable.cpp line 76:
>> 
>>> 74:   bool added;
>>> 75:   size_t* v = _table.put_if_absent(*entry, size, &added);
>>> 76:   assert(added, "must be");
>> 
>>> Since primitive_hash is now used, I also changed TsanOopMapTableKey::get_hash and TsanOopMapTableKey::equals to use entry.object_no_keepalive() instead of entry._obj. If entry._obj is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. 
>> 
>> I think the concurrent processing of `TsanOopMapTable` and another worker thread processing `OopStorageSet` to update TSAN's `WeakHandles` is a scary situation, when we start using `entry.object_no_keepalive()` or `WeakHandle::peek()`. `peek()` may return an updated or not-yet-updated value, which will likely cause hidden bugs.
>> 
>> I came up with the buggy scenario below:
>> Suppose object A was at 0x1200, object B was at 0x2300. During GC, A becomes dead, and GC moves B to A's location, so B is now at 0x1200.
>> Worker thread T1 processed part of the `OopStorageSet` and updated B's weak handle (wh_B).  The weak handle for A (wh_A) is not yet processed.
>> T1 starts to process `TsanOopMapTable`, and it will try adding the new B to the table, with `wh_B.peek()` returing 0x1200. It will fail to add this entry, because the new B looks identical to the entry for A, with `wh_A.peek()` still returning 0x1200.
>> Then object B will be lost from the `TsanOopMapTable`.
>> 
>> I think a better approach is ensure the `TsanOopMapTable` is only processed once after all of the `OopStorageSet` is updated.
>> One option is the alternative approach of inserting  `TsanOopMap::notify_tsan_for_freed_and_moved_objects()` to individual GCs as mentioned in https://github.com/openjdk/tsan/pull/19.
>> Another possibility is to develop a mechanism to check whether `OopStorageSet` has been fully processed, before doing actual work to update `TsanOopMapTable`.
>
> Thanks for the analysis. The scenario describes a potential rare case where an `oop` is moved to an address location containing a freed `oop` and tsan is notified about the moved `oop` before the freed `oop`. I agree that's a potential bug.
> 
> Doing `TsanOopMap::notify_tsan_for_freed_and_moved_objects()` to individual GCs has higher risks of not covering all different cases if GC changes. I like you suggestion of implementing "a mechanism to check whether OopStorageSet has been fully processed". Implementing within `OopStorageSet` could be complicated. I think it might be doable in `WeakProcessor::Task::work` to check if all tasks are done. I'm trying that approach.

See updated PR for changing to only call TsanOopMap::notify_tsan_for_freed_and_moved_objects() in the last worker that completes WeakProcessor::Task work.

-------------

PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1720533267

From manc at openjdk.org  Mon Aug 19 18:57:01 2024
From: manc at openjdk.org (Man Cao)
Date: Mon, 19 Aug 2024 18:57:01 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
Message-ID: <5CV0J_x9ZfULE0NbcKG1iZvPqAQ2JSBExhcwBRO1upY=.c9cf931d-bd03-4cfe-aa7b-389a58bbb1e1@github.com>

On Sat, 17 Aug 2024 01:51:22 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
>> 
>> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
>> 
>> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
>> 
>> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
>> 
>> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
>> 
>> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
>> 
>> The change is updated to only call `TsanOopMap::notify_tsan_for_freed_and_moved_objects` from the last worker thread that completes the parallel WeakProcessor::Task work.
>> 
>> Testing
>> 
>> Tsan jtreg tests on linux-x64:
>> 
>> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
>> 
>> 
>> ==============================
>> Test summary
>> ==============================
>>    TEST                                        ...
>
> Jiangli Zhou has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - - Release memory for the malloced new TsanOopMapTableKey after adding back to the TsanOopMap. Thanks manc@ for
>      noticing the memory leak.
>    - Add TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj).
>  - Change to call TsanOopMap::notify_tsan_for_freed_and_moved_objects from the last worker thread that completes
>    the parallel WeakProcessor::Task work:
>    - Add WeakProcessor::Task::_nworkers_completed to track the completed workers. This is only used if
>      INCLUDE_TSAN is true.
>    - Increment WeakProcessor::Task::_nworkers_completed in WeakProcessor::Task::work when each parallel worker
>      finishes work.
>    
>    With the change, TsanOopMap::notify_tsan_for_freed_and_moved_objects is only called from one thread. This is
>    to address the issue described in https://github.com/openjdk/tsan/pull/23#discussion_r1717793720.

Thanks. This looks pretty good, one last optimization feedback.

src/hotspot/share/gc/shared/weakProcessor.inline.hpp line 100:

> 98: 
> 99:   TSAN_RUNTIME_ONLY(
> 100:     MutexLocker mu(TsanOopMap_lock, Mutex::_no_safepoint_check_flag);

I think we can completely get rid of this lock during GC, by switching to Atomic on `_nworkers_completed`:


TSAN_RUNTIME_ONLY(
uint completed = Atomic::add(&_nworkers_completed, 1);
assert(_nworkers >= completed, "must be");
if (_nworkers == completed) {
  TsanOopMap::notify_tsan_for_freed_and_moved_objects();
}
)


We need to declared `_nworkers_completed` as `volatile uint` for using Atomic.

src/hotspot/share/tsan/tsanOopMapTable.cpp line 73:

> 71: 
> 72: bool TsanOopMapTable::add_entry(TsanOopMapTableKey *entry, size_t size) {
> 73:   assert(TsanOopMap_lock->is_locked(), "sanity check");

If we get rid of the locking in `WeakProcessor::Task::work()`, this assert could be `SafepointSynchronize::is_at_safepoint()` instead.

-------------

PR Review: https://git.openjdk.org/tsan/pull/23#pullrequestreview-2246218431
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1722201699
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1722211993

From jiangli at openjdk.org  Mon Aug 19 21:47:09 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Mon, 19 Aug 2024 21:47:09 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <5CV0J_x9ZfULE0NbcKG1iZvPqAQ2JSBExhcwBRO1upY=.c9cf931d-bd03-4cfe-aa7b-389a58bbb1e1@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
 <5CV0J_x9ZfULE0NbcKG1iZvPqAQ2JSBExhcwBRO1upY=.c9cf931d-bd03-4cfe-aa7b-389a58bbb1e1@github.com>
Message-ID: <cyIcCHE3DxWrPHqWcgYxwcx2tSCQOjTW1_4VfpniyvM=.3a0f1898-0257-47ab-93d3-b8d2daeab315@github.com>

On Mon, 19 Aug 2024 18:38:06 GMT, Man Cao <manc at openjdk.org> wrote:

>> Jiangli Zhou has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - - Release memory for the malloced new TsanOopMapTableKey after adding back to the TsanOopMap. Thanks manc@ for
>>      noticing the memory leak.
>>    - Add TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj).
>>  - Change to call TsanOopMap::notify_tsan_for_freed_and_moved_objects from the last worker thread that completes
>>    the parallel WeakProcessor::Task work:
>>    - Add WeakProcessor::Task::_nworkers_completed to track the completed workers. This is only used if
>>      INCLUDE_TSAN is true.
>>    - Increment WeakProcessor::Task::_nworkers_completed in WeakProcessor::Task::work when each parallel worker
>>      finishes work.
>>    
>>    With the change, TsanOopMap::notify_tsan_for_freed_and_moved_objects is only called from one thread. This is
>>    to address the issue described in https://github.com/openjdk/tsan/pull/23#discussion_r1717793720.
>
> src/hotspot/share/gc/shared/weakProcessor.inline.hpp line 100:
> 
>> 98: 
>> 99:   TSAN_RUNTIME_ONLY(
>> 100:     MutexLocker mu(TsanOopMap_lock, Mutex::_no_safepoint_check_flag);
> 
> I think we can completely get rid of this lock during GC, by switching to Atomic on `_nworkers_completed`:
> 
> 
> TSAN_RUNTIME_ONLY(
> uint completed = Atomic::add(&_nworkers_completed, 1);
> assert(_nworkers >= completed, "must be");
> if (_nworkers == completed) {
>   TsanOopMap::notify_tsan_for_freed_and_moved_objects();
> }
> )
> 
> 
> We need to declared `_nworkers_completed` as `volatile uint` for using Atomic.

I also considered with `Atomic::add` when making the change and decided to go with `TsanOopMap_lock`. The performance was not a concern for tsan. Using `Atomic::add` for tracking the completed worker count without a lock seemed also be a little simpler however. Since we already have the `TsanOopMap_lock` and there could be some potential cases that we uncover where we want to put the operations under the lock protection, I think it's better to keep using `TsanOopMap_lock` for now. We could optimize this later after more testing and fully stabilizing the system for tsan/JDK 21.

> src/hotspot/share/tsan/tsanOopMapTable.cpp line 73:
> 
>> 71: 
>> 72: bool TsanOopMapTable::add_entry(TsanOopMapTableKey *entry, size_t size) {
>> 73:   assert(TsanOopMap_lock->is_locked(), "sanity check");
> 
> If we get rid of the locking in `WeakProcessor::Task::work()`, this assert could be `SafepointSynchronize::is_at_safepoint()` instead.

Please see above comment.

-------------

PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1722380841
PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1722381207

From manc at openjdk.org  Mon Aug 19 22:12:00 2024
From: manc at openjdk.org (Man Cao)
Date: Mon, 19 Aug 2024 22:12:00 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
Message-ID: <0Degeh2pmZwgfNFULLEj-_r3rr2VgTyKrH3b1230kC8=.99d258dc-dd3a-42b7-bba1-fe8c86f8c95d@github.com>

On Sat, 17 Aug 2024 01:51:22 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
>> 
>> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
>> 
>> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
>> 
>> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
>> 
>> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
>> 
>> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
>> 
>> The change is updated to only call `TsanOopMap::notify_tsan_for_freed_and_moved_objects` from the last worker thread that completes the parallel WeakProcessor::Task work.
>> 
>> Testing
>> 
>> Tsan jtreg tests on linux-x64:
>> 
>> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
>> 
>> 
>> ==============================
>> Test summary
>> ==============================
>>    TEST                                        ...
>
> Jiangli Zhou has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - - Release memory for the malloced new TsanOopMapTableKey after adding back to the TsanOopMap. Thanks manc@ for
>      noticing the memory leak.
>    - Add TsanOopMapTableKey::TsanOopMapTableKey(const TsanOopMapTableKey& src, oop obj).
>  - Change to call TsanOopMap::notify_tsan_for_freed_and_moved_objects from the last worker thread that completes
>    the parallel WeakProcessor::Task work:
>    - Add WeakProcessor::Task::_nworkers_completed to track the completed workers. This is only used if
>      INCLUDE_TSAN is true.
>    - Increment WeakProcessor::Task::_nworkers_completed in WeakProcessor::Task::work when each parallel worker
>      finishes work.
>    
>    With the change, TsanOopMap::notify_tsan_for_freed_and_moved_objects is only called from one thread. This is
>    to address the issue described in https://github.com/openjdk/tsan/pull/23#discussion_r1717793720.

Marked as reviewed by manc (Reviewer).

-------------

PR Review: https://git.openjdk.org/tsan/pull/23#pullrequestreview-2246561031

From manc at openjdk.org  Mon Aug 19 22:12:01 2024
From: manc at openjdk.org (Man Cao)
Date: Mon, 19 Aug 2024 22:12:01 GMT
Subject: RFR: Tsan oop map table hash fix [v2]
In-Reply-To: <cyIcCHE3DxWrPHqWcgYxwcx2tSCQOjTW1_4VfpniyvM=.3a0f1898-0257-47ab-93d3-b8d2daeab315@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
 <ZoyPHY4t5bUprvU4vXT1MzYZ3o7Jci8myNiqip2yLS4=.3446b024-70b3-417d-a24d-108cb08fc37c@github.com>
 <5CV0J_x9ZfULE0NbcKG1iZvPqAQ2JSBExhcwBRO1upY=.c9cf931d-bd03-4cfe-aa7b-389a58bbb1e1@github.com>
 <cyIcCHE3DxWrPHqWcgYxwcx2tSCQOjTW1_4VfpniyvM=.3a0f1898-0257-47ab-93d3-b8d2daeab315@github.com>
Message-ID: <-A0l23xfOTth5_G-hva3-5X_KpjN95d2L859hOr73OQ=.2bbc8303-ed95-4967-8f93-7c56261e5a5e@github.com>

On Mon, 19 Aug 2024 21:43:35 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> src/hotspot/share/gc/shared/weakProcessor.inline.hpp line 100:
>> 
>>> 98: 
>>> 99:   TSAN_RUNTIME_ONLY(
>>> 100:     MutexLocker mu(TsanOopMap_lock, Mutex::_no_safepoint_check_flag);
>> 
>> I think we can completely get rid of this lock during GC, by switching to Atomic on `_nworkers_completed`:
>> 
>> 
>> TSAN_RUNTIME_ONLY(
>> uint completed = Atomic::add(&_nworkers_completed, 1);
>> assert(_nworkers >= completed, "must be");
>> if (_nworkers == completed) {
>>   TsanOopMap::notify_tsan_for_freed_and_moved_objects();
>> }
>> )
>> 
>> 
>> We need to declared `_nworkers_completed` as `volatile uint` for using Atomic.
>
> I also considered with `Atomic::add` when making the change and decided to go with `TsanOopMap_lock`. The performance was not a concern for tsan. Using `Atomic::add` for tracking the completed worker count without a lock seemed also be a little simpler however. Since we already have the `TsanOopMap_lock` and there could be some potential cases that we uncover where we want to put the operations under the lock protection, I think it's better to keep using `TsanOopMap_lock` for now. We could optimize this later after more testing and fully stabilizing the system for tsan/JDK 21.

Do this later SGTM.

-------------

PR Review Comment: https://git.openjdk.org/tsan/pull/23#discussion_r1722410990

From jiangli at openjdk.org  Mon Aug 19 22:18:57 2024
From: jiangli at openjdk.org (Jiangli Zhou)
Date: Mon, 19 Aug 2024 22:18:57 GMT
Subject: Integrated: Tsan oop map table hash fix
In-Reply-To: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
References: <y72SiiaHChPZsqw3zYX_Rivz_-hTxzpK1LcGSOM-igA=.f6a9efaa-c726-4316-81fb-fdc0eb42e6c3@github.com>
Message-ID: <31_M49sDdUkm3S_u42ENSaYk0FfHheZx5WlF8qbkmPQ=.0afa52c8-1dcc-41fb-857a-038f16ea487f@github.com>

On Thu, 15 Aug 2024 00:07:22 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

> Additional testing on JDK 21 with tsan found some crashes due to using `oop` `identity_hash` for computing the TsanOopMap entry hash. The crashes manifest as following two types:
> 
> 1) `guarantee(moves_this_cycle) failed: "Impossible to reconcile GC"`
> 
> 2) `ThreadSanitizer: CHECK failed: tsan_sync.cpp:261 "((*dst_meta)) == ((0))"`
> 
> With `-Xlog:tsan=trace` output, I found the same `oop` is added to the map twice in some cases. The second time when an `oop` is added occurs during `InterpreterMacroAssembler::lock_object` path. Apparently a new identity hash is computed for the `oop` with a previous computed identity hash. That causes the `oop` being added again. Later when notifying tsan if the `oop` is moved, it triggers the above crashes depending on the memory layout.
> 
> This fix computes `primitive_hash` using the oop address instead of `identity_hash` for entry hash. During GC, `TsanOopMapTable::collect_moved_objects_and_notify_freed` unlinks an entry from the map if the enclosing `oop` is moved. The moved entries are collected into an array and  are inserted back to the map after iterating the map (within a worker thread). That's needed since the entry hash would be different after an `oop` is moved. The operations are protected by the `TsanOopMap_lock`.
> 
> Since `primitive_hash` is now used, I also changed `TsanOopMapTableKey::get_hash` and `TsanOopMapTableKey::equals` to use `entry.object_no_keepalive()` instead of `entry._obj`. If `entry._obj` is outdated, it could result incorrect hash value for the current entry. So the updated oop address should be used instead. Running tsan jtreg tests on slowdebug binary run into new failures without the change.
> 
> The change is updated to only call `TsanOopMap::notify_tsan_for_freed_and_moved_objects` from the last worker thread that completes the parallel WeakProcessor::Task work.
> 
> Testing
> 
> Tsan jtreg tests on linux-x64:
> 
> $ bash configure --with-boot-jdk=/usr/local/google/home/jianglizhou/openjdk/jdk-21.0.1 --with-debug-level=release --disable-warnings-as-errors --with-jtreg=/usr/local/google/home/jianglizhou/github/jtreg/build/images/jtreg --with-stdc++lib=static --disable-precompiled-headers --enable-unlimited-crypto --with-native-debug-symbols=internal --with-default-make-target=jdk-image --disable-warnings-as-errors --with-toolchain-type=clang
> 
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hot...

This pull request has now been integrated.

Changeset: aa3821a2
Author:    Jiangli Zhou <jiangli at openjdk.org>
URL:       https://git.openjdk.org/tsan/commit/aa3821a24450081e41d495da2841dd15496727d3
Stats:     97 lines in 6 files changed: 72 ins; 13 del; 12 mod

Tsan oop map table hash fix

Reviewed-by: manc

-------------

PR: https://git.openjdk.org/tsan/pull/23

From manc at google.com  Thu Aug 22 19:18:44 2024
From: manc at google.com (Man Cao)
Date: Thu, 22 Aug 2024 12:18:44 -0700
Subject: Result: New Tsan Reviewer: Jiangli Zhou
Message-ID: <CA+w6Hxa1x42r_bGJRyj8BgC_65gNgS91Xxn-K63Fawp_VjgEjQ@mail.gmail.com>

Voting for Jiangli Zhou [1] is now closed.
Yes: 1 (including myself)
Veto: 0
Abstain: 0

According to the Bylaws definition of Lazy Consensus, this is sufficient to
approve the nomination.

Man Cao

[1] https://mail.openjdk.org/pipermail/tsan-dev/2024-August/000558.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/tsan-dev/attachments/20240822/c0456be1/attachment-0001.htm>

From manc at google.com  Wed Aug 28 18:05:55 2024
From: manc at google.com (Man Cao)
Date: Wed, 28 Aug 2024 11:05:55 -0700
Subject: Result: New Tsan Reviewer: Jiangli Zhou
In-Reply-To: <CA+w6Hxa1x42r_bGJRyj8BgC_65gNgS91Xxn-K63Fawp_VjgEjQ@mail.gmail.com>
References: <CA+w6Hxa1x42r_bGJRyj8BgC_65gNgS91Xxn-K63Fawp_VjgEjQ@mail.gmail.com>
Message-ID: <CA+w6Hxbw34T=NLAvLZzobNYafQDa6M15Nk21r4t0H_TBr2yG8g@mail.gmail.com>

Forgot to CC to registrar at openjdk.org.

Could you grant Jiangli the Reviewer role on project Tsan?

-Man


On Thu, Aug 22, 2024 at 12:18?PM Man Cao <manc at google.com> wrote:

> Voting for Jiangli Zhou [1] is now closed.
> Yes: 1 (including myself)
> Veto: 0
> Abstain: 0
>
> According to the Bylaws definition of Lazy Consensus, this is sufficient
> to approve the nomination.
>
> Man Cao
>
> [1] https://mail.openjdk.org/pipermail/tsan-dev/2024-August/000558.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/tsan-dev/attachments/20240828/14c70fab/attachment.htm>