<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Panama people, hello Maurizio,</p>
<p>sending this again to the mailing list, we had just private
discussion with Maurizio. Maybe anyone else has an idea or might
figure out what the problem is. We are not yet ready to open issue
against Java 19 till 22.<br>
</p>
<p>There were several issues reported by users of Apache Lucene (a
wrongly-written benchmark and also Solr users) about bad
performance in highly concurrent environments. Actually what was
found out is that when you have many threads closing shared
arenas, under some circumstances it causes all "reader threads"
(those accessing MemorySegment no matter which arena they use)
suddenly deoptimize. This causes immense slowdowns during Lucene
searches.</p>
<p>Lucily one of our committers found a workaround and we are
investigating to write a benchmark shoing the issue. But first let
me explain what happens:</p>
<ul>
<li>Lucene opens MemorySegments with a shared Arenas (one per
file) and accesses them by multiple threads. Basically for each
index file we have a shared arena which is closed when the file
is closed.</li>
<li>There are many shared arenas (one per index file)!!!</li>
<li>If you close a shared arena normally you see no large delay on
the thread calling the close and also no real effect on any
thread that reads from other MemorySegments</li>
<li>But under certain circumstances ALL reading threads accessing
any MemorySegment slow down dramatically! So once you close one
of our Arenas, all other MemorySegments using a different shared
arena suddelny get deoptimized (we have Hotspot logs showing
this). Of course, the MemorySegment belonging to the closed
arena is no longer used and this one should in reality the only
affected one (throwing a IllegalStateEx).<br>
</li>
<li>The problem seems to occur mainly when multiple Arenas are
closed in highly concurrent environments. This is why we did not
see the issue before.</li>
<li>If we put a gloval lock around all calls to Arena#close() the
issues seem to go away.<br>
</li>
</ul>
<p>I plan to write some benchmark showing this issue. Do you have an
idea what could go wrong? To me it looks like race in the
thread-local handshakes which may cause some crazy hotspot
behaviour causing of deotimization of all threads concurrently
accessing MemorySegments once Arena#close() is called in highly
concurrent environments.<br>
</p>
<p>This is the main issue where the observation is tracked: <a
class="moz-txt-link-freetext"
href="https://github.com/apache/lucene/issues/13325"
moz-do-not-send="true">https://github.com/apache/lucene/issues/13325</a></p>
<p>These are issues opened:</p>
<ul>
<li><a class="moz-txt-link-freetext"
href="https://github.com/dacapobench/dacapobench/issues/264"
moz-do-not-send="true">https://github.com/dacapobench/dacapobench/issues/264</a>
(the issue on this benchmark was of course that they wer
opening/closing too often, but actually it just showed the
problem, so it was very helpful). Funny detail: Alexey Shipilev
opened the issue!</li>
<li>This was the comment showing the issue at huge installations
of Apache Solr 9.7: <a class="moz-txt-link-freetext"
href="https://github.com/apache/lucene/pull/13146#pullrequestreview-2089347714"
moz-do-not-send="true">https://github.com/apache/lucene/pull/13146#pullrequestreview-2089347714</a>
(David Smiley also talked to me at berlinbuzzwords). They had to
disable MemorySegment usage in Apache SOlr/Lucene in their
environment. They have machines with thousands of indexes (and
therefor 10 thousands of Arenas) open at same time and the close
rate is very very high!</li>
</ul>
<p>Uwe<br>
</p>
<pre class="moz-signature" cols="72">--
Uwe Schindler
<a class="moz-txt-link-abbreviated" href="mailto:uschindler@apache.org">uschindler@apache.org</a>
ASF Member, Member of PMC and Committer of Apache Lucene and Apache Solr
Bremen, Germany
<a class="moz-txt-link-freetext" href="https://lucene.apache.org/">https://lucene.apache.org/</a>
<a class="moz-txt-link-freetext" href="https://solr.apache.org/">https://solr.apache.org/</a></pre>
</body>
</html>