<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-indent:21.0pt;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1022630823;
mso-list-type:hybrid;
mso-list-template-ids:151423764 -719807802 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:宋体;
mso-bidi-font-family:宋体;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:63.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:126.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:189.0pt;
text-indent:-21.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">>Otherwise if fixup_callers_callsite would call icache_flush() somewhere inside, then safepoint_ifence wouldn’t be needed here<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Yes, we called icache_flush in fixup_callers_callsite:
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">SharedRuntime::fixup_callers_callsite->NativeCall::set_destination_mt_safe->ICache::invalidate_range-> icache_flush.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">And I started a PR to fix the usage of fence.i in user space of RISC-V port:
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://github.com/openjdk/jdk/pull/9770">https://github.com/openjdk/jdk/pull/9770</a>.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ISBs used in Aarch64 port are following the AArch64 Reference Manual:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Ensuring the visibility of updates to instructions for a multiprocessor<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">The ARMv8 architecture requires a PE that performs an instruction cache maintenance operation to execute a DSB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">instruction to ensure completion of the maintenance operation. This ensures that the cache maintenance operation<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">is complete on all PEs in the Inner Shareable shareability domain.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">An ISB is not broadcast, and so does not affect other PEs. This means that any other PE must perform its own ISB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">synchronization after it knows that the update is visible, if it is necessary to ensure its synchronization with the<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">update. The following example shows how this might be done:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">AArch64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">P1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">STR X11, [X1] ;X11 contains a new instruction to stored in program memory<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">DC CVAU, X1 ; clean to PoU makes visible to instruction cache<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">DSB ISH ; ensure completion of the clean on all processors<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">IC IVAU, X1 ; ensure instruction cache/branch predictor discard stale data<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">DSB ISH ; ensure completion of the ICache and branch predictor<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:84.0pt;text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">; invalidation on all processors<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">STR W0, [X2] ; set flag to signal completion<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">ISB ; synchronize context on this processor<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">BR R1 ; branch to new code<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">P2-Px<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">WAIT ([X2] == 1) ; wait for flag signalling completion<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">ISB ; synchronize context on this processor<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">BR X1 ; branch to new code<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Vladimir Kempik [mailto:vladimir.kempik@gmail.com]
<br>
<b>Sent:</b> Saturday, August 6, 2022 6:07 AM<br>
<b>To:</b> wangyadong (E) <yadonn.wang@huawei.com><br>
<b>Cc:</b> Palmer Dabbelt <palmer@dabbelt.com>; riscv-port-dev@openjdk.org<br>
<b>Subject:</b> Re: The usage of fence.i in openjdk<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">More on this subject<br>
I can see the use of ifence() in the code is identical to the use of isb() in aarch64.<br>
Checking the documentation for fence.i and isb, I don’t see them to be 1:1 identical<br>
<br>
fence.i ( <a href="https://five-embeddev.com/riscv-isa-manual/latest/zifencei.html">
https://five-embeddev.com/riscv-isa-manual/latest/zifencei.html</a> ): <br>
FENCE.I instruction provides explicit synchronization between writes to instruction memory and instruction fetches on the same hart.<br>
<br>
ISB ( <a href="https://developer.arm.com/documentation/den0024/a/Memory-Ordering/Barriers/ISB-in-more-detail">https://developer.arm.com/documentation/den0024/a/Memory-Ordering/Barriers/ISB-in-more-detail</a> ):<br>
An ISB flushes the pipeline, and re-fetches the instructions from the cache or memory and ensures that the effects of any completed context-changing operation before the ISB are visible to any instruction after the ISB. It also ensures that any context-changing
operations after the ISB instruction only take effect after the ISB has been executed and are not seen by instructions before the ISB. <br>
And some info from the web:<br>
<br>
To me it sound like isb ( in aarch64) does the job a bit different than fence.i ( in rv64)<br>
<br>
So, I think here:<br>
<br>
__ la_patchable(t0, RuntimeAddress(CAST_FROM_FN_PTR(address, SharedRuntime::fixup_callers_callsite)), offset);<br>
__ jalr(x1, t0, offset);<br>
<br>
// Explicit fence.i required because fixup_callers_callsite may change the code<br>
// stream.<br>
__ safepoint_ifence();<br>
<br>
__ pop_CPU_state();<br>
// restore sp<br>
__ leave();<br>
__ bind(L);<br>
<br>
we still have a small chance to start executing invalid ( old) code from l1i if right after safepoint_ifence() our thread would be moved to another hart. Otherwise if fixup_callers_callsite would call icache_flush() somewhere inside, then safepoint_ifence
wouldn’t be needed here<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Regards, Vladimir<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">30 июля 2022 г., в 13:29, Vladimir Kempik <<a href="mailto:vladimir.kempik@gmail.com">vladimir.kempik@gmail.com</a>> написал(а):<br>
<br>
Hello<br>
Thanks for explanation.<br>
that sounds like the fence.i in userspace code is not needed at all<br>
Regards, Vladimir<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">30 июля 2022 г., в 05:41, wangyadong (E) <<a href="mailto:yadonn.wang@huawei.com">yadonn.wang@huawei.com</a>> написал(а):<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Lets say you have a thread A running on hart 1.<br>
You've changed some code in region 0x11223300 and need fence.i before executing that code.<br>
you execute fence.i in your thread A running on hart 1. <br>
right after that your thread ( for some reason) got rescheduled ( by kernel) to hart 2.<br>
if hart 2 had something in l1i corresponding to region 0x11223300, then you gonna have a problem: l1i on hart 2 has old code, it wasn’t refreshed, because fence.i was executed on hart 1 ( and never on hart 2). And you thread gonna execute old code, or mix of
old and new code.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
@vladimir Thanks for your explanation. I understand your concern now. We know the fence.i's scope, so the write hart does not rely solely on the fence.i in RISC-V port, but calls the icache_flush syscall in ICache::invalidate_range() every time after modifying
the code.<br>
<br>
For example:<br>
Hart 1<br>
void MacroAssembler::emit_static_call_stub() {<br>
// CompiledDirectStaticCall::set_to_interpreted knows the<br>
// exact layout of this stub.<br>
<br>
ifence();<br>
mov_metadata(xmethod, (Metadata*)NULL); <- patchable code here<br>
<br>
// Jump to the entry point of the i2c stub.<br>
int32_t offset = 0;<br>
movptr_with_offset(t0, 0, offset);<br>
jalr(x0, t0, offset);<br>
}<br>
<br>
Hart 2 (write hart)<br>
void NativeMovConstReg::set_data(intptr_t x) {<br>
// ...<br>
// Store x into the instruction stream.<br>
MacroAssembler::pd_patch_instruction_size(instruction_address(), (address)x); <- write code<br>
ICache::invalidate_range(instruction_address(), movptr_instruction_size); <- syscall here<br>
// ...<br>
} <o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>