<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<base href="x-msg://368/"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Igor,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">we didn’t find an easy and feasible way to ensure the ordering, either.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Grabbing the buffers and cleaning the cards at safepoints might be the best solution.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Maybe removing the barrier that flushes the store to the cardtable makes the problem more likely to occur.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I guess the purpose of the barrier was exactly to avoid this problem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(which should be working perfectly if the post barriers had StoreLoad barriers, too).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Igor Veresov [mailto:iggy.veresov@gmail.com]
<br>
<b>Sent:</b> Donnerstag, 27. Juni 2013 16:13<br>
<b>To:</b> Doerr, Martin<br>
<b>Cc:</b> John Cuthbertson; hotspot-gc-dev@openjdk.java.net; Braun, Matthias<br>
<b>Subject:</b> Re: G1 question: concurrent cleaning of dirty cards<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Yea, unless I'm forgetting something, this seems very fundamental. The probability of this happening is probably greatly reduced by the card cache, nevertheless it seems possible. The only solution that comes to mind is do periodic safepoint
to grab the already filled buffers and clean the corresponding entries in the card table. The processing of the grabbed buffers may of course be done concurrently.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">But there doesn't seem to be an easy way to ensure the ordering between the original store to the object and the cleaning store to the card table. The barrier that flushes the store to the cardtable doesn't in any way enforce the original
store to the object from the other thread to happen before that. So the failure case would be this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Mutator thread: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- store to object <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- load from cardtable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- compare the cardtable byte (it is dirty)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- bail from barrier<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Refinement thread:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- clear the card<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If clearing of the card occurs after the mutator loads the byte from the cardtable, the mutator won't enqueue the card, which is sort of the intended behavior. But there is no guarantee that the refinement thread would see the result of
"store to object", in which case the information will be lost.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">igor<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Jun 27, 2013, at 1:27 AM, "Doerr, Martin" <<a href="mailto:martin.doerr@sap.com">martin.doerr@sap.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Igor,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">we have seen crashes while testing our hotspot 23 based SAPJVM with G1.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However, there’s no evidence that these crashes are caused by this problem.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We basically found it by reading code.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Igor
Veresov [mailto:iggy.veresov@<a href="http://gmail.com">gmail.com</a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Donnerstag, 27. Juni 2013 08:15<br>
<b>To:</b><span class="apple-converted-space"> </span>Doerr, Martin<br>
<b>Cc:</b><span class="apple-converted-space"> </span>John Cuthbertson; <a href="mailto:hotspot-gc-dev@openjdk.java.net">
hotspot-gc-dev@openjdk.java.net</a>; Braun, Matthias<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: G1 question: concurrent cleaning of dirty cards</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Oh, re-read your letter, yup, there seems to be a problem. Have you observed that in practice?<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">igor<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Jun 26, 2013, at 9:27 PM, Igor Veresov <<a href="mailto:iggy.veresov@gmail.com"><span style="color:purple">iggy.veresov@gmail.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">The cards that are stored in the buffers are not available for concurrent processing right when they are enqueued. Instead they are passed to the processing threads when the buffer fills up. This passing of the buffer involves signaling
of a condition (like pthread_cond_signal(), literally) that has a write barrier for sure, which would guarantee that the cards in the buffer, and contents of the card table, and the contents of the object are "in sync".<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">The only place in the generated code where there has to be a store-store barrier (for non-TSO architectures) is between the actual field store and the dirtying of the card.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Does this make sense?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">igor<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On May 23, 2013, at 6:12 AM, "Doerr, Martin" <<a href="mailto:martin.doerr@sap.com"><span style="color:purple">martin.doerr@sap.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi John,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">thank you very much for your comments. Your last line explains exactly what we are concerned about.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Does anybody plan to prevent this situation?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I don’t want to propose adding StoreLoad barriers in all G1 post barriers because I’d expect undesired performance impact.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Would it be feasible to rescan all cards which have been dirtied (at least once) during the next stop-the-world phase?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Maybe anybody has a better idea.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Kind regards,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">John
Cuthbertson [mailto:john.cuthbertson@<a href="http://oracle.com/"><span style="color:purple">oracle.com</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Donnerstag, 23. Mai 2013 02:29<br>
<b>To:</b><span class="apple-converted-space"> </span>Doerr, Martin<br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:hotspot-gc-dev@openjdk.java.net"><span style="color:purple">hotspot-gc-dev@openjdk.java.net</span></a>; Mikael Gerdin; Braun, Matthias<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: G1 question: concurrent cleaning of dirty cards</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Hi Martin,<br>
<br>
An enqueued card let's the refinment threads know that the oops spanned by that card need to be walked but we're only interested in the latest contents of the fields in those oops. IOW the oop in (3') doesn't need to be the oop stored in (1). If there's a subsequent
store (3) to the same location then we want the load at (3') to see the lastest contents. For example suppose we have:<br>
<br>
x.f = a;<br>
x.f = b;<br>
<br>
If the application thread sees the card spanning x.f is dirty at the second store then we won't enqueue the card after the second store. As long as the refinement thread sees 'b' when the card is 'refined' then we're OK since we no longer need to add an entry
into the RSet for the region containing a - we do need an entry in the RSet for the region containing b.<br>
<br>
If the application thread sees the card as clean at the second store before the refinement thread loads x.f we have just needlessly enqueued the card again.<br>
<br>
<b>It is only if the application thread sees the card as dirty but the refinement thread reads 'a' then there could be a problem. We have a missing RSet entry for 'b'.</b><br>
<br>
JohnC</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">On 5/17/2013 1:29 AM, Doerr, Martin wrote:</span><o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">Hi all,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">we have a question about the interaction between G1 post barriers and the refinement thread's concurrent dirty card cleaning.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">The case in which the G1 post barrier sees a clean card is obviously not problematic, because it will add an entry in a dirty card queue.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">However, in case in which the Java thread (mutator thread) sees the card already dirtied, it won’t enqueue the card again. Which is safe as long as its stored oop (1)
is seen and processed (3’) by the parallel refinement after having cleaned the card (1’):</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">Java Thread (mutator) Refinement Thread (G1RemSet::concurrentRefineOneCard_impl calls oops_on_card_seq_iterate_careful)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">(1) store(oop)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">( StoreLoad required here ?)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">(2) load(card==dirty)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> (1’) store(card==clean)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> (2’) StoreLoad barrier</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> (3’) load(oop)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">So the refinement thread seems to rely on getting the oop which was written BEFORE the (2) load(card==dirty) was observed.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">We wonder how this ordering is guaranteed? There are no StoreLoad barriers in the Java Thread's path. (StoreLoad ordering needs explicit barriers even on TSO platforms.)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">Kind regards,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New"">Martin</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>