<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Severin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for your comments – hopefully, this can provide some clarity:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> This boils down to having different profiles for physical deployments (status quo) vs. deployment in containers (on k8s), right?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes, the default ergonomics for physical deployments (`shared` profile) would remain unchanged. The JVM would activate the `dedicated` profile by default when it detects that it is in that type of environment
(e.g., the JVM sees it is running inside a cgroups namespace or a container). A JVM flag could also override the ergonomics profile -XX:ErgonomicsProfile=<shared|dedicated>. This flag would be useful, especially for Cloud PaaS services where sometimes the
environment is a pure VM (not cgroups) but still dedicated to that JVM process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> Of course, such a feature would have a non-free cost item on the continuous test column. Any thoughts how you'd plan to ensure that both profiles behave as they're supposed to behave?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We could piggyback off existing container tests here to ensure that the dedicated profile is set correctly when the JVM runs in environments where we want the dedicated profile to be on by default (e.g., cgroups).
However, we will need to resolve a few known issues before this proposal lands. Most notably that OSContainer::is_containerized() returns true when run outside a container [1].
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> Perhaps it would sense from a high-level understanding perspective to sketch out what you envision such a 'dedicated' profile would amount to? Do you have some concrete ideas?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We believe the dedicated profile should set a larger default max heap size - likely 75% of the available memory. This 75% guideline aligns with what we already see in the .NET Runtime [2]. We are also looking
into changing the default GCs in dedicated, based on available memory, especially considering the remaining 25% of non-heap memory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">One possibility is to use ParallelGC by default within `dedicated` with 2+ processors and up to a certain amount of available memory. We are initially considering 2GB or 4GB as the threshold before switching
to G1GC. Then, after a certain amount of memory, we could set ZGC, with 32GB+ of memory available.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Other OpenJDK distributions could, of course, change these profiles and have different settings, with even different GCs (e.g., Shenandoah instead of ZGC in the Red Hat Build of OpenJDK; or Azul C4 in Azul
Platform Prime; and so on).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We still need to do some tests, and we will report back here, but we would love to hear your thoughts on the concept.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> Any thoughts why active processor counting would need adjustment for such a profile? Why would the current way how the container detection code abstracts that metric be insufficient?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We observed a few tests where bumping active processor counting to 2 when the JVM has a 1000millicore cpu limit yielded better results. After more thought into this and reviewing the executed tests, we are
no longer ready to proceed with this adjustment. Moreover, several other articles and advocates strongly suggest that application workloads on Kubernetes should have no CPU limit at all (only cpu request). If users follow this guidance, there should be no
need to tweak active processor counting, as JVM would consider them all.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> Greater than 2 profiles seem concerning. Why do you think more than two would be necessary?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">At this stage, we only have plans for the `shared` and `dedicated` profiles. However, while it is not a goal of the initial proposal, there is an opportunity for this feature to provide a framework for OpenJDK
to allow OpenJDK providers in the future to bundle a set of ergonomics for a broader set of workloads and hardware. For example, on Microsoft Azure, we have VMs that are “General purpose”, “Compute optimized” (higher CPU-to-memory ratio), “Memory optimized”
(higher Memory-to-CPU ratio), “Disk optimized” (high disk throughput and IO; for databases and data warehouse) [3].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Other JVM flags could be also enabled by default as well, such as UseNUMA.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">As for how to make the JVM observe these special VMs, the VM image could have a default JAVA_TOOL_OPTIONS=-XX:ErgonomicsProfile=`<profile>` to be picked up automatically by HotSpot while still maintaining
the `shared` as the default in the case of traditional VMs and bare metal (non-container/non-cgroups).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In summary, we believe ergonomics profiles are a step forward in JVM tuning without complexity, allowing developers to continue relying on JVM defaults.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[1] <a href="https://bugs.openjdk.org/browse/JDK-8261242">
[JDK-8261242] [Linux] OSContainer::is_containerized() returns true when run outside a container - Java Bug System (openjdk.org)</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2] <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#heap-limit">
Garbage collector config settings - .NET | Microsoft Learn</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[3] <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/sizes">
VM sizes - Azure Virtual Machines | Microsoft Learn</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Stephanie and Bruno<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Severin Gehwolf <sgehwolf@redhat.com><br>
<b>Date: </b>Friday, May 19, 2023 at 2:23 AM<br>
<b>To: </b>Stephanie Crater <scrater@microsoft.com>, hotspot-dev@openjdk.org <hotspot-dev@openjdk.org><br>
<b>Subject: </b>[EXTERNAL] Re: Proposed Ergonomics Profiles<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">Hi Stephanie,<br>
<br>
In principle it would be useful to have, so I'd be on board with such a<br>
proposal. It would free us from rolling our own tuning in downstream<br>
images. This boils down to having different profiles for physical<br>
deployments (status quo) vs. deployment in containers (on k8s), right?<br>
<br>
Of course, such a feature would have a non-free cost item on the<br>
continuous test column. Any thoughts how you'd plan to ensure that both<br>
profiles behave as they're supposed to behave?<br>
<br>
On Tue, 2023-05-16 at 20:18 +0000, Stephanie Crater wrote:<br>
> Hi, <br>
> <br>
> The Java Engineering Group at Microsoft is currently working on a JEP<br>
> to introduce Ergonomics Profiles as a new JVM feature, with a<br>
> `shared` profile for the existing JVM ergonomics and a `dedicated`<br>
> option for when the JVM is running on systems with dedicated<br>
> resources for the one JVM process.<br>
> <br>
> The current default JVM ergonomics were designed with the<br>
> understanding that the JVM must share resources with other processes.<br>
> However, a recent study done by an APM vendor (New Relic) identified<br>
> that more than 70% of monitored JVMs [1] in production are running in<br>
> dedicated environments (e.g., containers) as opposed to being shared.<br>
> Many of these JVMs are running without explicit JVM tuning flags,<br>
> once more confirming that JVM tuning is a challenging exercise many<br>
> developers have no experience with. Introducing updated ergonomics<br>
> for when the JVM is running in specific environments would allow the<br>
> JVM to consume available resources more effectively instead of<br>
> running with default ergonomics aimed at shared environments.<br>
> <br>
> For example, our customer data from Azure Spring Apps shows that 83%<br>
> of monitored JVMs do not use JVM flags to set the heap size. Using<br>
> the current JVM ergonomics, the default maximum heap size of the JVM<br>
> varies from 50% to 25%, depending on how much memory is available in<br>
> the environment: up to 256MB, or 512MB or more, respectively, with a<br>
> fixed amount of ~127MB for systems with anywhere between 256MB and<br>
> 512MB of memory. These amounts do not adequately map the intended<br>
> resource plan of dedicated environments. The user may have already<br>
> considered to allocating, e.g., 4GB of memory to the JVM and expect<br>
> it to use more than only 1GB of the heap (25%). <br>
> <br>
> The `dedicated` ergonomics profile will contain different heuristics<br>
> to increase resource consumption in the environment, compared to<br>
> `shared`.<br>
<br>
Perhaps it would sense from an high level understanding perspective to<br>
sketch out what you envision such a 'dedicated' profile would actually<br>
amount to? Do you have some concrete ideas?<br>
<br>
> The ergonomics we target include heuristics for maximum heap size, GC<br>
> selection, active processor counting, and thread pool sizes internal<br>
> to the JVM. If it would help, we have started writing this proposal<br>
> in a JEP format.<br>
<br>
Any thoughts why active processor counting would need adjustment for<br>
such a profile? Why would the current way how the container detection<br>
code abstracts that metric be insufficient?<br>
<br>
> We would love to hear what the community thinks about this proposed<br>
> enhancement and any suggestions you may have for the dedicated<br>
> ergonomics profile. For example, this profile will likely increase<br>
> heap size allocation to 60%-70% by default, but GC selection and<br>
> active processor counting are much more complex. This JEP would also<br>
> provide a framework for OpenJDK to include more ergonomics profiles<br>
> for specific machines, environments, or workloads.<br>
<br>
Greater than 2 profiles seem concerning. Why do you think more than two<br>
would be necessary?<br>
<br>
Thanks,<br>
Severin<br>
<br>
> Thank you for the feedback! <br>
> <br>
> [1]: <a href="https://newrelic.com/resources/report/2023-state-of-the-java-ecosystem">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnewrelic.com%2Fresources%2Freport%2F2023-state-of-the-java-ecosystem&data=05%7C01%7Cscrater%40microsoft.com%7C0b347b7060284548e33808db584aa219%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638200849798608285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iePSh5jvyA36bbRMOzUIUVodp9e8XEfYdonzTkMDrTI%3D&reserved=0</a><o:p></o:p></span></p>
</div>
</div>
</body>
</html>