<div dir="ltr"><p style="margin:0in;font-family:Calibri;font-size:11pt">Hi Ron,</p>
<p style="margin:0in;font-family:Calibri;font-size:11pt">I reviewed integrity
JEPs once again along with this email thread and I think there are several
flaws in the proposal that need to be addressed before implementation.</p>
<p style="margin:0in;font-family:Calibri;font-size:11pt"> </p>
<ol type="1" style="direction:ltr;unicode-bidi:embed;margin-top:0in;margin-bottom:0in;font-family:Calibri;font-size:11pt">
<li value="1" style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">First, the JEP draws equality
between an agent and an instrumenting agent, which is not true.
Instrumentation is just one of the capabilities that an agent needs to
request explicitly by calling JVM TI AddCapabilities function. There are
many other read-only features of JVM TI that observability and
troubleshooting agents can use without compromising application integrity.
Disabling all agents by default just to protect from a few ones that
modify application code is like cracking a nut with a sledgehammer,
especially when a more fine-grained approach is already built into JVM TI.<br><br></span></li>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">JEP states that most
serviceability tools do not require dynamic agents. This sounds weird to
me. How was that "most" measured? How can half a dozen JDK
builtin tools be compared to an infinite number of custom tools that
may be and already developed using JVM TI?<br><br></span></li>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">JEP assumes that existing JDK
tools are enough for troubleshooting. I wish they were. How, for example,
you would dump an object graph without sensitive user data from a live
service? With JVM TI agent, this is possible. Which builtin tool allows
you to find native memory leaks, sources of long time-to-safepoint pauses,
map perf counters to Java code? Unfortunately, none. Even worse,
when dynamic agents are disabled, development of new custom tools will
become meaningless.<br><br></span></li>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">You emphasized many times
that the proposal to disable dynamic agents appeared years ago. And that's
actually the problem with this JEP. It relies on outdated assumptions
and has not been adjusted to the modern trends. Technology didn't stay
still; new use cases became popular, which this proposal does not take
into account. Here are some examples:</span></li>
<ol type="a" style="direction:ltr;unicode-bidi:embed;margin-top:0in;margin-bottom:0in;font-size:11pt">
<li value="1" style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">Containers became
the standard way to ship and deploy applications (btw, a good thing
integrity-wise). Container image usually has the minimum amount of
software required to run the app: no additional tools, restricted
environment. Now consider that I want to monitor the application. Even if I'm
allowed to modify the command line, I can't simply add -agentpath, since
the agent library is not available in the container. A typical pattern for
using serviceability tools with containerized applications is to run a
sidecar container that has all required tools and capabilities. How would
you suggest attaching a tool to a running container?</span></li>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">In the last couple of years,
with the growing popularity of continuous profilers, a number of
solutions appeared for system-wide or infrastructure-wide
zero-configuration monitoring. The idea is that you install the
observability software, and it automatically discovers all supported
processes and starts monitoring/profiling them, regardless of how they
were deployed. gProfiler, Parca, Pyroscope, just to name a few examples.
The keyword here is "zero-configuration". Observability by
default is just as important nowadays as integrity by default.<br><br></span></li>
</ol>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">JEP outlines JFR as a
universal solution for profiling, claiming it is "far more efficient
than anything" in collecting stack traces. This is not true.
Async-profiler (6K stars on GitHub, 700+ forks, more than a million
downloads) can collect 1000 execution samples per second per core without
significant overhead, thanks to hardware performance counters. Scalability
of JFR sampling mechanism is inherently poor: it uses just one dedicated
thread to walk through all Java threads in a loop and stop them one by
one. JFR does not show non-Java threads in a profile, it is blind to
native frames, its notion of thread states is misleading (e.g.,
Socket.read can spend CPU time in the networking stack or just wait for
incoming data, but JFR has no clue). JFR fails to traverse valid Java
stacks and silently discards such samples, e.g., you will not see
arraycopy in a profile, although it's a common performance bottleneck. JFR
is misleading not only in CPU profiling but also in memory profiling, see
JDK-8307488. It's utopian to think that JFR can replace external profilers
sometime soon - there is no even progress on fixing smaller issues: open
bugs hang for years (JDK-8252417, JDK-8153167, JDK-8281677), some are
closed as will-not-fix (JDK-8191415). Is it fair to disallow valid usages
of profilers at runtime without providing a viable alternative?<br><br></span></li>
<li style="margin-top:0px;margin-bottom:0px;vertical-align:middle"><span style="font-size:11pt">You mentioned two goals: 1)
disallow libraries to grant themselves superpowers; 2) minimize the impact
on serviceability tools that have to be started by a human operator.
However, what this JEP actually suggests is the opposite: disabling
dynamic loading of agents does not prevent libraries from obtaining
superpowers - they can simply call System.load(). At the same time,
disabling dynamic loading of agents has a huge impact on serviceability, up
to the complete inability to use external tools at runtime. I understand
that the plan is to disallow JNI someday too (unless explicitly allowed
via a command line option) for the purpose of integrity. Following your
goals, it would be more logical to disallow JNI first, as it is an easier way
for libraries to break integrity.</span></li>
</ol>
<p style="margin:0in;font-family:Calibri;font-size:11pt"> </p>
<p style="margin:0in;font-family:Calibri;font-size:11pt">To summarize the
above, the current proposal does not seem to me elaborate enough for targeting to JDK 21. I would suggest improving it by 1) actualizing assumptions; 2) taking
mentioned use cases into account; 3) providing read-to-use alternatives; 4)
matching the plan with the goals.</p>
<p style="margin:0in;font-family:Calibri;font-size:11pt"> </p>
<p style="margin:0in;font-family:Calibri;font-size:11pt">Thank you,</p>
<p style="margin:0in;font-family:Calibri;font-size:11pt">Andrei Pangin</p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">пт, 19 мая 2023 г. в 15:44, Ron Pressler <<a href="mailto:ron.pressler@oracle.com">ron.pressler@oracle.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Because the discussion of this JEP has veered in many directions, let me summarise where we are:<br>
<br>
This JEP proposes to emit a suppressible warning when a JVM TI or Java agent is loaded into a JVM sometime after startup through the Attach mechanism.<br>
<br>
The warning helps make users aware that an agent has been injected into the JVM and identify deployments that may need adjustment in advance of any future changes to disallow agents from being dynamically loaded without the application's consent. The warning will also let us better judge the impact of such a future change.<br>
<br>
— Ron<br>
<br>
> On 8 May 2023, at 20:17, Mark Reinhold <<a href="mailto:mark.reinhold@oracle.com" target="_blank">mark.reinhold@oracle.com</a>> wrote:<br>
> <br>
> <a href="https://openjdk.org/jeps/451" rel="noreferrer" target="_blank">https://openjdk.org/jeps/451</a><br>
> <br>
> Summary: Issue warnings when agents are loaded dynamically into a<br>
> running JVM. These warnings aim to prepare users for a future release<br>
> which disallows the dynamic loading of agents by default in order to<br>
> improve integrity by default. Serviceability tools that load agents at<br>
> startup will not cause warnings to be issued in any release.<br>
> <br>
> - Mark<br>
<br>
</blockquote></div>