Call for Discussion: New Project: Hamburg

Tue Jun 23 15:02:46 UTC 2020

Hello OpenJDK developers.

[I apologise, this is a long email, but I thought sharing more details
would be useful for the discussion.]

I would like to start a discussion for a new Project, Project Hamburg
[1], whose primary goal is to address the issue of exposing and
organising JFR recordings within a container setup, where direct
access to the target JVM is not possible.

The recent discussion about the tentative backport of the Streaming
API for JFR to 11u and 8u has highlighted the desire to have a
standardised way to easily obtain the JFR recordings not only for
(quasi) real time monitoring but also as a mean of accessing event
metrics from OpenJDKs within remote installations, like those in
servers deployments and specifically from within containers - for
example Kubernetes/OpenShift-managed cloud.

In the last few months, some of us have been working on a project that
aims to make it easier to use JFR and JDK Mission Control in those
contexts, and I would like to start a discussion about potentially
promoting our code as an official OpenJDK Project, where it may be
useful for a larger community of developers.

At the current state, Project Hamburg lives in a github repository
under the name “Container-JFR”, and has several components designed to
work together at various [optional] levels of integration, the
proposed Project will maintain overall the same layout.

[Container JFR - https://github.com/rh-jmc-team/container-jfr]

A containerized web service that communicates with JVMs in the same
Kubernetes namespace. This communication involves invoking JFR
operations over remote JMX, which is made available by the VMs'
corresponding Kubernetes Services, and does not need to be exposed to
the external world. Instead, clients can interact with Container JFR
via a WebSocket command-based interface. These commands include
operations like starting and stopping recordings with given options,
listing in-memory recordings, listing event types, and saving a
recording to persistent storage, etc...

This web service also serves raw JFR files for download and analysis
with JMC, and generates HTML reports identical to JMC's automated
analysis.

[Container JFR Core - https://github.com/rh-jmc-team/container-jfr-core]

This project is a dependency of Container JFR. It contains interfaces
and utilities for calling JMC code to establish remote JMX connections
and communicate with JFR. The JMC dependencies are abstracted so we
can replace this module without requiring access to the JMC API
(although since JMC is also an OpenJDK Project we think this is not a
problem).

[Container JFR Web - https://github.com/rh-jmc-team/container-jfr-web]

A front-end to graphically communicate with Container JFR via web
browser. It allows the user to connect to a JVM from a drop down list
of available JVMs. Once connected, Container JFR Web displays a table
of available event types and their options. Users can then use a
wizard to create a recording, and see a list of recordings present in
that JVM. For each recording, the user can see the automated analysis
results provided by JMC code, save the recording to persistent
in-cluster storage, download the recording to local disk, or view
compatible metrics in a Grafana dashboard. This mimics some of the
functionality of JMC to provide an overview of the target JVM and
offers the option to download the JFR file further analysis in JMC.

[JFR Datasource - https://github.com/rh-jmc-team/jfr-datasource]

This project parses a provided JFR file and serves it as a Grafana
SimpleJson datasource in order to populate a Grafana dashboard with
metrics from the JFR file.

[Container JFR Operator - https://github.com/rh-jmc-team/container-jfr-operator]

A Kubernetes operator that deploys and manages all of the above
projects with a Kubernetes/OpenShift cluster. At this stage, this
project is built using Operator SDK.

In addition to the standard deployment of all Project Hamburg
components along with Grafana, the operator also supports a "minimal"
option. This deploys only Container JFR, and no web front-end or
Grafana components.

The operator also provides a Kubernetes API for JFR using Custom
Resource Definitions. This API consists of FlightRecorder objects
created by the operator for each compatible Kubernetes Service in its
namespace. These objects contain information like JFR event types for
the target JVM. A consumer of the API can create Recording objects for
a FlightRecorder object and view the status of them via kubectl/oc.

The operator reconciles these objects with Container JFR to retrieve
information and manage JFR recordings.

Furthermore, we plan to incorporate over time API to deploy and
control the JMC Agent [2], to allow existing applications to be
instrumented without the need to specifically write Events in the
application code.

Not all the components need to be used together, for example one can
easily change the Web part for direct integration into their own
framework management consoles or skip the Operator completely, but my
hope is that this Project will help in two key aspects: provide a
stable and standardised API to access Flight Recordings without
exposing the underlying JMX connections and offer a cross platform
functionality similar to that of the JFR Streaming that works cross
platform also for older releases.

The Project will live as a separate entity from OpenJDK, and we don't
expect changes to the OpenJDK code base to be necessary, although the
experience gathered may of course lead to fixes, additional APIs and
JFR Events to support a container aware OpenJDK over time; if this
happens to be the case, such changes will be discussed as usual on the
proper development mailing list and, when necessary, be addressed via
JEPs.

I propose to lead this Project with me and Elliott Baron as the
initial reviewers.

Any comments and feedback is welcomed!

Cheers,
Mario

[1] The project is formerly known as container-jfr, and is of course
meant to bridge the worlds of containers with the world of JFR.
Hamburg is a bridge city, it’s probably the city with the highest
number of bridges in the World - or certainly in Europe. It’s also one
of the homes for the largest container ship in the World, the HMM
Algeciras, so I thought this name would be perfectly fitting!

[2] https://wiki.openjdk.java.net/display/jmc/The+JMC+Agent

-- 
Mario Torre
Associate Manager, Software Engineering
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898