In memory or streaming JFR event implementation?

[Derek Thomson]

Hi Erik,

I think the idea of “flush points” is a useful one. For example, we’re only
interested in GC events at the moment, and right now it’s hard to know if
we’ve “seen” all events for a given gcId. I’ve often thought it would help
if there were events we could consider definitive “start” and “end” points
of a collection with a given GC id, too,.

I don’t think a file based solution would be right for us - we try to avoid
disk access if possible. We’d have to do extensive testing, but my feeling
is it’d be infeasible for us, and I’d be very surprised to experimentally
discover otherwise, honestly.

I’m not sure how to overcome the problems of an in-memory solution,
especially regarding flow control. We simply have a rolling buffer of “last
N events” in memory, which existed back when we used our own custom GC
event types. This has its own set of issues obviously, but for our specific
needs the trade-offs are okay here. It is reaching breaking point, and I
was already thinking of heading towards some more sophisticated
callback-like system. Which of course has its own set of problems e.g. you
don’t want slow callbacks to block writes.

Our API on the Java size is very simple. We can just read from this buffer,
in Java, as a set of Java objects of classes generated from the JFR XML
spec.

I think others (eg. Twitter) also have some in-memory JFR event mechanism,
I’ll see if I can get background from them or better yet have them chime in
here.

Thanks,
Derek



On Wed, Aug 1, 2018 at 7:06 PM Erik Gahlin <erik.gahlin at oracle.com> wrote:

> Hi Derek,
>
> We have been thinking of adding streaming capabilities to JFR for some
> time.
>
> In JDK 9, we added something to the file format we call a flush point.
> When a parser reaches a flush point it can be sure that all relevant
> constants, i.e stack traces, classes, methods etc., has been written down
> so they can be resolved in the events. We have however not implemented the
> flushing mechanism, but it should not be hard to do.
>
> We are thinking of adding a Java API for reading the data in the disk
> repository before a chunk is finished. It could perhaps  look something
> like this:
>
> new EventPublisher()
>        .subscribe(event -> System.out.println(event));
>
> new EventPublisher(“jdk.GarbageCollection”)
>        .maxAge(Duration.ofSeconds(200))
>        .subscribe(event -> System.out.print(event));
>
> new EventPublisher()
>         .flushInterval(Duration.ofSeconds(2)
>         .subscribe(event -> System.out.print(event));
>
> The layout of an event looks like this:
>
> struct Event {
>   int eventSize;
>   long eventTypeId;
>   long startTime;
>   long duration;
>   long threadId;
>   <user defined fields>
> };
>
> If we know a user are only interested in certain events, we can just read
> the eventTypeId and skip parsing the rest of the data.
>
> Our thinking is to make the mechanism disk-based. It makes it easier to
> handle flow control and we believe the overhead is negligible compared to
> an in-memory solution. If this turns out to be incorrect, we could always
> add in-memory support later.
>
> In your implementation, how do you consume data today? Is there a Java
> API, or do you use some other mechanism?
>
> Thanks
> Erik
>
>
> > On 1 Aug 2018, at 15:35, Derek Thomson <dthomson at google.com> wrote:
> >
> > Hi all,
> >
> > After talking with some people with Oracle at JVMLS I was pointed at this
> > list.
> >
> > We (at Google) basically have our own internal implementation of JFR
> event
> > in-memory writing (on the JDK side) and reading (in our own Java code).
> We
> > use the events in interesting ways, aggregated across large-scale jobs
> and
> > services that have many, many processes.
> >
> > It's a pretty simplistic implementation to be honest - we had this for
> our
> > own definitions of "GC events" for years (which weren't a great fit for
> CMS
> > let alone G1) and I just decided to move to the JFR definitions (as
> > described in the XML), rather than invent all our own event types for G1.
> >
> > Now that this is open source we'd like to move to this standard
> > implementation, and contribute where we can of course. One barrier for us
> > will be that currently it's file only, and I'd like to start talking
> about
> > the possibility of an in-memory or streaming API. Any thoughts on that so
> > far?
> >
> > Thanks,
> > Derek.
>
>