Proposal: should we add a 'clip' command to the jfr tool?

[Mat Carter]

Hi Erik

I agree that adding clip to scrub would make the commend cumbersome even though it would be performant.

Answering your questions:

Is the purpose to reduce the size of the recording to be able to open it in JMC? 

Answer: yes this is one of the benefits

Divide the recording into smaller pieces, for example, at most 5 minutes each so they are more convenient to work with? 

Answer: again yes, while I considered a 'slice' command here (slice into 10 pieces or slice into 6 minute clips, this can easily be achieved with some scripting once the clip functionality exists [and is in fact what I've done locally]); also the naming of the slices likely wouldn't satisfy everyone and so to keep it simple lets not try this

Pick out a certain time interval, for example where you have an indication the application/JVM didn't respond? 

Answer: yes, here I used three parameters to the clip command, they can be used individually and also in combinations of two; start, end and length (these where specified in s(econds), m(inutes). h(ours) and can have negative values in order to 'clip to last 5 minutes' etc).  Might make sense to also support datetime whilst more cumbersome to specify manully, can help automated clipping of JFR files based on an event time

Is there a need to use this in combination with options to limit the size, similar to exist today with jcmd <pid> JFR.dump

Answer: I feel that if you already have a JFR file then making it smaller should not present any issues; also asking for a clip of length N and then getting a clip of length N-m makes it hard to programmatically 'slice' a clip, i..e you'd have to account for the events in the new clip to understand where to make the next slice to avoid losing data

One artifact of the implementation to support 'scrub' is that the duration of the JFR file doesn't change, at first I thought this would be an issue but i actually like it esp. when visualizing in JMC (you can see where the clip was in the original recording).  

Hopefully that answers your questions so far

Thanks again
Mat

Sent from Outlook






From: Erik Gahlin <erik.gahlin at oracle.com>
Sent: Monday, July 25, 2022 3:29 AM
To: Mat Carter <Matthew.Carter at microsoft.com>; Erik Gahlin <egahlin at openjdk.java.net>; hotspot-jfr-dev at openjdk.java.net <hotspot-jfr-dev at openjdk.java.net>
Subject: Re: Proposal: should we add a 'clip' command to the jfr tool? 
 

Hi Mat,

I think there is a use case for removing events with respect to time. A common use case for the 'disassemble' command today is to reduce the size of the recording, but it's unintuitive and cumbersome to use as it is based on chunks.

I think the capability can be included in the existing 'scrub' command, but I can also see the use case for a separate command if 'scrub' gets too many options so it's hard for users to discover and understand how to use it.

Time-based filtering has been mentioned before:
https://mail.openjdk.org/pipermail/hotspot-jfr-dev/2022-February/003584.html

Before discussing the implementation, it would be interesting to hear how you want to use it.

Is the purpose to reduce the size of the recording to be able to open it in JMC? Divide the recording into smaller pieces, for example, at most 5 minutes each so they are more convenient to work with? Pick out a certain time interval, for example where you have an indication the application/JVM didn't respond? Is there a need to use this in combination with options to limit the size, similar to exist today with jcmd <pid> JFR.dump

Thanks
Erik


From: Mat Carter <Matthew.Carter at microsoft.com>
Sent: Tuesday, July 12, 2022 10:12 PM
To: Erik Gahlin <egahlin at openjdk.java.net>; hotspot-jfr-dev at openjdk.java.net <hotspot-jfr-dev at openjdk.java.net>
Subject: Proposal: should we add a 'clip' command to the jfr tool? 
 

I'd like to propose that we add a command that filters based on time (clip perhaps)

While working with large JFR files from jdk8 and jdk11 jvms I stumbled upon the 'jfr scrub' command that was introduced in 19.
I used the 'jfr summary' to get a list of events and programmatically split the JFR into a JFR for each eventtype, but the data can still be unwieldy especially when converted to JSON.

Given that 'jfr scrub' introduced the write(newJfrFile, predicate) method I wrote a java app to clip the JFR file based on a period of time (i hadn't seen the example in the JBS issue[1]), so now I can clip a JFR recording both horizontally (by events) and vertically (by time) and generate slices that are easier for JMC and other tools to parse (JMC would crash alot with large data)

I took a stab at writing 'jfr clip' locally (based on scrub) and can share that in a new JBS issue if it's felt by this group that its a worthy addition to the jfr tool

Again, I know that developers can write this themselves, but I feel it would be a nice convenience to have this in the jfr tool

Alternatively, we could extend 'jfr scrub' to handle both event filters and a time region

Cheers
Mat



[1] https://bugs.openjdk.org/browse/JDK-8271585 
[JDK-8271585] JFR: Scrub recording data - Java Bug System
JDK; JDK-8271585; JFR: Scrub recording data. Log In. Export
bugs.openjdk.org


Sent from Outlook

From: hotspot-jfr-dev <hotspot-jfr-dev-retn at openjdk.java.net> on behalf of Erik Gahlin <egahlin at openjdk.java.net>
Sent: Tuesday, May 17, 2022 9:47 PM
To: hotspot-jfr-dev at openjdk.java.net <hotspot-jfr-dev at openjdk.java.net>
Subject: Integrated: 8286706: JFR: 'jfr scrub' should overwrite output 
 
On Mon, 16 May 2022 14:04:43 GMT, Erik Gahlin <egahlin at openjdk.org> wrote:

> Could I have a review of a change that makes the 'jfr scrub' overwrite the output file if it already exists.
> 
> Testing: /test/jdk/jdk/jfr/tool  (Mac, Linux, Windows)
> 
> Thanks
> Erik

This pull request has now been integrated.

Changeset: ab144190
Author:    Erik Gahlin <egahlin at openjdk.org>
URL:       https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.openjdk.java.net%2Fjdk%2Fcommit%2Fab144190c9951f2a9a3acf30db4b570484d5f751&data=05%7C01%7Cmatthew.carter%40microsoft.com%7C6f907133f0514919728708da3889a4f5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637884461083849921%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zzPR1OA6evAeRWbmVBqXoOKgKb8wef7SVHZlxGF9xc4%3D&reserved=0
Stats:     35 lines in 2 files changed: 34 ins; 0 del; 1 mod

8286706: JFR: 'jfr scrub' should overwrite output

Reviewed-by: mgronlun

-------------

PR: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.openjdk.java.net%2Fjdk%2Fpull%2F8728&data=05%7C01%7Cmatthew.carter%40microsoft.com%7C6f907133f0514919728708da3889a4f5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637884461083849921%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QZ1ptTTj5JDIcqDPmdmK4BWSqq3JLKHMzBhT%2BbrM%2FDs%3D&reserved=0