From denghui.ddh at alibaba-inc.com Mon Mar 1 07:41:56 2021 From: denghui.ddh at alibaba-inc.com (Denghui Dong) Date: Mon, 01 Mar 2021 15:41:56 +0800 Subject: =?UTF-8?B?UmU6IFJGUjogODI2MDU4OTogQ3Jhc2ggaW4gSmZyVHJhY2VJZExvYWRCYXJyaWVyOjpsb2Fk?= =?UTF-8?B?KF9qY2xhc3MqKSBbdjJd?= In-Reply-To: References: , Message-ID: <86e62852-2836-4ce4-8234-e1c9efdb0921.denghui.ddh@alibaba-inc.com> Hi Markus, This patch is also worked in JDK 11 but needs some adjustments. I think a proper review should be done for the backport after this PR is integrated. Best Regards, Denghui ------------------------------------------------------------------ From:Markus Gr?nlund Send Time:2021?2?26?(???) 23:31 To:hotspot-jfr-dev Subject:Re: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v2] > Hi Denghui, > > I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. > > Can you do a test run with this suggestion to see if it works for you? > > Thanks > Markus Markus Gr?nlund has updated the pull request incrementally with two additional commits since the last revision: - cds hook - skip return value ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2738/files - new: https://git.openjdk.java.net/jdk/pull/2738/files/2d08073d..ce1ed381 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2738&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2738&range=00-01 Stats: 6 lines in 2 files changed: 4 ins; 1 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/2738.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2738/head:pull/2738 PR: https://git.openjdk.java.net/jdk/pull/2738 From ddong at openjdk.java.net Mon Mar 1 07:55:41 2021 From: ddong at openjdk.java.net (Denghui Dong) Date: Mon, 1 Mar 2021 07:55:41 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v2] In-Reply-To: References: Message-ID: On Fri, 26 Feb 2021 15:31:06 GMT, Markus Gr?nlund wrote: >> Hi Denghui, >> >> I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. >> >> Can you do a test run with this suggestion to see if it works for you? >> >> Thanks >> Markus > > Markus Gr?nlund has updated the pull request incrementally with two additional commits since the last revision: > > - cds hook > - skip return value src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.cpp line 99: > 97: assert(mirror != NULL, "invariant"); > 98: const Klass* const k = java_lang_Class::as_Klass(mirror); > 99: return k != NULL ? load(k) : load_primitive(mirror); I think the same modification should be made in JfrTraceId::load_raw. ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From ddong at openjdk.java.net Mon Mar 1 07:59:39 2021 From: ddong at openjdk.java.net (Denghui Dong) Date: Mon, 1 Mar 2021 07:59:39 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) In-Reply-To: References: <7Jps9Sq1KlTAc3TUYWGzD23tkuzO_DR0Jah1JMHE_JI=.20407a00-1795-46c8-a7e3-c3225636ccd6@github.com> Message-ID: On Sun, 7 Feb 2021 01:27:31 GMT, Denghui Dong wrote: >> JVM.getClassId() doesn't seem to be used, so I didn't modify the implementation of the corresponding intrinsics. > > Gentle ping. another PR(https://github.com/openjdk/jdk/pull/2738) is created for the same issue. ------------- PR: https://git.openjdk.java.net/jdk/pull/2295 From ddong at openjdk.java.net Mon Mar 1 07:59:39 2021 From: ddong at openjdk.java.net (Denghui Dong) Date: Mon, 1 Mar 2021 07:59:39 GMT Subject: Withdrawn: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) In-Reply-To: <7Jps9Sq1KlTAc3TUYWGzD23tkuzO_DR0Jah1JMHE_JI=.20407a00-1795-46c8-a7e3-c3225636ccd6@github.com> References: <7Jps9Sq1KlTAc3TUYWGzD23tkuzO_DR0Jah1JMHE_JI=.20407a00-1795-46c8-a7e3-c3225636ccd6@github.com> Message-ID: On Thu, 28 Jan 2021 16:55:41 GMT, Denghui Dong wrote: > hi, > > Could I have a review of this fix? > > If we run the following code snippet, will get a crash > class JFRDemo { > public static void main(String[] args) { > Recording r = new Recording(); > r.enable("MyEvent"); > r.start(); > > MyEvent event = new MyEvent(); > event.begin(); > event.clazz = long.class; > event.commit(); > } > } > > class MyEvent extends Event { > > @Label("Class") > public Class clazz; > } This pull request has been closed without being integrated. ------------- PR: https://git.openjdk.java.net/jdk/pull/2295 From jbachorik at openjdk.java.net Mon Mar 1 12:56:41 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 12:56:41 GMT Subject: RFR: 8258414: OldObjectSample events too expensive In-Reply-To: <1_LsNBt-Yy5NlHbfwtRSRNvGa2AbTuhMGYuiw3Hy8gU=.3b79e283-87fe-451e-8e60-25b59c5e837a@github.com> References: <1_LsNBt-Yy5NlHbfwtRSRNvGa2AbTuhMGYuiw3Hy8gU=.3b79e283-87fe-451e-8e60-25b59c5e837a@github.com> Message-ID: On Fri, 19 Feb 2021 14:45:00 GMT, Florian David wrote: > The purpose of this change is to reduce the size of JFR recordings when the OldObjectSample event is enabled. > > ## Problem > JFR recordings size blows up when the OldObjectSample is enabled. The memory allocation events are known to be very high traffic and will cause a lot of data, just the sheer number of events produced, and if stacktraces are added to this, the associated metadata can be huge as well. Sampled object are stored in a priority queue and their associated stack traces stored in JFRStackTraceRepository. When sample candidates are removed from the priority queue, their stacktraces remain in the repository, which will be later written at chunk rotation even if the sample has been removed. > > ## Implementation > This PR adds a JFRStackTraceRepository dedicated to store stack traces for the OldObjectSample event. At chunk rotation, every sample stack trace is looked up in this repository and is serialized. Other stack traces are simply removed. > > ## Benchmarks > On an AWS c5.metal instance (96 cores, 192 Gib), running SPECjvm2008 with default profile.jfc configuration with OldObjectSample event enabled gives: > - a recording size of 20.73Mb without the PR fix > - a recording size of 2.78Mb with the PR fix Ok, kicking off the review. The implementation is doing the right thing (AFAICT) and I have no strong objections, just minor things regarding slightly more detailed comments. src/hotspot/share/jfr/recorder/jfrRecorder.cpp line 284: > 282: return false; > 283: } > 284: if (!create_leak_profiler_stacktrace_repository()) { Can you add a comment here explaining the purpose of having a separate leak profiler stacktrace repository? src/hotspot/share/jfr/recorder/stacktrace/jfrStackTraceRepository.cpp line 27: > 25: #include "precompiled.hpp" > 26: #include "jfr/leakprofiler/sampling/objectSample.hpp" > 27: #include "jfr/leakprofiler/sampling/objectSampler.hpp" Are those 2 extra includes needed? src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleCheckpoint.cpp line 200: > 198: } > 199: > 200: class StackTraceChunkWriter { Perhaps, a detailed comment explaining how this is working with the regular stacktrace writer would be good to have here. ------------- PR: https://git.openjdk.java.net/jdk/pull/2645 From jbachorik at openjdk.java.net Mon Mar 1 13:15:49 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 13:15:49 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:18:03 GMT, Aleksey Shipilev wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/gc/shared/genCollectedHeap.cpp line 1144: > >> 1142: _old_gen->prepare_for_compaction(&cp); >> 1143: _young_gen->prepare_for_compaction(&cp); >> 1144: > > Stray newline? ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 13:15:48 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 13:15:48 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <30TKeg0tEzRXVOG5fcZp_Zsb9dTxU__Cn1VnpUUZJY4=.d7efe1f7-ebec-4b22-a5ac-9ba4b3d732eb@github.com> On Mon, 22 Feb 2021 16:50:48 GMT, Thomas Schatzl wrote: >> src/hotspot/share/gc/g1/g1CollectedHeap.cpp line 4578: >> >>> 4576: >>> 4577: void G1CollectedHeap::set_live(size_t bytes) { >>> 4578: Atomic::release_store(&_live_size, bytes); >> >> I don't think this requires `release_store`, regular `store` would be enough. G1 folks can say for sure. > > Not required. ?? >> src/hotspot/share/gc/shared/genCollectedHeap.hpp line 183: >> >>> 181: size_t live = _live_size; >>> 182: return live > 0 ? live : used(); >>> 183: }; >> >> I think the implementation belongs to `genCollectedHeap.cpp`. > > +1. Does not seem to be performance sensitive. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From mgronlun at openjdk.java.net Mon Mar 1 14:04:54 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Mon, 1 Mar 2021 14:04:54 GMT Subject: RFR: 8258414: OldObjectSample events too expensive Message-ID: Hi Florian, thanks for your great attempt to fix this issue, it is indeed a complex area with many interrelations. I think your suggestion has two problems that we need to solve: 1) If we separate the Leak Profiler stacktraces from the "regular" traces, there is a disconnect at the stacktrace record site, because the stacktrace id is cached to the thread. This means other events (ObjectAllocationSample and other memory allocation events) will attempt to reuse the id. Since this id is no longer valid for reuse, it must be handled. 2) The division into two separate stacktrace repositories now also separates the stacktrace id assignment, with a consequence that separate stacktraces can now end up having overlapping and conflicting ids. I also appreciated your effort in trying to incorporate the count of the traces written, to be tallied in the "regular" process. I do think we can simplify this a bit by not having them tallied (they are only a very small number, max 256 by default). Please take a look what you think about this suggestion. Thanks! Markus PS If we go with this suggestion, I will put you down as a contributor. ------------- Commit messages: - 8258414 Changes: https://git.openjdk.java.net/jdk/pull/2780/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=2780&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8258414 Stats: 122 lines in 7 files changed: 59 ins; 18 del; 45 mod Patch: https://git.openjdk.java.net/jdk/pull/2780.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2780/head:pull/2780 PR: https://git.openjdk.java.net/jdk/pull/2780 From jbachorik at openjdk.java.net Mon Mar 1 14:06:49 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 14:06:49 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:19:31 GMT, Aleksey Shipilev wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/gc/shared/generation.hpp line 140: > >> 138: virtual size_t used() const = 0; // The number of used bytes in the gen. >> 139: virtual size_t free() const = 0; // The number of free bytes in the gen. >> 140: virtual size_t live() const = 0; > > Needs a comment to match the lines above? Say, `// The estimate of live bytes in the gen.` ?? > src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 579: > >> 577: event.set_heapLive(heap->live()); >> 578: event.commit(); >> 579: } > > On the first sight, this belongs in `ShenandoahConcurrentMark::finish_mark()`. Placing the event here would fire the event when concurrent GC is cancelled, which is not what you want. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 14:21:42 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 14:21:42 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:22:58 GMT, Aleksey Shipilev wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.cpp line 265: > >> 263: ShenandoahHeap* const heap = ShenandoahHeap::heap(); >> 264: heap->set_concurrent_mark_in_progress(false); >> 265: heap->mark_finished(); > > Let's not rename this method. Introduce a new method, `ShenandoahHeap::update_live`, and call it every time after `mark_complete_marking_context()` is called. ?? > src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp line 494: > >> 492: mark_complete_marking_context(); >> 493: >> 494: class ShenandoahCollectLiveSizeClosure : public ShenandoahHeapRegionClosure { > > We don't usually use the in-method declarations like these, pull it out of the method. ?? > src/hotspot/share/gc/epsilon/epsilonHeap.hpp line 80: > >> 78: virtual size_t capacity() const { return _virtual_space.committed_size(); } >> 79: virtual size_t used() const { return _space->used(); } >> 80: virtual size_t live() const { return used(); } > > I'd prefer to call `_space->used()` directly here. Minor optimization, I know. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From shade at openjdk.java.net Mon Mar 1 14:21:40 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Mon, 1 Mar 2021 14:21:40 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 17:20:49 GMT, Thomas Schatzl wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > The change also misses liveness update after G1 Full GC: it should at least reset the internal liveness counter to 0 so that `used()` is used. > I think there is the same issue for Parallel Full GC. Serial seems to be handled. Another general comment about Shenandoah. It would seem easier to piggyback liveness summarization on region iteration that heuristics does at the end of mark anyway. See `ShenandoahHeuristics::choose_collection_set`. I can do that when you are done with your changes, or try it yourself. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 14:21:43 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 14:21:43 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 08:44:25 GMT, Per Liden wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/gc/z/zStat.hpp line 549: > >> 547: static size_t used_at_mark_start(); >> 548: static size_t used_at_relocate_end(); >> 549: static size_t live(); > > Please call this `live_at_mark_end()` to match the names of the neighboring functions. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 14:21:43 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 14:21:43 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Fri, 19 Feb 2021 08:21:36 GMT, Albert Mingkun Yang wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/jfr/periodic/jfrPeriodic.cpp line 649: > >> 647: TRACE_REQUEST_FUNC(HeapUsageSummary) { >> 648: EventHeapUsageSummary event; >> 649: if (event.should_commit()) { > > I believe the `should_commit` check is not needed; the period check is handle by the caller. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 14:27:41 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 14:27:41 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <6L-2nr_YRy2MikNW7fhxblbLswAZD3L05fliGk36WTM=.822e958d-3cc1-43cd-88fb-d4e6bbbed012@github.com> On Mon, 22 Feb 2021 17:12:43 GMT, Thomas Schatzl wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > src/hotspot/share/gc/shared/genCollectedHeap.cpp line 683: > >> 681: } >> 682: // update the live size after last GC >> 683: _live_size = _young_gen->live() + _old_gen->live(); > > I would prefer if that code were placed into `gc_epilogue`. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:11:41 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:11:41 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 1 Mar 2021 14:03:37 GMT, Jaroslav Bachorik wrote: >> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 579: >> >>> 577: event.set_heapLive(heap->live()); >>> 578: event.commit(); >>> 579: } >> >> On the first sight, this belongs in `ShenandoahConcurrentMark::finish_mark()`. Placing the event here would fire the event when concurrent GC is cancelled, which is not what you want. > > ?? Actually, this shouldn't even be here. `EventGCHeapSummary` is emitted via `trace_heap*` calls which should already be hooked into Shenandoah. Let me remove this. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:27:24 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:27:24 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> On Mon, 1 Mar 2021 14:17:17 GMT, Aleksey Shipilev wrote: >> The change also misses liveness update after G1 Full GC: it should at least reset the internal liveness counter to 0 so that `used()` is used. >> I think there is the same issue for Parallel Full GC. Serial seems to be handled. > > Another general comment about Shenandoah. It would seem easier to piggyback liveness summarization on region iteration that heuristics does at the end of mark anyway. See `ShenandoahHeuristics::choose_collection_set`. I can do that when you are done with your changes, or try it yourself. I have addressed comments with trivial fixes. Will take a look at the remainder of more complex ones next. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:27:22 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:27:22 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision: - Merge remote-tracking branch 'origin/master' into jb/live_set_1 - Change dead space calculation - Common PR fixes - Minor G1 related PR fixes - Epsilon related PR fixes - Shenandoah related PR fixes - Rename ZStatHeap::live() to live_at_mark_end() - Update event definition and emission - 8258431: Provide a JFR event with live set size estimate ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/ddc5b5c1..03a8617e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=00-01 Stats: 45701 lines in 1355 files changed: 27365 ins; 10881 del; 7455 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:27:26 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:27:26 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:27:21 GMT, Aleksey Shipilev wrote: >> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 627: >> >>> 625: >>> 626: size_t ShenandoahHeap::live() const { >>> 627: size_t live = Atomic::load_acquire(&_live); >> >> I understand you copy-pasted from the same file. We have removed `_acquire` with #2504. Do `Atomic::load` here. > > ...which also means you want to merge from master to get recent changes? Yep. Done. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:27:32 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:27:32 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:23:53 GMT, Aleksey Shipilev wrote: >> Jaroslav Bachorik has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision: >> >> - Merge remote-tracking branch 'origin/master' into jb/live_set_1 >> - Change dead space calculation >> - Common PR fixes >> - Minor G1 related PR fixes >> - Epsilon related PR fixes >> - Shenandoah related PR fixes >> - Rename ZStatHeap::live() to live_at_mark_end() >> - Update event definition and emission >> - 8258431: Provide a JFR event with live set size estimate > > src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 655: > >> 653: >> 654: void ShenandoahHeap::set_live(size_t bytes) { >> 655: Atomic::release_store_fence(&_live, bytes); > > Same, do `Atomic::store` here. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:27:38 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:27:38 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 17:16:46 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision: >> >> - Merge remote-tracking branch 'origin/master' into jb/live_set_1 >> - Change dead space calculation >> - Common PR fixes >> - Minor G1 related PR fixes >> - Epsilon related PR fixes >> - Shenandoah related PR fixes >> - Rename ZStatHeap::live() to live_at_mark_end() >> - Update event definition and emission >> - 8258431: Provide a JFR event with live set size estimate > > src/hotspot/share/gc/shared/space.inline.hpp line 189: > >> 187: oop obj = oop(cur_obj); >> 188: size_t obj_size = obj->size(); >> 189: live_offset += obj_size; > > It seems more natural to me to put this counting into the `DeadSpacer` as this is what this change does. Also, the actual dead space "used" can be calculated from the difference between the `_allowed_deadspace_words` and the maximum (calculated in the constructor of `DeadSpacer`) afaict at the end of evacuation. So there is no need to incur per-object costs during evacuation at all. Something like https://github.com/openjdk/jdk/pull/2579/commits/b28e7a0959a4655173bf1599bd035ab668196af6 would be ok? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:30:02 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:30:02 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v2] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 19:37:37 GMT, Erik Gahlin wrote: >> Jaroslav Bachorik has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision: >> >> - Merge remote-tracking branch 'origin/master' into jb/live_set_1 >> - Change dead space calculation >> - Common PR fixes >> - Minor G1 related PR fixes >> - Epsilon related PR fixes >> - Shenandoah related PR fixes >> - Rename ZStatHeap::live() to live_at_mark_end() >> - Update event definition and emission >> - 8258431: Provide a JFR event with live set size estimate > > src/hotspot/share/jfr/metadata/metadata.xml line 205: > >> 203: >> 204: >> 205: > > I think it would be good to mention in the description that it is an estimate, i.e. "Estimate of live bytes ....". ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:37:05 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:37:05 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v3] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Do not track young, eden and old live size separately ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/03a8617e..01c22ce6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=01-02 Stats: 13 lines in 3 files changed: 1 ins; 7 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:37:06 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:37:06 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v3] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 17:08:08 GMT, Thomas Schatzl wrote: >> src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp line 79: >> >>> 77: size_t _young_live; >>> 78: size_t _eden_live; >>> 79: size_t _old_live; >> >> It's only the sum that's ever exposed, right? I wonder if it makes sense to merge them into one var to only track the sum. > > I agree because they seem to be always read and written at the same time. The original idea was that the sum might be computed from different areas depending on the GC phase. But, apparently, that's not the case so we can have just one common sum. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 15:41:11 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 15:41:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v4] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Fix dangling space ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/01c22ce6..6a1aa73e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=02-03 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 16:34:04 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 16:34:04 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v5] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Fix syntax error ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/6a1aa73e..dd204d8c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=03-04 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 17:37:11 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 17:37:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v6] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Attempt to fix G1 live set size computation ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/dd204d8c..08c715ab Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=04-05 Stats: 19 lines in 1 file changed: 7 ins; 10 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 1 17:37:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 1 Mar 2021 17:37:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v6] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 17:00:19 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Attempt to fix G1 live set size computation > > src/hotspot/share/gc/g1/g1ConcurrentMark.cpp line 1114: > >> 1112: >> 1113: _g1h->set_live(live_size * HeapWordSize); >> 1114: > > This code is located in the wrong place. It will return only the live words for the areas that have been marked, not eden or objects allocated in old gen after the marking started. > > Further it iterates over all regions, which can be large compared to actually active regions. > > A better place is in `G1UpdateRemSetTrackingBeforeRebuild::do_heap_region()` after the last method call - at that point, `HeapRegion::live_bytes()` contains the per-region number of live data for all regions. > > `G1UpdateRemSetTrackingBeforeRebuild` is instantiated and then called by multiple threads. It's probably best that that `HeapClosure` locally sums up the live byte estimates and then in the caller `G1UpdateRemSetTrackingBeforeRebuildTask::work()` sums up the per thread results like is done for `G1UpdateRemSetTrackingBeforeRebuildTask::_total_selected_for_rebuild`, which is then set in the caller of the `G1UpdateRemSetTrackingBeforeRebuildTask`. Would something along the line of https://github.com/openjdk/jdk/pull/2579/commits/08c715abccddbd04ced58706b7a705670843b43a be ok? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From egahlin at openjdk.java.net Tue Mar 2 04:36:49 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Tue, 2 Mar 2021 04:36:49 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v12] In-Reply-To: References: Message-ID: On Thu, 18 Feb 2021 02:18:57 GMT, Yi Yang wrote: >> Hi all, >> >> May I please have a review for this minor patch? >> >> It simplifies the use of subcommand `metadata` of jfr tool. Recording >> file is no longer mandatory, users can directly use `jfr metadata` to >> output JDK builtin meta information. As JDK-8256156 mentioned, it also >> supports events and categories filters: >> $ jfr metadata >> $ jfr metadata recording.jfr # compatible >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr >> $ jfr metadata --categories GC,Detailed >> $ jfr metadata --categories GC,Detailed recording.jfr > > Yi Yang has updated the pull request incrementally with one additional commit since the last revision: > > fix I think it is easier to grasp what the code does without the continue statement. test/jdk/jdk/jfr/tool/TestMetadata.java line 106: > 104: expectedNames.add(eventType.getName()); > 105: } > 106: Asserts.assertGTE(eventNames.size(), expectedNames.size()); Seems something like this would be sufficient: public void testNumberOfEventTypes() { int count = 0; for (String line : output.asLines()) { if (line.contains("extends jdk.jfr.Event")) { count++; } Assert.assertEquals(count, FlightRecorder.getFlightRecorder().getEventTypes().size()); } ? test/jdk/jdk/jfr/tool/TestMetadata.java line 57: > 55: } > 56: > 57: static void testBasic() throws Throwable { Perhaps change the name of the method to testUnfiltered() to make it more clear what the test does and create a separate method for the --wrongOption, i.e. testIllegalOption() test/jdk/jdk/jfr/tool/TestMetadata.java line 109: > 107: } > 108: > 109: static void testDeterministic() throws Throwable { Change name of the method to testEventFilter(). test/jdk/jdk/jfr/tool/TestMetadata.java line 115: > 113: > 114: for (String line : lines) { > 115: if (line.startsWith("@Name(\"")) { Use contains "extends jdk.jfr.Event" as it is more stable test/jdk/jdk/jfr/tool/TestMetadata.java line 116: > 114: for (String line : lines) { > 115: if (line.startsWith("@Name(\"")) { > 116: eventNames.add(line.substring(7, line.indexOf("\"", 7))); Should be easy to check if the event is correct, not just the number. if (line.contains("XXZQZYY.") { event1 = true; } if (line.contains("ZQZPP") { event2 = true; } Regarding events names, see following comment. test/jdk/jdk/jfr/tool/TestMetadata.java line 123: > 121: > 122: static void testWildcardAndAcronym() throws Throwable { > 123: OutputAnalyzer output = ExecuteHelper.jfr("metadata", "--events", "Thread*"); Instead of depending on name of existing JVM events (which may change, be removed or interfere with wildcard expansion done by the OS) it would be better to create two custom Java events with an unlikely name pattern, i.e. com.example.XXZQZYY and com.stuff.ZQZPP, and then filter on "ZQZ". To make sure they are registered, you can do: FlightRecorder.register(XXZQZYY.class); FlightRecorder.register(ZQZPP.class); in static initializer so it is executed before calling ExecuteHelper.jfr("metadata", "--events", "ZQZ*"); test/jdk/jdk/jfr/tool/TestMetadata.java line 122: > 120: } > 121: > 122: static void testWildcardAndAcronym() throws Throwable { The name of the seems misleading since it doesn't test acronyms, testWildcard() shuld be sufficient. ------------- PR: https://git.openjdk.java.net/jdk/pull/1904 From jbachorik at openjdk.java.net Tue Mar 2 09:07:55 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 09:07:55 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v6] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 22 Feb 2021 17:10:28 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Attempt to fix G1 live set size computation > > src/hotspot/share/gc/parallel/parallelScavengeHeap.inline.hpp line 49: > >> 47: _young_live = young_gen()->used_in_bytes(); >> 48: _eden_live = young_gen()->eden_space()->used_in_bytes(); >> 49: _old_live = old_gen()->used_in_bytes(); > > `_young_live` already seems to contain `_eden_live` looking at the implementation of `PSYoungGen::used_in_bytes()`: > > I.e. > > `size_t PSYoungGen::used_in_bytes() const { > return eden_space()->used_in_bytes() > + from_space()->used_in_bytes(); // to_space() is only used during scavenge > } > ` > > but maybe I'm wrong here. This seems like a correct summary - I have updated the implementation. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 09:22:55 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 09:22:55 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v6] In-Reply-To: <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> Message-ID: On Mon, 1 Mar 2021 15:24:48 GMT, Jaroslav Bachorik wrote: >> Another general comment about Shenandoah. It would seem easier to piggyback liveness summarization on region iteration that heuristics does at the end of mark anyway. See `ShenandoahHeuristics::choose_collection_set`. I can do that when you are done with your changes, or try it yourself. > > I have addressed comments with trivial fixes. > Will take a look at the remainder of more complex ones next. Going over my initial changes I am starting to doubt the usefulness of defaulting to `used` value when `live` estimate is not available. It seems to be giving false information - perhaps it would be ok to return `0` as an invalid value to indicate that that particular information is not available and it would be up to the event consumer to fall back to using the `used` value or deal with the missing value by some other means. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From shade at openjdk.java.net Tue Mar 2 09:54:40 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Tue, 2 Mar 2021 09:54:40 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v6] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> Message-ID: <850jkqk912ORnuVtTM7JhVB4gP5BlxuggFktpkmJVvU=.f076b561-4b3a-4378-83b7-99b14b75e099@github.com> On Tue, 2 Mar 2021 09:20:21 GMT, Jaroslav Bachorik wrote: >> I have addressed comments with trivial fixes. >> Will take a look at the remainder of more complex ones next. > > Going over my initial changes I am starting to doubt the usefulness of defaulting to `used` value when `live` estimate is not available. It seems to be giving false information - perhaps it would be ok to return `0` as an invalid value to indicate that that particular information is not available and it would be up to the event consumer to fall back to using the `used` value or deal with the missing value by some other means. Shenandoah parts are better like this (applies on top of your PR): https://cr.openjdk.java.net/~shade/8258431/shenandoah.patch ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 11:44:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 11:44:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v7] In-Reply-To: <850jkqk912ORnuVtTM7JhVB4gP5BlxuggFktpkmJVvU=.f076b561-4b3a-4378-83b7-99b14b75e099@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> <850jkqk912ORnuVtTM7JhVB4gP5BlxuggFktpkmJVvU=.f076b561-4b3a-4378-83b7-99b14b75e099@github.com> Message-ID: On Tue, 2 Mar 2021 09:50:20 GMT, Aleksey Shipilev wrote: >> Going over my initial changes I am starting to doubt the usefulness of defaulting to `used` value when `live` estimate is not available. It seems to be giving false information - perhaps it would be ok to return `0` as an invalid value to indicate that that particular information is not available and it would be up to the event consumer to fall back to using the `used` value or deal with the missing value by some other means. > > Shenandoah parts are better like this (applies on top of your PR): https://cr.openjdk.java.net/~shade/8258431/shenandoah.patch @shipilev Thanks! The patch has been applied. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 11:44:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 11:44:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v7] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <5FpR9OM3pJ2JvMS0p9s7vHaft1oVb9wGdtmSqL7IlcM=.a5a51fde-ce07-4b9a-a4a1-5d9df52eb9df@github.com> > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Proper Shenandoah implementation of live size esitmate Co-authored-by: Aleksey Shipilev ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/08c715ab..aa180d11 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=05-06 Stats: 45 lines in 6 files changed: 16 ins; 27 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 12:15:17 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 12:15:17 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v8] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Use '0' to indicate unvailable live estimate ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/aa180d11..6c6f8a8a Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=06-07 Stats: 21 lines in 11 files changed: 1 ins; 4 del; 16 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 14:33:14 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 14:33:14 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Add tests for the heap usage summary event ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/6c6f8a8a..f6954186 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=07-08 Stats: 24 lines in 5 files changed: 24 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 14:33:14 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 14:33:14 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8Q_OEFTu-Npp_Pr74VPWQ6DrQUzcaDtVZy7nru2RCbU=.7e80a5b9-be94-4442-beaf-9593b067241d@github.com> <850jkqk912ORnuVtTM7JhVB4gP5BlxuggFktpkmJVvU=.f076b561-4b3a-4378-83b7-99b14b75e099@github.com> Message-ID: On Tue, 2 Mar 2021 11:41:19 GMT, Jaroslav Bachorik wrote: >> Shenandoah parts are better like this (applies on top of your PR): https://cr.openjdk.java.net/~shade/8258431/shenandoah.patch > > @shipilev Thanks! The patch has been applied. @pliden @tschatzl @albertnetymk @egahlin I believe I addressed all review comments. Can you, please, take a second look? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 2 14:33:15 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 2 Mar 2021 14:33:15 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 18 Feb 2021 10:25:34 GMT, Aleksey Shipilev wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Add tests for the heap usage summary event > > src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp line 511: > >> 509: >> 510: ShenandoahCollectLiveSizeClosure cl; >> 511: heap_region_iterate(&cl); > > I think you want `parallel_heap_region_iterate` on this path, and do `Atomic::add(&_live, r->get_live_data_bytes())` in the closure. We shall see if this makes sense to make fully concurrently... Outdated by the patch provided by @shipilev ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From ayang at openjdk.java.net Tue Mar 2 16:04:40 2021 From: ayang at openjdk.java.net (Albert Mingkun Yang) Date: Tue, 2 Mar 2021 16:04:40 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 2 Mar 2021 14:33:14 GMT, Jaroslav Bachorik wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: > > Add tests for the heap usage summary event Marked as reviewed by ayang (Author). ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From shade at openjdk.java.net Tue Mar 2 17:40:44 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Tue, 2 Mar 2021 17:40:44 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 2 Mar 2021 14:33:14 GMT, Jaroslav Bachorik wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: > > Add tests for the heap usage summary event Shenandoah parts look good. I have a few minor stylistic comments. src/hotspot/share/gc/shared/space.inline.hpp line 190: > 188: oop obj = oop(cur_obj); > 189: size_t obj_size = obj->size(); > 190: compact_top = cp->space->forward(obj, obj_size, cp, compact_top); This change seems superfluous now. Inline `obj_size` back? src/hotspot/share/gc/shared/space.hpp line 555: > 553: size_t live() const { > 554: return used() - _dead_space; > 555: } Move it a few lines down, so `capacity`, `used`, `live` line up? src/hotspot/share/gc/shared/collectedHeap.hpp line 218: > 216: virtual size_t capacity() const = 0; > 217: virtual size_t used() const = 0; > 218: // Returns the estimate of live set size. Because live set changes over time, I believe a blank line is in order here, look at other comments in the same header. src/hotspot/share/gc/shared/space.inline.hpp line 90: > 88: > 89: public: > 90: size_t _dead_space; Should this really be "public"? Maybe `friend`-ing with the only user is better? ------------- Marked as reviewed by shade (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/2579 From tschatzl at openjdk.java.net Wed Mar 3 12:19:50 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Wed, 3 Mar 2021 12:19:50 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 2 Mar 2021 14:33:14 GMT, Jaroslav Bachorik wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: > > Add tests for the heap usage summary event Fwiw, the change still does not capture G1 full gc `live_estimate()`. src/hotspot/share/gc/g1/g1ConcurrentMark.cpp line 1070: > 1068: > 1069: uint num_selected_for_rebuild() const { return _num_regions_selected_for_rebuild; } > 1070: size_t live_estimate() const { return _live; } Please sync the member name with the getter name. I.e. `_live` -> `_live_estimate` src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp line 60: > 58: class PSAdaptiveSizePolicy : public AdaptiveSizePolicy { > 59: friend class PSGCAdaptivePolicyCounters; > 60: friend class ParallelScavengeHeap; Delete this apparently unneeded friend declaration (compiled successfully without here) src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp line 87: > 85: > 86: // in order to provide accurate estimate this method must be called only when the heap has just been collected and compacted > 87: inline void capture_live(); Sentences should start with upper case in the comment. Also I'd prefer to name the method `update_live_estimate()` instead. src/hotspot/share/gc/g1/g1CollectedHeap.hpp line 182: > 180: G1BlockOffsetTable* _bot; > 181: > 182: volatile size_t _live; I'm not happy with naming this `_live`, better use `_live_estimate`. The contents are not continuously updated and basically out of date after the first following allocation. This includes the naming in all other instances too. src/hotspot/share/gc/serial/serialHeap.hpp line 44: > 42: MemoryPool* _old_pool; > 43: > 44: size_t _live_size; Please rename to `_live_estimate` like the others. Avoid having different names in different collectors for the same thing. src/hotspot/share/gc/shared/space.inline.hpp line 128: > 126: p2i(dead_start), p2i(dead_end), dead_length * HeapWordSize); > 127: > 128: _dead_space += dead_length; I do not think adding this to the counter here instead of the other method for every object makes a difference performance-wise. As mentioned before, `_allowed_deadspace_words` counts *down* from `(space->capacity() * ratio / 100) / HeapWordSize;` to whatever end value. So at the end of collection, `(space->capacity() * ratio / 100) / HeapWordSize - _allowed_deadspace_words` should be equal to what `_dead_space` is now. Please add a getter to `DeadSpacer` that calculates this (factoring out the calculation of the maximum allowed deadspace). src/hotspot/share/gc/shared/space.hpp line 553: > 551: size_t capacity() const { return byte_size(bottom(), end()); } > 552: size_t used() const { return byte_size(bottom(), top()); } > 553: size_t live() const { The code for serial gc, contrary to others, tries to give some resemblance of tracking actual liveness. I.e. calculating this anew every call to `SerialHeap::live()`. However if calling an `update_live_estimate()` in parallel and G1 (and the other collectors) is fine at certain places, this should be as good for serial gc. Doing so would reduce the footprint of this change quite a bit (for serial gc) ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From github.com+5010047+kelthuzadx at openjdk.java.net Fri Mar 5 07:15:05 2021 From: github.com+5010047+kelthuzadx at openjdk.java.net (Yi Yang) Date: Fri, 5 Mar 2021 07:15:05 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v13] In-Reply-To: References: Message-ID: <86TK9HtLVoY8-3vziKYtYEzZfWn6Rm1RA4FFxp0YO6Y=.8e5b91fd-3101-4335-aac6-3f5f5e328673@github.com> > Hi all, > > May I please have a review for this minor patch? > > It simplifies the use of subcommand `metadata` of jfr tool. Recording > file is no longer mandatory, users can directly use `jfr metadata` to > output JDK builtin meta information. As JDK-8256156 mentioned, it also > supports events and categories filters: > $ jfr metadata > $ jfr metadata recording.jfr # compatible > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr > $ jfr metadata --categories GC,Detailed > $ jfr metadata --categories GC,Detailed recording.jfr Yi Yang has updated the pull request incrementally with one additional commit since the last revision: reimpl test ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/1904/files - new: https://git.openjdk.java.net/jdk/pull/1904/files/a26202fc..690f4224 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=12 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=11-12 Stats: 95 lines in 1 file changed: 39 ins; 11 del; 45 mod Patch: https://git.openjdk.java.net/jdk/pull/1904.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/1904/head:pull/1904 PR: https://git.openjdk.java.net/jdk/pull/1904 From github.com+5010047+kelthuzadx at openjdk.java.net Fri Mar 5 07:15:06 2021 From: github.com+5010047+kelthuzadx at openjdk.java.net (Yi Yang) Date: Fri, 5 Mar 2021 07:15:06 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v12] In-Reply-To: References: Message-ID: On Tue, 2 Mar 2021 04:25:45 GMT, Erik Gahlin wrote: >> Yi Yang has updated the pull request incrementally with one additional commit since the last revision: >> >> fix > > test/jdk/jdk/jfr/tool/TestMetadata.java line 123: > >> 121: >> 122: static void testWildcardAndAcronym() throws Throwable { >> 123: OutputAnalyzer output = ExecuteHelper.jfr("metadata", "--events", "Thread*"); > > Instead of depending on name of existing JVM events (which may change, be removed or interfere with wildcard expansion done by the OS) it would be better to create two custom Java events with an unlikely name pattern, i.e. com.example.XXZQZYY and com.stuff.ZQZPP, and then filter on "ZQZ". > > To make sure they are registered, you can do: > > FlightRecorder.register(XXZQZYY.class); > FlightRecorder.register(ZQZPP.class); > > in static initializer so it is executed before calling ExecuteHelper.jfr("metadata", "--events", "ZQZ*"); Make sense, I re-implement the test, using customized events. > test/jdk/jdk/jfr/tool/TestMetadata.java line 122: > >> 120: } >> 121: >> 122: static void testWildcardAndAcronym() throws Throwable { > > The name of the seems misleading since it doesn't test acronyms, testWildcard() shuld be sufficient. `MyEv*` and `Custo*` are expected to acronym filter, so I keep this name, please let me know if I did wrong. ------------- PR: https://git.openjdk.java.net/jdk/pull/1904 From github.com+5010047+kelthuzadx at openjdk.java.net Sat Mar 6 11:45:26 2021 From: github.com+5010047+kelthuzadx at openjdk.java.net (Yi Yang) Date: Sat, 6 Mar 2021 11:45:26 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v14] In-Reply-To: References: Message-ID: > Hi all, > > May I please have a review for this minor patch? > > It simplifies the use of subcommand `metadata` of jfr tool. Recording > file is no longer mandatory, users can directly use `jfr metadata` to > output JDK builtin meta information. As JDK-8256156 mentioned, it also > supports events and categories filters: > $ jfr metadata > $ jfr metadata recording.jfr # compatible > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr > $ jfr metadata --categories GC,Detailed > $ jfr metadata --categories GC,Detailed recording.jfr Yi Yang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: reimpl test ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/1904/files - new: https://git.openjdk.java.net/jdk/pull/1904/files/690f4224..ff13850c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=13 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=12-13 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/1904.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/1904/head:pull/1904 PR: https://git.openjdk.java.net/jdk/pull/1904 From github.com+741251+turbanoff at openjdk.java.net Sun Mar 7 20:31:08 2021 From: github.com+741251+turbanoff at openjdk.java.net (Andrey Turbanov) Date: Sun, 7 Mar 2021 20:31:08 GMT Subject: RFR: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy [v11] In-Reply-To: References: <6zhT8ySP8rFE4wV5vva0oKRDSxDnSYYZ-sXAQJ6F_DM=.b83a0f88-7231-4796-a0b5-d51b5950d316@github.com> Message-ID: On Fri, 19 Feb 2021 15:03:11 GMT, Sean Mullan wrote: >> As I can see only ByteArrayInputStream is actually passed in `InputStream` in current JDK code: >> >> PKCS7 pkcs7 = new PKCS7(is.readAllBytes()); >> private static List parsePKCS7(InputStream is) >> certs = parsePKCS7(is); >> public X509CertPath(InputStream is, String encoding) >> return new X509CertPath(new ByteArrayInputStream(data), encoding); >> >> PKCS7 pkcs7 = new PKCS7(is.readAllBytes()); >> private static List parsePKCS7(InputStream is) >> certs = parsePKCS7(is); >> public X509CertPath(InputStream is, String encoding) >> this(is, PKIPATH_ENCODING); >> public X509CertPath(InputStream is) throws CertificateException { >> return new X509CertPath(new ByteArrayInputStream(encoding)); >> >> ![???????????](https://user-images.githubusercontent.com/741251/108475587-f4f61080-72a1-11eb-91e0-ac2b98c7c490.png) >> >> Perhaps original marking approach was lost during refactoring? > > You are right, the code was refactored (way back in 2010) [1] to read one block at a time, so this check on mark can be removed. So, in this case, I think it is probably safe to just pass the InputStream as-is to PKCS7(InputStream), but maybe you can add a comment that says this should always be a ByteArrayInputStream. We can look at refactoring this code and clean it up a bit more later. > > [1] https://hg.openjdk.java.net/jdk/jdk/rev/337ae296b6d6 I find implementation of `sun.security.pkcs.PKCS7#PKCS7(java.io.InputStream)` a bit confusing (or even buggy). It uses only `InputStream.available()` to parse block. So I would prefer to not use it. ------------- PR: https://git.openjdk.java.net/jdk/pull/1853 From denghui.ddh at alibaba-inc.com Mon Mar 8 03:12:39 2021 From: denghui.ddh at alibaba-inc.com (Denghui Dong) Date: Mon, 08 Mar 2021 11:12:39 +0800 Subject: =?UTF-8?B?SWdub3JlZCBJT0V4Y2VwdGlvbg==?= Message-ID: <875620cc-b632-47d3-833e-9871431f86f0.denghui.ddh@alibaba-inc.com> Hi, I noticed that there is an ignored IO Exception in jdk.jfr.internal.PlatformRecording#stop, but I cannot open the related issue(8925030). The problem I have is that when users use `java -XX:StartFlightRecording=name=test,duration=5m,filename=./dumped.jfr `, and the 'dumped.jfr' is accidentally deleted(e.g. by clean task), the final jfr file will not exist, and also no exception will be generated after the recording is ended. Users need an extra step to generate the jfr file. If there is a problem with throwing this exception, maybe we can print the exception information there, so that at least a reminder can be given to the user. Any input is appreciated. Thanks, Denghui From jbachorik at openjdk.java.net Mon Mar 8 15:34:23 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 15:34:23 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v10] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with two additional commits since the last revision: - Adjust the deadspace calculation - Minor cleanup ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/f6954186..f708023b Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=09 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=08-09 Stats: 28 lines in 7 files changed: 11 ins; 7 del; 10 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 8 17:08:27 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 17:08:27 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v11] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Cache live size estimate for memory spaces ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/f708023b..343e4809 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=09-10 Stats: 17 lines in 3 files changed: 13 ins; 1 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 8 17:17:11 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 17:17:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Wed, 3 Mar 2021 12:13:43 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Add tests for the heap usage summary event > > src/hotspot/share/gc/shared/space.hpp line 553: > >> 551: size_t capacity() const { return byte_size(bottom(), end()); } >> 552: size_t used() const { return byte_size(bottom(), top()); } >> 553: size_t live() const { > > The code for serial gc, contrary to others, tries to give some resemblance of tracking actual liveness. I.e. calculating this anew every call to `SerialHeap::live()`. > However if calling an `update_live_estimate()` in parallel and G1 (and the other collectors) is fine at certain places, this should be as good for serial gc. > Doing so would reduce the footprint of this change quite a bit (for serial gc) Ok. I am caching the live estimate per memory space now. Not sure how much it will change the footprint of this change, though, but it is good for consistency anyway. > src/hotspot/share/gc/shared/space.inline.hpp line 128: > >> 126: p2i(dead_start), p2i(dead_end), dead_length * HeapWordSize); >> 127: >> 128: _dead_space += dead_length; > > I do not think adding this to the counter here instead of the other method for every object makes a difference performance-wise. > > As mentioned before, `_allowed_deadspace_words` counts *down* from `(space->capacity() * ratio / 100) / HeapWordSize;` to whatever end value. > > So at the end of collection, `(space->capacity() * ratio / 100) / HeapWordSize - _allowed_deadspace_words` should be equal to what `_dead_space` is now. > > Please add a getter to `DeadSpacer` that calculates this (factoring out the calculation of the maximum allowed deadspace). Fixed in https://github.com/openjdk/jdk/pull/2579/commits/f708023b84e33aee34f92d183a0d03d2747646db ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 8 17:29:27 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 17:29:27 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Remove unused field ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/343e4809..67d78940 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=11 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=10-11 Stats: 2 lines in 1 file changed: 0 ins; 2 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 8 17:29:30 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 17:29:30 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Wed, 3 Mar 2021 12:03:01 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Add tests for the heap usage summary event > > src/hotspot/share/gc/g1/g1CollectedHeap.hpp line 182: > >> 180: G1BlockOffsetTable* _bot; >> 181: >> 182: volatile size_t _live; > > I'm not happy with naming this `_live`, better use `_live_estimate`. The contents are not continuously updated and basically out of date after the first following allocation. > This includes the naming in all other instances too. I see your point - but that would probably lead to renaming `live()` method to `live_estimate()` (to keep the variable and the accessor method in sync) and that would break the nice symmetry we have now with `free()`, `used()` and `live()`. I have no strong feelings about this and if we can get quorum on this change I will do the renaming pass. > src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp line 87: > >> 85: >> 86: // in order to provide accurate estimate this method must be called only when the heap has just been collected and compacted >> 87: inline void capture_live(); > > Sentences should start with upper case in the comment. Also I'd prefer to name the method `update_live_estimate()` instead. Done > src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp line 60: > >> 58: class PSAdaptiveSizePolicy : public AdaptiveSizePolicy { >> 59: friend class PSGCAdaptivePolicyCounters; >> 60: friend class ParallelScavengeHeap; > > Delete this apparently unneeded friend declaration (compiled successfully without here) Done > src/hotspot/share/gc/g1/g1ConcurrentMark.cpp line 1070: > >> 1068: >> 1069: uint num_selected_for_rebuild() const { return _num_regions_selected_for_rebuild; } >> 1070: size_t live_estimate() const { return _live; } > > Please sync the member name with the getter name. I.e. `_live` -> `_live_estimate` Done > src/hotspot/share/gc/serial/serialHeap.hpp line 44: > >> 42: MemoryPool* _old_pool; >> 43: >> 44: size_t _live_size; > > Please rename to `_live_estimate` like the others. Avoid having different names in different collectors for the same thing. This field is unused. Removed. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 8 17:34:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 8 Mar 2021 17:34:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 2 Mar 2021 17:34:32 GMT, Aleksey Shipilev wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Add tests for the heap usage summary event > > src/hotspot/share/gc/shared/space.inline.hpp line 190: > >> 188: oop obj = oop(cur_obj); >> 189: size_t obj_size = obj->size(); >> 190: compact_top = cp->space->forward(obj, obj_size, cp, compact_top); > > This change seems superfluous now. Inline `obj_size` back? Done > src/hotspot/share/gc/shared/space.hpp line 555: > >> 553: size_t live() const { >> 554: return used() - _dead_space; >> 555: } > > Move it a few lines down, so `capacity`, `used`, `live` line up? Ok. I did my best to make the code look nice there :) > src/hotspot/share/gc/shared/collectedHeap.hpp line 218: > >> 216: virtual size_t capacity() const = 0; >> 217: virtual size_t used() const = 0; >> 218: // Returns the estimate of live set size. Because live set changes over time, > > I believe a blank line is in order here, look at other comments in the same header. Done > src/hotspot/share/gc/shared/space.inline.hpp line 90: > >> 88: >> 89: public: >> 90: size_t _dead_space; > > Should this really be "public"? Maybe `friend`-ing with the only user is better? Yep. Befriended. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jaroslav.bachorik at datadoghq.com Tue Mar 9 12:14:15 2021 From: jaroslav.bachorik at datadoghq.com (=?UTF-8?Q?Jaroslav_Bachor=C3=ADk?=) Date: Tue, 9 Mar 2021 13:14:15 +0100 Subject: RFR: 8247471: Enhance CPU load events with the actual elapsed CPU time In-Reply-To: References: Message-ID: Gentle ping (again)? -JB- On Tue, Jan 26, 2021 at 2:43 PM Jaroslav Bachorik wrote: > > A continuation of an RFR thread started last year - https://mail.openjdk.java.net/pipermail/hotspot-jfr-dev/2020-June/001533.html > > This change adds the raw CPU time value to CPU load events (per-thread and per-process as well). > The CPU time value is already known and used to calculate the load so adding it to the events does not incur any extra overhead while making it much easier for the end users to eg. aggregate and compare the active execution time per time period without the detailed knowledge how JFR computes and normalizes the CPU load. > > ------------- > > Commit messages: > - Fix wording and remove unnecessary debug output > - Fix jcheck > - Merge branch 'master' into 8247471_cpuload_with_time > - 8247471: Enhance CPU load events with the actual elapsed CPU time > > Changes: https://git.openjdk.java.net/jdk/pull/2186/files > Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=2186&range=00 > Issue: https://bugs.openjdk.java.net/browse/JDK-8247471 > Stats: 543 lines in 11 files changed: 238 ins; 205 del; 100 mod > Patch: https://git.openjdk.java.net/jdk/pull/2186.diff > Fetch: git fetch https://git.openjdk.java.net/jdk pull/2186/head:pull/2186 > > PR: https://git.openjdk.java.net/jdk/pull/2186 From jbachorik at openjdk.java.net Tue Mar 9 12:16:17 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 9 Mar 2021 12:16:17 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v9] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <1SitAwn_HPwglwUiTJdjoACmIhD4U9My1RVGzGJz5Ho=.f3f6556f-3fb1-44c7-b522-5c15de734b26@github.com> On Wed, 3 Mar 2021 12:15:21 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Add tests for the heap usage summary event > > Fwiw, the change still does not capture G1 full gc `live_estimate()`. @tschatzl I think I have finished the changes you requested. Please, take a look once you have time. Thanks! ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jaroslav.bachorik at datadoghq.com Tue Mar 9 16:09:46 2021 From: jaroslav.bachorik at datadoghq.com (=?UTF-8?Q?Jaroslav_Bachor=C3=ADk?=) Date: Tue, 9 Mar 2021 17:09:46 +0100 Subject: [Discussion] Making the native JFR ticks getter function semi-public Message-ID: Hello, I would like to propose exporting a symbol for obtaining the current value of the JFR ticks counter (which is used to record the events' timestamps). The rationale behind this proposal is to make external profilers (eg. async profiler) to better integrate with the built-in JFR recordings. Nowadays the events generated from the external profilers are using their own timestamp sources and as a result the recordings may (and will) experience tearing. With the ability to query the JFR native ticks count directly one could easily correlate any built-in recording with any external recording within the resolution of the timestamp provider. I am not proposing to introduce a new public API here - rather going the way of 'AsyncGetCallTrace' when one needs to manually load the shared JVM library and lookup the symbol. While it takes slightly more effort for the consumers it allows graceful degradation when the symbol can not be resolved in older or newer (if that export will be removed) JVMs. A simplistic change is drafted at https://github.com/openjdk/jdk/compare/master...DataDog:jb/export_ticks With that change in place I am able to obtain the JFR ticks from a JVMTI agent. Please, let me know what you think about this proposal. If it will be considered useful I will finalize the change and open a corresponding PR. Cheers, -JB- From jbachorik at openjdk.java.net Wed Mar 10 15:07:09 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Wed, 10 Mar 2021 15:07:09 GMT Subject: RFR: 8258414: OldObjectSample events too expensive In-Reply-To: References: Message-ID: On Mon, 1 Mar 2021 13:59:51 GMT, Markus Gr?nlund wrote: > Hi Florian, > > thanks for your great attempt to fix this issue, it is indeed a complex area with many interrelations. > > I think your suggestion has two problems that we need to solve: > > 1) If we separate the Leak Profiler stacktraces from the "regular" traces, there is a disconnect at the stacktrace record site, because the stacktrace id is cached to the thread. This means other events (ObjectAllocationSample and other memory allocation events) will attempt to reuse the id. Since this id is no longer valid for reuse, it must be handled. > > 2) The division into two separate stacktrace repositories now also separates the stacktrace id assignment, with a consequence that separate stacktraces can now end up having overlapping and conflicting ids. > > I also appreciated your effort in trying to incorporate the count of the traces written, to be tallied in the "regular" process. I do think we can simplify this a bit by not having them tallied (they are only a very small number, max 256 by default). > > Please take a look and let me know what you think about this suggestion. > > Thanks! > Markus > > PS If we go with this suggestion, I will put you down as a contributor. src/hotspot/share/jfr/recorder/stacktrace/jfrStackTraceRepository.cpp line 223: > 221: > 222: // invariant is that the entry to be resolved actually exists in the table > 223: const JfrStackTrace* JfrStackTraceRepository::lookup(unsigned int hash, traceid id) { Could the comment be extended to make it explicit that the lookup will be done only in the leak profiler stacktrace repo? Or, better yet the method renamed to `lookup_in_leak_profiler()` (but it might be too long ...). ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From github.com+741251+turbanoff at openjdk.java.net Wed Mar 10 19:57:20 2021 From: github.com+741251+turbanoff at openjdk.java.net (Andrey Turbanov) Date: Wed, 10 Mar 2021 19:57:20 GMT Subject: RFR: 8263395: Incorrect use of Objects.nonNull Message-ID: <2A5gaEFSZX0ADVk300EjtZjtmsscE6H2mlYxb_kGRgA=.6825b7ad-81a1-4dbe-9caf-b71820c044ba@github.com> 8263395: Incorrect use of Objects.nonNull ------------- Commit messages: - 8263395: Incorrect use of Objects.nonNull Changes: https://git.openjdk.java.net/jdk/pull/2923/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=2923&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263395 Stats: 3 lines in 3 files changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/2923.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2923/head:pull/2923 PR: https://git.openjdk.java.net/jdk/pull/2923 From shade at openjdk.java.net Thu Mar 11 09:38:24 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Thu, 11 Mar 2021 09:38:24 GMT Subject: RFR: JDK-8263426: Reflow JfrNetworkUtilization::send_events Message-ID: SonarCloud instance reports a problem in JfrNetworkUtilization::send_events: "Identical sub-expressions on both sides of operator "-"." here: const JfrTickspan interval = last_sample_instant == 0 ? cur_time - cur_time : cur_time - last_sample_instant; Note `cur_time - cur_time`. It would seem that `interval` is effectively zero when `last_sample_instant` is not yet initialized. And I see that the subsequent code checks for `interval.value() > 0` for every interface. That can be optimized a bit: when `last_sample_instant` is not available, do not enter the loop, and don't check `interval.value()`. Additional testing: - [x] `jdk_jfr` on Linux x86_64 ------------- Commit messages: - JDK-8263426: Reflow JfrNetworkUtilization::send_events Changes: https://git.openjdk.java.net/jdk/pull/2935/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=2935&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263426 Stats: 6 lines in 1 file changed: 1 ins; 1 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/2935.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2935/head:pull/2935 PR: https://git.openjdk.java.net/jdk/pull/2935 From mgronlun at openjdk.java.net Thu Mar 11 09:59:11 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 11 Mar 2021 09:59:11 GMT Subject: RFR: JDK-8263426: Reflow JfrNetworkUtilization::send_events In-Reply-To: References: Message-ID: On Thu, 11 Mar 2021 09:32:58 GMT, Aleksey Shipilev wrote: > SonarCloud instance reports a problem in JfrNetworkUtilization::send_events: "Identical sub-expressions on both sides of operator "-"." here: > > const JfrTickspan interval = last_sample_instant == 0 ? cur_time - cur_time : cur_time - last_sample_instant; > > Note `cur_time - cur_time`. It would seem that `interval` is effectively zero when `last_sample_instant` is not yet initialized. And I see that the subsequent code checks for `interval.value() > 0` for every interface. That can be optimized a bit: when `last_sample_instant` is not available, do not enter the loop, and don't check `interval.value()`. > > Additional testing: > - [x] `jdk_jfr` on Linux x86_64 Looks good, thanks. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/2935 From mgronlun at openjdk.java.net Thu Mar 11 10:41:27 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 11 Mar 2021 10:41:27 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: References: Message-ID: > Hi Florian, > > thanks for your great attempt to fix this issue, it is indeed a complex area with many interrelations. > > I think your suggestion has two problems that we need to solve: > > 1) If we separate the Leak Profiler stacktraces from the "regular" traces, there is a disconnect at the stacktrace record site, because the stacktrace id is cached to the thread. This means other events (ObjectAllocationSample and other memory allocation events) will attempt to reuse the id. Since this id is no longer valid for reuse, it must be handled. > > 2) The division into two separate stacktrace repositories now also separates the stacktrace id assignment, with a consequence that separate stacktraces can now end up having overlapping and conflicting ids. > > I also appreciated your effort in trying to incorporate the count of the traces written, to be tallied in the "regular" process. I do think we can simplify this a bit by not having them tallied (they are only a very small number, max 256 by default). > > Please take a look and let me know what you think about this suggestion. > > Thanks! > Markus > > PS If we go with this suggestion, I will put you down as a contributor. Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: lookup_for_leak_profiler ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2780/files - new: https://git.openjdk.java.net/jdk/pull/2780/files/45c82f13..6757188a Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2780&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2780&range=00-01 Stats: 24 lines in 5 files changed: 5 ins; 13 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/2780.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2780/head:pull/2780 PR: https://git.openjdk.java.net/jdk/pull/2780 From mgronlun at openjdk.java.net Thu Mar 11 10:45:10 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 11 Mar 2021 10:45:10 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: References: Message-ID: On Wed, 10 Mar 2021 15:03:55 GMT, Jaroslav Bachorik wrote: >> Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: >> >> lookup_for_leak_profiler > > src/hotspot/share/jfr/recorder/stacktrace/jfrStackTraceRepository.cpp line 223: > >> 221: >> 222: // invariant is that the entry to be resolved actually exists in the table >> 223: const JfrStackTrace* JfrStackTraceRepository::lookup(unsigned int hash, traceid id) { > > Could the comment be extended to make it explicit that the lookup will be done only in the leak profiler stacktrace repo? Or, better yet the method renamed to `lookup_in_leak_profiler()` (but it might be too long ...). Thanks, Jaroslav - yes, indeed. ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From mgronlun at openjdk.java.net Thu Mar 11 10:59:12 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 11 Mar 2021 10:59:12 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: References: Message-ID: On Thu, 11 Mar 2021 10:42:46 GMT, Markus Gr?nlund wrote: >> src/hotspot/share/jfr/recorder/stacktrace/jfrStackTraceRepository.cpp line 223: >> >>> 221: >>> 222: // invariant is that the entry to be resolved actually exists in the table >>> 223: const JfrStackTrace* JfrStackTraceRepository::lookup(unsigned int hash, traceid id) { >> >> Could the comment be extended to make it explicit that the lookup will be done only in the leak profiler stacktrace repo? Or, better yet the method renamed to `lookup_in_leak_profiler()` (but it might be too long ...). > > Thanks, Jaroslav - yes, indeed. I also attempted to have the artefact tagging happen lazily, e.g. methods and classes, making it a part of the stack trace resolution process. There is a problem with that approach concerning class unloading because a referenced class can unload, but since it does not yet have the proper tag, the framework does not intercept it. I elaborated on setting an "interest" tag to handle this, but that approach opens additional challenges related to visibility and concurrency. Hard problem to solve; I was unable to do it (yet). ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From egahlin at openjdk.java.net Thu Mar 11 14:04:09 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Thu, 11 Mar 2021 14:04:09 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v14] In-Reply-To: References: Message-ID: On Sat, 6 Mar 2021 11:45:26 GMT, Yi Yang wrote: >> Hi all, >> >> May I please have a review for this minor patch? >> >> It simplifies the use of subcommand `metadata` of jfr tool. Recording >> file is no longer mandatory, users can directly use `jfr metadata` to >> output JDK builtin meta information. As JDK-8256156 mentioned, it also >> supports events and categories filters: >> $ jfr metadata >> $ jfr metadata recording.jfr # compatible >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr >> $ jfr metadata --categories GC,Detailed >> $ jfr metadata --categories GC,Detailed recording.jfr > > Yi Yang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. Looks good. I still think the name of the test method should be called testWildcard() instead of testWildcardAndAcronym() since it is not testing acronyms, i.e using "GC" instead of "GarbageCollection", which is fine, but the name of the method should be appropriate. ------------- Marked as reviewed by egahlin (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/1904 From jbachorik at openjdk.java.net Thu Mar 11 14:29:08 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Thu, 11 Mar 2021 14:29:08 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: References: Message-ID: <2DiAJ_2-1Q2PcFHY2rrFdXvqkFL7BNXpPtuSYWeWd7c=.26d7ea05-466e-4c8e-ae43-0187cd1d0f83@github.com> On Thu, 11 Mar 2021 10:41:27 GMT, Markus Gr?nlund wrote: >> Hi Florian, >> >> thanks for your great attempt to fix this issue, it is indeed a complex area with many interrelations. >> >> I think your suggestion has two problems that we need to solve: >> >> 1) If we separate the Leak Profiler stacktraces from the "regular" traces, there is a disconnect at the stacktrace record site, because the stacktrace id is cached to the thread. This means other events (ObjectAllocationSample and other memory allocation events) will attempt to reuse the id. Since this id is no longer valid for reuse, it must be handled. >> >> 2) The division into two separate stacktrace repositories now also separates the stacktrace id assignment, with a consequence that separate stacktraces can now end up having overlapping and conflicting ids. >> >> I also appreciated your effort in trying to incorporate the count of the traces written, to be tallied in the "regular" process. I do think we can simplify this a bit by not having them tallied (they are only a very small number, max 256 by default). >> >> Please take a look and let me know what you think about this suggestion. >> >> Thanks! >> Markus >> >> PS If we go with this suggestion, I will put you down as a contributor. > > Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: > > lookup_for_leak_profiler Marked as reviewed by jbachorik (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From jbachorik at openjdk.java.net Thu Mar 11 14:29:09 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Thu, 11 Mar 2021 14:29:09 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: References: Message-ID: <0zrKsiIR9hKaqOXaY9yQTfatNCm8G_1Dl18ZbFdZ1fw=.b7be40ea-d357-4a65-ba7b-1ff5613d177a@github.com> On Thu, 11 Mar 2021 10:55:54 GMT, Markus Gr?nlund wrote: >> Thanks, Jaroslav - yes, indeed. > > I also attempted to have the artefact tagging happen lazily, e.g. methods and classes, making it a part of the stack trace resolution process. There is a problem with that approach concerning class unloading because a referenced class can unload, but since it does not yet have the proper tag, the framework does not intercept it. I elaborated on setting an "interest" tag to handle this, but that approach opens additional challenges related to visibility and concurrency. Hard problem to solve; I was unable to do it (yet). Probably can live without lazy tagging in the first iteration - still it will be much better than the current state. And then we can keep on adding incremental improvements based on cost-benefit weighing. ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From tschatzl at openjdk.java.net Thu Mar 11 14:53:13 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Thu, 11 Mar 2021 14:53:13 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 8 Mar 2021 17:29:27 GMT, Jaroslav Bachorik wrote: >> The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. >> >> ## Introducing new JFR event >> >> While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. >> Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. >> >> ## Implementation >> >> The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. >> >> The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. >> >> ### Epsilon GC >> >> Trivial implementation - just return `used()` instead. >> >> ### Serial GC >> >> Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). >> >> ### Parallel GC >> >> For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). >> >> ### G1 GC >> >> Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. >> >> ### Shenandoah >> >> In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. >> This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. >> >> ### ZGC >> >> `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. > > Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: > > Remove unused field I am leaving this as "request changes" for now as the question I had earlier about that after G1 Full gc the value of `_live_estimate` still seems unanswered and there does not seem to be code in this change for this. Is this intentional? (Not even setting the live bytes to `used()` which at that point would be a good estimate) There is another PR (#2760) that implements something like that although I haven't looked at it in detail. Otherwise looks okay. src/hotspot/share/gc/shared/space.inline.hpp line 140: > 138: size_t get_dead_space() { > 139: return (_max_deadspace_words - _allowed_deadspace_words) * HeapWordSize; > 140: } Hotspot does not use a "get_" prefix for getters. Also not sure why this needs to be private (and the friend class), I would prefer this instead of the friending. Retrieving the actual amount of dead space from a class that calculates it does not seem something that needs hiding. ------------- Changes requested by tschatzl (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/2579 From tschatzl at openjdk.java.net Thu Mar 11 15:46:11 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Thu, 11 Mar 2021 15:46:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 11 Mar 2021 14:50:10 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Remove unused field > > I am leaving this as "request changes" for now as the question I had earlier about that after G1 Full gc the value of `_live_estimate` still seems unanswered and there does not seem to be code in this change for this. Is this intentional? (Not even setting the live bytes to `used()` which at that point would be a good estimate) > > There is another PR (#2760) that implements something like that although I haven't looked at it in detail. > > Otherwise looks okay. Started reviewing PR #2760, and it implements liveness calculation for G1 full gc. I also suggested [there](https://github.com/openjdk/jdk/pull/2760#discussion_r592449837) to extract this functionality out into an extra CR. Maybe you can work together. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From yyang at openjdk.java.net Fri Mar 12 02:07:23 2021 From: yyang at openjdk.java.net (Yi Yang) Date: Fri, 12 Mar 2021 02:07:23 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v14] In-Reply-To: References: Message-ID: <26mcMPKwrF_H9DyvmN7DX7Sbt2hGusHZfBM-xhpxfMs=.c24f03b3-b61f-495d-9907-03b7fa392611@github.com> On Thu, 11 Mar 2021 14:01:33 GMT, Erik Gahlin wrote: >> Yi Yang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. > > Looks good. > > I still think the name of the test method should be called testWildcard() instead of testWildcardAndAcronym() since it is not testing acronyms, i.e using "GC" instead of "GarbageCollection", which is fine, but the name of the method should be appropriate. Thank Erik for your patient and reviews. Your naming suggestion makes sense, I forgot to rename that method in the previous commit. Now it has been corrected. ------------- PR: https://git.openjdk.java.net/jdk/pull/1904 From yyang at openjdk.java.net Fri Mar 12 02:07:23 2021 From: yyang at openjdk.java.net (Yi Yang) Date: Fri, 12 Mar 2021 02:07:23 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v15] In-Reply-To: References: Message-ID: > Hi all, > > May I please have a review for this minor patch? > > It simplifies the use of subcommand `metadata` of jfr tool. Recording > file is no longer mandatory, users can directly use `jfr metadata` to > output JDK builtin meta information. As JDK-8256156 mentioned, it also > supports events and categories filters: > $ jfr metadata > $ jfr metadata recording.jfr # compatible > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr > $ jfr metadata --categories GC,Detailed > $ jfr metadata --categories GC,Detailed recording.jfr Yi Yang has updated the pull request incrementally with one additional commit since the last revision: rename ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/1904/files - new: https://git.openjdk.java.net/jdk/pull/1904/files/ff13850c..a2cedbf9 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=14 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=13-14 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/1904.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/1904/head:pull/1904 PR: https://git.openjdk.java.net/jdk/pull/1904 From mli at openjdk.java.net Fri Mar 12 05:25:15 2021 From: mli at openjdk.java.net (Hamlin Li) Date: Fri, 12 Mar 2021 05:25:15 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 11 Mar 2021 15:42:51 GMT, Thomas Schatzl wrote: > Started reviewing PR #2760, and it implements liveness calculation for G1 full gc. I also suggested [there](https://github.com/openjdk/jdk/pull/2760#discussion_r592449837) to extract this functionality out into an extra CR. Maybe you can work together. Hi Thomas, Jaroslav, How about we track jfr liveness event in g1 full gc in a separate bug, so this PR #2579 can go ahead without blocking? If you don't mind I can work on new bug later after liveness collection in g1 full gc is finished in another separate bug. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From shade at openjdk.java.net Fri Mar 12 08:01:18 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Fri, 12 Mar 2021 08:01:18 GMT Subject: Integrated: JDK-8263426: Reflow JfrNetworkUtilization::send_events In-Reply-To: References: Message-ID: On Thu, 11 Mar 2021 09:32:58 GMT, Aleksey Shipilev wrote: > SonarCloud instance reports a problem in JfrNetworkUtilization::send_events: "Identical sub-expressions on both sides of operator "-"." here: > > const JfrTickspan interval = last_sample_instant == 0 ? cur_time - cur_time : cur_time - last_sample_instant; > > Note `cur_time - cur_time`. It would seem that `interval` is effectively zero when `last_sample_instant` is not yet initialized. And I see that the subsequent code checks for `interval.value() > 0` for every interface. That can be optimized a bit: when `last_sample_instant` is not available, do not enter the loop, and don't check `interval.value()`. > > Additional testing: > - [x] `jdk_jfr` on Linux x86_64 This pull request has now been integrated. Changeset: ff259393 Author: Aleksey Shipilev URL: https://git.openjdk.java.net/jdk/commit/ff259393 Stats: 6 lines in 1 file changed: 1 ins; 1 del; 4 mod 8263426: Reflow JfrNetworkUtilization::send_events Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/2935 From mgronlun at openjdk.java.net Fri Mar 12 10:08:12 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 12 Mar 2021 10:08:12 GMT Subject: Integrated: 8258414: OldObjectSample events too expensive In-Reply-To: References: Message-ID: <-zAe3YSkkM2uQOYTN52q-nuI846J6C8gVlXQ-1LvO-A=.e03f5adc-69b6-40aa-9b74-cce102201820@github.com> On Mon, 1 Mar 2021 13:59:51 GMT, Markus Gr?nlund wrote: > Hi Florian, > > thanks for your great attempt to fix this issue, it is indeed a complex area with many interrelations. > > I think your suggestion has two problems that we need to solve: > > 1) If we separate the Leak Profiler stacktraces from the "regular" traces, there is a disconnect at the stacktrace record site, because the stacktrace id is cached to the thread. This means other events (ObjectAllocationSample and other memory allocation events) will attempt to reuse the id. Since this id is no longer valid for reuse, it must be handled. > > 2) The division into two separate stacktrace repositories now also separates the stacktrace id assignment, with a consequence that separate stacktraces can now end up having overlapping and conflicting ids. > > I also appreciated your effort in trying to incorporate the count of the traces written, to be tallied in the "regular" process. I do think we can simplify this a bit by not having them tallied (they are only a very small number, max 256 by default). > > Please take a look and let me know what you think about this suggestion. > > Thanks! > Markus > > PS If we go with this suggestion, I will put you down as a contributor. This pull request has now been integrated. Changeset: a9b156d3 Author: Markus Gr?nlund URL: https://git.openjdk.java.net/jdk/commit/a9b156d3 Stats: 142 lines in 9 files changed: 64 ins; 31 del; 47 mod 8258414: OldObjectSample events too expensive Co-authored-by: Florian David Reviewed-by: jbachorik ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From mgronlun at openjdk.java.net Fri Mar 12 10:16:10 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 12 Mar 2021 10:16:10 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v2] In-Reply-To: References: Message-ID: On Mon, 1 Mar 2021 07:52:37 GMT, Denghui Dong wrote: >> Markus Gr?nlund has updated the pull request incrementally with two additional commits since the last revision: >> >> - cds hook >> - skip return value > > src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.cpp line 99: > >> 97: assert(mirror != NULL, "invariant"); >> 98: const Klass* const k = java_lang_Class::as_Klass(mirror); >> 99: return k != NULL ? load(k) : load_primitive(mirror); > > I think the same modification should be made in JfrTraceId::load_raw. Thanks, Denghui, I will make the same change there. I will also put you down as a contributor to this change set. ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From jbachorik at openjdk.java.net Fri Mar 12 10:40:09 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Fri, 12 Mar 2021 10:40:09 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: <2DiAJ_2-1Q2PcFHY2rrFdXvqkFL7BNXpPtuSYWeWd7c=.26d7ea05-466e-4c8e-ae43-0187cd1d0f83@github.com> References: <2DiAJ_2-1Q2PcFHY2rrFdXvqkFL7BNXpPtuSYWeWd7c=.26d7ea05-466e-4c8e-ae43-0187cd1d0f83@github.com> Message-ID: <0BQCLfn_beiTjBcEg4A2HJVy6e-3YW4y9ebiwg4e2aY=.c00d3592-1bd1-48d9-a335-a988fe7f782f@github.com> On Thu, 11 Mar 2021 14:25:56 GMT, Jaroslav Bachorik wrote: >> Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: >> >> lookup_for_leak_profiler > > Marked as reviewed by jbachorik (Reviewer). Yahoo! Thanks, Markus! ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From mgronlun at openjdk.java.net Fri Mar 12 10:45:31 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 12 Mar 2021 10:45:31 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v3] In-Reply-To: References: Message-ID: > Hi Denghui, > > I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. > > Can you do a test run with this suggestion to see if it works for you? > > Thanks > Markus Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: update ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2738/files - new: https://git.openjdk.java.net/jdk/pull/2738/files/ce1ed381..ab8eed19 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2738&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2738&range=01-02 Stats: 70 lines in 5 files changed: 24 ins; 36 del; 10 mod Patch: https://git.openjdk.java.net/jdk/pull/2738.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2738/head:pull/2738 PR: https://git.openjdk.java.net/jdk/pull/2738 From mgronlun at openjdk.java.net Fri Mar 12 10:48:10 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 12 Mar 2021 10:48:10 GMT Subject: RFR: 8258414: OldObjectSample events too expensive [v2] In-Reply-To: <0BQCLfn_beiTjBcEg4A2HJVy6e-3YW4y9ebiwg4e2aY=.c00d3592-1bd1-48d9-a335-a988fe7f782f@github.com> References: <2DiAJ_2-1Q2PcFHY2rrFdXvqkFL7BNXpPtuSYWeWd7c=.26d7ea05-466e-4c8e-ae43-0187cd1d0f83@github.com> <0BQCLfn_beiTjBcEg4A2HJVy6e-3YW4y9ebiwg4e2aY=.c00d3592-1bd1-48d9-a335-a988fe7f782f@github.com> Message-ID: <9Q9-6Y4cKiZ7N0UZZDEVbrec7tgrhZApdaGgDu-GXFs=.753710e5-8d89-445c-a361-742142a598a6@github.com> On Fri, 12 Mar 2021 10:36:54 GMT, Jaroslav Bachorik wrote: >> Marked as reviewed by jbachorik (Reviewer). > > Yahoo! Thanks, Markus! Thank you, Jaroslav for contributing and reviewing! ------------- PR: https://git.openjdk.java.net/jdk/pull/2780 From yyang at openjdk.java.net Fri Mar 12 13:19:24 2021 From: yyang at openjdk.java.net (Yi Yang) Date: Fri, 12 Mar 2021 13:19:24 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v16] In-Reply-To: References: Message-ID: > Hi all, > > May I please have a review for this minor patch? > > It simplifies the use of subcommand `metadata` of jfr tool. Recording > file is no longer mandatory, users can directly use `jfr metadata` to > output JDK builtin meta information. As JDK-8256156 mentioned, it also > supports events and categories filters: > $ jfr metadata > $ jfr metadata recording.jfr # compatible > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr > $ jfr metadata --categories GC,Detailed > $ jfr metadata --categories GC,Detailed recording.jfr Yi Yang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: rename ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/1904/files - new: https://git.openjdk.java.net/jdk/pull/1904/files/a2cedbf9..c401df49 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=15 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1904&range=14-15 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/1904.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/1904/head:pull/1904 PR: https://git.openjdk.java.net/jdk/pull/1904 From egahlin at openjdk.java.net Fri Mar 12 20:45:08 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Fri, 12 Mar 2021 20:45:08 GMT Subject: RFR: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording [v16] In-Reply-To: References: Message-ID: On Fri, 12 Mar 2021 13:19:24 GMT, Yi Yang wrote: >> Hi all, >> >> May I please have a review for this minor patch? >> >> It simplifies the use of subcommand `metadata` of jfr tool. Recording >> file is no longer mandatory, users can directly use `jfr metadata` to >> output JDK builtin meta information. As JDK-8256156 mentioned, it also >> supports events and categories filters: >> $ jfr metadata >> $ jfr metadata recording.jfr # compatible >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd >> $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr >> $ jfr metadata --categories GC,Detailed >> $ jfr metadata --categories GC,Detailed recording.jfr > > Yi Yang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. Marked as reviewed by egahlin (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/1904 From yyang at openjdk.java.net Sat Mar 13 02:06:08 2021 From: yyang at openjdk.java.net (Yi Yang) Date: Sat, 13 Mar 2021 02:06:08 GMT Subject: Integrated: 8256156: JFR: Allow 'jfr' tool to show metadata without a recording In-Reply-To: References: Message-ID: On Tue, 29 Dec 2020 07:44:36 GMT, Yi Yang wrote: > Hi all, > > May I please have a review for this minor patch? > > It simplifies the use of subcommand `metadata` of jfr tool. Recording > file is no longer mandatory, users can directly use `jfr metadata` to > output JDK builtin meta information. As JDK-8256156 mentioned, it also > supports events and categories filters: > $ jfr metadata > $ jfr metadata recording.jfr # compatible > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd > $ jfr metadata --events jdk.ThreadStart,jdk.ThreadEnd recording.jfr > $ jfr metadata --categories GC,Detailed > $ jfr metadata --categories GC,Detailed recording.jfr This pull request has now been integrated. Changeset: 86e4c755 Author: Yi Yang Committer: Erik Gahlin URL: https://git.openjdk.java.net/jdk/commit/86e4c755 Stats: 478 lines in 5 files changed: 345 ins; 115 del; 18 mod 8256156: JFR: Allow 'jfr' tool to show metadata without a recording Reviewed-by: egahlin ------------- PR: https://git.openjdk.java.net/jdk/pull/1904 From jbachorik at openjdk.java.net Mon Mar 15 09:25:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 09:25:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 09:18:52 GMT, Jaroslav Bachorik wrote: >> src/hotspot/share/gc/shared/space.inline.hpp line 140: >> >>> 138: size_t get_dead_space() { >>> 139: return (_max_deadspace_words - _allowed_deadspace_words) * HeapWordSize; >>> 140: } >> >> Hotspot does not use a "get_" prefix for getters. Also not sure why this needs to be private (and the friend class), I would prefer this instead of the friending. Retrieving the actual amount of dead space from a class that calculates it does not seem something that needs hiding. > > The visibility for `_dead_space` was changed based on this comment: https://github.com/openjdk/jdk/pull/2579/files/f69541864e093bc5b250bf625ec75983764ba2bf#r585771280 by @shipilev Also, I see a bunch of methods named `get_*` in GC code alone. I have no problem renaming it to eg. `dead_space()` but it does not seem that this naming pattern is not used in hotspot. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 15 09:25:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 09:25:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 11 Mar 2021 14:44:10 GMT, Thomas Schatzl wrote: >> Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: >> >> Remove unused field > > src/hotspot/share/gc/shared/space.inline.hpp line 140: > >> 138: size_t get_dead_space() { >> 139: return (_max_deadspace_words - _allowed_deadspace_words) * HeapWordSize; >> 140: } > > Hotspot does not use a "get_" prefix for getters. Also not sure why this needs to be private (and the friend class), I would prefer this instead of the friending. Retrieving the actual amount of dead space from a class that calculates it does not seem something that needs hiding. The visibility for `_dead_space` was changed based on this comment: https://github.com/openjdk/jdk/pull/2579/files/f69541864e093bc5b250bf625ec75983764ba2bf#r585771280 by @shipilev ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 15 09:31:10 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 09:31:10 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Thu, 11 Mar 2021 15:42:51 GMT, Thomas Schatzl wrote: >> I am leaving this as "request changes" for now as the question I had earlier about that after G1 Full gc the value of `_live_estimate` still seems unanswered and there does not seem to be code in this change for this. Is this intentional? (Not even setting the live bytes to `used()` which at that point would be a good estimate) >> >> There is another PR (#2760) that implements something like that although I haven't looked at it in detail. >> >> Otherwise looks okay. > > Started reviewing PR #2760, and it implements liveness calculation for G1 full gc. I also suggested [there](https://github.com/openjdk/jdk/pull/2760#discussion_r592449837) to extract this functionality out into an extra CR. Maybe you can work together. @tschatzl @Hamlin-Li Would it be ok to set the live estimate to the `used()` value at the end of `G1FullCollector::phase4_do_compaction()` method to have something suboptimal but working and refine in https://github.com/openjdk/jdk/pull/2760 (or a subsequent ticket/PR once both parts are ready)? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From sjohanss at openjdk.java.net Mon Mar 15 12:27:17 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Mon, 15 Mar 2021 12:27:17 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 09:26:32 GMT, Jaroslav Bachorik wrote: >> Started reviewing PR #2760, and it implements liveness calculation for G1 full gc. I also suggested [there](https://github.com/openjdk/jdk/pull/2760#discussion_r592449837) to extract this functionality out into an extra CR. Maybe you can work together. > > @tschatzl @Hamlin-Li > Would it be ok to set the live estimate to the `used()` value at the end of `G1FullCollector::phase4_do_compaction()` method to have something suboptimal but working and refine in https://github.com/openjdk/jdk/pull/2760 (or a subsequent ticket/PR once both parts are ready)? Sorry for being a bit late to the party. Looking at the suggested implementation for G1 I see a problem with only updating this after concurrent mark (and the Full GC). Say for example you have a concurrent mark cycle before the heap has expanded a lot and you get a low value stored in `G1CollectedHeap::_live`. Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. In this case the same value will be reported for the entire run. For this to work the _live value would have to be updated at every GC, but this is a bit costly. Maybe the first version could just use `used()` for G1. Have you done any tests to see how off this would be compared to the other GCs? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jboes at openjdk.java.net Mon Mar 15 12:55:11 2021 From: jboes at openjdk.java.net (Julia Boes) Date: Mon, 15 Mar 2021 12:55:11 GMT Subject: RFR: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy In-Reply-To: References: <5s4i5725zJd8QrefwYyc8sn2fQXad3hAGLVflkFttmY=.abe55236-e258-482b-b8be-be75e1eb086c@github.com> Message-ID: On Thu, 18 Feb 2021 07:12:34 GMT, Andrey Turbanov wrote: >> Hi @turbanoff, I'm happy to sponsor but I see two comments by @marschall - have they been addressed? >> https://github.com/openjdk/jdk/pull/1853#discussion_r572815422 >> https://github.com/openjdk/jdk/pull/1853#discussion_r572380746 > > @FrauBoes fixed in the last commit. Is there any way to de-_integrate_ PR (to include last commit)? @turbanoff Given that this PR has been lingering for a while, you could drop the change in `X509CertPath.java` for now and integrate the remaining changes. I'm happy to sponsor in that case. ------------- PR: https://git.openjdk.java.net/jdk/pull/1853 From github.com+741251+turbanoff at openjdk.java.net Mon Mar 15 14:50:24 2021 From: github.com+741251+turbanoff at openjdk.java.net (Andrey Turbanov) Date: Mon, 15 Mar 2021 14:50:24 GMT Subject: RFR: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy [v13] In-Reply-To: References: Message-ID: > 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy Andrey Turbanov has updated the pull request incrementally with one additional commit since the last revision: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy drop changes in X509CertPath ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/1853/files - new: https://git.openjdk.java.net/jdk/pull/1853/files/1b30471d..96920ee6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1853&range=12 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1853&range=11-12 Stats: 25 lines in 1 file changed: 22 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/1853.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/1853/head:pull/1853 PR: https://git.openjdk.java.net/jdk/pull/1853 From jbachorik at openjdk.java.net Mon Mar 15 15:24:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 15:24:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <_1nv16MS2e-Ed1xHZyA8hewbCCCSCCkq0OafP4rRJq8=.75ee1e9d-17b3-46c3-87d1-c03ef2765cf3@github.com> On Mon, 15 Mar 2021 12:24:02 GMT, Stefan Johansson wrote: >> @tschatzl @Hamlin-Li >> Would it be ok to set the live estimate to the `used()` value at the end of `G1FullCollector::phase4_do_compaction()` method to have something suboptimal but working and refine in https://github.com/openjdk/jdk/pull/2760 (or a subsequent ticket/PR once both parts are ready)? > > Sorry for being a bit late to the party. Looking at the suggested implementation for G1 I see a problem with only updating this after concurrent mark (and the Full GC). Say for example you have a concurrent mark cycle before the heap has expanded a lot and you get a low value stored in `G1CollectedHeap::_live`. Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. In this case the same value will be reported for the entire run. > > For this to work the _live value would have to be updated at every GC, but this is a bit costly. Maybe the first version could just use `used()` for G1. Have you done any tests to see how off this would be compared to the other GCs? @kstefanj > Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. Is there a way to get the liveness info when the heap expands? If not that would mean we had no way to figure out the new live set size and would assume, conservatively, the last known value. As I mentioned in the PR description the live size value will be a 'best effort estimate' depending on what can each particular GC provide. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From tschatzl at openjdk.java.net Mon Mar 15 15:41:11 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Mon, 15 Mar 2021 15:41:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 09:21:50 GMT, Jaroslav Bachorik wrote: >> The visibility for `_dead_space` was changed based on this comment: https://github.com/openjdk/jdk/pull/2579/files/f69541864e093bc5b250bf625ec75983764ba2bf#r585771280 by @shipilev > > Also, I see a bunch of methods named `get_*` in GC code alone. I have no problem renaming it to eg. `dead_space()` but it does not seem that this naming pattern is not used in hotspot. https://wiki.openjdk.java.net/display/HotSpot/StyleGuide > Nearly all of the guidelines mentioned below have many counter-examples in the Hotspot code base. Finding a counterexample is not sufficient justification for new code to follow the counterexample as a precedent, since readers of your code will rightfully expect your code to follow the greater bulk of precedents documented here. For more on counterexamples, see the section at the bottom of this page. > > When changing pre-existing code, it is reasonable to adjust it to match these conventions. Exception: If the pre-existing code clearly conforms locally to its own peculiar conventions, it is not worth reformatting the whole thing. and > Getter accessor names are noun phrases, with no "get_" noise word. Boolean getters can also begin with "is_" or "has_". Unless there is a good reason, please keep to few rules the official style guide for new code. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 15 16:03:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 16:03:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 15:38:48 GMT, Thomas Schatzl wrote: >> Also, I see a bunch of methods named `get_*` in GC code alone. I have no problem renaming it to eg. `dead_space()` but it does not seem that this naming pattern is not used in hotspot. > > https://wiki.openjdk.java.net/display/HotSpot/StyleGuide > >> Nearly all of the guidelines mentioned below have many counter-examples in the Hotspot code base. Finding a counterexample is not sufficient justification for new code to follow the counterexample as a precedent, since readers of your code will rightfully expect your code to follow the greater bulk of precedents documented here. For more on counterexamples, see the section at the bottom of this page. >> >> When changing pre-existing code, it is reasonable to adjust it to match these conventions. Exception: If the pre-existing code clearly conforms locally to its own peculiar conventions, it is not worth reformatting the whole thing. > > and > >> Getter accessor names are noun phrases, with no "get_" noise word. Boolean getters can also begin with "is_" or "has_". > > Unless there is a good reason, please keep to few rules the official style guide for new code. ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 15 16:17:23 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 16:17:23 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v13] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Change get_dead_space() to dead_space() ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/67d78940..056f5fd7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=12 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=11-12 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Mon Mar 15 16:32:31 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 15 Mar 2021 16:32:31 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v14] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Capture live estimate for G1 full cycle ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/056f5fd7..81250d1c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=13 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=12-13 Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From jboes at openjdk.java.net Mon Mar 15 18:52:17 2021 From: jboes at openjdk.java.net (Julia Boes) Date: Mon, 15 Mar 2021 18:52:17 GMT Subject: RFR: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy In-Reply-To: References: <5s4i5725zJd8QrefwYyc8sn2fQXad3hAGLVflkFttmY=.abe55236-e258-482b-b8be-be75e1eb086c@github.com> Message-ID: <4Bae_Uh8Zakn4cnmbXkUcV9xRTus1hgHCEPVthHuEyc=.fb959963-ff7a-4067-b888-e178d9f70e91@github.com> On Thu, 18 Feb 2021 07:12:34 GMT, Andrey Turbanov wrote: >> Hi @turbanoff, I'm happy to sponsor but I see two comments by @marschall - have they been addressed? >> https://github.com/openjdk/jdk/pull/1853#discussion_r572815422 >> https://github.com/openjdk/jdk/pull/1853#discussion_r572380746 > > @FrauBoes fixed in the last commit. Is there any way to de-_integrate_ PR (to include last commit)? @turbanoff Tier 1-3 still all clear. If you /integrate, I will sponsor this tomorrow. ------------- PR: https://git.openjdk.java.net/jdk/pull/1853 From sjohanss at openjdk.java.net Mon Mar 15 21:25:10 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Mon, 15 Mar 2021 21:25:10 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 12:24:02 GMT, Stefan Johansson wrote: >> @tschatzl @Hamlin-Li >> Would it be ok to set the live estimate to the `used()` value at the end of `G1FullCollector::phase4_do_compaction()` method to have something suboptimal but working and refine in https://github.com/openjdk/jdk/pull/2760 (or a subsequent ticket/PR once both parts are ready)? > > Sorry for being a bit late to the party. Looking at the suggested implementation for G1 I see a problem with only updating this after concurrent mark (and the Full GC). Say for example you have a concurrent mark cycle before the heap has expanded a lot and you get a low value stored in `G1CollectedHeap::_live`. Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. In this case the same value will be reported for the entire run. > > For this to work the _live value would have to be updated at every GC, but this is a bit costly. Maybe the first version could just use `used()` for G1. Have you done any tests to see how off this would be compared to the other GCs? > @kstefanj > > > Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. > > Is there a way to get the liveness info when the heap expands? If not that would mean we had no way to figure out the new live set size and would assume, conservatively, the last known value. > > As I mentioned in the PR description the live size value will be a 'best effort estimate' depending on what can each particular GC provide. Sure, and this is fair, my concern is just that this 'best effort estimate' for G1 will often be worse than just using `used()`. This is not only a problem for when the heap expands, that was just an example, the live value will become more and more stale the longer an application run without triggering a new concurrent cycle. For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return `used()` in `live()` as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 09:40:11 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 09:40:11 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Mar 2021 21:22:44 GMT, Stefan Johansson wrote: > For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return used() in live() as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? Actually yes - this event was meant to be a cheap way to see whether the *known* live size is growing (has an upwards trend) which would using the `used()` value make more difficult and unreliable. I would prefer keeping the implementation to return the lower bound of the live size as is the case for other GCs as well. May add explanatory comments to the code and the event definition to make this clear. Also, the `used()` value is already captured in the event so we would have it duplicated. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From github.com+741251+turbanoff at openjdk.java.net Tue Mar 16 10:13:11 2021 From: github.com+741251+turbanoff at openjdk.java.net (Andrey Turbanov) Date: Tue, 16 Mar 2021 10:13:11 GMT Subject: Integrated: 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy In-Reply-To: References: Message-ID: On Sun, 20 Dec 2020 17:05:21 GMT, Andrey Turbanov wrote: > 8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy This pull request has now been integrated. Changeset: 68deb24b Author: Andrey Turbanov Committer: Julia Boes URL: https://git.openjdk.java.net/jdk/commit/68deb24b Stats: 105 lines in 7 files changed: 3 ins; 78 del; 24 mod 8080272: Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy Reviewed-by: mcimadamore, alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/1853 From sjohanss at openjdk.java.net Tue Mar 16 10:50:09 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Tue, 16 Mar 2021 10:50:09 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 16 Mar 2021 09:37:08 GMT, Jaroslav Bachorik wrote: >>> @kstefanj >>> >>> > Then the heap expands and your application get to a steady state that doesn't require any more marking cycles. >>> >>> Is there a way to get the liveness info when the heap expands? If not that would mean we had no way to figure out the new live set size and would assume, conservatively, the last known value. >>> >>> As I mentioned in the PR description the live size value will be a 'best effort estimate' depending on what can each particular GC provide. >> >> Sure, and this is fair, my concern is just that this 'best effort estimate' for G1 will often be worse than just using `used()`. This is not only a problem for when the heap expands, that was just an example, the live value will become more and more stale the longer an application run without triggering a new concurrent cycle. >> >> For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return `used()` in `live()` as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? > >> For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return used() in live() as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? > > Actually yes - this event was meant to be a cheap way to see whether the *known* live size is growing (has an upwards trend) which would using the `used()` value make more difficult and unreliable. > > I would prefer keeping the implementation to return the lower bound of the live size as is the case for other GCs as well. May add explanatory comments to the code and the event definition to make this clear. > > Also, the `used()` value is already captured in the event so we would have it duplicated. > > For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return used() in live() as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? > > Actually yes - this event was meant to be a cheap way to see whether the _known_ live size is growing (has an upwards trend) which would using the `used()` value make more difficult and unreliable. > > I would prefer keeping the implementation to return the lower bound of the live size as is the case for other GCs as well. May add explanatory comments to the code and the event definition to make this clear. > > Also, the `used()` value is already captured in the event so we would have it duplicated. Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be `used()` at the end of the GC. I know `used()` is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. So as a middle road I would suggest to update `G1CollectedHeap::gc_epilogue(bool full)` to include: set_live(used()); With this you don't need the changes for the `G1FullCollector`. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 11:09:09 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 11:09:09 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Tue, 16 Mar 2021 10:47:22 GMT, Stefan Johansson wrote: > Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. > So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: > > set_live(used()); With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. This sounds interesting. Let me try this out. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From sjohanss at openjdk.java.net Tue Mar 16 11:32:14 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Tue, 16 Mar 2021 11:32:14 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> On Tue, 16 Mar 2021 11:06:22 GMT, Jaroslav Bachorik wrote: > > Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. > > Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. > The other STW GCs do report the same, right? > > So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: > > set_live(used()); > > With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. > > This sounds interesting. Let me try this out. Glad you like the idea :) I did a quick test locally and it shows the trend ok, even if it is an over estimate of live: live = 1.1 GB live = 1.2 GB live = 1.5 GB live = 1.7 GB live = 2.1 GB name = "G1Old" live = 1.4 GB live = 1.6 GB live = 1.8 GB live = 2.0 GB live = 2.3 GB live = 2.5 GB live = 2.8 GB live = 3.1 GB live = 3.3 GB live = 3.7 GB live = 4.0 GB live = 4.3 GB name = "G1Old" live = 1.2 GB G1Old is from concurrent mark events. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 11:41:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 11:41:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi 8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> Message-ID: On Tue, 16 Mar 2021 11:28:50 GMT, Stefan Johansson wrote: > The other STW GCs do report the same, right? They report the lower bound - basically the real live size right at the end of a GC cycle (either gathered during marking or the used size after compaction). Of course, a few moments later it might (and probably will) not be 100% correct but that's why it is just an estimate. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From tschatzl at openjdk.java.net Tue Mar 16 11:54:09 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Tue, 16 Mar 2021 11:54:09 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> On Tue, 16 Mar 2021 10:47:22 GMT, Stefan Johansson wrote: > > For the liveness value to be useful it would have to be updated at each GC, and we need to investigate further to see how we can do that in a "cheap" way. I would prefer if G1 did just return used() in live() as a start and we can create a follow-up task to investigate how to best add a better estimate. Do you see any problem with this? > > Actually yes - this event was meant to be a cheap way to see whether the _known_ live size is growing (has an upwards trend) which would using the `used()` value make more difficult and unreliable. So one of the actual purposes seems to be some kind of leak detection: there is this JFR leak detector (I only know the feature name, not completely how it works and what its overhead is) for this purpose, wouldn't that work? Also, for this purpose, why would used() not be a good substitute for liveness? If e.g. used() average grows over time you can deduce the same I would assume (particularly used() after mixed gc phase in g1). Do you have any numbers on what the impact of using used() vs. this live() would be in such a use case? What I'm afraid of is that mixing values taken at different times - used and capacity are taken at the time of the event, and the liveness estimated updated at other, irregular intervals may cause significiant amount of confusion in interpreting this value. It might be obvious to you, but there will be other users. One option could be detaching the liveness estimate from used()/capacity() (I see a value in having some heap usage summary at regular intervals) and send the liveness estimate event just when they are generated? Then the various collectors could send this liveness value at times when they think they are fairly accurate, not when the collectors must and particularly not in conjunction with samples taken at completely different times. Independent of whether used/capacity and liveness are sent, the receiver needs to do statistics (trend lines) on those anyway. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 12:26:14 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 12:26:14 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> Message-ID: On Tue, 16 Mar 2021 11:51:45 GMT, Thomas Schatzl wrote: > So one of the actual purposes seems to be some kind of leak detection: there is this JFR leak detector (I only know the feature name, not completely how it works and what its overhead is) for this purpose, wouldn't that work? Yes. But enabling that comes with an extra price so it is more of a focused tool than something you could use in continuous monitoring/signal evaluation. > Also, for this purpose, why would used() not be a good substitute for liveness? If e.g. used() average grows over time you can deduce the same I would assume (particularly used() after mixed gc phase in g1). The major problem is that eg. for g1 given large enough heap the used value can keep on growing for quite long time, possibly generating wrong signal about potential memory leak. If the live estimate is set to `used()` after mixed gc phase in g1 I think it still will be a good estimate. The only thing I am opposing is having `live()` call return the current `used()` value which, IMO, might become rather confusing. > Do you have any numbers on what the impact of using used() vs. this live() would be in such a use case? Nope. Do you mean perf impact? > What I'm afraid of is that mixing values taken at different times - used and capacity are taken at the time of the event, and the liveness estimated updated at other, irregular intervals may cause significiant amount of confusion in interpreting this value. It might be obvious to you, but there will be other users. IDK. If the event field would explicitly mention that this is the **last known live size estimate** it should set the expectations right. > >One option could be detaching the liveness estimate from used()/capacity() (I see a value in having some heap usage summary at regular intervals) and send the liveness estimate event just when they are generated? Then the various collectors could send this liveness value at times when they think they are fairly accurate, not when the collectors must and particularly not in conjunction with samples taken at completely different times. The problem is the irregularity - when the live size is reported only when it is calculated there might be long periods in the recording missing the live size data at all. In order for this information to be useful it should be reported at least at the beginning and end of a JFR chunk. > Independent of whether used/capacity and liveness are sent, the receiver needs to do statistics (trend lines) on those anyway. Yes. It's just that with the live size estimate one wouldn't be getting the false positives one would get with used heap trend. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 12:32:12 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 12:32:12 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi 8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> Message-ID: On Tue, 16 Mar 2021 11:28:50 GMT, Stefan Johansson wrote: >>> Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. >> >> Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. >> >>> So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: >>> >>> set_live(used()); >> With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. >> >> This sounds interesting. Let me try this out. > >> > Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. >> >> Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. >> > The other STW GCs do report the same, right? > >> > So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: >> > set_live(used()); >> > With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. >> >> This sounds interesting. Let me try this out. > > Glad you like the idea :) I did a quick test locally and it shows the trend ok, even if it is an over estimate of live: > live = 1.1 GB > live = 1.2 GB > live = 1.5 GB > live = 1.7 GB > live = 2.1 GB > name = "G1Old" > live = 1.4 GB > live = 1.6 GB > live = 1.8 GB > live = 2.0 GB > live = 2.3 GB > live = 2.5 GB > live = 2.8 GB > live = 3.1 GB > live = 3.3 GB > live = 3.7 GB > live = 4.0 GB > live = 4.3 GB > name = "G1Old" > live = 1.2 GB > G1Old is from concurrent mark events. @kstefanj Just to make sure - `set_live(used())` should be the last call in `G1CollectedHeap::gc_prologue(bool full)` ? I am getting some funny numbers with this change - basically, last known live size is getting bigger than the current used size ?? ![Screen Shot 2021-03-16 at 1 28 27 PM](https://user-images.githubusercontent.com/738413/111308939-97b06d80-865b-11eb-9e2f-d595b49d6401.png) ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From sjohanss at openjdk.java.net Tue Mar 16 14:15:33 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Tue, 16 Mar 2021 14:15:33 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi 8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> Message-ID: On Tue, 16 Mar 2021 11:28:50 GMT, Stefan Johansson wrote: >>> Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. >> >> Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. >> >>> So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: >>> >>> set_live(used()); >> With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. >> >> This sounds interesting. Let me try this out. > >> > Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. >> >> Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. >> > The other STW GCs do report the same, right? > >> > So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: >> > set_live(used()); >> > With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. >> >> This sounds interesting. Let me try this out. > > Glad you like the idea :) I did a quick test locally and it shows the trend ok, even if it is an over estimate of live: > live = 1.1 GB > live = 1.2 GB > live = 1.5 GB > live = 1.7 GB > live = 2.1 GB > name = "G1Old" > live = 1.4 GB > live = 1.6 GB > live = 1.8 GB > live = 2.0 GB > live = 2.3 GB > live = 2.5 GB > live = 2.8 GB > live = 3.1 GB > live = 3.3 GB > live = 3.7 GB > live = 4.0 GB > live = 4.3 GB > name = "G1Old" > live = 1.2 GB > G1Old is from concurrent mark events. > @kstefanj > Just to make sure - `set_live(used())` should be the last call in `G1CollectedHeap::gc_prologue(bool full)` ? > Anywhere in there should be fine, we should look if there is anything related in there it could be grouped with. > I am getting some funny numbers with this change - basically, last known live size is getting bigger than the current used size ?? > > ![Screen Shot 2021-03-16 at 1 28 27 PM](https://user-images.githubusercontent.com/738413/111308939-97b06d80-865b-11eb-9e2f-d595b49d6401.png) This is strange, what kind of GCs are happening around those samples? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 14:15:32 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 14:15:32 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v15] In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: <2GhOBO4qaEDTMS1tOwAj3u_-_3__n-M_HVFLE9cSJ-s=.9d7830c1-1f65-46a3-8b16-ce6b77367559@github.com> > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Update liveness for G1 mixed GC ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2579/files - new: https://git.openjdk.java.net/jdk/pull/2579/files/81250d1c..f767f257 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=14 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2579&range=13-14 Stats: 5 lines in 2 files changed: 3 ins; 2 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/2579.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2579/head:pull/2579 PR: https://git.openjdk.java.net/jdk/pull/2579 From sjohanss at openjdk.java.net Tue Mar 16 14:15:33 2021 From: sjohanss at openjdk.java.net (Stefan Johansson) Date: Tue, 16 Mar 2021 14:15:33 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi 8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> Message-ID: On Tue, 16 Mar 2021 13:24:38 GMT, Stefan Johansson wrote: >>> > Sure, but for the event to be useful we want the reported value to be as close to the reality as possible. I don't understand why you want the lower bound, can you explain why? I would go for the upper bound, which in that case would be used() at the end of the GC. I know used() is not perfect, but for G1 this is the best "cheap" value we have for liveness at the end of any GC. >>> >>> Mostly because `used()` will report all live instances and potential garbage and will make it inconsistent with what the other GCs would report. >>> >> The other STW GCs do report the same, right? >> >>> > So as a middle road I would suggest to update G1CollectedHeap::gc_epilogue(bool full) to include: >>> > set_live(used()); >>> > With this you don't need the changes for the G1FullCollector. The liveness calculated at Remark would be used until the next young collection and I think here is where some improvements could be made. During the mixed phase a better solution would make use of the liveness information we have for the old regions together with what is newly allocated, but this needs further investigation. >>> >>> This sounds interesting. Let me try this out. >> >> Glad you like the idea :) I did a quick test locally and it shows the trend ok, even if it is an over estimate of live: >> live = 1.1 GB >> live = 1.2 GB >> live = 1.5 GB >> live = 1.7 GB >> live = 2.1 GB >> name = "G1Old" >> live = 1.4 GB >> live = 1.6 GB >> live = 1.8 GB >> live = 2.0 GB >> live = 2.3 GB >> live = 2.5 GB >> live = 2.8 GB >> live = 3.1 GB >> live = 3.3 GB >> live = 3.7 GB >> live = 4.0 GB >> live = 4.3 GB >> name = "G1Old" >> live = 1.2 GB >> G1Old is from concurrent mark events. > >> @kstefanj >> Just to make sure - `set_live(used())` should be the last call in `G1CollectedHeap::gc_prologue(bool full)` ? >> > Anywhere in there should be fine, we should look if there is anything related in there it could be grouped with. > >> I am getting some funny numbers with this change - basically, last known live size is getting bigger than the current used size ?? >> >> ![Screen Shot 2021-03-16 at 1 28 27 PM](https://user-images.githubusercontent.com/738413/111308939-97b06d80-865b-11eb-9e2f-d595b49d6401.png) > > This is strange, what kind of GCs are happening around those samples? Oh, sorry! I messed up, you should put the code in `gc_epilogue()` ?? ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Tue Mar 16 14:15:33 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 16 Mar 2021 14:15:33 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <8kTv1ENK1FbxIt7NVkNJNgT55bLw5ao4lWi 8uq0nktQ=.a969127a-351a-4393-a932-934fbe9b6924@github.com> Message-ID: <6T_A5WVMZCvYpceI82Ho8ltG-fLddego0zquojRykOI=.36314a7c-1e28-4e0c-8425-ab583d78c8d8@github.com> On Tue, 16 Mar 2021 13:25:35 GMT, Stefan Johansson wrote: > Oh, sorry! I messed up, you should put the code in gc_epilogue() ?? Np! Cool, this version works as expected. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From rkennke at redhat.com Tue Mar 16 15:18:50 2021 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 16 Mar 2021 16:18:50 +0100 Subject: RFR(11u): 8257621: JFR StringPool misses cached items across consecutive recordings In-Reply-To: References: Message-ID: Any takers? This is becoming relevant for some Graal work... :-) Cheers, Roman > Can I please get a review of the following backport: > > Original issue: > https://bugs.openjdk.java.net/browse/JDK-8257621 > > The original fix: > https://github.com/openjdk/jdk16/commit/7aac4dc1 > > JDK11u webrev: > http://cr.openjdk.java.net/~rkennke/JDK-8257621-jdk11/webrev.00/ > > None of the code that has been changed to use jfrSignal in jdk16 seems > to exist in jdk11, so I left those parts out. I am not totally sure if > JfrStringPool::add() is correct. > > I successfully ran JFR tests, including TestReEnableName.java. > > Thanks, > Roman From jaroslav.bachorik at datadoghq.com Tue Mar 16 17:16:11 2021 From: jaroslav.bachorik at datadoghq.com (=?UTF-8?Q?Jaroslav_Bachor=C3=ADk?=) Date: Tue, 16 Mar 2021 18:16:11 +0100 Subject: RFR(11u): 8257621: JFR StringPool misses cached items across consecutive recordings In-Reply-To: References: Message-ID: Hi Roman, AFAICT, this looks good. The changes related to jfrSignal usage seem to be only cosmetic (encapsulating a common logic) and not required for this backport. I checked the diffs and everything else in JDK 11 backport fully corresponds to the original patch. Looks good! -JB- -JB- On Tue, Mar 16, 2021 at 4:19 PM Roman Kennke wrote: > > Any takers? This is becoming relevant for some Graal work... :-) > > Cheers, > Roman > > > Can I please get a review of the following backport: > > > > Original issue: > > https://bugs.openjdk.java.net/browse/JDK-8257621 > > > > The original fix: > > https://github.com/openjdk/jdk16/commit/7aac4dc1 > > > > JDK11u webrev: > > http://cr.openjdk.java.net/~rkennke/JDK-8257621-jdk11/webrev.00/ > > > > None of the code that has been changed to use jfrSignal in jdk16 seems > > to exist in jdk11, so I left those parts out. I am not totally sure if > > JfrStringPool::add() is correct. > > > > I successfully ran JFR tests, including TestReEnableName.java. > > > > Thanks, > > Roman > From stefank at openjdk.java.net Wed Mar 17 11:10:59 2021 From: stefank at openjdk.java.net (Stefan Karlsson) Date: Wed, 17 Mar 2021 11:10:59 GMT Subject: RFR: 8263725: JFR oldobject tests are not run when GCs are specified explicitly Message-ID: The tests are tagged with @requires == "null", which has the effect that if you run with GC explicitly set, like: make -C ../build/fastdebug test TEST=test/jdk/jdk/jfr/event/oldobject/TestZ.java JTREG="JAVA_OPTIONS=-XX:+UseG1GC" then the tests won't run. The intention has probably been to only run the test with the default GC. I propose that we remove this and run the tests with other GCs as well. I've tested this locally with ZGC, G1, Parallel and Serial. The following test doesn't work with ZGC and has been disabled for that GC: test/jdk/jdk/jfr/event/oldobject/TestObjectDescription.java The following test doesn't work with Serial and has been disabled for that GC test/jdk/jdk/jfr/event/oldobject/TestClassLoaderLeak.java ------------- Commit messages: - 8263725: JFR oldobject tests are not run when GCs are specified explicitly Changes: https://git.openjdk.java.net/jdk/pull/3046/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3046&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263725 Stats: 19 lines in 19 files changed: 0 ins; 14 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/3046.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3046/head:pull/3046 PR: https://git.openjdk.java.net/jdk/pull/3046 From tschatzl at openjdk.java.net Wed Mar 17 11:38:47 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Wed, 17 Mar 2021 11:38:47 GMT Subject: RFR: 8263725: JFR oldobject tests are not run when GCs are specified explicitly In-Reply-To: References: Message-ID: On Wed, 17 Mar 2021 11:05:07 GMT, Stefan Karlsson wrote: > The tests are tagged with @requires == "null", which has the effect that if you run with GC explicitly set, like: > make -C ../build/fastdebug test TEST=test/jdk/jdk/jfr/event/oldobject/TestZ.java JTREG="JAVA_OPTIONS=-XX:+UseG1GC" > > then the tests won't run. The intention has probably been to only run the test with the default GC. I propose that we remove this and run the tests with other GCs as well. > > I've tested this locally with ZGC, G1, Parallel and Serial. > > The following test doesn't work with ZGC and has been disabled for that GC: > test/jdk/jdk/jfr/event/oldobject/TestObjectDescription.java > > The following test doesn't work with Serial and has been disabled for that GC > test/jdk/jdk/jfr/event/oldobject/TestClassLoaderLeak.java Lgtm. ------------- Marked as reviewed by tschatzl (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3046 From mgronlun at openjdk.java.net Wed Mar 17 12:16:50 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Wed, 17 Mar 2021 12:16:50 GMT Subject: RFR: 8263725: JFR oldobject tests are not run when GCs are specified explicitly In-Reply-To: References: Message-ID: <6HE6jSMHRM79tVnZ317Tn5_oZoV3_czWF7ktB2E9ED8=.74a72339-1939-4ddb-9f6a-18ca43528be8@github.com> On Wed, 17 Mar 2021 11:05:07 GMT, Stefan Karlsson wrote: > The tests are tagged with @requires == "null", which has the effect that if you run with GC explicitly set, like: > make -C ../build/fastdebug test TEST=test/jdk/jdk/jfr/event/oldobject/TestZ.java JTREG="JAVA_OPTIONS=-XX:+UseG1GC" > > then the tests won't run. The intention has probably been to only run the test with the default GC. I propose that we remove this and run the tests with other GCs as well. > > I've tested this locally with ZGC, G1, Parallel and Serial. > > The following test doesn't work with ZGC and has been disabled for that GC: > test/jdk/jdk/jfr/event/oldobject/TestObjectDescription.java > > The following test doesn't work with Serial and has been disabled for that GC > test/jdk/jdk/jfr/event/oldobject/TestClassLoaderLeak.java Looks good. Thanks. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3046 From tschatzl at openjdk.java.net Wed Mar 17 17:21:51 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Wed, 17 Mar 2021 17:21:51 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> Message-ID: <3FYRI-abe_Q_yAVKNJad9uB-TOTFD9p22wO7VW5ojH8=.6f73c0f0-aa0a-44fe-b38c-a8758c9cb46d@github.com> On Tue, 16 Mar 2021 12:23:52 GMT, Jaroslav Bachorik wrote: > > Also, for this purpose, why would used() not be a good substitute for liveness? If e.g. used() average grows over time you can deduce the same I would assume (particularly used() after mixed gc phase in g1). > > The major problem is that eg. for g1 given large enough heap the used value can keep on growing for quite long time, possibly generating wrong signal about potential memory leak. So can this liveness estimate - just filter used() a bit more. Actually I strongly believe that due to containing more details, used() could be more appropriate for this: first, it updates more often, second, information like distance between valleys and peaks of the sawtooth pattern are indicative of memory running away. If you really want to limit yourselves to something that is similar in update frequency to this liveness estimate, one could use just one value in a "tooth". However I think just strong enough low-pass filtering enough is as fine. > > If the live estimate is set to `used()` after mixed gc phase in g1 I think it still will be a good estimate. > The only thing I am opposing is having `live()` call return the current `used()` value which, IMO, might become rather confusing. > > > Do you have any numbers on what the impact of using used() vs. this live() would be in such a use case? > > Nope. Do you mean perf impact? Impact on false positives/negatives. > > > What I'm afraid of is that mixing values taken at different times - used and capacity are taken at the time of the event, and the liveness estimated updated at other, irregular intervals may cause significiant amount of confusion in interpreting this value. It might be obvious to you, but there will be other users. > > IDK. If the event field would explicitly mention that this is the **last known live size estimate** it should set the expectations right. > > > One option could be detaching the liveness estimate from used()/capacity() (I see a value in having some heap usage summary at regular intervals) and send the liveness estimate event just when they are generated? Then the various collectors could send this liveness value at times when they think they are fairly accurate, not when the collectors must and particularly not in conjunction with samples taken at completely different times. > > The problem is the irregularity - when the live size is reported only when it is calculated there might be long periods in the recording missing the live size data at all. In order for this information to be useful it should be reported at least at the beginning and end of a JFR chunk. > > > Independent of whether used/capacity and liveness are sent, the receiver needs to do statistics (trend lines) on those anyway. > > Yes. It's just that with the live size estimate one wouldn't be getting the false positives one would get with used heap trend. We've now discussed this issue within the gc team a bit and came to the following conclusions. Before going into that, our guidelines for adding new tracing code: generally we avoid providing functionality in the VM/GC that can be procured easily otherwise or there is a good substitute, particularly if their actual content is unclear. The VM is also generally not the place to store or accumulate data that is only used in external applications for their convenience, particularly where the general public usefulness is questionable or do not drive forward GC algorithms in some way. In the past in the cases we have done this a few times, and this has caused lots of maintenance burden (adding it, keeping it up to date, and finally removing it because nobody used it after all). We are open to providing raw data for events that fits this in the most painless way for the VM if they are well specified. The whole periodic HeapSummary event and its contents are questionable in this light: - used() and capacity() are provided "regularly", and it could either be retrieved at any time by other means (e.g. MXBeans), or even forced to be provided (cause a gc if you are really desperate). Particularly in cases of continuous monitoring, there should be no problem getting them even with existing JFR events. - it can be argued that it is *very* painless for GC to provide the periodic used/capacity, their values are well defined for all collectors. Still I believe particularly if you do continuous monitoring, this is kind of unnecessary and should be polled instead of pushed if required at higher frequency as provided now. - the suggested "liveness estimate" however goes against all of these guidelines: - the value is ill-defined if at all. The quality of the implementations is all over the place: - Epsilon the used() value - Serial the used() value and sometimes some attempt to actually return the live data using the dead-wood heuristic - Parallel returns used() always - G1 returns the amount of bytes marked plus the bytes allocated while marking, used() in other cases (although that may change) - Z returns the amount of bytes marked without the bytes allocated while marking (this is unclear to me actually) - Shenandoah seems to be fairly close to G1, I have no idea what the results are on the various additional modes it uses. - for those collectors that can not give you a good value, the application could as well easily generate it - just use used(). (That deadwood optimization does not change the situation a lot as the difference would be at most 5% or so difference, well within "estimate" range). - it forces gcs that do not use or need that value at all first calculate it and then keep it around for just this event - this liveness estimate, which is outdated a few instructions after the application runs, is coupled with current used()/capacity() values - there is no indication if that estimate in that event can actually used for the suggested purpose: It could have been calculated at any point in time, so its use for trend lines is limited (e.g.. regression etc). For such a regression you typically need multiple values anyway, and even more for some output with a significant amount of considence, so that single value without timestamp does not seem to help. If you track continuously, you would get all values anyway. So overall I believe the current suggestion to have the VM provide all these values and the event is just introducing complexity in the VM for convenience of the application. Still I think there might be need for the raw data if available, and if it's easily obtainable, then fine, do send some duplicate data. So my suggestion and what we in the gc team can support is to a) provide that HeapSummary event with capacity() and used() (but as mentioned, on a change they are sent out already so I do not see the exact situation in particular with continuous tracking...). b) provide some Liveness (or "Marked bytes" or similar) event when the value is generated (if they are generated) as a one-off event in places it can be derived without significant costs. I.e. some "send_marked_bytes()/live_bytes()" method to be called in Serial, Parallel, G1, Z and Shenandoah when generated and available. No additional storage of that value in the and repeated emission of that event by the VM. I'm intentionally using "marked bytes" here because this is a value that can actually be defined and verified by reviewers that it's actually returned. Some best effort estimate is just misleading, and is a pain to maintain and argue whether the goal has been met (and it's even worse to (dis-)prove that a change introduced a bug). Maybe it could be extended to "marked bytes plus allocated during marking, sent when marking finished" for all collectors that do marking - I do not know (particularly for Z), maybe it can. This approach also allows anyone to easily incrementally add new occurrences of that event as more code to support it has been written (e.g. with PR #2966 for g1 full gc), and allows leaving out collectors that do not support it, or places where this has not been gathered. We think this may be a useful value, that can be explained to others, will cause minimal misunderstanding, can be verified at least in the reviews (and tests where sending of that event is verified for the various situations it should be sent), and maintained. Returning "liveness" in a good way (both in accuracy and overhead) is a completely different issue, and probably worth a few PhDs. Just dodging the issue with appending "estimate" to the name is not the fix given alternatives. I would further ask you to at least create two different CRs for adding the two events (more for later additions of the event) for easier and faster review. You can provide a link to a "all-in" diff to let the reviewers see where you want to go with this. However having non-trivial changes across (at this point) 34 files for different collectors for different reasons is nontrivial and very exhausting to review and re-review (10 times so far at least for me). Thanks, Thomas ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From stefank at openjdk.java.net Thu Mar 18 06:55:47 2021 From: stefank at openjdk.java.net (Stefan Karlsson) Date: Thu, 18 Mar 2021 06:55:47 GMT Subject: RFR: 8263725: JFR oldobject tests are not run when GCs are specified explicitly In-Reply-To: <6HE6jSMHRM79tVnZ317Tn5_oZoV3_czWF7ktB2E9ED8=.74a72339-1939-4ddb-9f6a-18ca43528be8@github.com> References: <6HE6jSMHRM79tVnZ317Tn5_oZoV3_czWF7ktB2E9ED8=.74a72339-1939-4ddb-9f6a-18ca43528be8@github.com> Message-ID: On Wed, 17 Mar 2021 12:14:20 GMT, Markus Gr?nlund wrote: >> The tests are tagged with @requires == "null", which has the effect that if you run with GC explicitly set, like: >> make -C ../build/fastdebug test TEST=test/jdk/jdk/jfr/event/oldobject/TestZ.java JTREG="JAVA_OPTIONS=-XX:+UseG1GC" >> >> then the tests won't run. The intention has probably been to only run the test with the default GC. I propose that we remove this and run the tests with other GCs as well. >> >> I've tested this locally with ZGC, G1, Parallel and Serial. >> >> The following test doesn't work with ZGC and has been disabled for that GC: >> test/jdk/jdk/jfr/event/oldobject/TestObjectDescription.java >> >> The following test doesn't work with Serial and has been disabled for that GC >> test/jdk/jdk/jfr/event/oldobject/TestClassLoaderLeak.java > > Looks good. Thanks. Thanks for reviewing! ------------- PR: https://git.openjdk.java.net/jdk/pull/3046 From stefank at openjdk.java.net Thu Mar 18 06:55:49 2021 From: stefank at openjdk.java.net (Stefan Karlsson) Date: Thu, 18 Mar 2021 06:55:49 GMT Subject: Integrated: 8263725: JFR oldobject tests are not run when GCs are specified explicitly In-Reply-To: References: Message-ID: On Wed, 17 Mar 2021 11:05:07 GMT, Stefan Karlsson wrote: > The tests are tagged with @requires == "null", which has the effect that if you run with GC explicitly set, like: > make -C ../build/fastdebug test TEST=test/jdk/jdk/jfr/event/oldobject/TestZ.java JTREG="JAVA_OPTIONS=-XX:+UseG1GC" > > then the tests won't run. The intention has probably been to only run the test with the default GC. I propose that we remove this and run the tests with other GCs as well. > > I've tested this locally with ZGC, G1, Parallel and Serial. > > The following test doesn't work with ZGC and has been disabled for that GC: > test/jdk/jdk/jfr/event/oldobject/TestObjectDescription.java > > The following test doesn't work with Serial and has been disabled for that GC > test/jdk/jdk/jfr/event/oldobject/TestClassLoaderLeak.java This pull request has now been integrated. Changeset: 72b82fd7 Author: Stefan Karlsson URL: https://git.openjdk.java.net/jdk/commit/72b82fd7 Stats: 19 lines in 19 files changed: 0 ins; 14 del; 5 mod 8263725: JFR oldobject tests are not run when GCs are specified explicitly Reviewed-by: tschatzl, mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3046 From tschatzl at openjdk.java.net Thu Mar 18 08:36:50 2021 From: tschatzl at openjdk.java.net (Thomas Schatzl) Date: Thu, 18 Mar 2021 08:36:50 GMT Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <3FYRI-abe_Q_yAVKNJad9uB-TOTFD9p22wO7VW5ojH8=.6f73c0f0-aa0a-44fe-b38c-a8758c9cb46d@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> <3FYRI-abe_Q_yAVKNJad9uB-TOTFD9p22wO7VW5ojH8=.6f73c0f0-aa0a-44fe-b38c-a8758c9cb46d@github.com> Message-ID: On Wed, 17 Mar 2021 17:19:06 GMT, Thomas Schatzl wrote: > Still I think there might be need for the raw data if available, and if it's easily obtainable, then fine, do send some duplicate data. So my suggestion and what we in the gc team can support is to > > a) provide that HeapSummary event with capacity() and used() (but as mentioned, on a change they are sent out already so I do not see the exact situation in particular with continuous tracking...). There has been some question in the latter part of this statement: the "but as mentioned...." part refers to the situation that if you are already continuously monitoring, you will get all of the `used()` events the VM currently sends anyway (if subscribed). So this periodic event does not give you more information. There may be need for sending `used()` in particular more often as it is done now (and I am open to somebody improving this), not for convenience but because something interesting happens in the Java heap. I am not sure that a "JFR chunk" is the right periodicity though, because it can potentially mean sending (assuming that a "chunk" is some fixed amount of events) every ms to every hour or day. This "random" sampling may send events both too often and too infrequent and not when it matters. Thomas ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From jaroslav.bachorik at datadoghq.com Thu Mar 18 09:26:53 2021 From: jaroslav.bachorik at datadoghq.com (=?UTF-8?Q?Jaroslav_Bachor=C3=ADk?=) Date: Thu, 18 Mar 2021 10:26:53 +0100 Subject: RFR: 8258431: Provide a JFR event with live set size estimate [v12] In-Reply-To: <3FYRI-abe_Q_yAVKNJad9uB-TOTFD9p22wO7VW5ojH8=.6f73c0f0-aa0a-44fe-b38c-a8758c9cb46d@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> <_XwpqPW-sKwuvZG26bSgdW6RKtQaIJz-0fpRR5wGa0c=.d6c5394b-7fea-4f0e-ab5f-9092286d3db1@github.com> <3FYRI-abe_Q_yAVKNJad9uB-TOTFD9p22wO7VW5ojH8=.6f73c0f0-aa0a-44fe-b38c-a8758c9cb46d@github.com> Message-ID: First of all, it is a pity my [attempt to open a discussion regarding this topic](https://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2021-February/033705.html) didn't get more attention. On Wed, Mar 17, 2021 at 6:22 PM Thomas Schatzl wrote: > > On Tue, 16 Mar 2021 12:23:52 GMT, Jaroslav Bachorik wrote: > > > > Also, for this purpose, why would used() not be a good substitute for liveness? If e.g. used() average grows over time you can deduce the same I would assume (particularly used() after mixed gc phase in g1). > > > > The major problem is that eg. for g1 given large enough heap the used value can keep on growing for quite long time, possibly generating wrong signal about potential memory leak. > > So can this liveness estimate - just filter used() a bit more. > > Actually I strongly believe that due to containing more details, used() could be more appropriate for this: first, it updates more often, second, information like distance between valleys and peaks of the sawtooth pattern are indicative of memory running away. > If you really want to limit yourselves to something that is similar in update frequency to this liveness estimate, one could use just one value in a "tooth". However I think just strong enough low-pass filtering enough is as fine. > > > > > If the live estimate is set to `used()` after mixed gc phase in g1 I think it still will be a good estimate. > > The only thing I am opposing is having `live()` call return the current `used()` value which, IMO, might become rather confusing. > > > > > Do you have any numbers on what the impact of using used() vs. this live() would be in such a use case? > > > > Nope. Do you mean perf impact? > > Impact on false positives/negatives. > > > > > > What I'm afraid of is that mixing values taken at different times - used and capacity are taken at the time of the event, and the liveness estimated updated at other, irregular intervals may cause significiant amount of confusion in interpreting this value. It might be obvious to you, but there will be other users. > > > > IDK. If the event field would explicitly mention that this is the **last known live size estimate** it should set the expectations right. > > > > > One option could be detaching the liveness estimate from used()/capacity() (I see a value in having some heap usage summary at regular intervals) and send the liveness estimate event just when they are generated? Then the various collectors could send this liveness value at times when they think they are fairly accurate, not when the collectors must and particularly not in conjunction with samples taken at completely different times. > > > > The problem is the irregularity - when the live size is reported only when it is calculated there might be long periods in the recording missing the live size data at all. In order for this information to be useful it should be reported at least at the beginning and end of a JFR chunk. > > > > > Independent of whether used/capacity and liveness are sent, the receiver needs to do statistics (trend lines) on those anyway. > > > > Yes. It's just that with the live size estimate one wouldn't be getting the false positives one would get with used heap trend. > > We've now discussed this issue within the gc team a bit and came to the following conclusions. > > Before going into that, our guidelines for adding new tracing code: generally we avoid providing functionality in the VM/GC that can be procured easily otherwise or there is a good substitute, particularly if their actual content is unclear. The VM is also generally not the place to store or accumulate data that is only used in external applications for their convenience, particularly where the general public usefulness is questionable or do not drive forward GC algorithms in some way. > > In the past in the cases we have done this a few times, and this has caused lots of maintenance burden (adding it, keeping it up to date, and finally removing it because nobody used it after all). > > We are open to providing raw data for events that fits this in the most painless way for the VM if they are well specified. > > The whole periodic HeapSummary event and its contents are questionable in this light: > > - used() and capacity() are provided "regularly", and it could either be retrieved at any time by other means (e.g. MXBeans), or even forced to be provided (cause a gc if you are really desperate). Particularly in cases of continuous monitoring, there should be no problem getting them even with existing JFR events. These values might be quite imprecise - it is not uncommon for G1 to see heap usage increasing for long time which could falsely trigger a leak alarm. Also, triggering GC does not seem to be that great idea considering that it will cause a rather lengthy safepoint. > - it can be argued that it is *very* painless for GC to provide the periodic used/capacity, their values are well defined for all collectors. Still I believe particularly if you do continuous monitoring, this is kind of unnecessary and should be polled instead of pushed if required at higher frequency as provided now. Well, we are doing continuous monitoring and it is not unnecessary .. > > - the suggested "liveness estimate" however goes against all of these guidelines: > - the value is ill-defined if at all. The quality of the implementations is all over the place: > - Epsilon the used() value There is literally no other information for epsilon if I am not totally mistaken. > - Serial the used() value and sometimes some attempt to actually return the live data using the dead-wood heuristic > - Parallel returns used() always > - G1 returns the amount of bytes marked plus the bytes allocated while marking, used() in other cases (although that may change) > - Z returns the amount of bytes marked without the bytes allocated while marking (this is unclear to me actually) > - Shenandoah seems to be fairly close to G1, I have no idea what the results are on the various additional modes it uses. Yes. And this was my initial question in the email thread I started before even attempting this PR. Whatever limited response I got seemed to confirm this way of getting the liveness estimate. > - for those collectors that can not give you a good value, the application could as well easily generate it - just use used(). (That deadwood optimization does not change the situation a lot as the difference would be at most 5% or so difference, well within "estimate" range). > - it forces gcs that do not use or need that value at all first calculate it and then keep it around for just this event > - this liveness estimate, which is outdated a few instructions after the application runs, is coupled with current used()/capacity() values > - there is no indication if that estimate in that event can actually used for the suggested purpose: It could have been calculated at any point in time, so its use for trend lines is limited (e.g.. regression etc). For such a regression you typically need multiple values anyway, and even more for some output with a significant amount of considence, so that single value without timestamp does not seem to help. If you track continuously, you would get all values anyway. > > So overall I believe the current suggestion to have the VM provide all these values and the event is just introducing complexity in the VM for convenience of the application. > > Still I think there might be need for the raw data if available, and if it's easily obtainable, then fine, do send some duplicate data. So my suggestion and what we in the gc team can support is to > > a) provide that HeapSummary event with capacity() and used() (but as mentioned, on a change they are sent out already so I do not see the exact situation in particular with continuous tracking...). > > b) provide some Liveness (or "Marked bytes" or similar) event when the value is generated (if they are generated) as a one-off event in places it can be derived without significant costs. I.e. some "send_marked_bytes()/live_bytes()" method to be called in Serial, Parallel, G1, Z and Shenandoah when generated and available. No additional storage of that value in the and repeated emission of that event by the VM. > I'm intentionally using "marked bytes" here because this is a value that can actually be defined and verified by reviewers that it's actually returned. Some best effort estimate is just misleading, and is a pain to maintain and argue whether the goal has been met (and it's even worse to (dis-)prove that a change introduced a bug). Maybe it could be extended to "marked bytes plus allocated during marking, sent when marking finished" for all collectors that do marking - I do not know (particularly for Z), maybe it can. > This approach also allows anyone to easily incrementally add new occurrences of that event as more code to support it has been written (e.g. with PR #2966 for g1 full gc), and allows leaving out collectors that do not support it, or places where this has not been gathered. > > We think this may be a useful value, that can be explained to others, will cause minimal misunderstanding, can be verified at least in the reviews (and tests where sending of that event is verified for the various situations it should be sent), and maintained. > > Returning "liveness" in a good way (both in accuracy and overhead) is a completely different issue, and probably worth a few PhDs. Just dodging the issue with appending "estimate" to the name is not the fix given alternatives. > > I would further ask you to at least create two different CRs for adding the two events (more for later additions of the event) for easier and faster review. You can provide a link to a "all-in" diff to let the reviewers see where you want to go with this. However having non-trivial changes across (at this point) 34 files for different collectors for different reasons is nontrivial and very exhausting to review and re-review (10 times so far at least for me). I am sorry for wasting yours and other reviewers time. I really tried getting some consensus before coming with a half-baked PR but as you can see the responses start flowing only when the PR is 'almost done'. I am sensing quite negative attitude toward this change and the whole idea of capturing the liveness estimate. At the same time the solutions you propose give me nothing over just doing custom JFR event generation using data obtained via JMX - so I really don't see a point in doing this change in hotspot. I am going to close this PR and I apologize again for wasting time of all involved engineers. Regards, -JB- > > Thanks, > Thomas > > ------------- > > PR: https://git.openjdk.java.net/jdk/pull/2579 From jbachorik at openjdk.java.net Thu Mar 18 09:32:52 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Thu, 18 Mar 2021 09:32:52 GMT Subject: Withdrawn: 8258431: Provide a JFR event with live set size estimate In-Reply-To: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> References: <7HVs4jngEbNIQIPQByuE6IRYAxdijfa82uhEFWHld5U=.a7784482-d7e1-4d59-88ee-455d8691631e@github.com> Message-ID: On Mon, 15 Feb 2021 17:23:44 GMT, Jaroslav Bachorik wrote: > The purpose of this change is to expose a 'cheap' estimate of the current live set size (the meaning of 'current' is dependent on each particular GC implementation but in worst case 'at last full GC') in form of a periodically emitted JFR event. > > ## Introducing new JFR event > > While there is already 'GC Heap Summary' JFR event it does not fit the requirements as it is closely tied to GC cycle so eg. for ZGC or Shenandoah it may not happen for quite a long time, increasing the risk of not having the heap summary events being present in the JFR recording at all. > Because of this I am proposing to add a new 'Heap Usage Summary' event which will be emitted periodically, by default on each JFR chunk, and will contain the information abut the heap capacity, the used and live bytes. This information is available from all GC implementations and can be provided at literally any time. > > ## Implementation > > The implementation differs from GC to GC because each GC algorithm/implementation provides a slightly different way to track the liveness. The common part is `size_t live() const` method added to `CollectedHeap` superclass and the use of a cached 'liveness' value computed after the last GC cycle. If `liveness` hasn't been calculated yet the implementation will default to returning 'used' value. > > The implementations are based on my (rather shallow) knowledge of inner working of the respective GC engines and I am open to suggestions to make them better/correct. > > ### Epsilon GC > > Trivial implementation - just return `used()` instead. > > ### Serial GC > > Here we utilize the fact that mark-copy phase is naturally compacting so the number of bytes after copy is 'live' and that the mark-sweep implementation keeps an internal info about objects being 'dead' but excluded from the compaction effort and we can these numbers to derive the old-gen live set size (used bytes minus the cumulative size of the 'un-dead' objects). > > ### Parallel GC > > For Parallel GC the liveness is calculated as the sum of used bytes in all regions after the last GC cycle. This seems to be a safe bet because this collector is always compacting (AFAIK). > > ### G1 GC > > Using `G1ConcurrentMark::remark()` method the live set size is computed as a sum of `_live_words` from the associated `G1RegionMarkStats` objects. Here I am not 100% sure this approach covers all eventualities and it would be great to have someone skilled in G1 implementation to chime in so I can fix it. However, the numbers I am getting for G1 are comparable to other GCs for the same application. > > ### Shenandoah > > In Shenandoah, the regions are keeping the liveness info. However, the VM op that is used for iterating regions is a safe-pointing one so it would be great to run it in an already safe-pointed context. > This leads to hooking into `ShenandoahConcurrentMark::finish_mark()` and `ShenandoahSTWMark::mark()` where at the end of the marking process the liveness info is summarized and set to `ShenandoahHeap::_live` volatile field - which is later read by the event emitting code. > > ### ZGC > > `ZStatHeap` is already holding the liveness info - so this implementation is just making it accessible via `ZCollectedHeap::live()` method. This pull request has been closed without being integrated. ------------- PR: https://git.openjdk.java.net/jdk/pull/2579 From ddong at openjdk.java.net Fri Mar 19 02:31:39 2021 From: ddong at openjdk.java.net (Denghui Dong) Date: Fri, 19 Mar 2021 02:31:39 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) In-Reply-To: References: Message-ID: On Fri, 26 Feb 2021 00:19:12 GMT, Markus Gr?nlund wrote: > Hi Denghui, > > I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. > > Can you do a test run with this suggestion to see if it works for you? > > Thanks > Markus LGTM, although I'm not a reviewer :) ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From jbachorik at openjdk.java.net Fri Mar 19 11:35:40 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Fri, 19 Mar 2021 11:35:40 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v3] In-Reply-To: References: Message-ID: On Fri, 12 Mar 2021 10:45:31 GMT, Markus Gr?nlund wrote: >> Hi Denghui, >> >> I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. >> >> Can you do a test run with this suggestion to see if it works for you? >> >> Thanks >> Markus > > Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: > > update Marked as reviewed by jbachorik (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From egahlin at openjdk.java.net Fri Mar 19 13:10:40 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Fri, 19 Mar 2021 13:10:40 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v3] In-Reply-To: References: Message-ID: <7rAkbAZJqDuGDsLenxolc4hXPbVPxuwG8xJinqbqThg=.e4b186ed-b27f-4d94-a30d-3f88316c22bb@github.com> On Fri, 12 Mar 2021 10:45:31 GMT, Markus Gr?nlund wrote: >> Hi Denghui, >> >> I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. >> >> Can you do a test run with this suggestion to see if it works for you? >> >> Thanks >> Markus > > Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: > > update Marked as reviewed by egahlin (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From mgronlun at openjdk.java.net Fri Mar 19 16:33:45 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 19 Mar 2021 16:33:45 GMT Subject: RFR: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) [v3] In-Reply-To: <7rAkbAZJqDuGDsLenxolc4hXPbVPxuwG8xJinqbqThg=.e4b186ed-b27f-4d94-a30d-3f88316c22bb@github.com> References: <7rAkbAZJqDuGDsLenxolc4hXPbVPxuwG8xJinqbqThg=.e4b186ed-b27f-4d94-a30d-3f88316c22bb@github.com> Message-ID: <39BuZHklapGNXO8CwYjRJnQcO7o_Qg8G1mzYKCP3-Zk=.121697ce-ac01-4603-901c-3ad647dfe813@github.com> On Fri, 19 Mar 2021 13:07:24 GMT, Erik Gahlin wrote: >> Markus Gr?nlund has updated the pull request incrementally with one additional commit since the last revision: >> >> update > > Marked as reviewed by egahlin (Reviewer). Thanks for reviewing! ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From egahlin at openjdk.java.net Sun Mar 21 01:55:15 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Sun, 21 Mar 2021 01:55:15 GMT Subject: RFR: 8260862 JFR: New configure command for the jfr tool Message-ID: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> Hi, Could I have a review of a new configure command for the jfr tool. Purpose is to simplify changing event settings: $ jfr configure method-profiling=high gc=detailed Configuration written successfully to: For more details, see CSR: https://bugs.openjdk.java.net/browse/JDK-8260863 Testing: - jdk/jdk/jfr - JMC recording wizard Thanks Erik ------------- Commit messages: - Remove tabs - Remove carriage return - Improve formatting - Remove mistakenly added file - Fix typo - Initial Changes: https://git.openjdk.java.net/jdk/pull/3100/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3100&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8260862 Stats: 3232 lines in 43 files changed: 2961 ins; 54 del; 217 mod Patch: https://git.openjdk.java.net/jdk/pull/3100.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3100/head:pull/3100 PR: https://git.openjdk.java.net/jdk/pull/3100 From mgronlun at openjdk.java.net Mon Mar 22 11:55:51 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Mon, 22 Mar 2021 11:55:51 GMT Subject: Integrated: 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) In-Reply-To: References: Message-ID: On Fri, 26 Feb 2021 00:19:12 GMT, Markus Gr?nlund wrote: > Hi Denghui, > > I did not manage to find a solution that would could reduce verbosity, but i did find one, although a bit involved, that lets us keep it local to and integrated in the core of JFR itself. > > Can you do a test run with this suggestion to see if it works for you? > > Thanks > Markus This pull request has now been integrated. Changeset: a9d2267f Author: Markus Gr?nlund URL: https://git.openjdk.java.net/jdk/commit/a9d2267f Stats: 244 lines in 9 files changed: 211 ins; 19 del; 14 mod 8260589: Crash in JfrTraceIdLoadBarrier::load(_jclass*) Co-authored-by: Denghui Dong Reviewed-by: jbachorik, egahlin ------------- PR: https://git.openjdk.java.net/jdk/pull/2738 From jbachorik at openjdk.java.net Mon Mar 22 16:05:15 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Mon, 22 Mar 2021 16:05:15 GMT Subject: RFR: 8203359: Container level resources events Message-ID: With this change it becomes possible to surface various cgroup level metrics (available via `jdk.internal.platform.Metrics`) as JFR events. Only a subset of the metrics exposed by `jdk.internal.platform.Metrics` is turned into JFR events to start with. * CPU related metrics * Memory related metrics * I/O related metrics For each of those subsystems a configuration data will be emitted as well. The initial proposal is to emit the configuration data events at least once per chunk and the metrics values at 30 seconds interval. By using these values the emitted events seem to contain useful information without increasing overhead (the metrics values are read from `/proc` filesystem so that should not be done too frequently). ------------- Commit messages: - Formatting spaces - 8203359: Container level resources events Changes: https://git.openjdk.java.net/jdk/pull/3126/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3126&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8203359 Stats: 389 lines in 8 files changed: 386 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/3126.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3126/head:pull/3126 PR: https://git.openjdk.java.net/jdk/pull/3126 From egahlin at openjdk.java.net Mon Mar 22 18:47:02 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Mon, 22 Mar 2021 18:47:02 GMT Subject: RFR: 8260565: JFR: Fix copyright header in tests Message-ID: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> Hi, Could I have a review of a fix that removes "Classpath" exception in the copyright header from the JFR tests. I had to update jdk.jfr.event.compilerTestDeoptimization.java because it relied on the line number of the file. A missing line ending was added at the end to some files as well. Testing: jdk/jdk/jfr + tier1,tier2 Thanks Erik ------------- Commit messages: - Initial Changes: https://git.openjdk.java.net/jdk/pull/3130/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3130&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8260565 Stats: 1664 lines in 552 files changed: 0 ins; 1104 del; 560 mod Patch: https://git.openjdk.java.net/jdk/pull/3130.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3130/head:pull/3130 PR: https://git.openjdk.java.net/jdk/pull/3130 From mseledtsov at openjdk.java.net Mon Mar 22 21:57:42 2021 From: mseledtsov at openjdk.java.net (Mikhailo Seledtsov) Date: Mon, 22 Mar 2021 21:57:42 GMT Subject: RFR: 8260565: JFR: Fix copyright header in tests In-Reply-To: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> References: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> Message-ID: On Mon, 22 Mar 2021 18:26:08 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a fix that removes "Classpath" exception in the copyright header from the JFR tests. I had to update jdk.jfr.event.compilerTestDeoptimization.java because it relied on the line number of the file. A missing line ending was added at the end to some files as well. > > Testing: jdk/jdk/jfr + tier1,tier2 > > Thanks > Erik Changes look good to me ------------- Marked as reviewed by mseledtsov (Committer). PR: https://git.openjdk.java.net/jdk/pull/3130 From never at openjdk.java.net Tue Mar 23 08:36:53 2021 From: never at openjdk.java.net (Tom Rodriguez) Date: Tue, 23 Mar 2021 08:36:53 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling Message-ID: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> 8264017: Correctly report inlined frame in JFR sampling ------------- Commit messages: - 8264017: Correctly report inlined frame in JFR sampling Changes: https://git.openjdk.java.net/jdk/pull/3148/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3148&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264017 Stats: 20 lines in 1 file changed: 17 ins; 3 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/3148.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3148/head:pull/3148 PR: https://git.openjdk.java.net/jdk/pull/3148 From jbachorik at openjdk.java.net Tue Mar 23 10:20:41 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Tue, 23 Mar 2021 10:20:41 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> Message-ID: On Tue, 23 Mar 2021 08:30:48 GMT, Tom Rodriguez wrote: > 8264017: Correctly report inlined frame in JFR sampling Marked as reviewed by jbachorik (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From mgronlun at openjdk.java.net Tue Mar 23 10:37:39 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Tue, 23 Mar 2021 10:37:39 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> Message-ID: On Tue, 23 Mar 2021 08:30:48 GMT, Tom Rodriguez wrote: > 8264017: Correctly report inlined frame in JFR sampling Hi Tom, looks good, thank you for this improvement. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3148 From github.com+76791+alblue at openjdk.java.net Tue Mar 23 19:26:51 2021 From: github.com+76791+alblue at openjdk.java.net (Alex Blewitt) Date: Tue, 23 Mar 2021 19:26:51 GMT Subject: RFR: 8263854: Used blessed modifier order for jdk.jfr Message-ID: This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. ------------- Commit messages: - Additional JFR modifier order fixes - Used blessed modifier order for jdk.jfr Changes: https://git.openjdk.java.net/jdk/pull/3161/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3161&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263854 Stats: 143 lines in 52 files changed: 0 ins; 0 del; 143 mod Patch: https://git.openjdk.java.net/jdk/pull/3161.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3161/head:pull/3161 PR: https://git.openjdk.java.net/jdk/pull/3161 From egahlin at openjdk.java.net Tue Mar 23 20:07:48 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Tue, 23 Mar 2021 20:07:48 GMT Subject: RFR: 8264001: JFR: Modernize implementation Message-ID: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> Hi, Could I have a review of changes that updates the Java implementation of JFR with newer language constructs and library calls. I also removed unused imports, dead code and added @Override where approriate. Testing: jdk/jdk/jfr ------------- Commit messages: - Update - Update - Initial Changes: https://git.openjdk.java.net/jdk/pull/3138/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3138&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264001 Stats: 346 lines in 57 files changed: 24 ins; 107 del; 215 mod Patch: https://git.openjdk.java.net/jdk/pull/3138.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3138/head:pull/3138 PR: https://git.openjdk.java.net/jdk/pull/3138 From mgronlun at openjdk.java.net Tue Mar 23 20:14:38 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Tue, 23 Mar 2021 20:14:38 GMT Subject: RFR: 8260565: JFR: Fix copyright header in tests In-Reply-To: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> References: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> Message-ID: On Mon, 22 Mar 2021 18:26:08 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a fix that removes "Classpath" exception in the copyright header from the JFR tests. I had to update jdk.jfr.event.compilerTestDeoptimization.java because it relied on the line number of the file. A missing line ending was added at the end to some files as well. > > Testing: jdk/jdk/jfr + tier1,tier2 > > Thanks > Erik Marked as reviewed by mgronlun (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/3130 From mgronlun at openjdk.java.net Tue Mar 23 20:37:41 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Tue, 23 Mar 2021 20:37:41 GMT Subject: RFR: 8264001: JFR: Modernize implementation In-Reply-To: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> References: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> Message-ID: On Mon, 22 Mar 2021 22:35:32 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of changes that updates the Java implementation of JFR with newer language constructs and library calls. I also removed unused imports, dead code and added @Override where approriate. > > Testing: jdk/jdk/jfr Looks good. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3138 From egahlin at openjdk.java.net Wed Mar 24 04:09:39 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Wed, 24 Mar 2021 04:09:39 GMT Subject: Integrated: 8260565: JFR: Fix copyright header in tests In-Reply-To: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> References: <_tfEwUjRcpjp7htiVqVBasKWtp7qrMNboUOggW5LiT0=.b1e37d27-281c-4443-9641-76f6c04f7edf@github.com> Message-ID: <_i8DPVK0R_7nhAS8ZhrGenh7eWKIx1GJ8XT1n-z8r54=.6470a77a-490d-419f-8744-66d04f9f387a@github.com> On Mon, 22 Mar 2021 18:26:08 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a fix that removes "Classpath" exception in the copyright header from the JFR tests. I had to update jdk.jfr.event.compilerTestDeoptimization.java because it relied on the line number of the file. A missing line ending was added at the end to some files as well. > > Testing: jdk/jdk/jfr + tier1,tier2 > > Thanks > Erik This pull request has now been integrated. Changeset: 8d63bb6a Author: Erik Gahlin URL: https://git.openjdk.java.net/jdk/commit/8d63bb6a Stats: 1664 lines in 552 files changed: 0 ins; 1104 del; 560 mod 8260565: JFR: Fix copyright header in tests Reviewed-by: mseledtsov, mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3130 From never at openjdk.java.net Wed Mar 24 06:09:38 2021 From: never at openjdk.java.net (Tom Rodriguez) Date: Wed, 24 Mar 2021 06:09:38 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> Message-ID: On Tue, 23 Mar 2021 10:34:51 GMT, Markus Gr?nlund wrote: >> 8264017: Correctly report inlined frame in JFR sampling > > Hi Tom, looks good, thank you for this improvement. Should I perform any additional testing related to JFR? ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From egahlin at openjdk.java.net Wed Mar 24 09:57:42 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Wed, 24 Mar 2021 09:57:42 GMT Subject: Integrated: 8264001: JFR: Modernize implementation In-Reply-To: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> References: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> Message-ID: On Mon, 22 Mar 2021 22:35:32 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of changes that updates the Java implementation of JFR with newer language constructs and library calls. I also removed unused imports, dead code and added @Override where approriate. > > Testing: jdk/jdk/jfr This pull request has now been integrated. Changeset: ae9af57b Author: Erik Gahlin URL: https://git.openjdk.java.net/jdk/commit/ae9af57b Stats: 346 lines in 57 files changed: 24 ins; 107 del; 215 mod 8264001: JFR: Modernize implementation Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3138 From jbachorik at openjdk.java.net Wed Mar 24 10:34:24 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Wed, 24 Mar 2021 10:34:24 GMT Subject: RFR: 8203359: Container level resources events [v2] In-Reply-To: References: Message-ID: <9_2MB3MaDFpBiwoAAQkxaKOEa9YgYVcXh-JZCIcwdBs=.e2c15f2e-073b-4f7b-a680-4ece79990710@github.com> > With this change it becomes possible to surface various cgroup level metrics (available via `jdk.internal.platform.Metrics`) as JFR events. > > Only a subset of the metrics exposed by `jdk.internal.platform.Metrics` is turned into JFR events to start with. > * CPU related metrics > * Memory related metrics > * I/O related metrics > > For each of those subsystems a configuration data will be emitted as well. The initial proposal is to emit the configuration data events at least once per chunk and the metrics values at 30 seconds interval. > By using these values the emitted events seem to contain useful information without increasing overhead (the metrics values are read from `/proc` filesystem so that should not be done too frequently). Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Split off the CPU throttling metrics ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/3126/files - new: https://git.openjdk.java.net/jdk/pull/3126/files/9ba3c5a6..90de0d8c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3126&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3126&range=00-01 Stats: 88 lines in 3 files changed: 73 ins; 15 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/3126.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3126/head:pull/3126 PR: https://git.openjdk.java.net/jdk/pull/3126 From mgronlun at openjdk.java.net Wed Mar 24 11:05:41 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Wed, 24 Mar 2021 11:05:41 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> Message-ID: <8cRzhjuabWlcsYBXLfbqg2CqXtYCYEQmKGXjxrmBXzk=.a6158717-9494-4337-82fb-421dfa9138c4@github.com> On Wed, 24 Mar 2021 06:06:31 GMT, Tom Rodriguez wrote: >> Hi Tom, looks good, thank you for this improvement. > > Should I perform any additional testing related to JFR? The tests located in test/jdk/jdk/jfr should be sufficient. ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From jbachorik at openjdk.java.net Wed Mar 24 12:29:22 2021 From: jbachorik at openjdk.java.net (Jaroslav Bachorik) Date: Wed, 24 Mar 2021 12:29:22 GMT Subject: RFR: 8203359: Container level resources events [v3] In-Reply-To: References: Message-ID: > With this change it becomes possible to surface various cgroup level metrics (available via `jdk.internal.platform.Metrics`) as JFR events. > > Only a subset of the metrics exposed by `jdk.internal.platform.Metrics` is turned into JFR events to start with. > * CPU related metrics > * Memory related metrics > * I/O related metrics > > For each of those subsystems a configuration data will be emitted as well. The initial proposal is to emit the configuration data events at least once per chunk and the metrics values at 30 seconds interval. > By using these values the emitted events seem to contain useful information without increasing overhead (the metrics values are read from `/proc` filesystem so that should not be done too frequently). Jaroslav Bachorik has updated the pull request incrementally with one additional commit since the last revision: Update the JFR control files ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/3126/files - new: https://git.openjdk.java.net/jdk/pull/3126/files/90de0d8c..82570022 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3126&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3126&range=01-02 Stats: 10 lines in 2 files changed: 10 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/3126.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3126/head:pull/3126 PR: https://git.openjdk.java.net/jdk/pull/3126 From github.com+76791+alblue at openjdk.java.net Wed Mar 24 13:20:49 2021 From: github.com+76791+alblue at openjdk.java.net (Alex Blewitt) Date: Wed, 24 Mar 2021 13:20:49 GMT Subject: RFR: 8264062: Use the blessed modifier order in jdk.jfr [v2] In-Reply-To: References: Message-ID: > This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. Alex Blewitt has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains two commits: - Additional JFR modifier order fixes - Used blessed modifier order for jdk.jfr ------------- Changes: https://git.openjdk.java.net/jdk/pull/3161/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3161&range=01 Stats: 143 lines in 52 files changed: 0 ins; 0 del; 143 mod Patch: https://git.openjdk.java.net/jdk/pull/3161.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3161/head:pull/3161 PR: https://git.openjdk.java.net/jdk/pull/3161 From fdavid at openjdk.java.net Wed Mar 24 14:44:44 2021 From: fdavid at openjdk.java.net (Florian David) Date: Wed, 24 Mar 2021 14:44:44 GMT Subject: Withdrawn: 8258414: OldObjectSample events too expensive In-Reply-To: <1_LsNBt-Yy5NlHbfwtRSRNvGa2AbTuhMGYuiw3Hy8gU=.3b79e283-87fe-451e-8e60-25b59c5e837a@github.com> References: <1_LsNBt-Yy5NlHbfwtRSRNvGa2AbTuhMGYuiw3Hy8gU=.3b79e283-87fe-451e-8e60-25b59c5e837a@github.com> Message-ID: <5MSScgUTaCRsk8cAhcUyHFkJ59F9teapMWImRepZNPU=.1e5bae48-8816-4c40-a23d-ed89165bfc1e@github.com> On Fri, 19 Feb 2021 14:45:00 GMT, Florian David wrote: > The purpose of this change is to reduce the size of JFR recordings when the OldObjectSample event is enabled. > > ## Problem > JFR recordings size blows up when the OldObjectSample is enabled. The memory allocation events are known to be very high traffic and will cause a lot of data, just the sheer number of events produced, and if stacktraces are added to this, the associated metadata can be huge as well. Sampled object are stored in a priority queue and their associated stack traces stored in JFRStackTraceRepository. When sample candidates are removed from the priority queue, their stacktraces remain in the repository, which will be later written at chunk rotation even if the sample has been removed. > > ## Implementation > This PR adds a JFRStackTraceRepository dedicated to store stack traces for the OldObjectSample event. At chunk rotation, every sample stack trace is looked up in this repository and is serialized. Other stack traces are simply removed. > > ## Benchmarks > On an AWS c5.metal instance (96 cores, 192 Gib), running SPECjvm2008 with default profile.jfc configuration with OldObjectSample event enabled gives: > - a recording size of 20.73Mb without the PR fix > - a recording size of 2.78Mb with the PR fix This pull request has been closed without being integrated. ------------- PR: https://git.openjdk.java.net/jdk/pull/2645 From github.com+741251+turbanoff at openjdk.java.net Wed Mar 24 17:54:41 2021 From: github.com+741251+turbanoff at openjdk.java.net (Andrey Turbanov) Date: Wed, 24 Mar 2021 17:54:41 GMT Subject: RFR: 8264001: JFR: Modernize implementation In-Reply-To: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> References: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> Message-ID: On Mon, 22 Mar 2021 22:35:32 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of changes that updates the Java implementation of JFR with newer language constructs and library calls. I also removed unused imports, dead code and added @Override where approriate. > > Testing: jdk/jdk/jfr src/jdk.jfr/share/classes/jdk/jfr/consumer/RecordedStackTrace.java line 54: > 52: Object[] array = getTyped("frames", Object[].class, null); > 53: if (array == null) { > 54: return new ArrayList<>(0); Why ArrayList is used here and not List.of() as everywhere else? ------------- PR: https://git.openjdk.java.net/jdk/pull/3138 From sgehwolf at openjdk.java.net Wed Mar 24 18:41:40 2021 From: sgehwolf at openjdk.java.net (Severin Gehwolf) Date: Wed, 24 Mar 2021 18:41:40 GMT Subject: RFR: 8203359: Container level resources events In-Reply-To: References: Message-ID: <32-BO7tExbAStlYsma_g-oc8kw-kj3HSyhJOP-qAjLk=.6c4d8c45-1a64-4b5a-8767-7307c297b3d8@github.com> On Mon, 22 Mar 2021 15:57:12 GMT, Jaroslav Bachorik wrote: > With this change it becomes possible to surface various cgroup level metrics (available via `jdk.internal.platform.Metrics`) as JFR events. > > Only a subset of the metrics exposed by `jdk.internal.platform.Metrics` is turned into JFR events to start with. > * CPU related metrics > * Memory related metrics > * I/O related metrics > > For each of those subsystems a configuration data will be emitted as well. The initial proposal is to emit the configuration data events at least once per chunk and the metrics values at 30 seconds interval. > By using these values the emitted events seem to contain useful information without increasing overhead (the metrics values are read from `/proc` filesystem so that should not be done too frequently). @jbachorik Would it make sense for `ContainerConfigurationEvent` to include the underlying cgroup version info (v1 or legacy vs. v2 or unified)? `Metrics.getProvider()` should give that info. ------------- PR: https://git.openjdk.java.net/jdk/pull/3126 From egahlin at openjdk.java.net Wed Mar 24 21:16:40 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Wed, 24 Mar 2021 21:16:40 GMT Subject: RFR: 8264001: JFR: Modernize implementation In-Reply-To: References: <4IizrlLMO2CkulVg5-Q4NRvgLHWSXu_QXFskkT-QBfs=.d4b535f2-46f1-4d22-893e-92f2b538c900@github.com> Message-ID: <3T0GOh6BpCVsr4PGplrFWhQXyjLBrBK4Svta8xU8wdo=.238bd591-eef4-42d9-91ce-f83c331741f0@github.com> On Wed, 24 Mar 2021 17:52:13 GMT, Andrey Turbanov wrote: >> Hi, >> >> Could I have a review of changes that updates the Java implementation of JFR with newer language constructs and library calls. I also removed unused imports, dead code and added @Override where approriate. >> >> Testing: jdk/jdk/jfr > > src/jdk.jfr/share/classes/jdk/jfr/consumer/RecordedStackTrace.java line 54: > >> 52: Object[] array = getTyped("frames", Object[].class, null); >> 53: if (array == null) { >> 54: return new ArrayList<>(0); > > Why ArrayList is used here and not List.of() as everywhere else? I wanted it to be consistent. Now the list is always modifiable, regardless of how it is constructed. One could argue for always returning an unmodifiable list, but it would require a spec change. ------------- PR: https://git.openjdk.java.net/jdk/pull/3138 From shade at openjdk.java.net Thu Mar 25 13:29:17 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Thu, 25 Mar 2021 13:29:17 GMT Subject: RFR: 8259623: JfrTypeSet::_subsystem_callback is left dangling after use Message-ID: SonarCloud instance reports the bug in cases like: "Address of stack memory associated with local variable 'callback' is still referred to by the global variable '_subsystem_callback' upon returning to the caller. This will be a dangling reference" For example: static void clear_packages() { ClearArtifact clear; ClearPackageCallback callback(&clear); _subsystem_callback = &callback; do_packages(); } I understand that `_subsystem_callback` is assigned to be used in do_packages(), but it would indeed be left dangling. The patch moves callback installation into the callback superclass itself. This way, we also guarantee that `_subsystem_callback` is not set before installation (i.e. check for overlapping callback installations), and make sure it is `NULL` after use. ------------- Commit messages: - 8259623: JfrTypeSet::_subsystem_callback is left dangling after use Changes: https://git.openjdk.java.net/jdk/pull/3192/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3192&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8259623 Stats: 33 lines in 2 files changed: 8 ins; 12 del; 13 mod Patch: https://git.openjdk.java.net/jdk/pull/3192.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3192/head:pull/3192 PR: https://git.openjdk.java.net/jdk/pull/3192 From mgronlun at openjdk.java.net Thu Mar 25 14:41:27 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 25 Mar 2021 14:41:27 GMT Subject: RFR: 8259623: JfrTypeSet::_subsystem_callback is left dangling after use In-Reply-To: References: Message-ID: On Thu, 25 Mar 2021 13:15:26 GMT, Aleksey Shipilev wrote: > SonarCloud instance reports the bug in cases like: > "Address of stack memory associated with local variable 'callback' is still referred to by the global variable '_subsystem_callback' upon returning to the caller. This will be a dangling reference" > > For example: > > static void clear_packages() { > ClearArtifact clear; > ClearPackageCallback callback(&clear); > _subsystem_callback = &callback; > do_packages(); > } > > I understand that `_subsystem_callback` is assigned to be used in do_packages(), but it would indeed be left dangling. The patch moves callback installation into the callback superclass itself. This way, we also guarantee that `_subsystem_callback` is not set before installation (i.e. check for overlapping callback installations), and make sure it is `NULL` after use. Looks good, Thanks. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3192 From github.com+76791+alblue at openjdk.java.net Thu Mar 25 15:42:28 2021 From: github.com+76791+alblue at openjdk.java.net (Alex Blewitt) Date: Thu, 25 Mar 2021 15:42:28 GMT Subject: RFR: 8264062: Use the blessed modifier order in jdk.jfr In-Reply-To: References: Message-ID: <-t4GD0qzxH-PV5oVo2uBnGkw6YCdGN1ZO6yJoL_pYu4=.cf0747d8-da11-4406-a33a-506daef06066@github.com> On Tue, 23 Mar 2021 18:56:37 GMT, Alex Blewitt wrote: > This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. Would someone mind reviewing this? @mgronlun perhaps? ------------- PR: https://git.openjdk.java.net/jdk/pull/3161 From egahlin at openjdk.java.net Thu Mar 25 17:36:21 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Thu, 25 Mar 2021 17:36:21 GMT Subject: RFR: 8260862 JFR: New configure command for the jfr tool [v2] In-Reply-To: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> References: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> Message-ID: > Hi, > > Could I have a review of a new configure command for the jfr tool. Purpose is to simplify changing event settings: > > $ jfr configure method-profiling=high gc=detailed > Configuration written successfully to: > > > For more details, see CSR: > https://bugs.openjdk.java.net/browse/JDK-8260863 > > Testing: > - jdk/jdk/jfr > - JMC recording wizard > > Thanks > Erik Erik Gahlin has updated the pull request incrementally with two additional commits since the last revision: - Review feedback - Fix copyright and formatting ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/3100/files - new: https://git.openjdk.java.net/jdk/pull/3100/files/3a95d908..78f1df90 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3100&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3100&range=00-01 Stats: 947 lines in 40 files changed: 475 ins; 430 del; 42 mod Patch: https://git.openjdk.java.net/jdk/pull/3100.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3100/head:pull/3100 PR: https://git.openjdk.java.net/jdk/pull/3100 From mgronlun at openjdk.java.net Thu Mar 25 18:14:26 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 25 Mar 2021 18:14:26 GMT Subject: RFR: 8260862 JFR: New configure command for the jfr tool [v2] In-Reply-To: References: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> Message-ID: On Thu, 25 Mar 2021 17:36:21 GMT, Erik Gahlin wrote: >> Hi, >> >> Could I have a review of a new configure command for the jfr tool. Purpose is to simplify changing event settings: >> >> $ jfr configure method-profiling=high gc=detailed >> Configuration written successfully to: >> >> >> For more details, see CSR: >> https://bugs.openjdk.java.net/browse/JDK-8260863 >> >> Testing: >> - jdk/jdk/jfr >> - JMC recording wizard >> >> Thanks >> Erik > > Erik Gahlin has updated the pull request incrementally with two additional commits since the last revision: > > - Review feedback > - Fix copyright and formatting Looks good Erik, nice work. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3100 From egahlin at openjdk.java.net Thu Mar 25 20:47:28 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Thu, 25 Mar 2021 20:47:28 GMT Subject: Integrated: 8260862 JFR: New configure command for the jfr tool In-Reply-To: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> References: <9UdRmBLP8LkjHyHH0eSbNQ7x7vsVHYuGHoqtN6wrmQI=.1e831728-75b6-4289-8329-d6e85cc5ef90@github.com> Message-ID: On Fri, 19 Mar 2021 23:37:37 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a new configure command for the jfr tool. Purpose is to simplify changing event settings: > > $ jfr configure method-profiling=high gc=detailed > Configuration written successfully to: > > > For more details, see CSR: > https://bugs.openjdk.java.net/browse/JDK-8260863 > > Testing: > - jdk/jdk/jfr > - JMC recording wizard > > Thanks > Erik This pull request has now been integrated. Changeset: 4e708e58 Author: Erik Gahlin URL: https://git.openjdk.java.net/jdk/commit/4e708e58 Stats: 3276 lines in 43 files changed: 3006 ins; 53 del; 217 mod 8260862: JFR: New configure command for the jfr tool Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3100 From mgronlun at openjdk.java.net Thu Mar 25 22:01:30 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Thu, 25 Mar 2021 22:01:30 GMT Subject: RFR: 8264062: Use the blessed modifier order in jdk.jfr [v2] In-Reply-To: References: Message-ID: On Wed, 24 Mar 2021 13:20:49 GMT, Alex Blewitt wrote: >> This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. > > Alex Blewitt has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains two commits: > > - Additional JFR modifier order fixes > - Used blessed modifier order for jdk.jfr Hi Alex, looks good. Thank you. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3161 From github.com+76791+alblue at openjdk.java.net Thu Mar 25 22:08:42 2021 From: github.com+76791+alblue at openjdk.java.net (Alex Blewitt) Date: Thu, 25 Mar 2021 22:08:42 GMT Subject: RFR: 8264062: Use the blessed modifier order in jdk.jfr [v3] In-Reply-To: References: Message-ID: <0z3ac28vs_Lnl4bFMDsLvCqoi_LstLdyRRhPxPJqu_4=.024d4464-a7a4-4332-b040-7bdf0cbdd220@github.com> > This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. Alex Blewitt has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains two commits: - Additional JFR modifier order fixes - Used blessed modifier order for jdk.jfr ------------- Changes: https://git.openjdk.java.net/jdk/pull/3161/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3161&range=02 Stats: 143 lines in 52 files changed: 0 ins; 0 del; 143 mod Patch: https://git.openjdk.java.net/jdk/pull/3161.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3161/head:pull/3161 PR: https://git.openjdk.java.net/jdk/pull/3161 From egahlin at openjdk.java.net Thu Mar 25 23:31:26 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Thu, 25 Mar 2021 23:31:26 GMT Subject: RFR: 8203359: Container level resources events In-Reply-To: <32-BO7tExbAStlYsma_g-oc8kw-kj3HSyhJOP-qAjLk=.6c4d8c45-1a64-4b5a-8767-7307c297b3d8@github.com> References: <32-BO7tExbAStlYsma_g-oc8kw-kj3HSyhJOP-qAjLk=.6c4d8c45-1a64-4b5a-8767-7307c297b3d8@github.com> Message-ID: On Wed, 24 Mar 2021 18:39:06 GMT, Severin Gehwolf wrote: >> With this change it becomes possible to surface various cgroup level metrics (available via `jdk.internal.platform.Metrics`) as JFR events. >> >> Only a subset of the metrics exposed by `jdk.internal.platform.Metrics` is turned into JFR events to start with. >> * CPU related metrics >> * Memory related metrics >> * I/O related metrics >> >> For each of those subsystems a configuration data will be emitted as well. The initial proposal is to emit the configuration data events at least once per chunk and the metrics values at 30 seconds interval. >> By using these values the emitted events seem to contain useful information without increasing overhead (the metrics values are read from `/proc` filesystem so that should not be done too frequently). > > @jbachorik Would it make sense for `ContainerConfigurationEvent` to include the underlying cgroup version info (v1 or legacy vs. v2 or unified)? `Metrics.getProvider()` should give that info. Does each getter call result in parsing /proc, or do things aggregated over several calls or hooks? Do you have any data how expensive the invocations are? You could for example try to measure it by temporary making the events durational, and fetch the values between begin() and end(), and perhaps show a 'jfr print --events Container* recording.jfr' printout. If possible, it would be interesting to get some idea about the startup cost as well If not too much overhead, I think it would be nice to skip the "flag" in the .jfcs, and always record the events in a container environment. I know there is a way to test JFR using Docker, maybe @mseledts could provide information? Some sanity tests would be good to have. ------------- PR: https://git.openjdk.java.net/jdk/pull/3126 From sgehwolf at openjdk.java.net Fri Mar 26 11:08:25 2021 From: sgehwolf at openjdk.java.net (Severin Gehwolf) Date: Fri, 26 Mar 2021 11:08:25 GMT Subject: RFR: 8203359: Container level resources events In-Reply-To: References: <32-BO7tExbAStlYsma_g-oc8kw-kj3HSyhJOP-qAjLk=.6c4d8c45-1a64-4b5a-8767-7307c297b3d8@github.com> Message-ID: On Thu, 25 Mar 2021 23:28:18 GMT, Erik Gahlin wrote: > Does each getter call result in parsing /proc, or do things aggregated over several calls or hooks? >From the looks of it the event emitting code uses `Metrics.java` interface for retrieving the info. Each call to a method exposed by Metrics result in file IO on some cgroup (v1 or v2) interface file(s) in `/sys/fs/...`. I don't see any aggregation being done. On the hotspot side, we implemented some caching for frequent calls (JDK-8232207, JDK-8227006), but we didn't do that yet for the Java side since there wasn't any need (so far). If calls are becoming frequent with this it should be reconsidered. So +1 on getting some data on what the perf penalty of this is. ------------- PR: https://git.openjdk.java.net/jdk/pull/3126 From github.com+76791+alblue at openjdk.java.net Fri Mar 26 14:10:25 2021 From: github.com+76791+alblue at openjdk.java.net (Alex Blewitt) Date: Fri, 26 Mar 2021 14:10:25 GMT Subject: Integrated: 8264062: Use the blessed modifier order in jdk.jfr In-Reply-To: References: Message-ID: On Tue, 23 Mar 2021 18:56:37 GMT, Alex Blewitt wrote: > This replaces patterns such as "final public" with "public final" to bring the modifiers in the standard order for the jdk.jfr module. This pull request has now been integrated. Changeset: e47dfb8e Author: Alex Blewitt Committer: Markus Gr?nlund URL: https://git.openjdk.java.net/jdk/commit/e47dfb8e Stats: 143 lines in 52 files changed: 0 ins; 0 del; 143 mod 8264062: Use the blessed modifier order in jdk.jfr Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3161 From egahlin at openjdk.java.net Fri Mar 26 18:46:37 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Fri, 26 Mar 2021 18:46:37 GMT Subject: RFR: 8264309: JFR: Improve .jfc parser Message-ID: <1AQZbTPvbdu3zF5IYbAgYO-QFj45xtBdCd52TrqcoqE=.130a3bb8-2dd5-40e0-b285-1bbd0a7804f4@github.com> Hi, Could I have a review of a fix that removes unnecessary number of allocations of StringBuilders. Events are now referred to by name, not path. The code should reflect that. The method getSettings() is not used and can be removed. Testing: jdk/jdk/jfr Thanks Erik ------------- Commit messages: - Remove getSettings() - Initial Changes: https://git.openjdk.java.net/jdk/pull/3222/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3222&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264309 Stats: 15 lines in 1 file changed: 1 ins; 6 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/3222.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3222/head:pull/3222 PR: https://git.openjdk.java.net/jdk/pull/3222 From never at openjdk.java.net Fri Mar 26 19:02:31 2021 From: never at openjdk.java.net (Tom Rodriguez) Date: Fri, 26 Mar 2021 19:02:31 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: <8cRzhjuabWlcsYBXLfbqg2CqXtYCYEQmKGXjxrmBXzk=.a6158717-9494-4337-82fb-421dfa9138c4@github.com> References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> <8cRzhjuabWlcsYBXLfbqg2CqXtYCYEQmKGXjxrmBXzk=.a6158717-9494-4337-82fb-421dfa9138c4@github.com> Message-ID: On Wed, 24 Mar 2021 11:02:43 GMT, Markus Gr?nlund wrote: >> Should I perform any additional testing related to JFR? > > The tests located in test/jdk/jdk/jfr should be sufficient. The tests were clean. ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From mgronlun at openjdk.java.net Fri Mar 26 19:13:31 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Fri, 26 Mar 2021 19:13:31 GMT Subject: RFR: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> <8cRzhjuabWlcsYBXLfbqg2CqXtYCYEQmKGXjxrmBXzk=.a6158717-9494-4337-82fb-421dfa9138c4@github.com> Message-ID: On Fri, 26 Mar 2021 19:00:11 GMT, Tom Rodriguez wrote: >> The tests located in test/jdk/jdk/jfr should be sufficient. > > The tests were clean. Great, please proceed with integration. ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From never at openjdk.java.net Fri Mar 26 20:23:28 2021 From: never at openjdk.java.net (Tom Rodriguez) Date: Fri, 26 Mar 2021 20:23:28 GMT Subject: Integrated: 8264017: Correctly report inlined frame in JFR sampling In-Reply-To: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> References: <15CaJBMulHDismunA3PLg6yVfOH0POymbXIcdq4_fIY=.75c7e039-5a9c-45b4-a705-2bf74b427b84@github.com> Message-ID: <2vLjR5kQMqdFsuFU0PrxTL1O16A_qybKQ3r_XhuuX2s=.ea28062c-d986-4d6b-bab9-a960e75bf8f8@github.com> On Tue, 23 Mar 2021 08:30:48 GMT, Tom Rodriguez wrote: > 8264017: Correctly report inlined frame in JFR sampling This pull request has now been integrated. Changeset: 054e0a42 Author: Tom Rodriguez URL: https://git.openjdk.java.net/jdk/commit/054e0a42 Stats: 20 lines in 1 file changed: 17 ins; 3 del; 0 mod 8264017: Correctly report inlined frame in JFR sampling Reviewed-by: jbachorik, mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3148 From shade at openjdk.java.net Mon Mar 29 08:03:30 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Mon, 29 Mar 2021 08:03:30 GMT Subject: Integrated: 8259623: JfrTypeSet::_subsystem_callback is left dangling after use In-Reply-To: References: Message-ID: On Thu, 25 Mar 2021 13:15:26 GMT, Aleksey Shipilev wrote: > SonarCloud instance reports the bug in cases like: > "Address of stack memory associated with local variable 'callback' is still referred to by the global variable '_subsystem_callback' upon returning to the caller. This will be a dangling reference" > > For example: > > static void clear_packages() { > ClearArtifact clear; > ClearPackageCallback callback(&clear); > _subsystem_callback = &callback; > do_packages(); > } > > I understand that `_subsystem_callback` is assigned to be used in do_packages(), but it would indeed be left dangling. The patch moves callback installation into the callback superclass itself. This way, we also guarantee that `_subsystem_callback` is not set before installation (i.e. check for overlapping callback installations), and make sure it is `NULL` after use. This pull request has now been integrated. Changeset: 8ee9a05d Author: Aleksey Shipilev URL: https://git.openjdk.java.net/jdk/commit/8ee9a05d Stats: 33 lines in 2 files changed: 8 ins; 12 del; 13 mod 8259623: JfrTypeSet::_subsystem_callback is left dangling after use Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3192 From mgronlun at openjdk.java.net Mon Mar 29 12:42:31 2021 From: mgronlun at openjdk.java.net (Markus =?UTF-8?B?R3LDtm5sdW5k?=) Date: Mon, 29 Mar 2021 12:42:31 GMT Subject: RFR: 8264309: JFR: Improve .jfc parser In-Reply-To: <1AQZbTPvbdu3zF5IYbAgYO-QFj45xtBdCd52TrqcoqE=.130a3bb8-2dd5-40e0-b285-1bbd0a7804f4@github.com> References: <1AQZbTPvbdu3zF5IYbAgYO-QFj45xtBdCd52TrqcoqE=.130a3bb8-2dd5-40e0-b285-1bbd0a7804f4@github.com> Message-ID: On Fri, 26 Mar 2021 18:03:11 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a fix that removes unnecessary number of StringBuilder allocations. Events are now referred to by name, not path. The code should reflect that. The method getSettings() is not used and can be removed. > > Testing: jdk/jdk/jfr > > Thanks > Erik Looks good. ------------- Marked as reviewed by mgronlun (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3222 From egahlin at openjdk.java.net Mon Mar 29 14:00:29 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Mon, 29 Mar 2021 14:00:29 GMT Subject: Integrated: 8264309: JFR: Improve .jfc parser In-Reply-To: <1AQZbTPvbdu3zF5IYbAgYO-QFj45xtBdCd52TrqcoqE=.130a3bb8-2dd5-40e0-b285-1bbd0a7804f4@github.com> References: <1AQZbTPvbdu3zF5IYbAgYO-QFj45xtBdCd52TrqcoqE=.130a3bb8-2dd5-40e0-b285-1bbd0a7804f4@github.com> Message-ID: On Fri, 26 Mar 2021 18:03:11 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review of a fix that removes unnecessary number of StringBuilder allocations. Events are now referred to by name, not path. The code should reflect that. The method getSettings() is not used and can be removed. > > Testing: jdk/jdk/jfr > > Thanks > Erik This pull request has now been integrated. Changeset: 963f1fc6 Author: Erik Gahlin URL: https://git.openjdk.java.net/jdk/commit/963f1fc6 Stats: 15 lines in 1 file changed: 1 ins; 6 del; 8 mod 8264309: JFR: Improve .jfc parser Reviewed-by: mgronlun ------------- PR: https://git.openjdk.java.net/jdk/pull/3222 From egahlin at openjdk.java.net Tue Mar 30 23:55:35 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Tue, 30 Mar 2021 23:55:35 GMT Subject: RFR: 8263395: Incorrect use of Objects.nonNull Message-ID: Hi, Could I have a review that changes incorrect use of Objects.nonNull to Objects.requireNonNull. Testing: jdk/jdk/jfr Thanks Erik ------------- Commit messages: - Initial Changes: https://git.openjdk.java.net/jdk/pull/3278/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3278&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263395 Stats: 6 lines in 3 files changed: 0 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/3278.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/3278/head:pull/3278 PR: https://git.openjdk.java.net/jdk/pull/3278 From erik.gahlin at oracle.com Wed Mar 31 03:55:58 2021 From: erik.gahlin at oracle.com (Erik Gahlin) Date: Wed, 31 Mar 2021 03:55:58 +0000 Subject: Ignored IOException In-Reply-To: <875620cc-b632-47d3-833e-9871431f86f0.denghui.ddh@alibaba-inc.com> References: <875620cc-b632-47d3-833e-9871431f86f0.denghui.ddh@alibaba-inc.com> Message-ID: <527A0191-95E6-4350-BCF2-905409F59F54@oracle.com> Hi Denghui, Sorry for the late reply. I was able to track down the bug number to an issue in JRockit in 2009 where the IOException was written to jrcmd, the predecessor of jcmd. I think it?s fine to remove the comment and log the issue, perhaps with log level ERROR. We have tried to avoid printing file path to the log unless a user has opted in for more logging. So perhaps: Logger.log(LogTag.JFR, LogLevel.ERROR, "Unable to complete I/O operation when dump recording \"" + getName() + "\" (" + getId() + "); It should be sufficient information to dump it manually later. Thanks Erik On 2021-03-08, 04:12, "hotspot-jfr-dev on behalf of Denghui Dong" wrote: Hi, I noticed that there is an ignored IO Exception in jdk.jfr.internal.PlatformRecording#stop, but I cannot open the related issue(8925030). The problem I have is that when users use `java -XX:StartFlightRecording=name=test,duration=5m,filename=./dumped.jfr `, and the 'dumped.jfr' is accidentally deleted(e.g. by clean task), the final jfr file will not exist, and also no exception will be generated after the recording is ended. Users need an extra step to generate the jfr file. If there is a problem with throwing this exception, maybe we can print the exception information there, so that at least a reminder can be given to the user. Any input is appreciated. Thanks, Denghui From egahlin at openjdk.java.net Wed Mar 31 05:27:27 2021 From: egahlin at openjdk.java.net (Erik Gahlin) Date: Wed, 31 Mar 2021 05:27:27 GMT Subject: RFR: 8263395: Incorrect use of Objects.nonNull In-Reply-To: <2A5gaEFSZX0ADVk300EjtZjtmsscE6H2mlYxb_kGRgA=.6825b7ad-81a1-4dbe-9caf-b71820c044ba@github.com> References: <2A5gaEFSZX0ADVk300EjtZjtmsscE6H2mlYxb_kGRgA=.6825b7ad-81a1-4dbe-9caf-b71820c044ba@github.com> Message-ID: <17SpTt-2FYqZtx2es2YaYtlMpAVKD-7O-GEorNNVsfE=.1019fca5-076e-4838-abff-413e45e9e6af@github.com> On Wed, 10 Mar 2021 19:52:36 GMT, Andrey Turbanov wrote: > 8263395: Incorrect use of Objects.nonNull I missed the e-mail and that you had started working on this. Looks good. ------------- Marked as reviewed by egahlin (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/2923 From shade at openjdk.java.net Wed Mar 31 07:39:27 2021 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Wed, 31 Mar 2021 07:39:27 GMT Subject: RFR: 8263395: Incorrect use of Objects.nonNull In-Reply-To: References: Message-ID: On Tue, 30 Mar 2021 22:28:13 GMT, Erik Gahlin wrote: > Hi, > > Could I have a review that changes incorrect use of Objects.nonNull to Objects.requireNonNull. > > Testing: jdk/jdk/jfr > > Thanks > Erik Yup, looks good. ------------- Marked as reviewed by shade (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/3278