From yangang at ec.com.cn  Wed Sep 11 17:08:39 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Wed, 11 Sep 2019 17:08:39 +0800(CST)
Subject: =?gbk?B?cXVlc3Rpb24gb2YgUGFyTmV3R2VuZXJhdGlvbjo6cmVhbF9mb3J3YXJkZWVfc2xvdw==?=
Message-ID: <46Sx080g2wz5vNT@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190911/a895bbf2/attachment.html>

From ecki at zusammenkunft.net  Wed Sep 11 09:31:59 2019
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Wed, 11 Sep 2019 09:31:59 +0000
Subject: question of ParNewGeneration::real_forwardee_slow
In-Reply-To: <46Sx080g2wz5vNT@mx3.ec.com.cn>
References: <46Sx080g2wz5vNT@mx3.ec.com.cn>
Message-ID: <DB7PR08MB3307E71C88E57D560ECA4801FFB10@DB7PR08MB3307.eurprd08.prod.outlook.com>

Can you publish your test code, or describe what it is doing?


--
http://bernd.eckenfels.net

________________________________
Von: hotspot-gc-use <hotspot-gc-use-bounces at openjdk.java.net> im Auftrag von Yan Gang <yangang at ec.com.cn>
Gesendet: Mittwoch, September 11, 2019 11:10 AM
An: hotspot-gc-use
Betreff: question of ParNewGeneration::real_forwardee_slow

hi all:
I run 100  threads performance test,  use perf top display:

  71.13%  libjvm.so                [.] ParNewGeneration::real_forwardee_slow

ParNewGeneration::real_forwardee_slow waste some CPU time. I search  google

ParNewGeneration::real_forwardee_slow belong of  parNewGeneration.cpp, also is JDK bug: https://bugs.openjdk.java.net/browse/JDK-7171273.  replaced to G1?

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190911/1fadbde3/attachment.html>

From yangang at ec.com.cn  Wed Sep 11 18:49:43 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Wed, 11 Sep 2019 18:49:43 +0800(CST)
Subject: =?gbk?B?UmU6UmU6IHF1ZXN0aW9uIG9mIFBhck5ld0dlbmVyYXRpb246OnJlYWxfZm9yd2FyZGVlX3Nsb3c=?=
Message-ID: <46SzDm2JhJz5vND@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190911/14f3fa1d/attachment.html>

From thomas.schatzl at oracle.com  Wed Sep 11 11:57:35 2019
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Wed, 11 Sep 2019 13:57:35 +0200
Subject: question of ParNewGeneration::real_forwardee_slow
In-Reply-To: <46Sx080g2wz5vNT@mx3.ec.com.cn>
References: <46Sx080g2wz5vNT@mx3.ec.com.cn>
Message-ID: <2114cfc92bd469d7afb9d24ff83d78936bf5c3d4.camel@oracle.com>

Hi,

On Wed, 2019-09-11 at 17:08 +0000, Yan Gang wrote:
> hi all:
> I run 100  threads performance test,  use perf top display:
> 
>   71.13%  libjvm.so                [.]
> ParNewGeneration::real_forwardee_slow
> 
> ParNewGeneration::real_forwardee_slow waste some CPU time. I search 
> google
> 
> ParNewGeneration::real_forwardee_slow belong
> of  parNewGeneration.cpp, also is JDK bug: 
> https://bugs.openjdk.java.net/browse/JDK-7171273.

The referenced CR only says that this busy-loop is inefficient cpu-
wise, not that waiting for another thread should be made substantially
faster :)

The other problem is that the waste_some_time() is probably completely
optimized away with current compilers (constant folded), so hammering
that cacheline (with reads) quite a bit. (I believe that there is a CR
for that as well somewhere, but maybe we haven't because of CMS
obsolescence).

If that real_forwardee_slow() method shows up a lot, *maybe* the amount
of parallel gc threads used is too high, i.e. threads stepping on each
other's toes all the time.

>  replaced to G1?

Our (Oracle's) general advice when having problems with CMS is to try
G1, but recent JDKs ship with other GCs that may be suitable to your
application (ZGC, Shenandoah, potentially Parallel GC).

CMS is on the way out in _future_ releases (see _draft_ JEP 
https://openjdk.java.net/jeps/8229049). We at Oracle recommend looking
into alternatives.

Thanks,
  Thomas



From yangang at ec.com.cn  Thu Sep 12 11:41:24 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Thu, 12 Sep 2019 11:41:24 +0800(CST)
Subject: =?gbk?B?UmU6UmU6IHF1ZXN0aW9uIG9mIFBhck5ld0dlbmVyYXRpb246OnJlYWxfZm9yd2FyZGVlX3Nsb3c=?=
Message-ID: <46TPh45Pwtz5vNS@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190912/9bd96b80/attachment.html>

From thomas.schatzl at oracle.com  Thu Sep 12 08:56:37 2019
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Thu, 12 Sep 2019 10:56:37 +0200
Subject: question of ParNewGeneration::real_forwardee_slow
In-Reply-To: <46TPh45Pwtz5vNS@mx3.ec.com.cn>
References: <46TPh45Pwtz5vNS@mx3.ec.com.cn>
Message-ID: <690f5a7d-537b-f2eb-f4c0-5cf8dd1a5e68@oracle.com>

On 9/12/19 5:41 AM, Yan Gang wrote:
> hi Thomas:
> 
> Thanks you reply!
> 
> Follows is full jvm option:
> -Xms6G -Xmx6G -XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=512M 
> -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+HeapDumpOnOutOfMemoryError 
> -verbose:gc -Xloggc:/data/gc.log -XX:+UnlockCommercialFeatures 
> -XX:+FlightRecorder 
> -XX:StartFlightRecording=delay=60s,duration=1h,settings=profile,filename=/data/flight.jfr 
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime 
> -XX:PrintFLSStatistics=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> -XX:+PrintGCDateStamps -XX:-UseParNewGC -XX:ParallelGCThreads=32 
> -XX:+TraceReferenceGC -XX:+TraceScavenge -XX:MaxTenuringThreshold=1 
> -Dcom.sun.management.jmxremote.port=7091 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false gmcf-pmc-web-1.4.0.jar
> 
> JDK version:?Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed 
> mode)
> 16C64G
> I check gc.log:
> 2019-09-10T17:42:30.071+0800: 5683.360: [GC (Allocation Failure) 
> 2019-09-10T17:42:30.071+0800: 5683.361: [ParNew: 
> 897542K->13611K(996800K), 0.0213537 secs] 1040925K->157022K(6180736K), 
> 0.0219481 secs] [Times: user=0.17 sys=0.00, real=0.02 secs]
> 2019-09-10T17:42:35.716+0800: 5689.005: [GC (Allocation Failure) 
> 2019-09-10T17:42:35.717+0800: 5689.006: [ParNew: 
> 899691K->7416K(996800K), 4.6872684 secs] 1043102K->156944K(6180736K), 
> 4.6880854 secs] [Times: user=3.76 sys=1.99, real=4.69 secs]
> 2019-09-10T17:42:47.524+0800: 5700.813: [GC (Allocation Failure) 
> 2019-09-10T17:42:47.524+0800: 5700.813: [ParNew: 
> 893496K->3500K(996800K), 0.0396484 secs] 1043024K->153075K(6180736K), 
> 0.0406422 secs] [Times: user=0.52 sys=0.00, real=0.04 secs]
> 
> Maybe big object cause long ParNew GC?

I would check the following:

- do you really need 32 threads? Afaics the amount of data actually 
touched during GC is like 20MB / GC. It makes sense that these 32 
threads are fighting to evacuate these 20MB, potentially leading to the 
reported stalls.

Decreasing the number of parallel GC threads may help.

Contradictory to your setting of ParallelGCThreads=32, you try disable 
parallel evacuation anyway (-XX:-UseParNewGC), which looks weird because 
the user/real time ratio indicates that indeed parallel evacuation is 
going on. Potentially manually setting ParallelGCThreads after 
-XX:-UseParnewGC) re-enables ParNew.

Not sure if this is what you want; but I would look at the suggestion 
below first.

 > 2019-09-10T17:42:35.717+0800: 5689.006: [ParNew:
 > 899691K->7416K(996800K), 4.6872684 secs] 1043102K->156944K(6180736K),
 > 4.6880854 secs] [Times: user=3.76 sys=1.99, real=4.69 secs]

Whatever causes the high sys time may be the actual cause for your 
troubles. Did you try the setup recommendations outlined in the tuning 
guide [0]?

I.e. disable THP, and move the log file to a separate (ram-)disk? The 
options you used indicate that -XX:+AlwaysPreTouch is not enabled either.
Please also look if some swapping activity might be going on on your 
machine.

Thanks,
   Thomas


[0] 
https://docs.oracle.com/en/java/javase/12/gctuning/garbage-first-garbage-collector-tuning.html#GUID-8D9B2530-E370-4B8B-8ADD-A43674FC6658
> 
> Thanks!
> 
> 
> 
> 
> 
> ----- ???? -----
> *???:*Thomas Schatzl <thomas.schatzl at oracle.com 
> <mailto:thomas.schatzl at oracle.com>>
> *???:*"Yan Gang" <yangang at ec.com.cn 
> <mailto:yangang at ec.com.cn>>,"hotspot-gc-use" 
> <hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>>
> *????:*2019?09?11? 19?57?35?
> *????:*Re: question of ParNewGeneration::real_forwardee_slow
> 
> 
>     Hi,
> 
>     On Wed, 2019-09-11 at 17:08 +0000, Yan Gang wrote:
>      > hi all:
>      > I run 100 threads performance test, use perf top display:
>      >
>      > 71.13% libjvm.so [.]
>      > ParNewGeneration::real_forwardee_slow
>      >
>      > ParNewGeneration::real_forwardee_slow waste some CPU time. I search
>      > google
>      >
>      > ParNewGeneration::real_forwardee_slow belong
>      > of parNewGeneration.cpp, also is JDK bug:
>      > https://bugs.openjdk.java.net/browse/JDK-7171273.
> 
>     The referenced CR only says that this busy-loop is inefficient cpu-
>     wise, not that waiting for another thread should be made substantially
>     faster :)
> 
>     The other problem is that the waste_some_time() is probably completely
>     optimized away with current compilers (constant folded), so hammering
>     that cacheline (with reads) quite a bit. (I believe that there is a CR
>     for that as well somewhere, but maybe we haven't because of CMS
>     obsolescence).
> 
>     If that real_forwardee_slow() method shows up a lot, *maybe* the amount
>     of parallel gc threads used is too high, i.e. threads stepping on each
>     other's toes all the time.
> 
>      > replaced to G1?
> 
>     Our (Oracle's) general advice when having problems with CMS is to try
>     G1, but recent JDKs ship with other GCs that may be suitable to your
>     application (ZGC, Shenandoah, potentially Parallel GC).
> 
>     CMS is on the way out in _future_ releases (see _draft_ JEP
>     https://openjdk.java.net/jeps/8229049). We at Oracle recommend looking
>     into alternatives.
> 
>     Thanks,
>     Thomas
> 
> 


From yangang at ec.com.cn  Thu Sep 12 17:38:44 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Thu, 12 Sep 2019 17:38:44 +0800(CST)
Subject: =?gbk?B?UmU6UmU6IHF1ZXN0aW9uIG9mIFBhck5ld0dlbmVyYXRpb246OnJlYWxfZm9yd2FyZGVlX3Nsb3c=?=
Message-ID: <46TYcN6p1Lz5vNK@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190912/d36421c4/attachment.html>

From yangang at ec.com.cn  Thu Sep 12 19:04:46 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Thu, 12 Sep 2019 19:04:46 +0800(CST)
Subject: =?gbk?B?UmU6UmU6IHF1ZXN0aW9uIG9mIFBhck5ld0dlbmVyYXRpb246OnJlYWxfZm9yd2FyZGVlX3Nsb3c=?=
Message-ID: <46TbWg1JjDz5vNG@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190912/c2af836c/attachment-0001.html>

From thomas.schatzl at oracle.com  Fri Sep 13 09:24:33 2019
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Fri, 13 Sep 2019 11:24:33 +0200
Subject: question of ParNewGeneration::real_forwardee_slow
In-Reply-To: <46TYcN6p1Lz5vNK@mx3.ec.com.cn>
References: <46TYcN6p1Lz5vNK@mx3.ec.com.cn>
Message-ID: <aeff52a3-2438-c298-29da-faf3badd7048@oracle.com>

Hi,

On 12.09.19 11:38, Yan Gang wrote:
> Thomas:
> 
> I modify Java option :

   I really recommend only doing one thing at a time: i.e. not change 
options *and* the environment at the same time, or try to find new 
causes for the issues. That will most likely only confuse or scare you, 
and often these are connected in some way, so fixing one may solve the 
other (or may obviate the need to "fix" them).

The environment exhibits very long sys times in your log output which I 
recommend to investigate. They also seem to be causing most of the 
slowdown. I.e. it seems wise to understand and fix these first.

In addition to simply trying the fixes for the most common issues 
presented in the tuning guide, you can use system tools like 
iostat/vmstat/* to get to understand your environment behavior more.

Thanks,
   Thomas

From yangang at ec.com.cn  Thu Sep 19 10:20:48 2019
From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=)
Date: Thu, 19 Sep 2019 10:20:48 +0800(CST)
Subject: =?gbk?B?UmU6UmU6IHF1ZXN0aW9uIG9mIFBhck5ld0dlbmVyYXRpb246OnJlYWxfZm9yd2FyZGVlX3Nsb3c=?=
Message-ID: <46YgYs4blmz5vNb@mx3.ec.com.cn>

An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190919/99d93c54/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc.log
Type: application/octet-stream
Size: 39300 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190919/99d93c54/gc-0001.log>