From martinrb at google.com Fri May 1 19:55:09 2015 From: martinrb at google.com (Martin Buchholz) Date: Fri, 1 May 2015 12:55:09 -0700 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <5542740B.6080507@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> Message-ID: On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik < jaroslav.bachorik at oracle.com> wrote: > On 30.4.2015 19:18, Martin Buchholz wrote: > >> Tests that sleep can almost always be better written some other way. >> In this case, I would prefer busy-waiting with timeout until the >> expected condition becomes true. >> > > The thing is that in case of a real issue with the thread counters we > a/ would be busy-waiting till the test times out (using an arbitrary delay > is also problematic due to different performance of different machines > running with different configurations) > Far less problematic (performance-wise and reliability-wise) than the fixed sleep. > b/ would get a rather confusing message about the test timing out at the > end You can easily improve the error message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Mon May 4 06:13:49 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 04 May 2015 08:13:49 +0200 Subject: RFR 8072906: sun/management/jmxremote/bootstrap/CustomLauncherTest.java failing on embedded platform In-Reply-To: <554299C2.6040301@oracle.com> References: <554240A6.7070309@oracle.com> <554299C2.6040301@oracle.com> Message-ID: <55470E1D.1080508@oracle.com> On 30.4.2015 23:08, David Holmes wrote: > On 1/05/2015 12:48 AM, Jaroslav Bachorik wrote: >> Please, review the following test change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8072906 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8072906/webrev.00 >> >> The test tries to exercise a custom launcher on embedded platforms and >> fails because the corresponding binary is not available. The test should >> not require the presence of the binary and skip the execution gracefully >> if it can't find one - this was broken while addressing JDK-8029809. >> >> The fix brings back the ability just skip the test if the launcher >> binary does not exist. > > A message to that effect would be useful. The message is printed at L247. It's been there for some time and because of that is not the part of the generated webrev. -JB- > > Thanks, > David > >> Thanks, >> >> -JB- From staffan.larsen at oracle.com Mon May 4 06:52:26 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 4 May 2015 08:52:26 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" Message-ID: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. I will push the change below directly to jdk9/hs. Thanks, /Staffan diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk --- a/make/lib/Lib-java.management.gmk +++ b/make/lib/Lib-java.management.gmk @@ -38,11 +38,6 @@ $(LIBJAVA_HEADER_FLAGS) \ # -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate -# a binary that is compatible with windows versions older than 7/2008R2. -# See MSDN documentation for GetProcessMemoryInfo for more information. -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 - LIBMANAGEMENT_OPTIMIZATION := HIGH ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) ifeq ($(ENABLE_DEBUG_SYMBOLS), true) diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk --- a/make/lib/Lib-jdk.management.gmk +++ b/make/lib/Lib-jdk.management.gmk @@ -39,6 +39,11 @@ $(LIBJAVA_HEADER_FLAGS) \ # +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate +# a binary that is compatible with windows versions older than 7/2008R2. +# See MSDN documentation for GetProcessMemoryInfo for more information. +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 + LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) ifeq ($(ENABLE_DEBUG_SYMBOLS), true) From david.holmes at oracle.com Mon May 4 07:23:21 2015 From: david.holmes at oracle.com (David Holmes) Date: Mon, 04 May 2015 17:23:21 +1000 Subject: RFR 8072906: sun/management/jmxremote/bootstrap/CustomLauncherTest.java failing on embedded platform In-Reply-To: <55470E1D.1080508@oracle.com> References: <554240A6.7070309@oracle.com> <554299C2.6040301@oracle.com> <55470E1D.1080508@oracle.com> Message-ID: <55471E69.4050404@oracle.com> On 4/05/2015 4:13 PM, Jaroslav Bachorik wrote: > On 30.4.2015 23:08, David Holmes wrote: >> On 1/05/2015 12:48 AM, Jaroslav Bachorik wrote: >>> Please, review the following test change >>> >>> Issue : https://bugs.openjdk.java.net/browse/JDK-8072906 >>> Webrev: http://cr.openjdk.java.net/~jbachorik/8072906/webrev.00 >>> >>> The test tries to exercise a custom launcher on embedded platforms and >>> fails because the corresponding binary is not available. The test should >>> not require the presence of the binary and skip the execution gracefully >>> if it can't find one - this was broken while addressing JDK-8029809. >>> >>> The fix brings back the ability just skip the test if the launcher >>> binary does not exist. >> >> A message to that effect would be useful. > > The message is printed at L247. It's been there for some time and > because of that is not the part of the generated webrev. Okay - thanks. David > -JB- > >> >> Thanks, >> David >> >>> Thanks, >>> >>> -JB- > From jaroslav.bachorik at oracle.com Mon May 4 07:26:48 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 04 May 2015 09:26:48 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> Message-ID: <55471F38.2000003@oracle.com> Looks ok. -JB- On 4.5.2015 08:52, Staffan Larsen wrote: > This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. > > I will push the change below directly to jdk9/hs. > > Thanks, > /Staffan > > > diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk > --- a/make/lib/Lib-java.management.gmk > +++ b/make/lib/Lib-java.management.gmk > @@ -38,11 +38,6 @@ > $(LIBJAVA_HEADER_FLAGS) \ > # > > -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate > -# a binary that is compatible with windows versions older than 7/2008R2. > -# See MSDN documentation for GetProcessMemoryInfo for more information. > -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 > - > LIBMANAGEMENT_OPTIMIZATION := HIGH > ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) > ifeq ($(ENABLE_DEBUG_SYMBOLS), true) > diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk > --- a/make/lib/Lib-jdk.management.gmk > +++ b/make/lib/Lib-jdk.management.gmk > @@ -39,6 +39,11 @@ > $(LIBJAVA_HEADER_FLAGS) \ > # > > +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate > +# a binary that is compatible with windows versions older than 7/2008R2. > +# See MSDN documentation for GetProcessMemoryInfo for more information. > +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 > + > LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH > ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) > ifeq ($(ENABLE_DEBUG_SYMBOLS), true) > From david.holmes at oracle.com Mon May 4 07:27:16 2015 From: david.holmes at oracle.com (David Holmes) Date: Mon, 04 May 2015 17:27:16 +1000 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> Message-ID: <55471F54.6090804@oracle.com> Hi Staffan, Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? Thanks, David On 4/05/2015 4:52 PM, Staffan Larsen wrote: > This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. > > I will push the change below directly to jdk9/hs. > > Thanks, > /Staffan > > > diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk > --- a/make/lib/Lib-java.management.gmk > +++ b/make/lib/Lib-java.management.gmk > @@ -38,11 +38,6 @@ > $(LIBJAVA_HEADER_FLAGS) \ > # > > -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate > -# a binary that is compatible with windows versions older than 7/2008R2. > -# See MSDN documentation for GetProcessMemoryInfo for more information. > -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 > - > LIBMANAGEMENT_OPTIMIZATION := HIGH > ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) > ifeq ($(ENABLE_DEBUG_SYMBOLS), true) > diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk > --- a/make/lib/Lib-jdk.management.gmk > +++ b/make/lib/Lib-jdk.management.gmk > @@ -39,6 +39,11 @@ > $(LIBJAVA_HEADER_FLAGS) \ > # > > +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate > +# a binary that is compatible with windows versions older than 7/2008R2. > +# See MSDN documentation for GetProcessMemoryInfo for more information. > +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 > + > LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH > ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) > ifeq ($(ENABLE_DEBUG_SYMBOLS), true) > From staffan.larsen at oracle.com Mon May 4 07:33:18 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 4 May 2015 09:33:18 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: <55471F54.6090804@oracle.com> References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> <55471F54.6090804@oracle.com> Message-ID: <2108C368-9023-4AFA-B35C-C954A63C391A@oracle.com> > On 4 maj 2015, at 09:27, David Holmes wrote: > > Hi Staffan, > > Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? I?ll let the build team comment on that. /Staffan > > Thanks, > David > > On 4/05/2015 4:52 PM, Staffan Larsen wrote: >> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. >> >> I will push the change below directly to jdk9/hs. >> >> Thanks, >> /Staffan >> >> >> diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk >> --- a/make/lib/Lib-java.management.gmk >> +++ b/make/lib/Lib-java.management.gmk >> @@ -38,11 +38,6 @@ >> $(LIBJAVA_HEADER_FLAGS) \ >> # >> >> -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >> -# a binary that is compatible with windows versions older than 7/2008R2. >> -# See MSDN documentation for GetProcessMemoryInfo for more information. >> -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 >> - >> LIBMANAGEMENT_OPTIMIZATION := HIGH >> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >> diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk >> --- a/make/lib/Lib-jdk.management.gmk >> +++ b/make/lib/Lib-jdk.management.gmk >> @@ -39,6 +39,11 @@ >> $(LIBJAVA_HEADER_FLAGS) \ >> # >> >> +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >> +# a binary that is compatible with windows versions older than 7/2008R2. >> +# See MSDN documentation for GetProcessMemoryInfo for more information. >> +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 >> + >> LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH >> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >> From magnus.ihse.bursie at oracle.com Mon May 4 07:54:38 2015 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Mon, 4 May 2015 09:54:38 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: <55471F54.6090804@oracle.com> References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> <55471F54.6090804@oracle.com> Message-ID: > 4 maj 2015 kl. 09:27 skrev David Holmes : > > Hi Staffan, > > Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? Or maybe as a define in the source code before the include section for the specific source code that needs a legacy version of GetProcessMemoryInfo? That seems more prudent to me. /Magnus > > Thanks, > David > >> On 4/05/2015 4:52 PM, Staffan Larsen wrote: >> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. >> >> I will push the change below directly to jdk9/hs. >> >> Thanks, >> /Staffan >> >> >> diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk >> --- a/make/lib/Lib-java.management.gmk >> +++ b/make/lib/Lib-java.management.gmk >> @@ -38,11 +38,6 @@ >> $(LIBJAVA_HEADER_FLAGS) \ >> # >> >> -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >> -# a binary that is compatible with windows versions older than 7/2008R2. >> -# See MSDN documentation for GetProcessMemoryInfo for more information. >> -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 >> - >> LIBMANAGEMENT_OPTIMIZATION := HIGH >> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >> diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk >> --- a/make/lib/Lib-jdk.management.gmk >> +++ b/make/lib/Lib-jdk.management.gmk >> @@ -39,6 +39,11 @@ >> $(LIBJAVA_HEADER_FLAGS) \ >> # >> >> +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >> +# a binary that is compatible with windows versions older than 7/2008R2. >> +# See MSDN documentation for GetProcessMemoryInfo for more information. >> +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 >> + >> LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH >> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >> From staffan.larsen at oracle.com Mon May 4 08:25:05 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 4 May 2015 10:25:05 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> <55471F54.6090804@oracle.com> Message-ID: > On 4 maj 2015, at 09:54, Magnus Ihse Bursie wrote: > > >> 4 maj 2015 kl. 09:27 skrev David Holmes : >> >> Hi Staffan, >> >> Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? > > Or maybe as a define in the source code before the include section for the specific source code that needs a legacy version of GetProcessMemoryInfo? That seems more prudent to me. In this case the function is called the same and works the same in all versions of Windows (there is no ?legacy version?). What differs is that it is linked against different names in different libs for different versions of Windows. So from a source code perspective there is no difference, but from a build perspective there is. /Staffan > > /Magnus > >> >> Thanks, >> David >> >>> On 4/05/2015 4:52 PM, Staffan Larsen wrote: >>> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. >>> >>> I will push the change below directly to jdk9/hs. >>> >>> Thanks, >>> /Staffan >>> >>> >>> diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk >>> --- a/make/lib/Lib-java.management.gmk >>> +++ b/make/lib/Lib-java.management.gmk >>> @@ -38,11 +38,6 @@ >>> $(LIBJAVA_HEADER_FLAGS) \ >>> # >>> >>> -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>> -# a binary that is compatible with windows versions older than 7/2008R2. >>> -# See MSDN documentation for GetProcessMemoryInfo for more information. >>> -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 >>> - >>> LIBMANAGEMENT_OPTIMIZATION := HIGH >>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >>> diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk >>> --- a/make/lib/Lib-jdk.management.gmk >>> +++ b/make/lib/Lib-jdk.management.gmk >>> @@ -39,6 +39,11 @@ >>> $(LIBJAVA_HEADER_FLAGS) \ >>> # >>> >>> +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>> +# a binary that is compatible with windows versions older than 7/2008R2. >>> +# See MSDN documentation for GetProcessMemoryInfo for more information. >>> +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 >>> + >>> LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH >>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >>> From erik.joelsson at oracle.com Mon May 4 09:11:12 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Mon, 04 May 2015 11:11:12 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> <55471F54.6090804@oracle.com> Message-ID: <554737B0.7030507@oracle.com> On 2015-05-04 10:25, Staffan Larsen wrote: >> On 4 maj 2015, at 09:54, Magnus Ihse Bursie wrote: >> >> >>> 4 maj 2015 kl. 09:27 skrev David Holmes : >>> >>> Hi Staffan, >>> >>> Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? >> Or maybe as a define in the source code before the include section for the specific source code that needs a legacy version of GetProcessMemoryInfo? That seems more prudent to me. > In this case the function is called the same and works the same in all versions of Windows (there is no ?legacy version?). What differs is that it is linked against different names in different libs for different versions of Windows. So from a source code perspective there is no difference, but from a build perspective there is. I think the placement suggested here is fine. It could be moved to configure for global application, but I doubt it will ever be needed. As soon as we drop support for Windows versions older than 7, we can remove this option. /Erik > /Staffan > >> /Magnus >> >>> Thanks, >>> David >>> >>>> On 4/05/2015 4:52 PM, Staffan Larsen wrote: >>>> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. >>>> >>>> I will push the change below directly to jdk9/hs. >>>> >>>> Thanks, >>>> /Staffan >>>> >>>> >>>> diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk >>>> --- a/make/lib/Lib-java.management.gmk >>>> +++ b/make/lib/Lib-java.management.gmk >>>> @@ -38,11 +38,6 @@ >>>> $(LIBJAVA_HEADER_FLAGS) \ >>>> # >>>> >>>> -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>>> -# a binary that is compatible with windows versions older than 7/2008R2. >>>> -# See MSDN documentation for GetProcessMemoryInfo for more information. >>>> -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 >>>> - >>>> LIBMANAGEMENT_OPTIMIZATION := HIGH >>>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >>>> diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk >>>> --- a/make/lib/Lib-jdk.management.gmk >>>> +++ b/make/lib/Lib-jdk.management.gmk >>>> @@ -39,6 +39,11 @@ >>>> $(LIBJAVA_HEADER_FLAGS) \ >>>> # >>>> >>>> +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>>> +# a binary that is compatible with windows versions older than 7/2008R2. >>>> +# See MSDN documentation for GetProcessMemoryInfo for more information. >>>> +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 >>>> + >>>> LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH >>>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >>>> From staffan.larsen at oracle.com Mon May 4 09:43:42 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 4 May 2015 11:43:42 +0200 Subject: RFR: JDK-8079248 JDK fails with "jdk\\bin\\management_ext.dll: The specified procedure could not be found" In-Reply-To: <554737B0.7030507@oracle.com> References: <480D0346-4F6B-492F-B243-6A045FC40193@oracle.com> <55471F54.6090804@oracle.com> <554737B0.7030507@oracle.com> Message-ID: <1B30D60F-7C14-4BB3-875D-181DC47B85BE@oracle.com> Thanks for the reviews - the fix is now in the JPRT queue. /Staffan > On 4 maj 2015, at 11:11, Erik Joelsson wrote: > > > On 2015-05-04 10:25, Staffan Larsen wrote: >>> On 4 maj 2015, at 09:54, Magnus Ihse Bursie wrote: >>> >>> >>>> 4 maj 2015 kl. 09:27 skrev David Holmes : >>>> >>>> Hi Staffan, >>>> >>>> Seems fine as a spot fix but I'm wondering if this shouldn't be a common option for all the dlls now we are building with VS2013? >>> Or maybe as a define in the source code before the include section for the specific source code that needs a legacy version of GetProcessMemoryInfo? That seems more prudent to me. >> In this case the function is called the same and works the same in all versions of Windows (there is no ?legacy version?). What differs is that it is linked against different names in different libs for different versions of Windows. So from a source code perspective there is no difference, but from a build perspective there is. > I think the placement suggested here is fine. It could be moved to configure for global application, but I doubt it will ever be needed. As soon as we drop support for Windows versions older than 7, we can remove this option. > > /Erik >> /Staffan >> >>> /Magnus >>> >>>> Thanks, >>>> David >>>> >>>>> On 4/05/2015 4:52 PM, Staffan Larsen wrote: >>>>> This is a P1 bug that surfaced when changes from jdk9/dev and jdk9/hs-rt met in jdk9/hs. In this case the windows compiler upgrades in jdk9/dev met changes in jdk9/hs-rt that moved a call to GetProcessMemoryInfo from management.dll to management_ext.dll. With the compiler upgrades PSAPI_VERSION=1 is needed when compiling the library calling GetProcessMemoryInfo. This fix simply moves that patch from make/lib/Lib-java.management.gmk to make/lib/Lib-jdk.management.gmk. The patch was introduced in JDK-8076557. >>>>> >>>>> I will push the change below directly to jdk9/hs. >>>>> >>>>> Thanks, >>>>> /Staffan >>>>> >>>>> >>>>> diff --git a/make/lib/Lib-java.management.gmk b/make/lib/Lib-java.management.gmk >>>>> --- a/make/lib/Lib-java.management.gmk >>>>> +++ b/make/lib/Lib-java.management.gmk >>>>> @@ -38,11 +38,6 @@ >>>>> $(LIBJAVA_HEADER_FLAGS) \ >>>>> # >>>>> >>>>> -# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>>>> -# a binary that is compatible with windows versions older than 7/2008R2. >>>>> -# See MSDN documentation for GetProcessMemoryInfo for more information. >>>>> -BUILD_LIBMANAGEMENT_CFLAGS += -DPSAPI_VERSION=1 >>>>> - >>>>> LIBMANAGEMENT_OPTIMIZATION := HIGH >>>>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>>>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) >>>>> diff --git a/make/lib/Lib-jdk.management.gmk b/make/lib/Lib-jdk.management.gmk >>>>> --- a/make/lib/Lib-jdk.management.gmk >>>>> +++ b/make/lib/Lib-jdk.management.gmk >>>>> @@ -39,6 +39,11 @@ >>>>> $(LIBJAVA_HEADER_FLAGS) \ >>>>> # >>>>> >>>>> +# In (at least) VS2013 and later, -DPSAPI_VERSION=1 is needed to generate >>>>> +# a binary that is compatible with windows versions older than 7/2008R2. >>>>> +# See MSDN documentation for GetProcessMemoryInfo for more information. >>>>> +BUILD_LIBMANAGEMENT_EXT_CFLAGS += -DPSAPI_VERSION=1 >>>>> + >>>>> LIBMANAGEMENT_EXT_OPTIMIZATION := HIGH >>>>> ifneq ($(findstring $(OPENJDK_TARGET_OS), solaris linux), ) >>>>> ifeq ($(ENABLE_DEBUG_SYMBOLS), true) -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Mon May 4 10:09:49 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 04 May 2015 03:09:49 -0700 Subject: RFR 8072906: sun/management/jmxremote/bootstrap/CustomLauncherTest.java failing on embedded platform In-Reply-To: <55471E69.4050404@oracle.com> References: <554240A6.7070309@oracle.com> <554299C2.6040301@oracle.com> <55470E1D.1080508@oracle.com> <55471E69.4050404@oracle.com> Message-ID: <5547456D.7030905@oracle.com> On 5/4/15 12:23 AM, David Holmes wrote: > On 4/05/2015 4:13 PM, Jaroslav Bachorik wrote: >> On 30.4.2015 23:08, David Holmes wrote: >>> On 1/05/2015 12:48 AM, Jaroslav Bachorik wrote: >>>> Please, review the following test change >>>> >>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8072906 >>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8072906/webrev.00 >>>> >>>> The test tries to exercise a custom launcher on embedded platforms and >>>> fails because the corresponding binary is not available. The test >>>> should >>>> not require the presence of the binary and skip the execution >>>> gracefully >>>> if it can't find one - this was broken while addressing JDK-8029809. >>>> >>>> The fix brings back the ability just skip the test if the launcher >>>> binary does not exist. >>> >>> A message to that effect would be useful. >> >> The message is printed at L247. It's been there for some time and >> because of that is not the part of the generated webrev. > > Okay - thanks. Ok from me too. :) Thanks, Serguei > > David > > >> -JB- >> >>> >>> Thanks, >>> David >>> >>>> Thanks, >>>> >>>> -JB- >> From erik.joelsson at oracle.com Mon May 4 13:46:19 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Mon, 04 May 2015 15:46:19 +0200 Subject: RFR: JDK-8077422: hprof agent: Build failed with VS2013 Update 4 In-Reply-To: <55381A5D.1020900@oracle.com> References: <55380D4E.7080700@oracle.com> <55380EA4.7030901@oracle.com> <55381A5D.1020900@oracle.com> Message-ID: <5547782B.2040804@oracle.com> Hello, This bug has been marked as won't fix since the code it affects is going to be removed. However, until that removal happens, we can't build using VS2013 SP4. I'm in the process of updating the compilers used internally at Oracle to that specific version and to be able to continue that work, I need this to be buildable now. For this reason I propose Peter Brunet's suggested solution to the problem, just adding a C macro definition that makes it compile. I slightly adjusted his initial proposal. Bug: https://bugs.openjdk.java.net/browse/JDK-8077422 Patch: diff -r 83ff0dedf9e1 make/lib/Lib-jdk.hprof.agent.gmk --- a/make/lib/Lib-jdk.hprof.agent.gmk +++ b/make/lib/Lib-jdk.hprof.agent.gmk @@ -31,7 +31,7 @@ BUILD_LIBHPROF_CFLAGS := $(addprefix -I, $(BUILD_LIBHPROF_SRC)) \ -I$(JDK_TOPDIR)/src/demo/share/jvmti/java_crw_demo BUILD_LIBHPROF_LDFLAGS := LIBHPROF_OPTIMIZATION := HIGHEST @@ -49,6 +49,7 @@ CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ $(BUILD_LIBHPROF_CFLAGS), \ CFLAGS_debug := -DHPROF_LOGGING, \ + CFLAGS_windows := -D_WINSOCK_DEPRECATED_NO_WARNINGS, \ MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ LDFLAGS := $(LDFLAGS_JDKLIB) \ $(call SET_SHARED_LIBRARY_ORIGIN), \ /Erik On 2015-04-23 00:02, Pete Brunet wrote: > This hack to jdk/make/lib/Lib-jdk.hprof.agent.gmk works for now: > > LANG := C, \ > OPTIMIZATION := $(LIBHPROF_OPTIMIZATION), \ > CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ > + -D_WINSOCK_DEPRECATED_NO_WARNINGS \ > $(BUILD_LIBHPROF_CFLAGS), \ > CFLAGS_debug := -DHPROF_LOGGING, \ > MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ > > Pete > > On 4/22/15 4:12 PM, Pete Brunet wrote: >> p.s. I also had done a make reconfigure and make clean. I'll try make >> clean and reconfigure from bash. >> >> On 4/22/15 4:06 PM, Pete Brunet wrote: >>> I was able to build 9 OK then I switched from VS2010 to VS2013 and now >>> get the following. I tried hg tpull -u but that didn't help. >>> >>> Is there something I need to do besides installing VS Pro 2013 with >>> Update 4? >>> >>> $ make images 2>&1 | tee make-64.log >>> Building target 'images' in configuration >>> 'windows-x86_64-normal-server-release' >>> >>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>> : error C2220: warning treated as error - no 'object' file generated >>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>> : warning C4996: 'gethostbyname': Use getaddrinfo() or GetAddrInfoW() >>> instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated >>> API warnings >>> c:\progra~2\wi3cf2~1\8.1\include\um\winsock2.h(2238) : see >>> declaration of 'gethostbyname' >>> Lib-jdk.hprof.agent.gmk:44: recipe for target >>> '/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.h >>> prof.agent/libhprof_jvmti/hprof_md.obj' failedmake[3]: *** >>> [/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.hprof.agent/libhprof_jvmti/hprof_md.obj] >>> Error 2 >>> make/Main.gmk:168: recipe for target 'jdk.hprof.agent-libs' failed >>> make[2]: *** [jdk.hprof.agent-libs] Error 1 >>> make[2]: *** Waiting for unfinished jobs.... >>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:255: >>> recipe for target 'main' failed >>> make[1]: *** [main] Error 1 >>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:171: >>> recipe for target 'images' failed >>> make: *** [images] Error 2 From staffan.larsen at oracle.com Mon May 4 15:13:22 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 4 May 2015 17:13:22 +0200 Subject: RFR: JDK-8077422: hprof agent: Build failed with VS2013 Update 4 In-Reply-To: <5547782B.2040804@oracle.com> References: <55380D4E.7080700@oracle.com> <55380EA4.7030901@oracle.com> <55381A5D.1020900@oracle.com> <5547782B.2040804@oracle.com> Message-ID: Looks good! Thanks, /Staffan > On 4 maj 2015, at 15:46, Erik Joelsson wrote: > > Hello, > > This bug has been marked as won't fix since the code it affects is going to be removed. However, until that removal happens, we can't build using VS2013 SP4. I'm in the process of updating the compilers used internally at Oracle to that specific version and to be able to continue that work, I need this to be buildable now. For this reason I propose Peter Brunet's suggested solution to the problem, just adding a C macro definition that makes it compile. I slightly adjusted his initial proposal. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8077422 > Patch: > diff -r 83ff0dedf9e1 make/lib/Lib-jdk.hprof.agent.gmk > --- a/make/lib/Lib-jdk.hprof.agent.gmk > +++ b/make/lib/Lib-jdk.hprof.agent.gmk > @@ -31,7 +31,7 @@ > > BUILD_LIBHPROF_CFLAGS := $(addprefix -I, $(BUILD_LIBHPROF_SRC)) \ > -I$(JDK_TOPDIR)/src/demo/share/jvmti/java_crw_demo > > BUILD_LIBHPROF_LDFLAGS := > > LIBHPROF_OPTIMIZATION := HIGHEST > @@ -49,6 +49,7 @@ > CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ > $(BUILD_LIBHPROF_CFLAGS), \ > CFLAGS_debug := -DHPROF_LOGGING, \ > + CFLAGS_windows := -D_WINSOCK_DEPRECATED_NO_WARNINGS, \ > MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ > LDFLAGS := $(LDFLAGS_JDKLIB) \ > $(call SET_SHARED_LIBRARY_ORIGIN), \ > > /Erik > > On 2015-04-23 00:02, Pete Brunet wrote: >> This hack to jdk/make/lib/Lib-jdk.hprof.agent.gmk works for now: >> >> LANG := C, \ >> OPTIMIZATION := $(LIBHPROF_OPTIMIZATION), \ >> CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ >> + -D_WINSOCK_DEPRECATED_NO_WARNINGS \ >> $(BUILD_LIBHPROF_CFLAGS), \ >> CFLAGS_debug := -DHPROF_LOGGING, \ >> MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ >> >> Pete >> >> On 4/22/15 4:12 PM, Pete Brunet wrote: >>> p.s. I also had done a make reconfigure and make clean. I'll try make >>> clean and reconfigure from bash. >>> >>> On 4/22/15 4:06 PM, Pete Brunet wrote: >>>> I was able to build 9 OK then I switched from VS2010 to VS2013 and now >>>> get the following. I tried hg tpull -u but that didn't help. >>>> >>>> Is there something I need to do besides installing VS Pro 2013 with >>>> Update 4? >>>> >>>> $ make images 2>&1 | tee make-64.log >>>> Building target 'images' in configuration >>>> 'windows-x86_64-normal-server-release' >>>> >>>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>>> : error C2220: warning treated as error - no 'object' file generated >>>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>>> : warning C4996: 'gethostbyname': Use getaddrinfo() or GetAddrInfoW() >>>> instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated >>>> API warnings >>>> c:\progra~2\wi3cf2~1\8.1\include\um\winsock2.h(2238) : see >>>> declaration of 'gethostbyname' >>>> Lib-jdk.hprof.agent.gmk:44: recipe for target >>>> '/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.h >>>> prof.agent/libhprof_jvmti/hprof_md.obj' failedmake[3]: *** >>>> [/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.hprof.agent/libhprof_jvmti/hprof_md.obj] >>>> Error 2 >>>> make/Main.gmk:168: recipe for target 'jdk.hprof.agent-libs' failed >>>> make[2]: *** [jdk.hprof.agent-libs] Error 1 >>>> make[2]: *** Waiting for unfinished jobs.... >>>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:255: >>>> recipe for target 'main' failed >>>> make[1]: *** [main] Error 1 >>>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:171: >>>> recipe for target 'images' failed >>>> make: *** [images] Error 2 > From tim.bell at oracle.com Mon May 4 15:17:29 2015 From: tim.bell at oracle.com (Tim Bell) Date: Mon, 04 May 2015 08:17:29 -0700 Subject: RFR: JDK-8077422: hprof agent: Build failed with VS2013 Update 4 In-Reply-To: References: <55380D4E.7080700@oracle.com> <55380EA4.7030901@oracle.com> <55381A5D.1020900@oracle.com> <5547782B.2040804@oracle.com> Message-ID: <55478D89.70708@oracle.com> Erik: Looks good to me as well. Tim On 05/04/15 08:13, Staffan Larsen wrote: > Looks good! > > Thanks, > /Staffan > >> On 4 maj 2015, at 15:46, Erik Joelsson wrote: >> >> Hello, >> >> This bug has been marked as won't fix since the code it affects is going to be removed. However, until that removal happens, we can't build using VS2013 SP4. I'm in the process of updating the compilers used internally at Oracle to that specific version and to be able to continue that work, I need this to be buildable now. For this reason I propose Peter Brunet's suggested solution to the problem, just adding a C macro definition that makes it compile. I slightly adjusted his initial proposal. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8077422 >> Patch: >> diff -r 83ff0dedf9e1 make/lib/Lib-jdk.hprof.agent.gmk >> --- a/make/lib/Lib-jdk.hprof.agent.gmk >> +++ b/make/lib/Lib-jdk.hprof.agent.gmk >> @@ -31,7 +31,7 @@ >> >> BUILD_LIBHPROF_CFLAGS := $(addprefix -I, $(BUILD_LIBHPROF_SRC)) \ >> -I$(JDK_TOPDIR)/src/demo/share/jvmti/java_crw_demo >> >> BUILD_LIBHPROF_LDFLAGS := >> >> LIBHPROF_OPTIMIZATION := HIGHEST >> @@ -49,6 +49,7 @@ >> CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ >> $(BUILD_LIBHPROF_CFLAGS), \ >> CFLAGS_debug := -DHPROF_LOGGING, \ >> + CFLAGS_windows := -D_WINSOCK_DEPRECATED_NO_WARNINGS, \ >> MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ >> LDFLAGS := $(LDFLAGS_JDKLIB) \ >> $(call SET_SHARED_LIBRARY_ORIGIN), \ >> >> /Erik >> >> On 2015-04-23 00:02, Pete Brunet wrote: >>> This hack to jdk/make/lib/Lib-jdk.hprof.agent.gmk works for now: >>> >>> LANG := C, \ >>> OPTIMIZATION := $(LIBHPROF_OPTIMIZATION), \ >>> CFLAGS := $(CFLAGS_JDKLIB) $(CFLAGS_WARNINGS_ARE_ERRORS) \ >>> + -D_WINSOCK_DEPRECATED_NO_WARNINGS \ >>> $(BUILD_LIBHPROF_CFLAGS), \ >>> CFLAGS_debug := -DHPROF_LOGGING, \ >>> MAPFILE := $(JDK_TOPDIR)/make/mapfiles/libhprof/mapfile-vers, \ >>> >>> Pete >>> >>> On 4/22/15 4:12 PM, Pete Brunet wrote: >>>> p.s. I also had done a make reconfigure and make clean. I'll try make >>>> clean and reconfigure from bash. >>>> >>>> On 4/22/15 4:06 PM, Pete Brunet wrote: >>>>> I was able to build 9 OK then I switched from VS2010 to VS2013 and now >>>>> get the following. I tried hg tpull -u but that didn't help. >>>>> >>>>> Is there something I need to do besides installing VS Pro 2013 with >>>>> Update 4? >>>>> >>>>> $ make images 2>&1 | tee make-64.log >>>>> Building target 'images' in configuration >>>>> 'windows-x86_64-normal-server-release' >>>>> >>>>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>>>> : error C2220: warning treated as error - no 'object' file generated >>>>> c:/Users/Pete/JDK9/INTJDK-7615864/client/jdk/src/jdk.hprof.agent/windows/native/libhprof/hprof_md.c(86) >>>>> : warning C4996: 'gethostbyname': Use getaddrinfo() or GetAddrInfoW() >>>>> instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated >>>>> API warnings >>>>> c:\progra~2\wi3cf2~1\8.1\include\um\winsock2.h(2238) : see >>>>> declaration of 'gethostbyname' >>>>> Lib-jdk.hprof.agent.gmk:44: recipe for target >>>>> '/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.h >>>>> prof.agent/libhprof_jvmti/hprof_md.obj' failedmake[3]: *** >>>>> [/cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/build/windows-x86_64-normal-server-release/support/native/jdk.hprof.agent/libhprof_jvmti/hprof_md.obj] >>>>> Error 2 >>>>> make/Main.gmk:168: recipe for target 'jdk.hprof.agent-libs' failed >>>>> make[2]: *** [jdk.hprof.agent-libs] Error 1 >>>>> make[2]: *** Waiting for unfinished jobs.... >>>>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:255: >>>>> recipe for target 'main' failed >>>>> make[1]: *** [main] Error 1 >>>>> /cygdrive/c/Users/Pete/JDK9/INTJDK-7615864/client/make/Init.gmk:171: >>>>> recipe for target 'images' failed >>>>> make: *** [images] Error 2 From shanliang.jiang at oracle.com Mon May 4 17:03:56 2015 From: shanliang.jiang at oracle.com (Shanliang Jiang) Date: Mon, 04 May 2015 19:03:56 +0200 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: References: Message-ID: <5547A67C.70304@oracle.com> Hi Yuji, (I reply to serviceability alias) When you create a RMI server connector, you can specify a RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE , this allows you to specify your SoTimeout. Hope this helps. Shanliang KUBOTA Yuji wrote: > Hi all, > > I want to contribute this issue. > If there are a problem about this patch or a better way for openjdk > community, please advise me. > > Thanks for > > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji : > >> Hi all, >> >> I found an infinite waiting at TCPChannel#createConnection. >> This method flushes the DataOutputStream without the socket timeout settings >> when choose stream protocol [1]. >> >> If connection lost (the destination server do no return response) >> during the flush, >> this method has possibilities to take long time beyond the expectations >> at java.net.SocketInputStream.socketRead0 as following stack trace. >> >> stack trace : >> at java.net.SocketInputStream.socketRead0(SocketInputStream.java) >> at java.net.SocketInputStream.read(SocketInputStream.java) >> at java.net.SocketInputStream.read(SocketInputStream.java) >> at sun.security.ssl.InputRecord.readFully(InputRecord.java) >> at sun.security.ssl.InputRecord.read(InputRecord.java) >> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) >> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) >> at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) >> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) >> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) >> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) >> at java.io.DataOutputStream.flush(DataOutputStream.java) >> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) >> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) >> at javax.management.remote.rmi.RMIServerImpl_Stub.newClient >> at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) >> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) >> at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) >> >> When create connection, we cannot set the timeout by properties. >> Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. >> So I wrote a patch to fix this infinite waiting by using property-configured value: >> sun.rmi.transport.tcp.responseTimeout. >> >> Please review this patch. :) >> >> Note: My OCA has been processed a few hour ago, so my name may take a >> short time to >> appear on the OCA signatories page. >> >> Thanks, >> KUBOTA Yuji >> >> [1]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 >> >> diff --git a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> @@ -222,20 +222,34 @@ >> // choose protocol (single op if not reusable socket) >> if (!conn.isReusable()) { >> out.writeByte(TransportConstants.SingleOpProtocol); >> } else { >> out.writeByte(TransportConstants.StreamProtocol); >> + >> + int usableSoTimeout = 0; >> + try { >> + /* >> + * If socket factory had set a zero timeout on its own, >> + * then set the property-configured value to prevent >> + * an infinite waiting. >> + */ >> + usableSoTimeout = sock.getSoTimeout(); >> + if (usableSoTimeout == 0) { >> + usableSoTimeout = responseTimeout; >> + } >> + sock.setSoTimeout(usableSoTimeout); >> + } catch (Exception e) { >> + // if we fail to set this, ignore and proceed anyway >> + } >> out.flush(); >> >> /* >> * Set socket read timeout to configured value for JRMP >> * connection handshake; this also serves to guard against >> * non-JRMP servers that do not respond (see 4322806). >> */ >> - int originalSoTimeout = 0; >> try { >> - originalSoTimeout = sock.getSoTimeout(); >> sock.setSoTimeout(handshakeTimeout); >> } catch (Exception e) { >> // if we fail to set this, ignore and proceed anyway >> } >> >> @@ -279,18 +293,11 @@ >> * connection. NOTE: this timeout, if configured to a >> * finite duration, places an upper bound on the time >> * that a remote method call is permitted to execute. >> */ >> try { >> - /* >> - * If socket factory had set a non-zero timeout on its >> - * own, then restore it instead of using the property- >> - * configured value. >> - */ >> - sock.setSoTimeout((originalSoTimeout != 0 ? >> - originalSoTimeout : >> - responseTimeout)); >> + sock.setSoTimeout(usableSoTimeout); >> } catch (Exception e) { >> // if we fail to set this, ignore and proceed anyway >> } >> >> out.flush(); >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From yekaterina.kantserova at oracle.com Tue May 5 09:04:20 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Tue, 05 May 2015 11:04:20 +0200 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests Message-ID: <55488794.3060008@oracle.com> Hi, Could I please have a review of this fix. bug: https://bugs.openjdk.java.net/browse/JDK-8078896 webrev: http://cr.openjdk.java.net/~ykantser/8078896 The push will be pushed to jdk9/dev. Thanks, Katja Alan's clarification from same change in hotspot (RFR: JDK-8075586: add @modules as needed to the open hotspot tests) ========== Just to put more context on this patch and similar patches that will be proposed for tests in other repos. The new @modules tag can be used for test selection (e.g. no point running a test that make use of types in module java.management if the runtime under test does not have the management module), and additionally to declare a dependency on JDK-internal APIs. If my test uses sun.misc.Unsafe for example then the runtime under test needs to export that API for the test to compile and run. Clearly the latter requires the module system in JDK 9 so consider this part as just preparing the tests for when that day comes. One other thing to say to say is that Alex has tooling to examine the test tree and create or change the @modules tag as needed. The intention is that this should run periodically to refresh the @modules tags. In this period before the module system then we can't expect everyone that is adding or changing tests to have internalized the module graph [1] and furthermore know which APIs are exported or not. In other words, the intention is not to needlessly burden anyone that adds or changes tests. Clearly making use of a new jtreg tag means everyone with a local copy of jtreg should grab a recent build. Finally, I hope in time that the TEST.groups files can be cleaned up to remove the needs_* groups, this is important for the group definitions in the jdk repo mostly (but there are some in the hotspot TEST.groups too). -Alan From dmitry.samersoff at oracle.com Tue May 5 10:01:13 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 05 May 2015 13:01:13 +0300 Subject: RFR(M): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo Message-ID: <554894E9.8020908@oracle.com> Everyone, Please review the fix: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ heap dcmd outputs the same information as SIGBREAK finalizerinfo dcmd outputs a list of all classes in finalization queue with count -Dmitry -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From staffan.larsen at oracle.com Tue May 5 10:38:37 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 5 May 2015 12:38:37 +0200 Subject: RFR(M): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <554894E9.8020908@oracle.com> References: <554894E9.8020908@oracle.com> Message-ID: <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> Dmitry, I think this should be reviewed on hotspot-gc and core-libs-dev as well considering the changes to Finalizer. I?m a little worried about the potentially very large String that is returned from printFinalizationQueue(). A possible different approach would be to write printFinalizationQueue() in C++ inside Hotspot. That would allow for outputting one line at a time. The output would still be saved in memory (since the stream is buffered), but at least the data is only saved once in memory, then. It would make the code a bit harder to write, so its a question of how scared we are of running out of memory. I see you are traversing the ?unfinalized? list in Finalizer, whereas the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am not sure of the difference, perhaps someone from the GC team can help out? For the output, it would be a nice touch to sort it on the number of references for each type. Perhaps outputting it more like a table (see the old code again) would also make it easier to digest. In diagnosticCommand.hpp, the new commands should override permission() and limit access: static const JavaPermission permission() { JavaPermission p = {"java.lang.management.ManagementPermission", "monitor", NULL}; return p; } The two tests don?t validate the output in any way. Would it be possible to add validation? Perhaps hard to make sure an object is on the finalizer queue? Hmm. Thanks, /Staffan > On 5 maj 2015, at 12:01, Dmitry Samersoff wrote: > > Everyone, > > Please review the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ > > heap dcmd outputs the same information as SIGBREAK > > finalizerinfo dcmd outputs a list of all classes in finalization queue > with count > > -Dmitry > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From yekaterina.kantserova at oracle.com Tue May 5 13:24:12 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Tue, 05 May 2015 15:24:12 +0200 Subject: RFR(XS): 8076998: BadHandshakeTest.java fails due to warnings in output Message-ID: <5548C47C.6070300@oracle.com> Hi, Could I please have a review of this fix. bug: https://bugs.openjdk.java.net/browse/JDK-8076998 webrev: http://cr.openjdk.java.net/~ykantser/8076998/webrev.00/ The test has been checking the first line of output for either 'Listening for transport dt_socket at address:' or 'Address already in use'. But different VM warnings can show up before these lines. The fix is to read output until one of these lines is found. Thanks, Katja From staffan.larsen at oracle.com Tue May 5 13:30:12 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 5 May 2015 15:30:12 +0200 Subject: RFR(XS): 8076998: BadHandshakeTest.java fails due to warnings in output In-Reply-To: <5548C47C.6070300@oracle.com> References: <5548C47C.6070300@oracle.com> Message-ID: <66002065-C6C5-4C46-9552-2D30431C6E26@oracle.com> Looks good! Thanks, /Staffan > On 5 maj 2015, at 15:24, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8076998 > webrev: http://cr.openjdk.java.net/~ykantser/8076998/webrev.00/ > > The test has been checking the first line of output for either 'Listening for transport dt_socket at address:' or 'Address already in use'. But different VM warnings can show up before these lines. The fix is to read output until one of these lines is found. > > Thanks, > Katja From Alan.Bateman at oracle.com Tue May 5 13:30:30 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 05 May 2015 14:30:30 +0100 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <55488794.3060008@oracle.com> References: <55488794.3060008@oracle.com> Message-ID: <5548C5F6.50102@oracle.com> On 05/05/2015 10:04, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8078896 > webrev: http://cr.openjdk.java.net/~ykantser/8078896 > > The push will be pushed to jdk9/dev. > Thanks Katja, this looks good. One thing that we should as part of this is rev the requiredVersion in jdk/test/TEST.ROOT in case people are using older versions of jtreg. I think you did that as part of the patch for the hotspot repo. -Alan. From yekaterina.kantserova at oracle.com Tue May 5 13:36:18 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Tue, 05 May 2015 15:36:18 +0200 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <5548C5F6.50102@oracle.com> References: <55488794.3060008@oracle.com> <5548C5F6.50102@oracle.com> Message-ID: <5548C752.5030308@oracle.com> Alan, Thanks for the review! And for the catch - I'll fix it. // Katja On 05/05/2015 03:30 PM, Alan Bateman wrote: > Thanks Katja, this looks good. One thing that we should as part of > this is rev the requiredVersion in jdk/test/TEST.ROOT in case people > are using older versions of jtreg. I think you did that as part of the > patch for the hotspot repo. > > -Alan. From yekaterina.kantserova at oracle.com Tue May 5 13:36:56 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Tue, 05 May 2015 15:36:56 +0200 Subject: RFR(XS): 8076998: BadHandshakeTest.java fails due to warnings in output In-Reply-To: <66002065-C6C5-4C46-9552-2D30431C6E26@oracle.com> References: <5548C47C.6070300@oracle.com> <66002065-C6C5-4C46-9552-2D30431C6E26@oracle.com> Message-ID: <5548C778.4080407@oracle.com> Thanks! // Katja On 05/05/2015 03:30 PM, Staffan Larsen wrote: > Looks good! > > Thanks, > /Staffan > >> On 5 maj 2015, at 15:24, Yekaterina Kantserova wrote: >> >> Hi, >> >> Could I please have a review of this fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8076998 >> webrev: http://cr.openjdk.java.net/~ykantser/8076998/webrev.00/ >> >> The test has been checking the first line of output for either 'Listening for transport dt_socket at address:' or 'Address already in use'. But different VM warnings can show up before these lines. The fix is to read output until one of these lines is found. >> >> Thanks, >> Katja From peter.levart at gmail.com Tue May 5 14:40:28 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 05 May 2015 16:40:28 +0200 Subject: RFR(M): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> Message-ID: <5548D65C.8040304@gmail.com> Hi Dmitry, Staffan, On 05/05/2015 12:38 PM, Staffan Larsen wrote: > Dmitry, > > I think this should be reviewed on hotspot-gc and core-libs-dev as well considering the changes to Finalizer. > > I?m a little worried about the potentially very large String that is returned from printFinalizationQueue(). A possible different approach would be to write printFinalizationQueue() in C++ inside Hotspot. That would allow for outputting one line at a time. The output would still be saved in memory (since the stream is buffered), but at least the data is only saved once in memory, then. It would make the code a bit harder to write, so its a question of how scared we are of running out of memory. If the output is just a histogram of # of instances per class name, then it should not be too large, as there are not many different classes overriding finalize() method (I counted 72 overriddings of finalize() method in the whole jdk/src tree). I'm more concerned about the fact that while traversing the list, a lock is held, which might impact prompt finalization(). Is it acceptable for diagnostic output to impact performance and/or interfere with synchronization? It might be possible to scan the list without holding a lock for diagnostic purposes if Finalizer.remove() didn't set the removed Finalizer.next pointer to point back to itself: private void remove() { synchronized (lock) { if (unfinalized == this) { if (this.next != null) { unfinalized = this.next; } else { unfinalized = this.prev; } } if (this.next != null) { this.next.prev = this.prev; } if (this.prev != null) { this.prev.next = this.next; } // this.next = this; must not be set so that we can traverse the list unsynchronized this.prev = this; /* Indicates that this has been finalized */ } } For detecting whether a Finalizer is already processed, the 'prev' pointer could be used instead: private boolean hasBeenFinalized() { return (prev == this); } Also, to make sure a data race dereferencing 'unfinalized' in unsynchronized printFinalizationQueue() would get you a fully initialized Finalizer instance (in particular the next pointer), you would have to make the 'unfinalized' field volatile or insert an Unsafe.storeFence() before assigning to 'unfinalized': private void add() { synchronized (lock) { if (unfinalized != null) { this.next = unfinalized; unfinalized.prev = this; } // make sure a data race dereferencing 'unfinalized' // in printFinalizationQueue() does see the Finalizer // instance fully initialized // (in particular the 'next' pointer) U.storeFence(); unfinalized = this; } } By doing these modifications, I think you can remove synchronized(lock) {} from printFinalizationQueue(). > > I see you are traversing the ?unfinalized? list in Finalizer, whereas the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am not sure of the difference, perhaps someone from the GC team can help out? The 'queue' is a ReferenceQueue of Finalizer (FinalReference) instances pending processing by finalizer thread because their referents are eligible for finalization (they are not reachable any more). The 'unfinalized' is a doubly-linked list of all Finalizer instances for which their referents have not been finalized yet (including those that are still reachable and alive). The later serves two purposes: - it keeps Finalizer instances reachable until they are processed - it is a source of unfinalized instances for running-finalizers-on-exit if requested by System.runFinalizersOnExit(true); So it really depends on what one would like to see. Showing the queue may be interesting if one wants to see how the finalizer thread is coping with processing the finalize() invocations. Showing unfinalized list may be interesting if one wants to know how many live + finalization pending instances are there on the heap that override finalize() method. Regards, Peter > > For the output, it would be a nice touch to sort it on the number of references for each type. Perhaps outputting it more like a table (see the old code again) would also make it easier to digest. > > In diagnosticCommand.hpp, the new commands should override permission() and limit access: > > static const JavaPermission permission() { > JavaPermission p = {"java.lang.management.ManagementPermission", > "monitor", NULL}; > return p; > } > > The two tests don?t validate the output in any way. Would it be possible to add validation? Perhaps hard to make sure an object is on the finalizer queue? Hmm. > > Thanks, > /Staffan > > >> On 5 maj 2015, at 12:01, Dmitry Samersoff wrote: >> >> Everyone, >> >> Please review the fix: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >> >> heap dcmd outputs the same information as SIGBREAK >> >> finalizerinfo dcmd outputs a list of all classes in finalization queue >> with count >> >> -Dmitry >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. From mandy.chung at oracle.com Tue May 5 20:33:27 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 05 May 2015 13:33:27 -0700 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <55488794.3060008@oracle.com> References: <55488794.3060008@oracle.com> Message-ID: <55492917.6010200@oracle.com> On 05/05/2015 02:04 AM, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8078896 > webrev: http://cr.openjdk.java.net/~ykantser/8078896 > com.sun.management has been moved to jdk.management module. The patch for JDK-8042901 is just integrated in jdk9/dev today. Most, if not all, test/com/sun/management tests need updates to require @modules jdk.management. There are several test/java/lang/management tests dependng on both java.lang.management and com.sun.management. For tests only using java.lang.management API, @modules java.management is adequate. test/com/sun/jdi/InterfaceMethodsTest.java test/com/sun/jdi/InterruptHangTest.java Since you have updated the end year of the copyright header in most tests, I spot a couple and so mention it to you. Otherwise looks good. Thanks for doing this. Mandy From mandy.chung at oracle.com Tue May 5 21:00:35 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 05 May 2015 14:00:35 -0700 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <55492917.6010200@oracle.com> References: <55488794.3060008@oracle.com> <55492917.6010200@oracle.com> Message-ID: <55492F73.2020806@oracle.com> On 05/05/2015 01:33 PM, Mandy Chung wrote: > > > On 05/05/2015 02:04 AM, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8078896 >> webrev: http://cr.openjdk.java.net/~ykantser/8078896 >> > > com.sun.management has been moved to jdk.management module. The patch > for JDK-8042901 is just integrated in jdk9/dev today. Most, if not > all, test/com/sun/management tests need updates to require @modules > jdk.management. > > There are several test/java/lang/management tests dependng on both > java.lang.management and com.sun.management. For tests only using > java.lang.management API, @modules java.management is adequate. > > test/com/sun/jdi/InterfaceMethodsTest.java > test/com/sun/jdi/InterruptHangTest.java > Since you have updated the end year of the copyright header in most > tests, I spot a couple and so mention it to you. > > Otherwise looks good. Thanks for doing this. About the test selection, one typical aspect of svc tests is to run a j* tool in a child process (e.g. jinfo, jstack, jstat, jstatd,jcmd, jps etc that are in jdk.jcmd module). I would expect all test/sun/tools/jcmd tests should have @modules jdk.jcmd and similiar for other sun/tools/j* tests. Are you thinking to come back in the next round of test selection update? Mandy From serguei.spitsyn at oracle.com Tue May 5 22:48:43 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 05 May 2015 15:48:43 -0700 Subject: RFR(XS): 8076998: BadHandshakeTest.java fails due to warnings in output In-Reply-To: <5548C47C.6070300@oracle.com> References: <5548C47C.6070300@oracle.com> Message-ID: <554948CB.2010601@oracle.com> Looks good. Thanks, Serguei On 5/5/15 6:24 AM, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8076998 > webrev: http://cr.openjdk.java.net/~ykantser/8076998/webrev.00/ > > The test has been checking the first line of output for either > 'Listening for transport dt_socket at address:' or 'Address already in > use'. But different VM warnings can show up before these lines. The > fix is to read output until one of these lines is found. > > Thanks, > Katja From Alan.Bateman at oracle.com Wed May 6 07:17:42 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 06 May 2015 08:17:42 +0100 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <55492F73.2020806@oracle.com> References: <55488794.3060008@oracle.com> <55492917.6010200@oracle.com> <55492F73.2020806@oracle.com> Message-ID: <5549C016.20002@oracle.com> On 05/05/2015 22:00, Mandy Chung wrote: > : > > > About the test selection, one typical aspect of svc tests is to run a > j* tool in a child process (e.g. jinfo, jstack, jstat, jstatd,jcmd, > jps etc that are in jdk.jcmd module). I would expect all > test/sun/tools/jcmd tests should have @modules jdk.jcmd and similiar > for other sun/tools/j* tests. Are you thinking to come back in the > next round of test selection update? Alexander Kulyakhtin's tool hasn't been pushed to jdk9/dev yet but I expect it will be run regularly to keep the tests updated. For modules then we're mostly interested in the tests that make use of JDK-internal APIs of course, the test selection aspect is for later when jtreg has support for this. -Alan From yekaterina.kantserova at oracle.com Wed May 6 07:57:41 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Wed, 06 May 2015 09:57:41 +0200 Subject: RFR(XS): 8076998: BadHandshakeTest.java fails due to warnings in output In-Reply-To: <554948CB.2010601@oracle.com> References: <5548C47C.6070300@oracle.com> <554948CB.2010601@oracle.com> Message-ID: <5549C975.1000904@oracle.com> Thanks Serguei! On 05/06/2015 12:48 AM, serguei.spitsyn at oracle.com wrote: > Looks good. > > Thanks, > Serguei > > On 5/5/15 6:24 AM, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8076998 >> webrev: http://cr.openjdk.java.net/~ykantser/8076998/webrev.00/ >> >> The test has been checking the first line of output for either >> 'Listening for transport dt_socket at address:' or 'Address already >> in use'. But different VM warnings can show up before these lines. >> The fix is to read output until one of these lines is found. >> >> Thanks, >> Katja > From kubota.yuji at gmail.com Wed May 6 09:13:25 2015 From: kubota.yuji at gmail.com (KUBOTA Yuji) Date: Wed, 6 May 2015 18:13:25 +0900 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: <5547A67C.70304@oracle.com> References: <5547A67C.70304@oracle.com> Message-ID: My apologies for re-post, I forgot to register serviceability-dev before the last post. Hi Shanliang, Thanks you for your help! RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. However, many users believe sun.rmi.transport.tcp.responseTimeout to specify the timeout, e.g. the second flush() of TCPChannel#createConnection [2]. In really, the first flush() [3] is not affected by sun.rmi.transport.tcp.responseTimeout, and will be the (potential) infinite waiting by bad luck. So I think openjdk should fix it for users. [2]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 [3]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 Thanks, Yuji 2015-05-05 2:03 GMT+09:00 Shanliang Jiang : > Hi Yuji, > > (I reply to serviceability alias) > > When you create a RMI server connector, you can specify a > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, this allows > you to specify your SoTimeout. > > Hope this helps. > > Shanliang > > > > KUBOTA Yuji wrote: > > Hi all, > > I want to contribute this issue. > If there are a problem about this patch or a better way for openjdk > community, please advise me. > > Thanks for > > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji : > > > Hi all, > > I found an infinite waiting at TCPChannel#createConnection. > This method flushes the DataOutputStream without the socket timeout settings > when choose stream protocol [1]. > > If connection lost (the destination server do no return response) > during the flush, > this method has possibilities to take long time beyond the expectations > at java.net.SocketInputStream.socketRead0 as following stack trace. > > stack trace : > at java.net.SocketInputStream.socketRead0(SocketInputStream.java) > at java.net.SocketInputStream.read(SocketInputStream.java) > at java.net.SocketInputStream.read(SocketInputStream.java) > at sun.security.ssl.InputRecord.readFully(InputRecord.java) > at sun.security.ssl.InputRecord.read(InputRecord.java) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) > at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) > at java.io.DataOutputStream.flush(DataOutputStream.java) > at > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient > at > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) > at > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) > at > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) > > When create connection, we cannot set the timeout by properties. > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. > So I wrote a patch to fix this infinite waiting by using property-configured > value: > sun.rmi.transport.tcp.responseTimeout. > > Please review this patch. :) > > Note: My OCA has been processed a few hour ago, so my name may take a > short time to > appear on the OCA signatories page. > > Thanks, > KUBOTA Yuji > > [1]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 > > diff --git > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > @@ -222,20 +222,34 @@ > // choose protocol (single op if not reusable socket) > if (!conn.isReusable()) { > out.writeByte(TransportConstants.SingleOpProtocol); > } else { > out.writeByte(TransportConstants.StreamProtocol); > + > + int usableSoTimeout = 0; > + try { > + /* > + * If socket factory had set a zero timeout on its > own, > + * then set the property-configured value to > prevent > + * an infinite waiting. > + */ > + usableSoTimeout = sock.getSoTimeout(); > + if (usableSoTimeout == 0) { > + usableSoTimeout = responseTimeout; > + } > + sock.setSoTimeout(usableSoTimeout); > + } catch (Exception e) { > + // if we fail to set this, ignore and proceed > anyway > + } > out.flush(); > > /* > * Set socket read timeout to configured value for JRMP > * connection handshake; this also serves to guard > against > * non-JRMP servers that do not respond (see 4322806). > */ > - int originalSoTimeout = 0; > try { > - originalSoTimeout = sock.getSoTimeout(); > sock.setSoTimeout(handshakeTimeout); > } catch (Exception e) { > // if we fail to set this, ignore and proceed > anyway > } > > @@ -279,18 +293,11 @@ > * connection. NOTE: this timeout, if configured to a > * finite duration, places an upper bound on the time > * that a remote method call is permitted to execute. > */ > try { > - /* > - * If socket factory had set a non-zero timeout on > its > - * own, then restore it instead of using the > property- > - * configured value. > - */ > - sock.setSoTimeout((originalSoTimeout != 0 ? > - originalSoTimeout : > - responseTimeout)); > + sock.setSoTimeout(usableSoTimeout); > } catch (Exception e) { > // if we fail to set this, ignore and proceed > anyway > } > > out.flush(); > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shanliang.jiang at oracle.com Wed May 6 09:51:24 2015 From: shanliang.jiang at oracle.com (Shanliang Jiang) Date: Wed, 06 May 2015 11:51:24 +0200 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: References: <5547A67C.70304@oracle.com> Message-ID: <5549E41C.1020202@oracle.com> Hi Yuji, I think better at first to create a bug at: https://bugs.openjdk.java.net/secure/Dashboard.jspa It looks like an issue for me, it must be possible to have a test to reproduce the issue. It is helpful to attach the test and present your solution in the bug. I can help if you need any help to create the bug. Shanliang KUBOTA Yuji wrote: > My apologies for re-post, I forgot to register serviceability-dev > before the last post. > > Hi Shanliang, > > Thanks you for your help! > > RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. > > However, many users believe sun.rmi.transport.tcp.responseTimeout > to specify the timeout, > e.g. the second flush() of TCPChannel#createConnection [2]. > In really, the first flush() [3] is not affected by > sun.rmi.transport.tcp.responseTimeout, > and will be the (potential) infinite waiting by bad luck. So I think > openjdk should fix it for users. > > [2]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 > [3]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 > > Thanks, > Yuji > > 2015-05-05 2:03 GMT+09:00 Shanliang Jiang >: > > Hi Yuji, > > > > (I reply to serviceability alias) > > > > When you create a RMI server connector, you can specify a > > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, this > allows > > you to specify your SoTimeout. > > > > Hope this helps. > > > > Shanliang > > > > > > > > KUBOTA Yuji wrote: > > > > Hi all, > > > > I want to contribute this issue. > > If there are a problem about this patch or a better way for openjdk > > community, please advise me. > > > > Thanks for > > > > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji >: > > > > > > Hi all, > > > > I found an infinite waiting at TCPChannel#createConnection. > > This method flushes the DataOutputStream without the socket timeout > settings > > when choose stream protocol [1]. > > > > If connection lost (the destination server do no return response) > > during the flush, > > this method has possibilities to take long time beyond the expectations > > at java.net.SocketInputStream.socketRead0 as following stack trace. > > > > stack trace : > > at > java.net.SocketInputStream.socketRead0(SocketInputStream.java) > > at java.net.SocketInputStream.read(SocketInputStream.java) > > at java.net.SocketInputStream.read(SocketInputStream.java) > > at sun.security.ssl.InputRecord.readFully(InputRecord.java) > > at sun.security.ssl.InputRecord.read(InputRecord.java) > > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) > > at > > > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) > > at > sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) > > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) > > at > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) > > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) > > at java.io.DataOutputStream.flush(DataOutputStream.java) > > at > > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) > > at > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) > > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) > > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient > > at > > > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) > > at > > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) > > at > > > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) > > > > When create connection, we cannot set the timeout by properties. > > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. > > So I wrote a patch to fix this infinite waiting by using > property-configured > > value: > > sun.rmi.transport.tcp.responseTimeout. > > > > Please review this patch. :) > > > > Note: My OCA has been processed a few hour ago, so my name may take a > > short time to > > appear on the OCA signatories page. > > > > Thanks, > > KUBOTA Yuji > > > > [1]: > > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 > > > > diff --git > > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > @@ -222,20 +222,34 @@ > > // choose protocol (single op if not reusable socket) > > if (!conn.isReusable()) { > > out.writeByte(TransportConstants.SingleOpProtocol); > > } else { > > out.writeByte(TransportConstants.StreamProtocol); > > + > > + int usableSoTimeout = 0; > > + try { > > + /* > > + * If socket factory had set a zero timeout > on its > > own, > > + * then set the property-configured value to > > prevent > > + * an infinite waiting. > > + */ > > + usableSoTimeout = sock.getSoTimeout(); > > + if (usableSoTimeout == 0) { > > + usableSoTimeout = responseTimeout; > > + } > > + sock.setSoTimeout(usableSoTimeout); > > + } catch (Exception e) { > > + // if we fail to set this, ignore and proceed > > anyway > > + } > > out.flush(); > > > > /* > > * Set socket read timeout to configured value > for JRMP > > * connection handshake; this also serves to guard > > against > > * non-JRMP servers that do not respond (see > 4322806). > > */ > > - int originalSoTimeout = 0; > > try { > > - originalSoTimeout = sock.getSoTimeout(); > > sock.setSoTimeout(handshakeTimeout); > > } catch (Exception e) { > > // if we fail to set this, ignore and proceed > > anyway > > } > > > > @@ -279,18 +293,11 @@ > > * connection. NOTE: this timeout, if > configured to a > > * finite duration, places an upper bound on > the time > > * that a remote method call is permitted to > execute. > > */ > > try { > > - /* > > - * If socket factory had set a non-zero > timeout on > > its > > - * own, then restore it instead of using the > > property- > > - * configured value. > > - */ > > - sock.setSoTimeout((originalSoTimeout != 0 ? > > - originalSoTimeout : > > - responseTimeout)); > > + sock.setSoTimeout(usableSoTimeout); > > } catch (Exception e) { > > // if we fail to set this, ignore and proceed > > anyway > > } > > > > out.flush(); > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kubota.yuji at gmail.com Wed May 6 10:43:12 2015 From: kubota.yuji at gmail.com (KUBOTA Yuji) Date: Wed, 6 May 2015 19:43:12 +0900 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: <5549E41C.1020202@oracle.com> References: <5547A67C.70304@oracle.com> <5549E41C.1020202@oracle.com> Message-ID: Hi Shanliang, Many thanks for your help! I do not have any role yet. So I can not create a new bug at JBS. It's a reason why I submitted a mail with my patch at first. This issue is caused by a rare network problem during the flush() [3] . I got this infinite loop only once. So I will try to write test or/and client codes with BCI for reproduction. Thanks, Yuji 2015-05-06 18:51 GMT+09:00 Shanliang Jiang : > Hi Yuji, > > I think better at first to create a bug at: > https://bugs.openjdk.java.net/secure/Dashboard.jspa > > It looks like an issue for me, it must be possible to have a test to > reproduce the issue. It is helpful to attach the test and present your > solution in the bug. > > I can help if you need any help to create the bug. > > Shanliang > > > > KUBOTA Yuji wrote: > > My apologies for re-post, I forgot to register serviceability-dev before > the last post. > > Hi Shanliang, > > Thanks you for your help! > > RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. > > However, many users believe sun.rmi.transport.tcp.responseTimeout > to specify the timeout, > e.g. the second flush() of TCPChannel#createConnection [2]. > In really, the first flush() [3] is not affected by > sun.rmi.transport.tcp.responseTimeout, > and will be the (potential) infinite waiting by bad luck. So I think > openjdk should fix it for users. > > [2]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 > [3]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 > > Thanks, > Yuji > > 2015-05-05 2:03 GMT+09:00 Shanliang Jiang : > > Hi Yuji, > > > > (I reply to serviceability alias) > > > > When you create a RMI server connector, you can specify a > > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, this > allows > > you to specify your SoTimeout. > > > > Hope this helps. > > > > Shanliang > > > > > > > > KUBOTA Yuji wrote: > > > > Hi all, > > > > I want to contribute this issue. > > If there are a problem about this patch or a better way for openjdk > > community, please advise me. > > > > Thanks for > > > > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji : > > > > > > Hi all, > > > > I found an infinite waiting at TCPChannel#createConnection. > > This method flushes the DataOutputStream without the socket timeout > settings > > when choose stream protocol [1]. > > > > If connection lost (the destination server do no return response) > > during the flush, > > this method has possibilities to take long time beyond the expectations > > at java.net.SocketInputStream.socketRead0 as following stack trace. > > > > stack trace : > > at java.net.SocketInputStream.socketRead0(SocketInputStream.java) > > at java.net.SocketInputStream.read(SocketInputStream.java) > > at java.net.SocketInputStream.read(SocketInputStream.java) > > at sun.security.ssl.InputRecord.readFully(InputRecord.java) > > at sun.security.ssl.InputRecord.read(InputRecord.java) > > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) > > at > > > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) > > at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) > > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) > > at > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) > > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) > > at java.io.DataOutputStream.flush(DataOutputStream.java) > > at > > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) > > at > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) > > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) > > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient > > at > > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) > > at > > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) > > at > > > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) > > > > When create connection, we cannot set the timeout by properties. > > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. > > So I wrote a patch to fix this infinite waiting by using > property-configured > > value: > > sun.rmi.transport.tcp.responseTimeout. > > > > Please review this patch. :) > > > > Note: My OCA has been processed a few hour ago, so my name may take a > > short time to > > appear on the OCA signatories page. > > > > Thanks, > > KUBOTA Yuji > > > > [1]: > > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 > > > > diff --git > > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > > @@ -222,20 +222,34 @@ > > // choose protocol (single op if not reusable socket) > > if (!conn.isReusable()) { > > out.writeByte(TransportConstants.SingleOpProtocol); > > } else { > > out.writeByte(TransportConstants.StreamProtocol); > > + > > + int usableSoTimeout = 0; > > + try { > > + /* > > + * If socket factory had set a zero timeout on > its > > own, > > + * then set the property-configured value to > > prevent > > + * an infinite waiting. > > + */ > > + usableSoTimeout = sock.getSoTimeout(); > > + if (usableSoTimeout == 0) { > > + usableSoTimeout = responseTimeout; > > + } > > + sock.setSoTimeout(usableSoTimeout); > > + } catch (Exception e) { > > + // if we fail to set this, ignore and proceed > > anyway > > + } > > out.flush(); > > > > /* > > * Set socket read timeout to configured value for > JRMP > > * connection handshake; this also serves to guard > > against > > * non-JRMP servers that do not respond (see > 4322806). > > */ > > - int originalSoTimeout = 0; > > try { > > - originalSoTimeout = sock.getSoTimeout(); > > sock.setSoTimeout(handshakeTimeout); > > } catch (Exception e) { > > // if we fail to set this, ignore and proceed > > anyway > > } > > > > @@ -279,18 +293,11 @@ > > * connection. NOTE: this timeout, if configured > to a > > * finite duration, places an upper bound on the > time > > * that a remote method call is permitted to > execute. > > */ > > try { > > - /* > > - * If socket factory had set a non-zero timeout > on > > its > > - * own, then restore it instead of using the > > property- > > - * configured value. > > - */ > > - sock.setSoTimeout((originalSoTimeout != 0 ? > > - originalSoTimeout : > > - responseTimeout)); > > + sock.setSoTimeout(usableSoTimeout); > > } catch (Exception e) { > > // if we fail to set this, ignore and proceed > > anyway > > } > > > > out.flush(); > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shanliang.jiang at oracle.com Wed May 6 10:48:49 2015 From: shanliang.jiang at oracle.com (Shanliang Jiang) Date: Wed, 06 May 2015 12:48:49 +0200 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: References: <5547A67C.70304@oracle.com> <5549E41C.1020202@oracle.com> Message-ID: <5549F191.2000601@oracle.com> KUBOTA Yuji wrote: > Hi Shanliang, > > Many thanks for your help! > > I do not have any role yet. So I can not create a new bug at JBS. It's > a reason why I submitted a mail with my patch at first. > > This issue is caused by a rare network problem during the flush() [3] > . I got this infinite loop only once. So I will try to write test > or/and client codes with BCI for reproduction. To reproduce the bug, I was thinking to use RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE to specify a user socket server, which will not response any client connection request, I did not yet test this solution. Shanliang > > Thanks, > Yuji > > 2015-05-06 18:51 GMT+09:00 Shanliang Jiang >: > > Hi Yuji, > > I think better at first to create a bug at: > https://bugs.openjdk.java.net/secure/Dashboard.jspa > > It looks like an issue for me, it must be possible to have a test > to reproduce the issue. It is helpful to attach the test and > present your solution in the bug. > > I can help if you need any help to create the bug. > > Shanliang > > > > KUBOTA Yuji wrote: >> My apologies for re-post, I forgot to register serviceability-dev >> before the last post. >> >> Hi Shanliang, >> >> Thanks you for your help! >> >> RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. >> >> However, many users believe sun.rmi.transport.tcp.responseTimeout >> to specify the timeout, >> e.g. the second flush() of TCPChannel#createConnection [2]. >> In really, the first flush() [3] is not affected by >> sun.rmi.transport.tcp.responseTimeout, >> and will be the (potential) infinite waiting by bad luck. So I >> think openjdk should fix it for users. >> >> [2]: >> http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 >> [3]: >> http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 >> >> Thanks, >> Yuji >> >> 2015-05-05 2:03 GMT+09:00 Shanliang Jiang >> >: >> > Hi Yuji, >> > >> > (I reply to serviceability alias) >> > >> > When you create a RMI server connector, you can specify a >> > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, >> this allows >> > you to specify your SoTimeout. >> > >> > Hope this helps. >> > >> > Shanliang >> > >> > >> > >> > KUBOTA Yuji wrote: >> > >> > Hi all, >> > >> > I want to contribute this issue. >> > If there are a problem about this patch or a better way for openjdk >> > community, please advise me. >> > >> > Thanks for >> > >> > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji > >: >> > >> > >> > Hi all, >> > >> > I found an infinite waiting at TCPChannel#createConnection. >> > This method flushes the DataOutputStream without the socket >> timeout settings >> > when choose stream protocol [1]. >> > >> > If connection lost (the destination server do no return response) >> > during the flush, >> > this method has possibilities to take long time beyond the >> expectations >> > at java.net.SocketInputStream.socketRead0 as following stack trace. >> > >> > stack trace : >> > at >> java.net.SocketInputStream.socketRead0(SocketInputStream.java) >> > at java.net.SocketInputStream.read(SocketInputStream.java) >> > at java.net.SocketInputStream.read(SocketInputStream.java) >> > at sun.security.ssl.InputRecord.readFully(InputRecord.java) >> > at sun.security.ssl.InputRecord.read(InputRecord.java) >> > at >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) >> > at >> > >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) >> > at >> sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) >> > at >> sun.security.ssl.AppOutputStream.write(AppOutputStream.java) >> > at >> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) >> > at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java) >> > at java.io.DataOutputStream.flush(DataOutputStream.java) >> > at >> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) >> > at >> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) >> > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) >> > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient >> > at >> > >> javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) >> > at >> > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) >> > at >> > >> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) >> > >> > When create connection, we cannot set the timeout by properties. >> > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., >> infinite. >> > So I wrote a patch to fix this infinite waiting by using >> property-configured >> > value: >> > sun.rmi.transport.tcp.responseTimeout. >> > >> > Please review this patch. :) >> > >> > Note: My OCA has been processed a few hour ago, so my name may >> take a >> > short time to >> > appear on the OCA signatories page. >> > >> > Thanks, >> > KUBOTA Yuji >> > >> > [1]: >> > >> http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 >> > >> > diff --git >> > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > --- >> a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > +++ >> b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > @@ -222,20 +222,34 @@ >> > // choose protocol (single op if not reusable >> socket) >> > if (!conn.isReusable()) { >> > >> out.writeByte(TransportConstants.SingleOpProtocol); >> > } else { >> > >> out.writeByte(TransportConstants.StreamProtocol); >> > + >> > + int usableSoTimeout = 0; >> > + try { >> > + /* >> > + * If socket factory had set a zero >> timeout on its >> > own, >> > + * then set the property-configured >> value to >> > prevent >> > + * an infinite waiting. >> > + */ >> > + usableSoTimeout = sock.getSoTimeout(); >> > + if (usableSoTimeout == 0) { >> > + usableSoTimeout = responseTimeout; >> > + } >> > + sock.setSoTimeout(usableSoTimeout); >> > + } catch (Exception e) { >> > + // if we fail to set this, ignore and >> proceed >> > anyway >> > + } >> > out.flush(); >> > >> > /* >> > * Set socket read timeout to configured >> value for JRMP >> > * connection handshake; this also serves >> to guard >> > against >> > * non-JRMP servers that do not respond >> (see 4322806). >> > */ >> > - int originalSoTimeout = 0; >> > try { >> > - originalSoTimeout = sock.getSoTimeout(); >> > sock.setSoTimeout(handshakeTimeout); >> > } catch (Exception e) { >> > // if we fail to set this, ignore and >> proceed >> > anyway >> > } >> > >> > @@ -279,18 +293,11 @@ >> > * connection. NOTE: this timeout, if >> configured to a >> > * finite duration, places an upper bound >> on the time >> > * that a remote method call is permitted >> to execute. >> > */ >> > try { >> > - /* >> > - * If socket factory had set a >> non-zero timeout on >> > its >> > - * own, then restore it instead of >> using the >> > property- >> > - * configured value. >> > - */ >> > - sock.setSoTimeout((originalSoTimeout >> != 0 ? >> > - originalSoTimeout : >> > - responseTimeout)); >> > + sock.setSoTimeout(usableSoTimeout); >> > } catch (Exception e) { >> > // if we fail to set this, ignore and >> proceed >> > anyway >> > } >> > >> > out.flush(); >> > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kubota.yuji at gmail.com Wed May 6 11:07:44 2015 From: kubota.yuji at gmail.com (KUBOTA Yuji) Date: Wed, 6 May 2015 20:07:44 +0900 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: <5549F191.2000601@oracle.com> References: <5547A67C.70304@oracle.com> <5549E41C.1020202@oracle.com> <5549F191.2000601@oracle.com> Message-ID: 2015-05-06 19:48 GMT+09:00 Shanliang Jiang : > To reproduce the bug, I was thinking to use RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE to specify a user socket server, which will not response any client connection request, I did not yet test this solution. Thanks, I will try to reproduce with the attribute at first. Yuji > Shanliang > > > Thanks, > Yuji > > 2015-05-06 18:51 GMT+09:00 Shanliang Jiang : >> >> Hi Yuji, >> >> I think better at first to create a bug at: >> https://bugs.openjdk.java.net/secure/Dashboard.jspa >> >> It looks like an issue for me, it must be possible to have a test to reproduce the issue. It is helpful to attach the test and present your solution in the bug. >> >> I can help if you need any help to create the bug. >> >> Shanliang >> >> >> >> KUBOTA Yuji wrote: >> >> My apologies for re-post, I forgot to register serviceability-dev before the last post. >> >> Hi Shanliang, >> >> Thanks you for your help! >> >> RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. >> >> However, many users believe sun.rmi.transport.tcp.responseTimeout to specify the timeout, >> e.g. the second flush() of TCPChannel#createConnection [2]. >> In really, the first flush() [3] is not affected by sun.rmi.transport.tcp.responseTimeout, >> and will be the (potential) infinite waiting by bad luck. So I think openjdk should fix it for users. >> >> [2]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 >> [3]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 >> >> Thanks, >> Yuji >> >> 2015-05-05 2:03 GMT+09:00 Shanliang Jiang : >> > Hi Yuji, >> > >> > (I reply to serviceability alias) >> > >> > When you create a RMI server connector, you can specify a >> > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, this allows >> > you to specify your SoTimeout. >> > >> > Hope this helps. >> > >> > Shanliang >> > >> > >> > >> > KUBOTA Yuji wrote: >> > >> > Hi all, >> > >> > I want to contribute this issue. >> > If there are a problem about this patch or a better way for openjdk >> > community, please advise me. >> > >> > Thanks for >> > >> > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji : >> > >> > >> > Hi all, >> > >> > I found an infinite waiting at TCPChannel#createConnection. >> > This method flushes the DataOutputStream without the socket timeout settings >> > when choose stream protocol [1]. >> > >> > If connection lost (the destination server do no return response) >> > during the flush, >> > this method has possibilities to take long time beyond the expectations >> > at java.net.SocketInputStream.socketRead0 as following stack trace. >> > >> > stack trace : >> > at java.net.SocketInputStream.socketRead0(SocketInputStream.java) >> > at java.net.SocketInputStream.read(SocketInputStream.java) >> > at java.net.SocketInputStream.read(SocketInputStream.java) >> > at sun.security.ssl.InputRecord.readFully(InputRecord.java) >> > at sun.security.ssl.InputRecord.read(InputRecord.java) >> > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) >> > at >> > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) >> > at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) >> > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) >> > at >> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) >> > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) >> > at java.io.DataOutputStream.flush(DataOutputStream.java) >> > at >> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) >> > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) >> > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) >> > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient >> > at >> > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) >> > at >> > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) >> > at >> > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) >> > >> > When create connection, we cannot set the timeout by properties. >> > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. >> > So I wrote a patch to fix this infinite waiting by using property-configured >> > value: >> > sun.rmi.transport.tcp.responseTimeout. >> > >> > Please review this patch. :) >> > >> > Note: My OCA has been processed a few hour ago, so my name may take a >> > short time to >> > appear on the OCA signatories page. >> > >> > Thanks, >> > KUBOTA Yuji >> > >> > [1]: >> > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 >> > >> > diff --git >> > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java >> > @@ -222,20 +222,34 @@ >> > // choose protocol (single op if not reusable socket) >> > if (!conn.isReusable()) { >> > out.writeByte(TransportConstants.SingleOpProtocol); >> > } else { >> > out.writeByte(TransportConstants.StreamProtocol); >> > + >> > + int usableSoTimeout = 0; >> > + try { >> > + /* >> > + * If socket factory had set a zero timeout on its >> > own, >> > + * then set the property-configured value to >> > prevent >> > + * an infinite waiting. >> > + */ >> > + usableSoTimeout = sock.getSoTimeout(); >> > + if (usableSoTimeout == 0) { >> > + usableSoTimeout = responseTimeout; >> > + } >> > + sock.setSoTimeout(usableSoTimeout); >> > + } catch (Exception e) { >> > + // if we fail to set this, ignore and proceed >> > anyway >> > + } >> > out.flush(); >> > >> > /* >> > * Set socket read timeout to configured value for JRMP >> > * connection handshake; this also serves to guard >> > against >> > * non-JRMP servers that do not respond (see 4322806). >> > */ >> > - int originalSoTimeout = 0; >> > try { >> > - originalSoTimeout = sock.getSoTimeout(); >> > sock.setSoTimeout(handshakeTimeout); >> > } catch (Exception e) { >> > // if we fail to set this, ignore and proceed >> > anyway >> > } >> > >> > @@ -279,18 +293,11 @@ >> > * connection. NOTE: this timeout, if configured to a >> > * finite duration, places an upper bound on the time >> > * that a remote method call is permitted to execute. >> > */ >> > try { >> > - /* >> > - * If socket factory had set a non-zero timeout on >> > its >> > - * own, then restore it instead of using the >> > property- >> > - * configured value. >> > - */ >> > - sock.setSoTimeout((originalSoTimeout != 0 ? >> > - originalSoTimeout : >> > - responseTimeout)); >> > + sock.setSoTimeout(usableSoTimeout); >> > } catch (Exception e) { >> > // if we fail to set this, ignore and proceed >> > anyway >> > } >> > >> > out.flush(); >> > >> > >> > >> >> > > From yekaterina.kantserova at oracle.com Wed May 6 11:21:45 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Wed, 06 May 2015 13:21:45 +0200 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <55492F73.2020806@oracle.com> References: <55488794.3060008@oracle.com> <55492917.6010200@oracle.com> <55492F73.2020806@oracle.com> Message-ID: <5549F949.4010300@oracle.com> Mandy, Thanks fro your review! Please see my comment inlined. On 05/05/2015 11:00 PM, Mandy Chung wrote: >> com.sun.management has been moved to jdk.management module. The >> patch for JDK-8042901 is just integrated in jdk9/dev today. Most, if >> not all, test/com/sun/management tests need updates to require >> @modules jdk.management. >> >> There are several test/java/lang/management tests dependng on both >> java.lang.management and com.sun.management. For tests only using >> java.lang.management API, @modules java.management is adequate. >> >> test/com/sun/jdi/InterfaceMethodsTest.java The copyright header contains already 2015. >> test/com/sun/jdi/InterruptHangTest.java The test has been missing copyrights, I've updated it with the copyright header. >> Since you have updated the end year of the copyright header in >> most tests, I spot a couple and so mention it to you. >> >> Otherwise looks good. Thanks for doing this. The new webrev can be found here: http://cr.openjdk.java.net/~ykantser/8078896/webrev.02/ > > About the test selection, one typical aspect of svc tests is to run a > j* tool in a child process (e.g. jinfo, jstack, jstat, jstatd,jcmd, > jps etc that are in jdk.jcmd module). I would expect all > test/sun/tools/jcmd tests should have @modules jdk.jcmd and similiar > for other sun/tools/j* tests. Are you thinking to come back in the > next round of test selection update? As Alan pointed in his answer to this mail JTreg is not ready yet. I prefer to do it when it will be possible to test this change. Best regards, Katja > > Mandy From aph at redhat.com Wed May 6 14:36:05 2015 From: aph at redhat.com (Andrew Haley) Date: Wed, 06 May 2015 15:36:05 +0100 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <55408B69.8050108@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> Message-ID: <554A26D5.4040003@redhat.com> On 04/29/2015 08:42 AM, Andrew Haley wrote: > > http://cr.openjdk.java.net/~aph/8078521-2/ Any news on this? It shouldn't be controversial at this point. Thanks, Andrew. From volker.simonis at gmail.com Wed May 6 16:12:24 2015 From: volker.simonis at gmail.com (Volker Simonis) Date: Wed, 6 May 2015 18:12:24 +0200 Subject: RFR(XS): 8079510: AIX: fix build after '8042901: Allow com.sun.management to be in a different module...' Message-ID: Hi, can somebody please review this trivial AIX-only change which fixes an AIX build error: http://cr.openjdk.java.net/~simonis/webrevs/2015/8079510/ https://bugs.openjdk.java.net/browse/JDK-8079510 Here are the details: Change '8042901: Allow com.sun.management to be in a different module to java.lang.management' has refactored the management package and moved the files: jdk/src/java.management/unix/native/libmanagement/OperatingSystem.c to: jdk/src/jdk.management//native/libmanagement_ext/UnixOperatingSystem.c Unfortunately we've never had a AIXOperatingSystem.c file on AIX (see 8030957). This was no problem with the old setup but with the new one the build complains about a missing directory 'jdk/src/jdk.management/aix/native/libmanagement_ext'. This bug is just for fixing the build, by introducing an emty stub file. The actual implementation of the corresponding OperatingSystemMXBean is still up to 8030957. Thanks, Volker From mandy.chung at oracle.com Wed May 6 16:55:12 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 06 May 2015 09:55:12 -0700 Subject: RFR(M): 8078896: Add @modules as needed to the jdk_svc tests In-Reply-To: <5549F949.4010300@oracle.com> References: <55488794.3060008@oracle.com> <55492917.6010200@oracle.com> <55492F73.2020806@oracle.com> <5549F949.4010300@oracle.com> Message-ID: <554A4770.2060305@oracle.com> On 05/06/2015 04:21 AM, Yekaterina Kantserova wrote: > The new webrev can be found here: > http://cr.openjdk.java.net/~ykantser/8078896/webrev.02/ > Looks good. >> >> About the test selection, one typical aspect of svc tests is to run a >> j* tool in a child process (e.g. jinfo, jstack, jstat, jstatd,jcmd, >> jps etc that are in jdk.jcmd module). I would expect all >> test/sun/tools/jcmd tests should have @modules jdk.jcmd and similiar >> for other sun/tools/j* tests. Are you thinking to come back in the >> next round of test selection update? > > As Alan pointed in his answer to this mail JTreg is not ready yet. I > prefer to do it when it will be possible to test this change. Right the test selection is yet to be supported by jtreg and it's okay to add those in the next round. As the tool analyzing the test dependencies doesn't detect that, these multi-process tests would need to be analyzed in a different way. Anyway, this patch looks good. Mandy From staffan.larsen at oracle.com Wed May 6 18:51:18 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Wed, 6 May 2015 20:51:18 +0200 Subject: RFR: 8079360 AttachProviderImpl could not be instantiated Message-ID: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> This is another library that needs to be compiled with -DPSAPI_VERSION=1 due to the recent Windows compiler upgrade. I have also fixed a better error message that prints the underlaying exception if something like this happens again. webrev: http://cr.openjdk.java.net/~sla/8079360/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8079360 Thanks, /Staffan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dean.long at oracle.com Wed May 6 18:53:57 2015 From: dean.long at oracle.com (Dean Long) Date: Wed, 06 May 2015 11:53:57 -0700 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <554A26D5.4040003@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> Message-ID: <554A6345.4040004@oracle.com> I added hotspot-dev at openjdk.java.net again. It looks reasonable to me, but I'm not a Reviewer. dl On 5/6/2015 7:36 AM, Andrew Haley wrote: > On 04/29/2015 08:42 AM, Andrew Haley wrote: >> http://cr.openjdk.java.net/~aph/8078521-2/ > Any news on this? It shouldn't be controversial at this point. > > Thanks, > Andrew. > > From dmitry.samersoff at oracle.com Wed May 6 19:47:53 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 06 May 2015 22:47:53 +0300 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <554A26D5.4040003@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> Message-ID: <554A6FE9.4010107@oracle.com> Andrew, I'm looking into it and come back tomorrow. (not a reviewer but current SA owner) -Dmitry On 2015-05-06 17:36, Andrew Haley wrote: > On 04/29/2015 08:42 AM, Andrew Haley wrote: >> >> http://cr.openjdk.java.net/~aph/8078521-2/ > > Any news on this? It shouldn't be controversial at this point. > > Thanks, > Andrew. > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Wed May 6 19:50:36 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 06 May 2015 22:50:36 +0300 Subject: RFR: 8079360 AttachProviderImpl could not be instantiated In-Reply-To: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> References: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> Message-ID: <554A708C.5090206@oracle.com> Staffan, Looks good for me. -Dmitry On 2015-05-06 21:51, Staffan Larsen wrote: > This is another library that needs to be compiled with -DPSAPI_VERSION=1 > due to the recent Windows compiler upgrade. > > I have also fixed a better error message that prints the underlaying > exception if something like this happens again. > > webrev: http://cr.openjdk.java.net/~sla/8079360/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8079360 > > Thanks, > /Staffan > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From kubota.yuji at gmail.com Mon May 4 18:20:56 2015 From: kubota.yuji at gmail.com (KUBOTA Yuji) Date: Tue, 5 May 2015 03:20:56 +0900 Subject: Potential infinite waiting at JMXConnection#createConnection In-Reply-To: <5547A67C.70304@oracle.com> References: <5547A67C.70304@oracle.com> Message-ID: Hi Shanliang, Thanks you for your help! RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE is a nice workaround. However, many users believe sun.rmi.transport.tcp.responseTimeout to specify the timeout, e.g. the second flush() of TCPChannel#createConnection [2]. In really, the first flush()[3] is not affected by sun.rmi.transport.tcp.responseTimeout, and will be the (potential) infinite waiting by bad luck. So I think openjdk should fix it. [2]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l296 [3]: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java#l227 Thanks, Yuji 2015-05-05 2:03 GMT+09:00 Shanliang Jiang : > Hi Yuji, > > (I reply to serviceability alias) > > When you create a RMI server connector, you can specify a > RMIClientSocketFactory by RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, this allows > you to specify your SoTimeout. > > Hope this helps. > > Shanliang > > > > KUBOTA Yuji wrote: > > Hi all, > > I want to contribute this issue. > If there are a problem about this patch or a better way for openjdk > community, please advise me. > > Thanks for > > 2015-04-22 0:31 GMT+09:00 KUBOTA Yuji : > > > Hi all, > > I found an infinite waiting at TCPChannel#createConnection. > This method flushes the DataOutputStream without the socket timeout settings > when choose stream protocol [1]. > > If connection lost (the destination server do no return response) > during the flush, > this method has possibilities to take long time beyond the expectations > at java.net.SocketInputStream.socketRead0 as following stack trace. > > stack trace : > at java.net.SocketInputStream.socketRead0(SocketInputStream.java) > at java.net.SocketInputStream.read(SocketInputStream.java) > at java.net.SocketInputStream.read(SocketInputStream.java) > at sun.security.ssl.InputRecord.readFully(InputRecord.java) > at sun.security.ssl.InputRecord.read(InputRecord.java) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java) > at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java) > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java) > at java.io.DataOutputStream.flush(DataOutputStream.java) > at > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java) > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java) > at javax.management.remote.rmi.RMIServerImpl_Stub.newClient > at > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java) > at > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java) > at > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java) > > When create connection, we cannot set the timeout by properties. > Therefore, JMX sets the default value of SO_TIMEOUT, i.e., infinite. > So I wrote a patch to fix this infinite waiting by using property-configured > value: > sun.rmi.transport.tcp.responseTimeout. > > Please review this patch. :) > > Note: My OCA has been processed a few hour ago, so my name may take a > short time to > appear on the OCA signatories page. > > Thanks, > KUBOTA Yuji > > [1]: > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/c5b5d9045728/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPConnection.java#l191 > > diff --git > a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > --- a/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > +++ b/src/java.rmi/share/classes/sun/rmi/transport/tcp/TCPChannel.java > @@ -222,20 +222,34 @@ > // choose protocol (single op if not reusable socket) > if (!conn.isReusable()) { > out.writeByte(TransportConstants.SingleOpProtocol); > } else { > out.writeByte(TransportConstants.StreamProtocol); > + > + int usableSoTimeout = 0; > + try { > + /* > + * If socket factory had set a zero timeout on its > own, > + * then set the property-configured value to > prevent > + * an infinite waiting. > + */ > + usableSoTimeout = sock.getSoTimeout(); > + if (usableSoTimeout == 0) { > + usableSoTimeout = responseTimeout; > + } > + sock.setSoTimeout(usableSoTimeout); > + } catch (Exception e) { > + // if we fail to set this, ignore and proceed > anyway > + } > out.flush(); > > /* > * Set socket read timeout to configured value for JRMP > * connection handshake; this also serves to guard > against > * non-JRMP servers that do not respond (see 4322806). > */ > - int originalSoTimeout = 0; > try { > - originalSoTimeout = sock.getSoTimeout(); > sock.setSoTimeout(handshakeTimeout); > } catch (Exception e) { > // if we fail to set this, ignore and proceed > anyway > } > > @@ -279,18 +293,11 @@ > * connection. NOTE: this timeout, if configured to a > * finite duration, places an upper bound on the time > * that a remote method call is permitted to execute. > */ > try { > - /* > - * If socket factory had set a non-zero timeout on > its > - * own, then restore it instead of using the > property- > - * configured value. > - */ > - sock.setSoTimeout((originalSoTimeout != 0 ? > - originalSoTimeout : > - responseTimeout)); > + sock.setSoTimeout(usableSoTimeout); > } catch (Exception e) { > // if we fail to set this, ignore and proceed > anyway > } > > out.flush(); > > > From derek.white at oracle.com Wed May 6 21:51:42 2015 From: derek.white at oracle.com (Derek White) Date: Wed, 06 May 2015 17:51:42 -0400 Subject: RFR(M): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5548D65C.8040304@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> Message-ID: <554A8CEE.2090009@oracle.com> Hi Dmitry, Staffan, Lots of good comments here. On the topic of what list should be printed out, I think we should focus on objects waiting to be finalized - e.g. the contents of the ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you could add a summerizeQueue(TreeMap) method, or a general iterator and lambda. A histogram of objects with finalize methods might also be interesting, but you can get most of the same information from a heap histogram (ClassHistogramDCmd, and search for count of Finalizer instances). BTW, by only locking the ReferenceQueue, we should only be blocking the FinalizerThread from making progress (and I think there's a GC thread that runs after GC that sorts found References objects that need processing into their respective ReferenceQueues). But locking the "unfinalized" list blocks initializing any object with a finalize() method. The sorting suggestion is a nice touch. - Derek White, GC team On 5/5/15 10:40 AM, Peter Levart wrote: > Hi Dmitry, Staffan, > > On 05/05/2015 12:38 PM, Staffan Larsen wrote: >> Dmitry, >> >> I think this should be reviewed on hotspot-gc and core-libs-dev as >> well considering the changes to Finalizer. >> >> I?m a little worried about the potentially very large String that is >> returned from printFinalizationQueue(). A possible different approach >> would be to write printFinalizationQueue() in C++ inside Hotspot. >> That would allow for outputting one line at a time. The output would >> still be saved in memory (since the stream is buffered), but at least >> the data is only saved once in memory, then. It would make the code a >> bit harder to write, so its a question of how scared we are of >> running out of memory. > > If the output is just a histogram of # of instances per class name, > then it should not be too large, as there are not many different > classes overriding finalize() method (I counted 72 overriddings of > finalize() method in the whole jdk/src tree). > > I'm more concerned about the fact that while traversing the list, a > lock is held, which might impact prompt finalization(). Is it > acceptable for diagnostic output to impact performance and/or > interfere with synchronization? > > It might be possible to scan the list without holding a lock for > diagnostic purposes if Finalizer.remove() didn't set the removed > Finalizer.next pointer to point back to itself: > > private void remove() { > synchronized (lock) { > if (unfinalized == this) { > if (this.next != null) { > unfinalized = this.next; > } else { > unfinalized = this.prev; > } > } > if (this.next != null) { > this.next.prev = this.prev; > } > if (this.prev != null) { > this.prev.next = this.next; > } > // this.next = this; must not be set so that we can > traverse the list unsynchronized > this.prev = this; /* Indicates that this has been > finalized */ > } > } > > For detecting whether a Finalizer is already processed, the 'prev' > pointer could be used instead: > > private boolean hasBeenFinalized() { > return (prev == this); > } > > Also, to make sure a data race dereferencing 'unfinalized' in > unsynchronized printFinalizationQueue() would get you a fully > initialized Finalizer instance (in particular the next pointer), you > would have to make the 'unfinalized' field volatile or insert an > Unsafe.storeFence() before assigning to 'unfinalized': > > private void add() { > synchronized (lock) { > if (unfinalized != null) { > this.next = unfinalized; > unfinalized.prev = this; > } > // make sure a data race dereferencing 'unfinalized' > // in printFinalizationQueue() does see the Finalizer > // instance fully initialized > // (in particular the 'next' pointer) > U.storeFence(); > unfinalized = this; > } > } > > > By doing these modifications, I think you can remove > synchronized(lock) {} from printFinalizationQueue(). > >> >> I see you are traversing the ?unfinalized? list in Finalizer, whereas >> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >> not sure of the difference, perhaps someone from the GC team can help >> out? > > The 'queue' is a ReferenceQueue of Finalizer (FinalReference) > instances pending processing by finalizer thread because their > referents are eligible for finalization (they are not reachable any > more). The 'unfinalized' is a doubly-linked list of all Finalizer > instances for which their referents have not been finalized yet > (including those that are still reachable and alive). The later serves > two purposes: > - it keeps Finalizer instances reachable until they are processed > - it is a source of unfinalized instances for > running-finalizers-on-exit if requested by > System.runFinalizersOnExit(true); > > So it really depends on what one would like to see. Showing the queue > may be interesting if one wants to see how the finalizer thread is > coping with processing the finalize() invocations. Showing unfinalized > list may be interesting if one wants to know how many live + > finalization pending instances are there on the heap that override > finalize() method. > > Regards, Peter > >> >> For the output, it would be a nice touch to sort it on the number of >> references for each type. Perhaps outputting it more like a table >> (see the old code again) would also make it easier to digest. >> >> In diagnosticCommand.hpp, the new commands should override >> permission() and limit access: >> >> static const JavaPermission permission() { >> JavaPermission p = {"java.lang.management.ManagementPermission", >> "monitor", NULL}; >> return p; >> } >> >> The two tests don?t validate the output in any way. Would it be >> possible to add validation? Perhaps hard to make sure an object is on >> the finalizer queue? Hmm. >> >> Thanks, >> /Staffan >> >> >>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>> wrote: >>> >>> Everyone, >>> >>> Please review the fix: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>> >>> heap dcmd outputs the same information as SIGBREAK >>> >>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>> with count >>> >>> -Dmitry >>> >>> -- >>> Dmitry Samersoff >>> Oracle Java development team, Saint Petersburg, Russia >>> * I would love to change the world, but they won't give me the sources. > From serguei.spitsyn at oracle.com Wed May 6 22:06:10 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 06 May 2015 15:06:10 -0700 Subject: RFR: 8079360 AttachProviderImpl could not be instantiated In-Reply-To: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> References: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> Message-ID: <554A9052.6010709@oracle.com> Looks good. Thanks, Serguei On 5/6/15 11:51 AM, Staffan Larsen wrote: > This is another library that needs to be compiled with > -DPSAPI_VERSION=1 due to the recent Windows compiler upgrade. > > I have also fixed a better error message that prints the underlaying > exception if something like this happens again. > > webrev: http://cr.openjdk.java.net/~sla/8079360/webrev.00/ > > bug: https://bugs.openjdk.java.net/browse/JDK-8079360 > > Thanks, > /Staffan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus.ihse.bursie at oracle.com Thu May 7 07:08:30 2015 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Thu, 7 May 2015 09:08:30 +0200 Subject: RFR: 8079360 AttachProviderImpl could not be instantiated In-Reply-To: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> References: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> Message-ID: <9E353370-E0C9-43F0-8A76-398FE0F1E82F@oracle.com> Build changes look good to me. /Magnus > 6 maj 2015 kl. 20:51 skrev Staffan Larsen : > > This is another library that needs to be compiled with -DPSAPI_VERSION=1 due to the recent Windows compiler upgrade. > > I have also fixed a better error message that prints the underlaying exception if something like this happens again. > > webrev: http://cr.openjdk.java.net/~sla/8079360/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8079360 > > Thanks, > /Staffan > From erik.joelsson at oracle.com Thu May 7 08:02:12 2015 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 07 May 2015 10:02:12 +0200 Subject: RFR: 8079360 AttachProviderImpl could not be instantiated In-Reply-To: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> References: <64EEA73D-FF3A-4868-8F93-530D419A7D04@oracle.com> Message-ID: <554B1C04.2020709@oracle.com> Looks good to me. /Erik On 2015-05-06 20:51, Staffan Larsen wrote: > This is another library that needs to be compiled with -DPSAPI_VERSION=1 due to the recent Windows compiler upgrade. > > I have also fixed a better error message that prints the underlaying exception if something like this happens again. > > webrev: http://cr.openjdk.java.net/~sla/8079360/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8079360 > > Thanks, > /Staffan > From staffan.larsen at oracle.com Thu May 7 08:06:16 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 7 May 2015 10:06:16 +0200 Subject: RFR: 8079559 Exclude demo/jvmti/hprof tests Message-ID: <020F8146-6E57-4613-92AE-CA6EFA96AF8A@oracle.com> A couple of hprof demos fail because of lambdas (see JDK-8079273). Since we are soon removing hprof, there is no use updating the code or tests, so instead I want to exclude the tests. bug: https://bugs.openjdk.java.net/browse/JDK-8079559 Thanks, /Staffan diff --git a/test/ProblemList.txt b/test/ProblemList.txt --- a/test/ProblemList.txt +++ b/test/ProblemList.txt @@ -385,4 +385,10 @@ # 8064572 8060736 8062938 sun/jvmstat/monitor/MonitoredVm/CR6672135.java generic-all +# 8079273 +demo/jvmti/hprof/CpuOldTest.java generic-all +demo/jvmti/hprof/CpuTimesTest.java generic-all +demo/jvmti/hprof/OptionsTest.java generic-all +demo/jvmti/hprof/StackMapTableTest.java generic-all + ############################################################################ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu May 7 08:07:24 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 07 May 2015 09:07:24 +0100 Subject: RFR: 8079559 Exclude demo/jvmti/hprof tests In-Reply-To: <020F8146-6E57-4613-92AE-CA6EFA96AF8A@oracle.com> References: <020F8146-6E57-4613-92AE-CA6EFA96AF8A@oracle.com> Message-ID: <554B1D3C.5060308@oracle.com> On 07/05/2015 09:06, Staffan Larsen wrote: > A couple of hprof demos fail because of lambdas (see JDK-8079273). > Since we are soon removing hprof, there is no use updating the code or > tests, so instead I want to exclude the tests. > > bug: https://bugs.openjdk.java.net/browse/JDK-8079559 > > Looks okay, the alternative is to just remove the tests early. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yekaterina.kantserova at oracle.com Thu May 7 08:54:10 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 07 May 2015 10:54:10 +0200 Subject: 8079200: Fix heapdump tests to validate heapdump after jhat is removed Message-ID: <554B2832.1030400@oracle.com> Hi, Could I please have a review of this fix. bug: https://bugs.openjdk.java.net/browse/JDK-8079200 webrev: http://cr.openjdk.java.net/~ykantser/8079200/webrev.00 The fix makes sure the HprofParser is available for all types of test frameworks, not only JTreg. It will be a part of test-lib.jar. Thanks, Katja From staffan.larsen at oracle.com Thu May 7 09:01:02 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 7 May 2015 11:01:02 +0200 Subject: 8079200: Fix heapdump tests to validate heapdump after jhat is removed In-Reply-To: <554B2832.1030400@oracle.com> References: <554B2832.1030400@oracle.com> Message-ID: Looks good! Thanks, /Staffan > On 7 maj 2015, at 10:54, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8079200 > webrev: http://cr.openjdk.java.net/~ykantser/8079200/webrev.00 > > The fix makes sure the HprofParser is available for all types of test frameworks, not only JTreg. It will be a part of test-lib.jar. > > Thanks, > Katja > > From goetz.lindenmaier at sap.com Thu May 7 13:39:58 2015 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Thu, 7 May 2015 13:39:58 +0000 Subject: RFR(XS): 8079510: AIX: fix build after '8042901: Allow com.sun.management to be in a different module...' In-Reply-To: References: Message-ID: <4295855A5C1DE049A61835A1887419CC2CFDBD0F@DEWDFEMB12A.global.corp.sap> Hi Volker, the change looks good. Thanks for fixing this. Best regards, Goetz. -----Original Message----- From: serviceability-dev [mailto:serviceability-dev-bounces at openjdk.java.net] On Behalf Of Volker Simonis Sent: Mittwoch, 6. Mai 2015 18:12 To: jdk9-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: RFR(XS): 8079510: AIX: fix build after '8042901: Allow com.sun.management to be in a different module...' Hi, can somebody please review this trivial AIX-only change which fixes an AIX build error: http://cr.openjdk.java.net/~simonis/webrevs/2015/8079510/ https://bugs.openjdk.java.net/browse/JDK-8079510 Here are the details: Change '8042901: Allow com.sun.management to be in a different module to java.lang.management' has refactored the management package and moved the files: jdk/src/java.management/unix/native/libmanagement/OperatingSystem.c to: jdk/src/jdk.management//native/libmanagement_ext/UnixOperatingSystem.c Unfortunately we've never had a AIXOperatingSystem.c file on AIX (see 8030957). This was no problem with the old setup but with the new one the build complains about a missing directory 'jdk/src/jdk.management/aix/native/libmanagement_ext'. This bug is just for fixing the build, by introducing an emty stub file. The actual implementation of the corresponding OperatingSystemMXBean is still up to 8030957. Thanks, Volker From yekaterina.kantserova at oracle.com Thu May 7 15:04:55 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 07 May 2015 17:04:55 +0200 Subject: RFR(S): 6755586: Test com/sun/jdi/NoLaunchOptionTest.java may erroneously fail Message-ID: <554B7F17.2040900@oracle.com> Hi, Could I please have a review of this small fix. bug: https://bugs.openjdk.java.net/browse/JDK-6755586 webrev: http://cr.openjdk.java.net/~ykantser/6755586/webrev.00/ This bug is actually already solved. But I'll use it as an opportunity to re-write the test using test library. There is no need to retrieve a free port since the error occurs before binding to the address. Thanks, Katja From dmitry.samersoff at oracle.com Thu May 7 15:20:09 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 07 May 2015 18:20:09 +0300 Subject: 8079200: Fix heapdump tests to validate heapdump after jhat is removed In-Reply-To: <554B2832.1030400@oracle.com> References: <554B2832.1030400@oracle.com> Message-ID: <554B82A9.9090702@oracle.com> Katja, Looks good for me. -Dmitry On 2015-05-07 11:54, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8079200 > webrev: http://cr.openjdk.java.net/~ykantser/8079200/webrev.00 > > The fix makes sure the HprofParser is available for all types of test > frameworks, not only JTreg. It will be a part of test-lib.jar. > > Thanks, > Katja > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Thu May 7 16:32:01 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 07 May 2015 19:32:01 +0300 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <554A26D5.4040003@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> Message-ID: <554B9381.8090907@oracle.com> Andrew, Looks good, except couple of minor nits. Please, take a look at LinuxAARCH64CFrame.java:70, it might be a bug - probably, you should check against fp, not rsp here. See also below. LinuxDebuggerLocal.c: 375: Please, use NPRGREG from AARCH64ThreadContext.java, the same way as other platforms does. HSDB.java 998: it's better to add String cpu = VM.getVM().getCPU(); and shorten if. But JDK 7 and later doesn't support Itanium, so you can remove this if entirely. LinuxAARCH64CFrame.java: 60 - please, remove space after/before brackets. 65,69,73 - please, remove space after bracket. 65 - it's better to add brackets around 2 * ADDRESS_SIZE 70 - probably, you should check against fp, not rsp here. RemoteAARCH64ThreadContext.java: 40, 46 Please,remove stale comments. LinuxAARCH64JavaThreadPDAccess.java: 67 and below - only one space should be after public -Dmitry On 2015-05-06 17:36, Andrew Haley wrote: > On 04/29/2015 08:42 AM, Andrew Haley wrote: >> >> http://cr.openjdk.java.net/~aph/8078521-2/ > > Any news on this? It shouldn't be controversial at this point. > > Thanks, > Andrew. > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From aph at redhat.com Thu May 7 16:43:03 2015 From: aph at redhat.com (Andrew Haley) Date: Thu, 07 May 2015 17:43:03 +0100 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <554B9381.8090907@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> Message-ID: <554B9617.2090406@redhat.com> On 05/07/2015 05:32 PM, Dmitry Samersoff wrote: > Andrew, > > Looks good, except couple of minor nits. > > Please, take a look at LinuxAARCH64CFrame.java:70, > it might be a bug - probably, you should check against fp, > not rsp here. > > See also below. > > LinuxDebuggerLocal.c: > > 375: Please, use NPRGREG from AARCH64ThreadContext.java, > the same way as other platforms does. OK. > HSDB.java > > 998: it's better to add String cpu = VM.getVM().getCPU(); > and shorten if. OK. > But JDK 7 and later doesn't support Itanium, so you can remove this if > entirely. > > LinuxAARCH64CFrame.java: > > 60 - please, remove space after/before brackets. > 65,69,73 - please, remove space after bracket. > 65 - it's better to add brackets around 2 * ADDRESS_SIZE OK. This is all stuff inherited from x86. > 70 - probably, you should check against fp, not rsp here. Hmmm. I would have thought it's a good sanity check either way. > RemoteAARCH64ThreadContext.java: > > 40, 46 Please,remove stale comments. OK. > LinuxAARCH64JavaThreadPDAccess.java: > > 67 and below - only one space should be after public OK. Again, it's all inherited from x86 and I didn't want to make any unnecessary whitespace changes, but OK. Thanks, Andrew. From staffan.larsen at oracle.com Thu May 7 18:35:09 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 7 May 2015 20:35:09 +0200 Subject: RFR(S): 6755586: Test com/sun/jdi/NoLaunchOptionTest.java may erroneously fail In-Reply-To: <554B7F17.2040900@oracle.com> References: <554B7F17.2040900@oracle.com> Message-ID: <97012654-E803-4210-A077-4D2FC7663A94@oracle.com> Nice and clean. Reviewed. /Staffan > On 7 maj 2015, at 17:04, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this small fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-6755586 > webrev: http://cr.openjdk.java.net/~ykantser/6755586/webrev.00/ > > This bug is actually already solved. But I'll use it as an opportunity to re-write the test using test library. There is no need to retrieve a free port since the error occurs before binding to the address. > > Thanks, > Katja From dmitry.samersoff at oracle.com Thu May 7 20:40:00 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 07 May 2015 23:40:00 +0300 Subject: RFR(S): 6755586: Test com/sun/jdi/NoLaunchOptionTest.java may erroneously fail In-Reply-To: <554B7F17.2040900@oracle.com> References: <554B7F17.2040900@oracle.com> Message-ID: <554BCDA0.7020600@oracle.com> Looks good for me! -Dmitry On 2015-05-07 18:04, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this small fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-6755586 > webrev: http://cr.openjdk.java.net/~ykantser/6755586/webrev.00/ > > This bug is actually already solved. But I'll use it as an opportunity > to re-write the test using test library. There is no need to retrieve a > free port since the error occurs before binding to the address. > > Thanks, > Katja -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From yekaterina.kantserova at oracle.com Fri May 8 06:44:05 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Fri, 08 May 2015 08:44:05 +0200 Subject: RFR(S): 6755586: Test com/sun/jdi/NoLaunchOptionTest.java may erroneously fail In-Reply-To: <554BCDA0.7020600@oracle.com> References: <554B7F17.2040900@oracle.com> <554BCDA0.7020600@oracle.com> Message-ID: <554C5B35.5030205@oracle.com> Staffan, Dmitry, thank you for the reviews! // Katja On 05/07/2015 10:40 PM, Dmitry Samersoff wrote: > Looks good for me! > > -Dmitry > > On 2015-05-07 18:04, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this small fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-6755586 >> webrev: http://cr.openjdk.java.net/~ykantser/6755586/webrev.00/ >> >> This bug is actually already solved. But I'll use it as an opportunity >> to re-write the test using test library. There is no need to retrieve a >> free port since the error occurs before binding to the address. >> >> Thanks, >> Katja > From nijiaben at gmail.com Mon May 11 11:55:07 2015 From: nijiaben at gmail.com (=?UTF-8?B?5p2O5ZiJ6bmP?=) Date: Mon, 11 May 2015 19:55:07 +0800 Subject: serviceability-dev Message-ID: nijiaben at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From aph at redhat.com Tue May 12 14:47:37 2015 From: aph at redhat.com (Andrew Haley) Date: Tue, 12 May 2015 15:47:37 +0100 Subject: sun/jvm/hotspot/oops/Symbol.java In-Reply-To: <553FE6E4.5010502@oracle.com> References: <553FD10A.7020006@redhat.com> <553FE6E4.5010502@oracle.com> Message-ID: <55521289.2050402@redhat.com> On 04/28/2015 09:00 PM, Dmitry Samersoff wrote: > Thank you for the fix, I'll take a look at it Hi, Any thoughts? HSDB doesn't really do anything without this fix, and there is no doubt that the bottom two bits of a Symbol address must be masked out. The only problem I have is deciding where to do the masking. Doing in when we create an instance of Symbol from an address always works, but I'm not sure it's the right thing to do. Andrew. From aph at redhat.com Tue May 12 15:24:24 2015 From: aph at redhat.com (Andrew Haley) Date: Tue, 12 May 2015 16:24:24 +0100 Subject: sun/jvm/hotspot/oops/Symbol.java In-Reply-To: <55521289.2050402@redhat.com> References: <553FD10A.7020006@redhat.com> <553FE6E4.5010502@oracle.com> <55521289.2050402@redhat.com> Message-ID: <55521B28.7000505@redhat.com> On 05/12/2015 03:47 PM, Andrew Haley wrote: > On 04/28/2015 09:00 PM, Dmitry Samersoff wrote: >> Thank you for the fix, I'll take a look at it > > Hi, > > Any thoughts? HSDB doesn't really do anything without this fix, and > there is no doubt that the bottom two bits of a Symbol address must be > masked out. > > The only problem I have is deciding where to do the masking. Doing in > when we create an instance of Symbol from an address always works, but > I'm not sure it's the right thing to do. For what it's worth, this may be a better patch. It works just as well, and it is only applied to cpool entries. Andrew. diff -r f39a03c0ced3 agent/src/share/classes/sun/jvm/hotspot/oops/ConstantPool.java --- a/agent/src/share/classes/sun/jvm/hotspot/oops/ConstantPool.java Tue May 12 16:14:34 2015 +0100 +++ b/agent/src/share/classes/sun/jvm/hotspot/oops/ConstantPool.java Tue May 12 16:23:00 2015 +0100 @@ -170,7 +170,7 @@ } public Symbol getSymbolAt(long index) { - return Symbol.create(getAddressAtRaw(index)); + return Symbol.create(getAddressAtRaw(index).andWithMask(~3)); } public int getIntAt(long index){ From jaroslav.bachorik at oracle.com Tue May 12 15:36:32 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 12 May 2015 17:36:32 +0200 Subject: RFR 8046869: Several java/lang/instrument/PremainClass/* tests fail due to timeout Message-ID: <55521E00.7060900@oracle.com> Please, review the following test change Issue : https://bugs.openjdk.java.net/browse/JDK-8046869 Webrev: http://cr.openjdk.java.net/~jbachorik/8046869/webrev.00 These two tests are timing out because they are deliberately crashing the virtual machine. This crash used to lead to coredump generation and the tests used to time out while dumping the core. The fix is to add '-XX:-CreateCoredumpOnCrash' VM option to suppress the core generation. Thanks, -JB- From staffan.larsen at oracle.com Tue May 12 16:46:34 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 12 May 2015 18:46:34 +0200 Subject: RFR 8046869: Several java/lang/instrument/PremainClass/* tests fail due to timeout In-Reply-To: <55521E00.7060900@oracle.com> References: <55521E00.7060900@oracle.com> Message-ID: <0BB0F347-7B37-4FC1-B113-D3327AF8AA5E@oracle.com> They only generate coredump in fastdebug mode, right? Anyway: Looks good. /Staffan > On 12 maj 2015, at 17:36, Jaroslav Bachorik wrote: > > Please, review the following test change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8046869 > Webrev: http://cr.openjdk.java.net/~jbachorik/8046869/webrev.00 > > These two tests are timing out because they are deliberately crashing the virtual machine. This crash used to lead to coredump generation and the tests used to time out while dumping the core. > > The fix is to add '-XX:-CreateCoredumpOnCrash' VM option to suppress the core generation. > > Thanks, > > -JB- From aph at redhat.com Tue May 12 17:05:12 2015 From: aph at redhat.com (Andrew Haley) Date: Tue, 12 May 2015 18:05:12 +0100 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <554B9617.2090406@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> Message-ID: <555232C8.4000704@redhat.com> http://cr.openjdk.java.net/~aph/8078521-4/ Thanks, Andrew. From dmitry.samersoff at oracle.com Tue May 12 17:10:48 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 12 May 2015 20:10:48 +0300 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <554A8CEE.2090009@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> Message-ID: <55523418.5010708@oracle.com> Everybody, Updated version: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ Now it iterates over queue and output result sorted by number of instances. -Dmitry On 2015-05-07 00:51, Derek White wrote: > Hi Dmitry, Staffan, > > Lots of good comments here. > > On the topic of what list should be printed out, I think we should focus > on objects waiting to be finalized - e.g. the contents of the > ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you > could add a summerizeQueue(TreeMap) method, or a > general iterator and lambda. > > A histogram of objects with finalize methods might also be interesting, > but you can get most of the same information from a heap histogram > (ClassHistogramDCmd, and search for count of Finalizer instances). > > BTW, by only locking the ReferenceQueue, we should only be blocking the > FinalizerThread from making progress (and I think there's a GC thread > that runs after GC that sorts found References objects that need > processing into their respective ReferenceQueues). But locking the > "unfinalized" list blocks initializing any object with a finalize() method. > > The sorting suggestion is a nice touch. > > - Derek White, GC team > > On 5/5/15 10:40 AM, Peter Levart wrote: >> Hi Dmitry, Staffan, >> >> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>> Dmitry, >>> >>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>> well considering the changes to Finalizer. >>> >>> I?m a little worried about the potentially very large String that is >>> returned from printFinalizationQueue(). A possible different approach >>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>> That would allow for outputting one line at a time. The output would >>> still be saved in memory (since the stream is buffered), but at least >>> the data is only saved once in memory, then. It would make the code a >>> bit harder to write, so its a question of how scared we are of >>> running out of memory. >> >> If the output is just a histogram of # of instances per class name, >> then it should not be too large, as there are not many different >> classes overriding finalize() method (I counted 72 overriddings of >> finalize() method in the whole jdk/src tree). >> >> I'm more concerned about the fact that while traversing the list, a >> lock is held, which might impact prompt finalization(). Is it >> acceptable for diagnostic output to impact performance and/or >> interfere with synchronization? >> >> It might be possible to scan the list without holding a lock for >> diagnostic purposes if Finalizer.remove() didn't set the removed >> Finalizer.next pointer to point back to itself: >> >> private void remove() { >> synchronized (lock) { >> if (unfinalized == this) { >> if (this.next != null) { >> unfinalized = this.next; >> } else { >> unfinalized = this.prev; >> } >> } >> if (this.next != null) { >> this.next.prev = this.prev; >> } >> if (this.prev != null) { >> this.prev.next = this.next; >> } >> // this.next = this; must not be set so that we can >> traverse the list unsynchronized >> this.prev = this; /* Indicates that this has been >> finalized */ >> } >> } >> >> For detecting whether a Finalizer is already processed, the 'prev' >> pointer could be used instead: >> >> private boolean hasBeenFinalized() { >> return (prev == this); >> } >> >> Also, to make sure a data race dereferencing 'unfinalized' in >> unsynchronized printFinalizationQueue() would get you a fully >> initialized Finalizer instance (in particular the next pointer), you >> would have to make the 'unfinalized' field volatile or insert an >> Unsafe.storeFence() before assigning to 'unfinalized': >> >> private void add() { >> synchronized (lock) { >> if (unfinalized != null) { >> this.next = unfinalized; >> unfinalized.prev = this; >> } >> // make sure a data race dereferencing 'unfinalized' >> // in printFinalizationQueue() does see the Finalizer >> // instance fully initialized >> // (in particular the 'next' pointer) >> U.storeFence(); >> unfinalized = this; >> } >> } >> >> >> By doing these modifications, I think you can remove >> synchronized(lock) {} from printFinalizationQueue(). >> >>> >>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>> not sure of the difference, perhaps someone from the GC team can help >>> out? >> >> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >> instances pending processing by finalizer thread because their >> referents are eligible for finalization (they are not reachable any >> more). The 'unfinalized' is a doubly-linked list of all Finalizer >> instances for which their referents have not been finalized yet >> (including those that are still reachable and alive). The later serves >> two purposes: >> - it keeps Finalizer instances reachable until they are processed >> - it is a source of unfinalized instances for >> running-finalizers-on-exit if requested by >> System.runFinalizersOnExit(true); >> >> So it really depends on what one would like to see. Showing the queue >> may be interesting if one wants to see how the finalizer thread is >> coping with processing the finalize() invocations. Showing unfinalized >> list may be interesting if one wants to know how many live + >> finalization pending instances are there on the heap that override >> finalize() method. >> >> Regards, Peter >> >>> >>> For the output, it would be a nice touch to sort it on the number of >>> references for each type. Perhaps outputting it more like a table >>> (see the old code again) would also make it easier to digest. >>> >>> In diagnosticCommand.hpp, the new commands should override >>> permission() and limit access: >>> >>> static const JavaPermission permission() { >>> JavaPermission p = {"java.lang.management.ManagementPermission", >>> "monitor", NULL}; >>> return p; >>> } >>> >>> The two tests don?t validate the output in any way. Would it be >>> possible to add validation? Perhaps hard to make sure an object is on >>> the finalizer queue? Hmm. >>> >>> Thanks, >>> /Staffan >>> >>> >>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>> wrote: >>>> >>>> Everyone, >>>> >>>> Please review the fix: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>> >>>> heap dcmd outputs the same information as SIGBREAK >>>> >>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>> with count >>>> >>>> -Dmitry >>>> >>>> -- >>>> Dmitry Samersoff >>>> Oracle Java development team, Saint Petersburg, Russia >>>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jaroslav.bachorik at oracle.com Tue May 12 17:36:33 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 12 May 2015 19:36:33 +0200 Subject: RFR 8046869: Several java/lang/instrument/PremainClass/* tests fail due to timeout In-Reply-To: <0BB0F347-7B37-4FC1-B113-D3327AF8AA5E@oracle.com> References: <55521E00.7060900@oracle.com> <0BB0F347-7B37-4FC1-B113-D3327AF8AA5E@oracle.com> Message-ID: <55523A21.4070308@oracle.com> On 12.5.2015 18:46, Staffan Larsen wrote: > They only generate coredump in fastdebug mode, right? I think so. But it is just a side effect and the test does not inspect the coredump anyway. > > Anyway: Looks good. Thanks! -JB- > > /Staffan > >> On 12 maj 2015, at 17:36, Jaroslav Bachorik wrote: >> >> Please, review the following test change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8046869 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8046869/webrev.00 >> >> These two tests are timing out because they are deliberately crashing the virtual machine. This crash used to lead to coredump generation and the tests used to time out while dumping the core. >> >> The fix is to add '-XX:-CreateCoredumpOnCrash' VM option to suppress the core generation. >> >> Thanks, >> >> -JB- > From jaroslav.bachorik at oracle.com Tue May 12 18:07:08 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 12 May 2015 20:07:08 +0200 Subject: RFR 8029098: Exclude javax/management/remote/mandatory/notif/ListenerScaleTest.java from running on fastdebug builds Message-ID: <5552414C.4080707@oracle.com> Please, review the following simple change Issue : https://bugs.openjdk.java.net/browse/JDK-8029098 Webrev: http://cr.openjdk.java.net/~jbachorik/8029098/webrev.00 The 'ListenerScaleTest' is meant to check the proper scaling of the MBean notification dispatching. This doesn't make much sense for fastdebug builds where all the timing is rather skewed. The patch checks for the fastdebug build and just skips the test and outputs a notification message. Thanks, -JB- From shanliang.jiang at oracle.com Tue May 12 18:50:28 2015 From: shanliang.jiang at oracle.com (Shanliang Jiang) Date: Tue, 12 May 2015 20:50:28 +0200 Subject: RFR 8029098: Exclude javax/management/remote/mandatory/notif/ListenerScaleTest.java from running on fastdebug builds In-Reply-To: <5552414C.4080707@oracle.com> References: <5552414C.4080707@oracle.com> Message-ID: <55524B74.2050905@oracle.com> Looks good to me! Shanliang Jaroslav Bachorik wrote: > Please, review the following simple change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8029098 > Webrev: http://cr.openjdk.java.net/~jbachorik/8029098/webrev.00 > > The 'ListenerScaleTest' is meant to check the proper scaling of the > MBean notification dispatching. This doesn't make much sense for > fastdebug builds where all the timing is rather skewed. > > The patch checks for the fastdebug build and just skips the test and > outputs a notification message. > > Thanks, > > -JB- From peter.levart at gmail.com Tue May 12 22:05:41 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 13 May 2015 00:05:41 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55523418.5010708@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> Message-ID: <55527935.5010408@gmail.com> Hi Dmitry, You iterate the queue then, not the unfinalized list. That's more logical. Holding the queue's lock may pause reference handler and finalizer threads for the entire time of iteration. This can blow up the application. Suppose one wants to diagnose the application because he suspects that finalizer thread hardly keeps up with production of finalizable instances. This can happen if the allocation rate of finalizable objects is very high and/or finalize() methods are not coded to be fast enough. Suppose the queue of Finalizer(s) builds up gradually and is already holding several million objects. Now you initiate the diagnostic command to print the queue. The iteration over and grouping/counting of several millions of Finalizer(s) takes some time. This blocks finalizer thread and reference handler thread. But does not block the threads that allocate finalizable objects. By the time the iteration is over, the JVM blows up and application slows down to a halt doing GCs most of the time, getting OOMEs, etc... It is possible to iterate the elements of the queue for diagnostic purposes without holding a lock. The modification required to do that is the following (havent tested this, but I think it should work): http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ I also suggest the addition to the ReferenceQueue to be contained (package-private) and as generic as possible. That's why I suggest forEach iteration method with no concrete logic. It would be possible to encapsulate the entire logic into a special package-private class (say java.lang.ref.DiagnosticCommands) with static method(s) (printFinalizationQueue)... You could just expose a package-private forEach static method from Finalizer and code the rest in DiagnosticCommands. Regards, Peter On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: > Everybody, > > Updated version: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ > > Now it iterates over queue and output result sorted by number of instances. > > -Dmitry > > On 2015-05-07 00:51, Derek White wrote: >> Hi Dmitry, Staffan, >> >> Lots of good comments here. >> >> On the topic of what list should be printed out, I think we should focus >> on objects waiting to be finalized - e.g. the contents of the >> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >> could add a summerizeQueue(TreeMap) method, or a >> general iterator and lambda. >> >> A histogram of objects with finalize methods might also be interesting, >> but you can get most of the same information from a heap histogram >> (ClassHistogramDCmd, and search for count of Finalizer instances). >> >> BTW, by only locking the ReferenceQueue, we should only be blocking the >> FinalizerThread from making progress (and I think there's a GC thread >> that runs after GC that sorts found References objects that need >> processing into their respective ReferenceQueues). But locking the >> "unfinalized" list blocks initializing any object with a finalize() method. >> >> The sorting suggestion is a nice touch. >> >> - Derek White, GC team >> >> On 5/5/15 10:40 AM, Peter Levart wrote: >>> Hi Dmitry, Staffan, >>> >>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>> Dmitry, >>>> >>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>> well considering the changes to Finalizer. >>>> >>>> I?m a little worried about the potentially very large String that is >>>> returned from printFinalizationQueue(). A possible different approach >>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>> That would allow for outputting one line at a time. The output would >>>> still be saved in memory (since the stream is buffered), but at least >>>> the data is only saved once in memory, then. It would make the code a >>>> bit harder to write, so its a question of how scared we are of >>>> running out of memory. >>> If the output is just a histogram of # of instances per class name, >>> then it should not be too large, as there are not many different >>> classes overriding finalize() method (I counted 72 overriddings of >>> finalize() method in the whole jdk/src tree). >>> >>> I'm more concerned about the fact that while traversing the list, a >>> lock is held, which might impact prompt finalization(). Is it >>> acceptable for diagnostic output to impact performance and/or >>> interfere with synchronization? >>> >>> It might be possible to scan the list without holding a lock for >>> diagnostic purposes if Finalizer.remove() didn't set the removed >>> Finalizer.next pointer to point back to itself: >>> >>> private void remove() { >>> synchronized (lock) { >>> if (unfinalized == this) { >>> if (this.next != null) { >>> unfinalized = this.next; >>> } else { >>> unfinalized = this.prev; >>> } >>> } >>> if (this.next != null) { >>> this.next.prev = this.prev; >>> } >>> if (this.prev != null) { >>> this.prev.next = this.next; >>> } >>> // this.next = this; must not be set so that we can >>> traverse the list unsynchronized >>> this.prev = this; /* Indicates that this has been >>> finalized */ >>> } >>> } >>> >>> For detecting whether a Finalizer is already processed, the 'prev' >>> pointer could be used instead: >>> >>> private boolean hasBeenFinalized() { >>> return (prev == this); >>> } >>> >>> Also, to make sure a data race dereferencing 'unfinalized' in >>> unsynchronized printFinalizationQueue() would get you a fully >>> initialized Finalizer instance (in particular the next pointer), you >>> would have to make the 'unfinalized' field volatile or insert an >>> Unsafe.storeFence() before assigning to 'unfinalized': >>> >>> private void add() { >>> synchronized (lock) { >>> if (unfinalized != null) { >>> this.next = unfinalized; >>> unfinalized.prev = this; >>> } >>> // make sure a data race dereferencing 'unfinalized' >>> // in printFinalizationQueue() does see the Finalizer >>> // instance fully initialized >>> // (in particular the 'next' pointer) >>> U.storeFence(); >>> unfinalized = this; >>> } >>> } >>> >>> >>> By doing these modifications, I think you can remove >>> synchronized(lock) {} from printFinalizationQueue(). >>> >>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>> not sure of the difference, perhaps someone from the GC team can help >>>> out? >>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>> instances pending processing by finalizer thread because their >>> referents are eligible for finalization (they are not reachable any >>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>> instances for which their referents have not been finalized yet >>> (including those that are still reachable and alive). The later serves >>> two purposes: >>> - it keeps Finalizer instances reachable until they are processed >>> - it is a source of unfinalized instances for >>> running-finalizers-on-exit if requested by >>> System.runFinalizersOnExit(true); >>> >>> So it really depends on what one would like to see. Showing the queue >>> may be interesting if one wants to see how the finalizer thread is >>> coping with processing the finalize() invocations. Showing unfinalized >>> list may be interesting if one wants to know how many live + >>> finalization pending instances are there on the heap that override >>> finalize() method. >>> >>> Regards, Peter >>> >>>> For the output, it would be a nice touch to sort it on the number of >>>> references for each type. Perhaps outputting it more like a table >>>> (see the old code again) would also make it easier to digest. >>>> >>>> In diagnosticCommand.hpp, the new commands should override >>>> permission() and limit access: >>>> >>>> static const JavaPermission permission() { >>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>> "monitor", NULL}; >>>> return p; >>>> } >>>> >>>> The two tests don?t validate the output in any way. Would it be >>>> possible to add validation? Perhaps hard to make sure an object is on >>>> the finalizer queue? Hmm. >>>> >>>> Thanks, >>>> /Staffan >>>> >>>> >>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>> wrote: >>>>> >>>>> Everyone, >>>>> >>>>> Please review the fix: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>> >>>>> heap dcmd outputs the same information as SIGBREAK >>>>> >>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>> with count >>>>> >>>>> -Dmitry >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.larsen at oracle.com Wed May 13 06:46:31 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Wed, 13 May 2015 08:46:31 +0200 Subject: RFR 8029098: Exclude javax/management/remote/mandatory/notif/ListenerScaleTest.java from running on fastdebug builds In-Reply-To: <5552414C.4080707@oracle.com> References: <5552414C.4080707@oracle.com> Message-ID: <5A0000F6-F6C5-4177-82EB-62CFA20DDCC0@oracle.com> Looks good! Thanks, /Staffan > On 12 maj 2015, at 20:07, Jaroslav Bachorik wrote: > > Please, review the following simple change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8029098 > Webrev: http://cr.openjdk.java.net/~jbachorik/8029098/webrev.00 > > The 'ListenerScaleTest' is meant to check the proper scaling of the MBean notification dispatching. This doesn't make much sense for fastdebug builds where all the timing is rather skewed. > > The patch checks for the fastdebug build and just skips the test and outputs a notification message. > > Thanks, > > -JB- From jaroslav.bachorik at oracle.com Wed May 13 11:46:09 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Wed, 13 May 2015 13:46:09 +0200 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> Message-ID: <55533981.70204@oracle.com> On 1.5.2015 21:55, Martin Buchholz wrote: > > > On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik > > wrote: > > On 30.4.2015 19:18, Martin Buchholz wrote: > > Tests that sleep can almost always be better written some other way. > In this case, I would prefer busy-waiting with timeout until the > expected condition becomes true. > > > The thing is that in case of a real issue with the thread counters we > a/ would be busy-waiting till the test times out (using an arbitrary > delay is also problematic due to different performance of different > machines running with different configurations) > > > Far less problematic (performance-wise and reliability-wise) than the > fixed sleep. > > b/ would get a rather confusing message about the test timing out at > the end > > > You can easily improve the error message. Well, not that easily. It is not possible to get a notification when JTREG decides to timeout the test. So you will get the standard JTREG message and that's all. I was able to modify the test to wait for a given condition and provide useful messages in case of mismatch and retry. For the price of an increased complexity. On the other hand, the test should be much more resilient to timing errors caused by slow setups. Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 Thanks, -JB- From jaroslav.bachorik at oracle.com Wed May 13 15:39:26 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Wed, 13 May 2015 17:39:26 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state Message-ID: <5553702E.8050507@oracle.com> Please, review the following change Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. The perf counters show stale data and mislead the users. Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. Thanks, -JB- From martinrb at google.com Wed May 13 17:40:01 2015 From: martinrb at google.com (Martin Buchholz) Date: Wed, 13 May 2015 10:40:01 -0700 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <55533981.70204@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> Message-ID: toString() should never return null, I think. 52 @Override 53 public String toString() { 54 T resolved = val.get(); 55 return resolved != null ? resolved.toString() : null; 56 } I expected methods like waitForCondition to include a timeout with failure. I like 10 seconds, being large enough to never be hit spuriously in tests. Why not () -> (long) mbean.getThreadCount(), 169 ()->{ 170 curLiveThreadCount = mbean.getThreadCount(); 171 return (long)curLiveThreadCount; 172 }, I worry that mbean.getThreadCount() is hard to test since the "system" may spin up and shut down utility threads at any time. On Wed, May 13, 2015 at 4:46 AM, Jaroslav Bachorik < jaroslav.bachorik at oracle.com> wrote: > On 1.5.2015 21:55, Martin Buchholz wrote: > >> >> >> On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik >> > >> wrote: >> >> On 30.4.2015 19:18, Martin Buchholz wrote: >> >> Tests that sleep can almost always be better written some other >> way. >> In this case, I would prefer busy-waiting with timeout until the >> expected condition becomes true. >> >> >> The thing is that in case of a real issue with the thread counters we >> a/ would be busy-waiting till the test times out (using an arbitrary >> delay is also problematic due to different performance of different >> machines running with different configurations) >> >> >> Far less problematic (performance-wise and reliability-wise) than the >> fixed sleep. >> >> b/ would get a rather confusing message about the test timing out at >> the end >> >> >> You can easily improve the error message. >> > > Well, not that easily. It is not possible to get a notification when JTREG > decides to timeout the test. So you will get the standard JTREG message and > that's all. > > I was able to modify the test to wait for a given condition and provide > useful messages in case of mismatch and retry. For the price of an > increased complexity. On the other hand, the test should be much more > resilient to timing errors caused by slow setups. > > Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 > > Thanks, > > -JB- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Wed May 13 18:07:39 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Wed, 13 May 2015 20:07:39 +0200 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> Message-ID: <555392EB.8020506@oracle.com> On 13.5.2015 19:40, Martin Buchholz wrote: > toString()should never return null, I think. It doesn't matter much here. The test would fail with an NPE and it would be right to do so. None of the suppliers should ever return null. > > 52 @Override > 53 public String toString() { > 54 T resolved = val.get(); > 55 return resolved != null ? resolved.toString() : null; > 56 } > > > I expected methods like waitForCondition to include a timeout with > failure. I like 10 seconds, being large enough to never be hit > spuriously in tests. It's difficult to find a value 'large enough'. Imagine the test running on a small embedded device and fastdebug build. I had my fun fixing tests failing intermittently because it was thought that the original timeout was large enough. I better leave it to the harness. > > Why not > () -> (long) mbean.getThreadCount(), Because curLiveThreadCount needs to be set to mbean.getThreadCount() value. > > 169 ()->{ > 170 curLiveThreadCount = mbean.getThreadCount(); > 171 return (long)curLiveThreadCount; > 172 }, > > > I worry that > mbean.getThreadCount() > is hard to test since the "system" may spin up and shut down utility > threads at any time. The 'system' threads are not reported by this method. And the current understanding is that once VM is fully initialized no user-observable threads are randomly started on behalf of the system. -JB- > > On Wed, May 13, 2015 at 4:46 AM, Jaroslav Bachorik > > wrote: > > On 1.5.2015 21:55, Martin Buchholz wrote: > > > > On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik > > >> wrote: > > On 30.4.2015 19:18, Martin Buchholz wrote: > > Tests that sleep can almost always be better written > some other way. > In this case, I would prefer busy-waiting with timeout > until the > expected condition becomes true. > > > The thing is that in case of a real issue with the thread > counters we > a/ would be busy-waiting till the test times out (using an > arbitrary > delay is also problematic due to different performance of > different > machines running with different configurations) > > > Far less problematic (performance-wise and reliability-wise) > than the > fixed sleep. > > b/ would get a rather confusing message about the test > timing out at > the end > > > You can easily improve the error message. > > > Well, not that easily. It is not possible to get a notification when > JTREG decides to timeout the test. So you will get the standard > JTREG message and that's all. > > I was able to modify the test to wait for a given condition and > provide useful messages in case of mismatch and retry. For the price > of an increased complexity. On the other hand, the test should be > much more resilient to timing errors caused by slow setups. > > Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 > > Thanks, > > -JB- > > From martinrb at google.com Wed May 13 19:14:44 2015 From: martinrb at google.com (Martin Buchholz) Date: Wed, 13 May 2015 12:14:44 -0700 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <555392EB.8020506@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <555392EB.8020506@oracle.com> Message-ID: David has given you an approval; feel free to ignore me! I tried running the test against jdk9, but it timed out! It's common for users to introduce parallelism in classloaders or jar-loaders (we do this at google) which may cause arbitrary thread fluctuations. That makes this particular API rather difficult to test robustly, especially if you are only testing against the spec. On Wed, May 13, 2015 at 11:07 AM, Jaroslav Bachorik < jaroslav.bachorik at oracle.com> wrote: > On 13.5.2015 19:40, Martin Buchholz wrote: > >> toString()should never return null, I think. >> > > It doesn't matter much here. The test would fail with an NPE and it would > be right to do so. None of the suppliers should ever return null. > > >> 52 @Override >> 53 public String toString() { >> 54 T resolved = val.get(); >> 55 return resolved != null ? resolved.toString() : null; >> 56 } >> >> >> I expected methods like waitForCondition to include a timeout with >> failure. I like 10 seconds, being large enough to never be hit >> spuriously in tests. >> > > It's difficult to find a value 'large enough'. Imagine the test running on > a small embedded device and fastdebug build. I had my fun fixing tests > failing intermittently because it was thought that the original timeout was > large enough. I better leave it to the harness. > > >> Why not >> () -> (long) mbean.getThreadCount(), >> > > Because curLiveThreadCount needs to be set to mbean.getThreadCount() value. > > >> 169 ()->{ >> 170 curLiveThreadCount = mbean.getThreadCount(); >> 171 return (long)curLiveThreadCount; >> 172 }, >> >> >> I worry that >> mbean.getThreadCount() >> is hard to test since the "system" may spin up and shut down utility >> threads at any time. >> > > The 'system' threads are not reported by this method. And the current > understanding is that once VM is fully initialized no user-observable > threads are randomly started on behalf of the system. > > -JB- > > >> On Wed, May 13, 2015 at 4:46 AM, Jaroslav Bachorik >> > >> wrote: >> >> On 1.5.2015 21:55, Martin Buchholz wrote: >> >> >> >> On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik >> > >> > >> >> wrote: >> >> On 30.4.2015 19:18, Martin Buchholz wrote: >> >> Tests that sleep can almost always be better written >> some other way. >> In this case, I would prefer busy-waiting with timeout >> until the >> expected condition becomes true. >> >> >> The thing is that in case of a real issue with the thread >> counters we >> a/ would be busy-waiting till the test times out (using an >> arbitrary >> delay is also problematic due to different performance of >> different >> machines running with different configurations) >> >> >> Far less problematic (performance-wise and reliability-wise) >> than the >> fixed sleep. >> >> b/ would get a rather confusing message about the test >> timing out at >> the end >> >> >> You can easily improve the error message. >> >> >> Well, not that easily. It is not possible to get a notification when >> JTREG decides to timeout the test. So you will get the standard >> JTREG message and that's all. >> >> I was able to modify the test to wait for a given condition and >> provide useful messages in case of mismatch and retry. For the price >> of an increased complexity. On the other hand, the test should be >> much more resilient to timing errors caused by slow setups. >> >> Updated webrev: >> http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 >> >> Thanks, >> >> -JB- >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysr1729 at gmail.com Wed May 13 20:08:15 2015 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Wed, 13 May 2015 13:08:15 -0700 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 Message-ID: With perm gen going away (and being replaced by metaspace) in JDK 8, it makes sense that the counter sun.gc.policy.generations should be "2", rather than "3". However, in JDK 8 that counter still says 3. As I understand, the intention was that this counter would allow you to (for example) know the range of the sun.gc.generation.$num.* counters describing each of $num < sun.gc.policy.generations in the heap. Recall that the erstwhile perm gen in JDK 7 used to be synonymous with sun.gc.generation.2, but the JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. The fix is simple, and I can submit a patch. Is there an existing bug for this? thanks! -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek.white at oracle.com Wed May 13 20:34:26 2015 From: derek.white at oracle.com (Derek White) Date: Wed, 13 May 2015 16:34:26 -0400 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55527935.5010408@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> Message-ID: <5553B552.9060900@oracle.com> Hi Peter, I don't have smoking gun evidence that your change introduces a bug, but I have some concerns... On 5/12/15 6:05 PM, Peter Levart wrote: > Hi Dmitry, > > You iterate the queue then, not the unfinalized list. That's more logical. > > Holding the queue's lock may pause reference handler and finalizer > threads for the entire time of iteration. This can blow up the > application. Suppose one wants to diagnose the application because he > suspects that finalizer thread hardly keeps up with production of > finalizable instances. This can happen if the allocation rate of > finalizable objects is very high and/or finalize() methods are not > coded to be fast enough. Suppose the queue of Finalizer(s) builds up > gradually and is already holding several million objects. Now you > initiate the diagnostic command to print the queue. The iteration over > and grouping/counting of several millions of Finalizer(s) takes some > time. This blocks finalizer thread and reference handler thread. But > does not block the threads that allocate finalizable objects. By the > time the iteration is over, the JVM blows up and application slows > down to a halt doing GCs most of the time, getting OOMEs, etc... > > It is possible to iterate the elements of the queue for diagnostic > purposes without holding a lock. The modification required to do that > is the following (havent tested this, but I think it should work): > > http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ > One issue to watch out for is the garbage collectors inspect the Reference.next field from C code directly (to distinguish active vs. pending, iterate over oops, etc.). You can search hotspot for java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. Your change makes "inactive" References superficially look like "enqueued" References. The GC code is rather subtle and I haven't yet seen a case where it would get confused by this change, but there could easily be something like that lurking in the GC code. > I also suggest the addition to the ReferenceQueue to be contained > (package-private) and as generic as possible. That's why I suggest > forEach iteration method with no concrete logic. > > It would be possible to encapsulate the entire logic into a special > package-private class (say java.lang.ref.DiagnosticCommands) with > static method(s) (printFinalizationQueue)... You could just expose a > package-private forEach static method from Finalizer and code the rest > in DiagnosticCommands. That's good for encapsulation. But I'm concerned that if "forEach" got exposed beyond careful use in DiagnosticCommands, and the Referents were leaked back into the heap, then we could get unexpected object resurrection after finalization. This isn't a bug on it's own - any finalizer might resurrect its object if not careful, but ordinarily /only/ the finalizer could resurrect the object. This could invalidate application invariants? I agree that using a lock over the ReferenceQueue iteration opens up /another/ case of diagnostics affecting application behavior. And there are pathological scenarios where this gets severe. This is unfortunate but not uncommon. There is enough complication here that you should be sure that the fix for diagnostics performance doesn't introduce subtle bugs to the system in general. A change in this area should get a thorough review from both the runtime and GC sides. Better yet, the reference handling code in GC and runtime should probably be thrown out and re-written, but that's another issue :-) - Derek > Regards, Peter > > > On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >> Everybody, >> >> Updated version: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >> >> Now it iterates over queue and output result sorted by number of instances. >> >> -Dmitry >> >> On 2015-05-07 00:51, Derek White wrote: >>> Hi Dmitry, Staffan, >>> >>> Lots of good comments here. >>> >>> On the topic of what list should be printed out, I think we should focus >>> on objects waiting to be finalized - e.g. the contents of the >>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>> could add a summerizeQueue(TreeMap) method, or a >>> general iterator and lambda. >>> >>> A histogram of objects with finalize methods might also be interesting, >>> but you can get most of the same information from a heap histogram >>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>> >>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>> FinalizerThread from making progress (and I think there's a GC thread >>> that runs after GC that sorts found References objects that need >>> processing into their respective ReferenceQueues). But locking the >>> "unfinalized" list blocks initializing any object with a finalize() method. >>> >>> The sorting suggestion is a nice touch. >>> >>> - Derek White, GC team >>> >>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>> Hi Dmitry, Staffan, >>>> >>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>> Dmitry, >>>>> >>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>> well considering the changes to Finalizer. >>>>> >>>>> I?m a little worried about the potentially very large String that is >>>>> returned from printFinalizationQueue(). A possible different approach >>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>> That would allow for outputting one line at a time. The output would >>>>> still be saved in memory (since the stream is buffered), but at least >>>>> the data is only saved once in memory, then. It would make the code a >>>>> bit harder to write, so its a question of how scared we are of >>>>> running out of memory. >>>> If the output is just a histogram of # of instances per class name, >>>> then it should not be too large, as there are not many different >>>> classes overriding finalize() method (I counted 72 overriddings of >>>> finalize() method in the whole jdk/src tree). >>>> >>>> I'm more concerned about the fact that while traversing the list, a >>>> lock is held, which might impact prompt finalization(). Is it >>>> acceptable for diagnostic output to impact performance and/or >>>> interfere with synchronization? >>>> >>>> It might be possible to scan the list without holding a lock for >>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>> Finalizer.next pointer to point back to itself: >>>> >>>> private void remove() { >>>> synchronized (lock) { >>>> if (unfinalized == this) { >>>> if (this.next != null) { >>>> unfinalized = this.next; >>>> } else { >>>> unfinalized = this.prev; >>>> } >>>> } >>>> if (this.next != null) { >>>> this.next.prev = this.prev; >>>> } >>>> if (this.prev != null) { >>>> this.prev.next = this.next; >>>> } >>>> // this.next = this; must not be set so that we can >>>> traverse the list unsynchronized >>>> this.prev = this; /* Indicates that this has been >>>> finalized */ >>>> } >>>> } >>>> >>>> For detecting whether a Finalizer is already processed, the 'prev' >>>> pointer could be used instead: >>>> >>>> private boolean hasBeenFinalized() { >>>> return (prev == this); >>>> } >>>> >>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>> unsynchronized printFinalizationQueue() would get you a fully >>>> initialized Finalizer instance (in particular the next pointer), you >>>> would have to make the 'unfinalized' field volatile or insert an >>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>> >>>> private void add() { >>>> synchronized (lock) { >>>> if (unfinalized != null) { >>>> this.next = unfinalized; >>>> unfinalized.prev = this; >>>> } >>>> // make sure a data race dereferencing 'unfinalized' >>>> // in printFinalizationQueue() does see the Finalizer >>>> // instance fully initialized >>>> // (in particular the 'next' pointer) >>>> U.storeFence(); >>>> unfinalized = this; >>>> } >>>> } >>>> >>>> >>>> By doing these modifications, I think you can remove >>>> synchronized(lock) {} from printFinalizationQueue(). >>>> >>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>> not sure of the difference, perhaps someone from the GC team can help >>>>> out? >>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>> instances pending processing by finalizer thread because their >>>> referents are eligible for finalization (they are not reachable any >>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>> instances for which their referents have not been finalized yet >>>> (including those that are still reachable and alive). The later serves >>>> two purposes: >>>> - it keeps Finalizer instances reachable until they are processed >>>> - it is a source of unfinalized instances for >>>> running-finalizers-on-exit if requested by >>>> System.runFinalizersOnExit(true); >>>> >>>> So it really depends on what one would like to see. Showing the queue >>>> may be interesting if one wants to see how the finalizer thread is >>>> coping with processing the finalize() invocations. Showing unfinalized >>>> list may be interesting if one wants to know how many live + >>>> finalization pending instances are there on the heap that override >>>> finalize() method. >>>> >>>> Regards, Peter >>>> >>>>> For the output, it would be a nice touch to sort it on the number of >>>>> references for each type. Perhaps outputting it more like a table >>>>> (see the old code again) would also make it easier to digest. >>>>> >>>>> In diagnosticCommand.hpp, the new commands should override >>>>> permission() and limit access: >>>>> >>>>> static const JavaPermission permission() { >>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>> "monitor", NULL}; >>>>> return p; >>>>> } >>>>> >>>>> The two tests don?t validate the output in any way. Would it be >>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>> the finalizer queue? Hmm. >>>>> >>>>> Thanks, >>>>> /Staffan >>>>> >>>>> >>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>> wrote: >>>>>> >>>>>> Everyone, >>>>>> >>>>>> Please review the fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>> >>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>> >>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>> with count >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> -- >>>>>> Dmitry Samersoff >>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>> * I would love to change the world, but they won't give me the sources. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek.white at oracle.com Wed May 13 21:52:33 2015 From: derek.white at oracle.com (Derek White) Date: Wed, 13 May 2015 17:52:33 -0400 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55523418.5010708@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> Message-ID: <5553C7A1.8060004@oracle.com> Hi Dmitry, Some review comments below... On 5/12/15 1:10 PM, Dmitry Samersoff wrote: > Everybody, > > Updated version: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ > > Now it iterates over queue and output result sorted by number of instances. FinalReference.java - Update copyright to 2015. ReferenceQueue.java - Update copyright to 2015. class "mutableInt": - Should be "MutableInt" - Not a collections lawyer, but should MutableInt implement Comparable? countInstances(): - Javadoc should mention may return null. - Can countInstances() be package-private? - After you get the lock in line 138, you should recheck for "head == null", and return null if so. Otherwise it might sometimes return null and sometimes return an empty map. - BIG: Is loop missing? Should "if" at line 140 be "while"? - Merge lines 147, 148: "} else {" - Check for consistent line spacing. - Derek > > -Dmitry > > On 2015-05-07 00:51, Derek White wrote: >> Hi Dmitry, Staffan, >> >> Lots of good comments here. >> >> On the topic of what list should be printed out, I think we should focus >> on objects waiting to be finalized - e.g. the contents of the >> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >> could add a summerizeQueue(TreeMap) method, or a >> general iterator and lambda. >> >> A histogram of objects with finalize methods might also be interesting, >> but you can get most of the same information from a heap histogram >> (ClassHistogramDCmd, and search for count of Finalizer instances). >> >> BTW, by only locking the ReferenceQueue, we should only be blocking the >> FinalizerThread from making progress (and I think there's a GC thread >> that runs after GC that sorts found References objects that need >> processing into their respective ReferenceQueues). But locking the >> "unfinalized" list blocks initializing any object with a finalize() method. >> >> The sorting suggestion is a nice touch. >> >> - Derek White, GC team >> >> On 5/5/15 10:40 AM, Peter Levart wrote: >>> Hi Dmitry, Staffan, >>> >>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>> Dmitry, >>>> >>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>> well considering the changes to Finalizer. >>>> >>>> I?m a little worried about the potentially very large String that is >>>> returned from printFinalizationQueue(). A possible different approach >>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>> That would allow for outputting one line at a time. The output would >>>> still be saved in memory (since the stream is buffered), but at least >>>> the data is only saved once in memory, then. It would make the code a >>>> bit harder to write, so its a question of how scared we are of >>>> running out of memory. >>> If the output is just a histogram of # of instances per class name, >>> then it should not be too large, as there are not many different >>> classes overriding finalize() method (I counted 72 overriddings of >>> finalize() method in the whole jdk/src tree). >>> >>> I'm more concerned about the fact that while traversing the list, a >>> lock is held, which might impact prompt finalization(). Is it >>> acceptable for diagnostic output to impact performance and/or >>> interfere with synchronization? >>> >>> It might be possible to scan the list without holding a lock for >>> diagnostic purposes if Finalizer.remove() didn't set the removed >>> Finalizer.next pointer to point back to itself: >>> >>> private void remove() { >>> synchronized (lock) { >>> if (unfinalized == this) { >>> if (this.next != null) { >>> unfinalized = this.next; >>> } else { >>> unfinalized = this.prev; >>> } >>> } >>> if (this.next != null) { >>> this.next.prev = this.prev; >>> } >>> if (this.prev != null) { >>> this.prev.next = this.next; >>> } >>> // this.next = this; must not be set so that we can >>> traverse the list unsynchronized >>> this.prev = this; /* Indicates that this has been >>> finalized */ >>> } >>> } >>> >>> For detecting whether a Finalizer is already processed, the 'prev' >>> pointer could be used instead: >>> >>> private boolean hasBeenFinalized() { >>> return (prev == this); >>> } >>> >>> Also, to make sure a data race dereferencing 'unfinalized' in >>> unsynchronized printFinalizationQueue() would get you a fully >>> initialized Finalizer instance (in particular the next pointer), you >>> would have to make the 'unfinalized' field volatile or insert an >>> Unsafe.storeFence() before assigning to 'unfinalized': >>> >>> private void add() { >>> synchronized (lock) { >>> if (unfinalized != null) { >>> this.next = unfinalized; >>> unfinalized.prev = this; >>> } >>> // make sure a data race dereferencing 'unfinalized' >>> // in printFinalizationQueue() does see the Finalizer >>> // instance fully initialized >>> // (in particular the 'next' pointer) >>> U.storeFence(); >>> unfinalized = this; >>> } >>> } >>> >>> >>> By doing these modifications, I think you can remove >>> synchronized(lock) {} from printFinalizationQueue(). >>> >>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>> not sure of the difference, perhaps someone from the GC team can help >>>> out? >>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>> instances pending processing by finalizer thread because their >>> referents are eligible for finalization (they are not reachable any >>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>> instances for which their referents have not been finalized yet >>> (including those that are still reachable and alive). The later serves >>> two purposes: >>> - it keeps Finalizer instances reachable until they are processed >>> - it is a source of unfinalized instances for >>> running-finalizers-on-exit if requested by >>> System.runFinalizersOnExit(true); >>> >>> So it really depends on what one would like to see. Showing the queue >>> may be interesting if one wants to see how the finalizer thread is >>> coping with processing the finalize() invocations. Showing unfinalized >>> list may be interesting if one wants to know how many live + >>> finalization pending instances are there on the heap that override >>> finalize() method. >>> >>> Regards, Peter >>> >>>> For the output, it would be a nice touch to sort it on the number of >>>> references for each type. Perhaps outputting it more like a table >>>> (see the old code again) would also make it easier to digest. >>>> >>>> In diagnosticCommand.hpp, the new commands should override >>>> permission() and limit access: >>>> >>>> static const JavaPermission permission() { >>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>> "monitor", NULL}; >>>> return p; >>>> } >>>> >>>> The two tests don?t validate the output in any way. Would it be >>>> possible to add validation? Perhaps hard to make sure an object is on >>>> the finalizer queue? Hmm. >>>> >>>> Thanks, >>>> /Staffan >>>> >>>> >>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>> wrote: >>>>> >>>>> Everyone, >>>>> >>>>> Please review the fix: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>> >>>>> heap dcmd outputs the same information as SIGBREAK >>>>> >>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>> with count >>>>> >>>>> -Dmitry >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. > From david.holmes at oracle.com Thu May 14 01:57:36 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2015 11:57:36 +1000 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <55533981.70204@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> Message-ID: <55540110.3050008@oracle.com> On 13/05/2015 9:46 PM, Jaroslav Bachorik wrote: > On 1.5.2015 21:55, Martin Buchholz wrote: >> >> >> On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik >> > >> wrote: >> >> On 30.4.2015 19:18, Martin Buchholz wrote: >> >> Tests that sleep can almost always be better written some >> other way. >> In this case, I would prefer busy-waiting with timeout until the >> expected condition becomes true. >> >> >> The thing is that in case of a real issue with the thread counters we >> a/ would be busy-waiting till the test times out (using an arbitrary >> delay is also problematic due to different performance of different >> machines running with different configurations) >> >> >> Far less problematic (performance-wise and reliability-wise) than the >> fixed sleep. >> >> b/ would get a rather confusing message about the test timing out at >> the end >> >> >> You can easily improve the error message. > > Well, not that easily. It is not possible to get a notification when > JTREG decides to timeout the test. So you will get the standard JTREG > message and that's all. > > I was able to modify the test to wait for a given condition and provide > useful messages in case of mismatch and retry. For the price of an > increased complexity. On the other hand, the test should be much more > resilient to timing errors caused by slow setups. > > Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 Certainly increased complexity - took me quite a while to figure out what it all meant. :) waitForCondition should be waitTillEqual as that is the only condition checked (unless you'd like to make it more complex and pass in a Predicate ;-) ). I think the Thread.yield would be better as a short sleep Thanks, David ----- > Thanks, > > -JB- From peter.levart at gmail.com Thu May 14 07:50:18 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 14 May 2015 09:50:18 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5553B552.9060900@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> Message-ID: <555453BA.2070205@gmail.com> Hi Derek, On 05/13/2015 10:34 PM, Derek White wrote: > Hi Peter, > > I don't have smoking gun evidence that your change introduces a bug, > but I have some concerns... I did have a concern too, ... > > On 5/12/15 6:05 PM, Peter Levart wrote: >> Hi Dmitry, >> >> You iterate the queue then, not the unfinalized list. That's more >> logical. >> >> Holding the queue's lock may pause reference handler and finalizer >> threads for the entire time of iteration. This can blow up the >> application. Suppose one wants to diagnose the application because he >> suspects that finalizer thread hardly keeps up with production of >> finalizable instances. This can happen if the allocation rate of >> finalizable objects is very high and/or finalize() methods are not >> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >> gradually and is already holding several million objects. Now you >> initiate the diagnostic command to print the queue. The iteration >> over and grouping/counting of several millions of Finalizer(s) takes >> some time. This blocks finalizer thread and reference handler thread. >> But does not block the threads that allocate finalizable objects. By >> the time the iteration is over, the JVM blows up and application >> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >> >> It is possible to iterate the elements of the queue for diagnostic >> purposes without holding a lock. The modification required to do that >> is the following (havent tested this, but I think it should work): >> >> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >> > One issue to watch out for is the garbage collectors inspect the > Reference.next field from C code directly (to distinguish active vs. > pending, iterate over oops, etc.). You can search hotspot for > java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. > > Your change makes "inactive" References superficially look like > "enqueued" References. The GC code is rather subtle and I haven't yet > seen a case where it would get confused by this change, but there > could easily be something like that lurking in the GC code. ...but then I thought that GC can in no way treat a Reference differently whether it is still enqueued in a ReferenceQueue or already dequeued (inactive) - responsibility is already on the Java side. Currently the definition of Reference.next is this: /* When active: NULL * pending: this * Enqueued: next reference in queue (or this if last) * Inactive: this */ @SuppressWarnings("rawtypes") Reference next; We see that, unless GC inspects all ReferenceQueue instances and scans their lists too, the state of a Reference that is enqueued as last in list is indistinguishable from the state of inactive Reference. So I deduced that this distinction (enqueued/inactive) can't be established solely on the value of .next field ( == this or != this)... But I should inspect the GC code too to build a better understanding of that part of the story... Anyway. It turns out that there is already enough state in Reference to destinguish between it being enqueued as last in list and already dequeued (inactive) - the additional state is Reference.queue that transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in ReferenceQueue.reallyPoll. We need to just make sure the two fields (r.next and r.queue) are assigned and read in correct order. This can be achieved either by making Reference.next a volatile field or by a couple of explicit fences: http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ The assignment of r.queue to ReferenceQueue.ENQUEUED already happens before linking the reference into the queue's head in ReferenceQueue.enqueue(): r.queue = ENQUEUED; r.next = (head == null) ? r : head; head = r; Both stores are volatile. > >> I also suggest the addition to the ReferenceQueue to be contained >> (package-private) and as generic as possible. That's why I suggest >> forEach iteration method with no concrete logic. >> >> It would be possible to encapsulate the entire logic into a special >> package-private class (say java.lang.ref.DiagnosticCommands) with >> static method(s) (printFinalizationQueue)... You could just expose a >> package-private forEach static method from Finalizer and code the >> rest in DiagnosticCommands. > That's good for encapsulation. But I'm concerned that if "forEach" got > exposed beyond careful use in DiagnosticCommands, and the Referents > were leaked back into the heap, then we could get unexpected object > resurrection after finalization. This isn't a bug on it's own - any > finalizer might resurrect its object if not careful, but ordinarily > /only/ the finalizer could resurrect the object. This could invalidate > application invariants? Well, it all stays in the confines of package-private API - internal to JDK. Any future additional use should be reviewed carefully. Comments on the forEach() method should warn about that. > > I agree that using a lock over the ReferenceQueue iteration opens up > /another/ case of diagnostics affecting application behavior. And > there are pathological scenarios where this gets severe. This is > unfortunate but not uncommon. There is enough complication here that > you should be sure that the fix for diagnostics performance doesn't > introduce subtle bugs to the system in general. A change in this area > should get a thorough review from both the runtime and GC sides. Is the webrev.02 proposed above more acceptable in that respect? It does not introduce any logical changes to existing code. > > Better yet, the reference handling code in GC and runtime should > probably be thrown out and re-written, but that's another issue :-) I may have some proposals in that direction. Stay tuned. Regards, Peter > > - Derek > >> Regards, Peter >> >> >> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>> Everybody, >>> >>> Updated version: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>> >>> Now it iterates over queue and output result sorted by number of instances. >>> >>> -Dmitry >>> >>> On 2015-05-07 00:51, Derek White wrote: >>>> Hi Dmitry, Staffan, >>>> >>>> Lots of good comments here. >>>> >>>> On the topic of what list should be printed out, I think we should focus >>>> on objects waiting to be finalized - e.g. the contents of the >>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>> could add a summerizeQueue(TreeMap) method, or a >>>> general iterator and lambda. >>>> >>>> A histogram of objects with finalize methods might also be interesting, >>>> but you can get most of the same information from a heap histogram >>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>> >>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>> FinalizerThread from making progress (and I think there's a GC thread >>>> that runs after GC that sorts found References objects that need >>>> processing into their respective ReferenceQueues). But locking the >>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>> >>>> The sorting suggestion is a nice touch. >>>> >>>> - Derek White, GC team >>>> >>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>> Hi Dmitry, Staffan, >>>>> >>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>> Dmitry, >>>>>> >>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>> well considering the changes to Finalizer. >>>>>> >>>>>> I?m a little worried about the potentially very large String that is >>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>> That would allow for outputting one line at a time. The output would >>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>> the data is only saved once in memory, then. It would make the code a >>>>>> bit harder to write, so its a question of how scared we are of >>>>>> running out of memory. >>>>> If the output is just a histogram of # of instances per class name, >>>>> then it should not be too large, as there are not many different >>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>> finalize() method in the whole jdk/src tree). >>>>> >>>>> I'm more concerned about the fact that while traversing the list, a >>>>> lock is held, which might impact prompt finalization(). Is it >>>>> acceptable for diagnostic output to impact performance and/or >>>>> interfere with synchronization? >>>>> >>>>> It might be possible to scan the list without holding a lock for >>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>> Finalizer.next pointer to point back to itself: >>>>> >>>>> private void remove() { >>>>> synchronized (lock) { >>>>> if (unfinalized == this) { >>>>> if (this.next != null) { >>>>> unfinalized = this.next; >>>>> } else { >>>>> unfinalized = this.prev; >>>>> } >>>>> } >>>>> if (this.next != null) { >>>>> this.next.prev = this.prev; >>>>> } >>>>> if (this.prev != null) { >>>>> this.prev.next = this.next; >>>>> } >>>>> // this.next = this; must not be set so that we can >>>>> traverse the list unsynchronized >>>>> this.prev = this; /* Indicates that this has been >>>>> finalized */ >>>>> } >>>>> } >>>>> >>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>> pointer could be used instead: >>>>> >>>>> private boolean hasBeenFinalized() { >>>>> return (prev == this); >>>>> } >>>>> >>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>> initialized Finalizer instance (in particular the next pointer), you >>>>> would have to make the 'unfinalized' field volatile or insert an >>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>> >>>>> private void add() { >>>>> synchronized (lock) { >>>>> if (unfinalized != null) { >>>>> this.next = unfinalized; >>>>> unfinalized.prev = this; >>>>> } >>>>> // make sure a data race dereferencing 'unfinalized' >>>>> // in printFinalizationQueue() does see the Finalizer >>>>> // instance fully initialized >>>>> // (in particular the 'next' pointer) >>>>> U.storeFence(); >>>>> unfinalized = this; >>>>> } >>>>> } >>>>> >>>>> >>>>> By doing these modifications, I think you can remove >>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>> >>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>> out? >>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>> instances pending processing by finalizer thread because their >>>>> referents are eligible for finalization (they are not reachable any >>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>> instances for which their referents have not been finalized yet >>>>> (including those that are still reachable and alive). The later serves >>>>> two purposes: >>>>> - it keeps Finalizer instances reachable until they are processed >>>>> - it is a source of unfinalized instances for >>>>> running-finalizers-on-exit if requested by >>>>> System.runFinalizersOnExit(true); >>>>> >>>>> So it really depends on what one would like to see. Showing the queue >>>>> may be interesting if one wants to see how the finalizer thread is >>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>> list may be interesting if one wants to know how many live + >>>>> finalization pending instances are there on the heap that override >>>>> finalize() method. >>>>> >>>>> Regards, Peter >>>>> >>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>> references for each type. Perhaps outputting it more like a table >>>>>> (see the old code again) would also make it easier to digest. >>>>>> >>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>> permission() and limit access: >>>>>> >>>>>> static const JavaPermission permission() { >>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>> "monitor", NULL}; >>>>>> return p; >>>>>> } >>>>>> >>>>>> The two tests don?t validate the output in any way. Would it be >>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>> the finalizer queue? Hmm. >>>>>> >>>>>> Thanks, >>>>>> /Staffan >>>>>> >>>>>> >>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>> wrote: >>>>>>> >>>>>>> Everyone, >>>>>>> >>>>>>> Please review the fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>> >>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>> >>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>> with count >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> -- >>>>>>> Dmitry Samersoff >>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>> * I would love to change the world, but they won't give me the sources. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Thu May 14 08:35:58 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 14 May 2015 10:35:58 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5553C7A1.8060004@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <5553C7A1.8060004@oracle.com> Message-ID: <55545E6E.9030304@gmail.com> I also have some comments on Dmitry's code... On 05/13/2015 11:52 PM, Derek White wrote: > Hi Dmitry, > > Some review comments below... > > On 5/12/15 1:10 PM, Dmitry Samersoff wrote: >> Everybody, >> >> Updated version: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >> >> Now it iterates over queue and output result sorted by number of >> instances. > FinalReference.java > - Update copyright to 2015. > > ReferenceQueue.java > - Update copyright to 2015. > > class "mutableInt": > - Should be "MutableInt" > - Not a collections lawyer, but should MutableInt implement Comparable? An alternative is a one-element int[] and the need for additional MutableInt class is eliminated. If you must have it, then perhaps you could make it Comparable and contain an additional String className field. Why? You could then dump the values() of the Map into an array and sort the array without the need for custom Comparator and then iterate the sorted array to build a message. > > countInstances(): > - Javadoc should mention may return null. > - Can countInstances() be package-private? It must be package-private, since ReferenceQueue is public API. Regards, Peter > - After you get the lock in line 138, you should recheck for "head == > null", and return null if so. Otherwise it might sometimes return null > and sometimes return an empty map. > - BIG: Is loop missing? Should "if" at line 140 be "while"? > - Merge lines 147, 148: "} else {" > - Check for consistent line spacing. > > - Derek >> >> -Dmitry >> >> On 2015-05-07 00:51, Derek White wrote: >>> Hi Dmitry, Staffan, >>> >>> Lots of good comments here. >>> >>> On the topic of what list should be printed out, I think we should >>> focus >>> on objects waiting to be finalized - e.g. the contents of the >>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>> could add a summerizeQueue(TreeMap) method, or a >>> general iterator and lambda. >>> >>> A histogram of objects with finalize methods might also be interesting, >>> but you can get most of the same information from a heap histogram >>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>> >>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>> FinalizerThread from making progress (and I think there's a GC thread >>> that runs after GC that sorts found References objects that need >>> processing into their respective ReferenceQueues). But locking the >>> "unfinalized" list blocks initializing any object with a finalize() >>> method. >>> >>> The sorting suggestion is a nice touch. >>> >>> - Derek White, GC team >>> >>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>> Hi Dmitry, Staffan, >>>> >>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>> Dmitry, >>>>> >>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>> well considering the changes to Finalizer. >>>>> >>>>> I?m a little worried about the potentially very large String that is >>>>> returned from printFinalizationQueue(). A possible different approach >>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>> That would allow for outputting one line at a time. The output would >>>>> still be saved in memory (since the stream is buffered), but at least >>>>> the data is only saved once in memory, then. It would make the code a >>>>> bit harder to write, so its a question of how scared we are of >>>>> running out of memory. >>>> If the output is just a histogram of # of instances per class name, >>>> then it should not be too large, as there are not many different >>>> classes overriding finalize() method (I counted 72 overriddings of >>>> finalize() method in the whole jdk/src tree). >>>> >>>> I'm more concerned about the fact that while traversing the list, a >>>> lock is held, which might impact prompt finalization(). Is it >>>> acceptable for diagnostic output to impact performance and/or >>>> interfere with synchronization? >>>> >>>> It might be possible to scan the list without holding a lock for >>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>> Finalizer.next pointer to point back to itself: >>>> >>>> private void remove() { >>>> synchronized (lock) { >>>> if (unfinalized == this) { >>>> if (this.next != null) { >>>> unfinalized = this.next; >>>> } else { >>>> unfinalized = this.prev; >>>> } >>>> } >>>> if (this.next != null) { >>>> this.next.prev = this.prev; >>>> } >>>> if (this.prev != null) { >>>> this.prev.next = this.next; >>>> } >>>> // this.next = this; must not be set so that we can >>>> traverse the list unsynchronized >>>> this.prev = this; /* Indicates that this has been >>>> finalized */ >>>> } >>>> } >>>> >>>> For detecting whether a Finalizer is already processed, the 'prev' >>>> pointer could be used instead: >>>> >>>> private boolean hasBeenFinalized() { >>>> return (prev == this); >>>> } >>>> >>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>> unsynchronized printFinalizationQueue() would get you a fully >>>> initialized Finalizer instance (in particular the next pointer), you >>>> would have to make the 'unfinalized' field volatile or insert an >>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>> >>>> private void add() { >>>> synchronized (lock) { >>>> if (unfinalized != null) { >>>> this.next = unfinalized; >>>> unfinalized.prev = this; >>>> } >>>> // make sure a data race dereferencing 'unfinalized' >>>> // in printFinalizationQueue() does see the Finalizer >>>> // instance fully initialized >>>> // (in particular the 'next' pointer) >>>> U.storeFence(); >>>> unfinalized = this; >>>> } >>>> } >>>> >>>> >>>> By doing these modifications, I think you can remove >>>> synchronized(lock) {} from printFinalizationQueue(). >>>> >>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>> not sure of the difference, perhaps someone from the GC team can help >>>>> out? >>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>> instances pending processing by finalizer thread because their >>>> referents are eligible for finalization (they are not reachable any >>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>> instances for which their referents have not been finalized yet >>>> (including those that are still reachable and alive). The later serves >>>> two purposes: >>>> - it keeps Finalizer instances reachable until they are processed >>>> - it is a source of unfinalized instances for >>>> running-finalizers-on-exit if requested by >>>> System.runFinalizersOnExit(true); >>>> >>>> So it really depends on what one would like to see. Showing the queue >>>> may be interesting if one wants to see how the finalizer thread is >>>> coping with processing the finalize() invocations. Showing unfinalized >>>> list may be interesting if one wants to know how many live + >>>> finalization pending instances are there on the heap that override >>>> finalize() method. >>>> >>>> Regards, Peter >>>> >>>>> For the output, it would be a nice touch to sort it on the number of >>>>> references for each type. Perhaps outputting it more like a table >>>>> (see the old code again) would also make it easier to digest. >>>>> >>>>> In diagnosticCommand.hpp, the new commands should override >>>>> permission() and limit access: >>>>> >>>>> static const JavaPermission permission() { >>>>> JavaPermission p = >>>>> {"java.lang.management.ManagementPermission", >>>>> "monitor", NULL}; >>>>> return p; >>>>> } >>>>> >>>>> The two tests don?t validate the output in any way. Would it be >>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>> the finalizer queue? Hmm. >>>>> >>>>> Thanks, >>>>> /Staffan >>>>> >>>>> >>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>> wrote: >>>>>> >>>>>> Everyone, >>>>>> >>>>>> Please review the fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>> >>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>> >>>>>> finalizerinfo dcmd outputs a list of all classes in finalization >>>>>> queue >>>>>> with count >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> -- >>>>>> Dmitry Samersoff >>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>> * I would love to change the world, but they won't give me the >>>>>> sources. >> > From cheleswer.sahu at oracle.com Thu May 14 09:26:32 2015 From: cheleswer.sahu at oracle.com (cheleswer sahu) Date: Thu, 14 May 2015 14:56:32 +0530 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <55546325.2090304@oracle.com> References: <55546325.2090304@oracle.com> Message-ID: <55546A48.6050108@oracle.com> Hi, Please review the code changes for https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and tested JDK9 with fix successfully. As I do not have an account for OpenJDK, David Buck will push the fix into jdk9/hs-rt/. Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ Regards, Cheleswer From cheleswer.sahu at oracle.com Thu May 14 09:34:28 2015 From: cheleswer.sahu at oracle.com (cheleswer sahu) Date: Thu, 14 May 2015 15:04:28 +0530 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <55546A48.6050108@oracle.com> References: <55546A48.6050108@oracle.com> Message-ID: <55546C24.6050701@oracle.com> Hi, Please review the code changes for https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and tested JDK9 with fix successfully. As I do not have an account for OpenJDK, David Buck will push the fix into jdk9/hs-rt/. Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ Regards, Cheleswer From dmitry.samersoff at oracle.com Thu May 14 10:06:40 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 14 May 2015 13:06:40 +0300 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <55546A48.6050108@oracle.com> References: <55546325.2090304@oracle.com> <55546A48.6050108@oracle.com> Message-ID: <555473B0.20807@oracle.com> Cheleswer, I would recommend to simple change ll. 222 of original code to uid_t euid = geteuid(); if (euid != 0 && statp->st_uid != euid) { return false; } -Dmitry On 2015-05-14 12:26, cheleswer sahu wrote: > Hi, > Please review the code changes for > https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and > tested JDK9 with fix successfully. As I do not have an account for OpenJDK, > David Buck will push the fix into jdk9/hs-rt/. > > Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ > > Regards, > Cheleswer > > > > > > > > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jaroslav.bachorik at oracle.com Thu May 14 10:08:27 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 14 May 2015 12:08:27 +0200 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <55540110.3050008@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <55540110.3050008@oracle.com> Message-ID: <5554741B.3040000@oracle.com> On 14.5.2015 03:57, David Holmes wrote: > On 13/05/2015 9:46 PM, Jaroslav Bachorik wrote: >> On 1.5.2015 21:55, Martin Buchholz wrote: >>> >>> >>> On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik >>> > >>> wrote: >>> >>> On 30.4.2015 19:18, Martin Buchholz wrote: >>> >>> Tests that sleep can almost always be better written some >>> other way. >>> In this case, I would prefer busy-waiting with timeout until the >>> expected condition becomes true. >>> >>> >>> The thing is that in case of a real issue with the thread >>> counters we >>> a/ would be busy-waiting till the test times out (using an arbitrary >>> delay is also problematic due to different performance of different >>> machines running with different configurations) >>> >>> >>> Far less problematic (performance-wise and reliability-wise) than the >>> fixed sleep. >>> >>> b/ would get a rather confusing message about the test timing out at >>> the end >>> >>> >>> You can easily improve the error message. >> >> Well, not that easily. It is not possible to get a notification when >> JTREG decides to timeout the test. So you will get the standard JTREG >> message and that's all. >> >> I was able to modify the test to wait for a given condition and provide >> useful messages in case of mismatch and retry. For the price of an >> increased complexity. On the other hand, the test should be much more >> resilient to timing errors caused by slow setups. >> >> Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 > > Certainly increased complexity - took me quite a while to figure out > what it all meant. :) waitForCondition should be waitTillEqual as that > is the only condition checked (unless you'd like to make it more complex > and pass in a Predicate ;-) ). I started with Predicate and then realized it brought more problems than benefits ... and didn't rename the method afterwords. 'waitTillEquals' sounds good. > > I think the Thread.yield would be better as a short sleep You mean not to use Thread.yield() and do Thread.sleep(1) instead? I've moved the logical blocks into separate methods to make the test more explicit about the steps it takes. Incidentally, this made the webrev also more readable ;) Update: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.02 -JB- > > Thanks, > David > ----- > > >> Thanks, >> >> -JB- From jaroslav.bachorik at oracle.com Thu May 14 10:11:12 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 14 May 2015 12:11:12 +0200 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <555392EB.8020506@oracle.com> Message-ID: <555474C0.30108@oracle.com> On 13.5.2015 21:14, Martin Buchholz wrote: > David has given you an approval; feel free to ignore me! > > I tried running the test against jdk9, but it timed out! ... and it did print nice messages, didn't it? ;) I managed to leave in a piece of code I used for testing the error messages. Sorry about that. > > It's common for users to introduce parallelism in classloaders or > jar-loaders (we do this at google) which may cause arbitrary thread > fluctuations. That makes this particular API rather difficult to test > robustly, especially if you are only testing against the spec. Well, in such an environment this test makes absolutely no sense - there is nothing deterministic in the thread counters we could assert against. -JB- > > On Wed, May 13, 2015 at 11:07 AM, Jaroslav Bachorik > > wrote: > > On 13.5.2015 19:40, Martin Buchholz wrote: > > toString()should never return null, I think. > > > It doesn't matter much here. The test would fail with an NPE and it > would be right to do so. None of the suppliers should ever return null. > > > 52 @Override > 53 public String toString() { > 54 T resolved = val.get(); > 55 return resolved != null ? resolved.toString() > : null; > 56 } > > > I expected methods like waitForCondition to include a timeout with > failure. I like 10 seconds, being large enough to never be hit > spuriously in tests. > > > It's difficult to find a value 'large enough'. Imagine the test > running on a small embedded device and fastdebug build. I had my fun > fixing tests failing intermittently because it was thought that the > original timeout was large enough. I better leave it to the harness. > > > Why not > () -> (long) mbean.getThreadCount(), > > > Because curLiveThreadCount needs to be set to mbean.getThreadCount() > value. > > > 169 ()->{ > 170 curLiveThreadCount = mbean.getThreadCount(); > 171 return (long)curLiveThreadCount; > 172 }, > > > I worry that > mbean.getThreadCount() > is hard to test since the "system" may spin up and shut down utility > threads at any time. > > > The 'system' threads are not reported by this method. And the > current understanding is that once VM is fully initialized no > user-observable threads are randomly started on behalf of the system. > > -JB- > > > On Wed, May 13, 2015 at 4:46 AM, Jaroslav Bachorik > > >> wrote: > > On 1.5.2015 21:55, Martin Buchholz wrote: > > > > On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik > > > > > > >>> wrote: > > On 30.4.2015 19:18, Martin Buchholz wrote: > > Tests that sleep can almost always be better > written > some other way. > In this case, I would prefer busy-waiting with > timeout > until the > expected condition becomes true. > > > The thing is that in case of a real issue with the > thread > counters we > a/ would be busy-waiting till the test times out > (using an > arbitrary > delay is also problematic due to different > performance of > different > machines running with different configurations) > > > Far less problematic (performance-wise and > reliability-wise) > than the > fixed sleep. > > b/ would get a rather confusing message about the test > timing out at > the end > > > You can easily improve the error message. > > > Well, not that easily. It is not possible to get a > notification when > JTREG decides to timeout the test. So you will get the standard > JTREG message and that's all. > > I was able to modify the test to wait for a given condition and > provide useful messages in case of mismatch and retry. For > the price > of an increased complexity. On the other hand, the test > should be > much more resilient to timing errors caused by slow setups. > > Updated webrev: > http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 > > Thanks, > > -JB- > > > > From daniel.daugherty at oracle.com Thu May 14 16:00:11 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 14 May 2015 10:00:11 -0600 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <55546C24.6050701@oracle.com> References: <55546A48.6050108@oracle.com> <55546C24.6050701@oracle.com> Message-ID: <5554C68B.5090702@oracle.com> On 5/14/15 3:34 AM, cheleswer sahu wrote: > Hi, > Please review the code changes for > https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and > tested JDK9 with fix successfully. As I do not have an account for > OpenJDK, > David Buck will push the fix into jdk9/hs-rt/. > > Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ > > Regards, > Cheleswer Cheleswer, Sorry for the lengthy review, but since this is security related, I have to be complete. The goal: Add a policy by-pass for the 'root' user in order to fix the regression in jps(1) behavior. The core of this policy by-pass is the change to this function: 205 static bool is_statbuf_secure(struct stat *statp, int mode) { 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { 207 // The path represents a link or some non-directory file type, 208 // which is not what we expected. Declare it insecure. 209 // 210 return false; 211 } 212 // If the directory is going to be opened readonly, we consider this as secure operation 213 // we do not need to do any more checks. 214 // 215 if ((mode & O_ACCMODE) == O_RDONLY) { 216 return true; 217 } 218 // We have an existing directory, check if the permissions are safe. 219 // 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { 221 // The directory is open for writing and could be subjected 222 // to a symlink or a hard link attack. Declare it insecure. 223 // 224 return false; 225 } 226 // See if the uid of the directory matches the effective uid of the process. 227 // 228 if (statp->st_uid != geteuid()) { 229 // The directory was not created by this user, declare it insecure. 230 // 231 return false; 232 } 233 return true; 234 } Lines 212-217 are added which allows a caller that passes in O_RDONLY to by-pass the security checks on lines 220-225 and 228-232. This implementation is using an attribute of _how_ the data is accessed to override security policy instead of an attribute of _who_ is accessing the data. Here are the code paths that access the modified policy code: is_statbuf_secure() is called by: - is_directory_secure() - is_dirfd_secure() is_directory_secure() is called by: - get_user_name_slow() with O_RDONLY - make_user_tmp_dir() with O_RDWR - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) is_dirfd_secure() is called by: - open_directory_secure() with a mode parameter open_directory_secure() is called by: - open_directory_secure_cwd() with O_RDWR - get_user_name_slow() with O_RDONLY Only the code paths that specify O_RDWR make use of the new policy by-pass code so it looks like only - get_user_name_slow() with O_RDONLY - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) are interesting. The new security policy by-pass will allow get_user_name_slow(): - to process directory entries in a directory that is writable which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that the calling user does not own; the intent of the policy by-pass is to allow this for the 'root' user, but this implementation allows the by-pass for any user. It looks like the get_user_name_slow() code is written safely enough such that any symlink or hard link attack should not cause any issues. The new policy by-pass will allow any user to determine the user name associated with VMs owned by another user. This is a broader policy by-pass than was intended. The new security policy by-pass will allow mmap_attach_shared(): - to process directory entries in a directory that is writable which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that the calling user does not own; the intent of the policy by-pass is to allow this for the 'root' user, but this implementation allows the by-pass for any user. The mmap_attach_shared() code protects itself from a symlink attack by including the 'O_NOFOLLOW' flag when opening the PerfData file and it protects itself from a hardlink attack by checking the hard link count after opening the file. It does not protect itself against being handed a corrupted file or even a very large file that would cause an OOM when the VM tries to map what is supposed to be a PerfData file. The new policy by-pass will allow any user to access the PerfData file associated with VMs owned by another user. This is a broader policy by-pass than was intended. Summary: This implementation of the new security policy by-pass is using an attribute of _how_ the data is accessed to override security policy instead of an attribute of _who_ is accessing the data. This allows the VM to be exposed to some of the attacks that the following fix was designed to prevent: JDK-8050807 Better performing performance data handling Dmitry's response to the code review provides a solution that is based on who is accessing the data and that solution or one like it should be considered. Again, sorry for the lengthy review. Dan From ysr1729 at gmail.com Thu May 14 16:12:54 2015 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Thu, 14 May 2015 09:12:54 -0700 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: References: Message-ID: https://bugs.openjdk.java.net/browse/JDK-8080345 On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna wrote: > > With perm gen going away (and being replaced by metaspace) in JDK 8, it > makes sense that the counter > sun.gc.policy.generations should be "2", rather than "3". However, in JDK > 8 that counter still says 3. > As I understand, the intention was that this counter would allow you to > (for example) know the range of > the sun.gc.generation.$num.* counters describing each of $num < > sun.gc.policy.generations in the heap. > Recall that the erstwhile perm gen in JDK 7 used to be synonymous with > sun.gc.generation.2, but the > JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. > > The fix is simple, and I can submit a patch. Is there an existing bug for > this? > > thanks! > -- ramki > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerald.thornbrugh at oracle.com Thu May 14 16:16:53 2015 From: gerald.thornbrugh at oracle.com (Gerald Thornbrugh) Date: Thu, 14 May 2015 09:16:53 -0700 (PDT) Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 Message-ID: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> Hi Dan, When Cheleswer and I discussed this fix my interpretation had a slightly different goal: Prior to the initial security fix any user could execute "jps" and get the user names associated with other user's perf data (i.e. the call to get_user_name_slow() would succeed.). My initial thought was that this was a regression for all users not just "root" and this goal led to this fix. At the time I did not see this as a security vulnerability, your review has changed my mind. I agree that Dmitry's fix is a more secure fix for the issue and I think we should use it. Let me know if you have any questions. Thanks! Jerry > Hi, > Please review the code changes for > https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and > tested JDK9 with fix successfully. As I do not have an account for > OpenJDK, > David Buck will push the fix into jdk9/hs-rt/. > > Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ > > Regards, > Cheleswer Cheleswer, Sorry for the lengthy review, but since this is security related, I have to be complete. The goal: Add a policy by-pass for the 'root' user in order to fix the regression in jps(1) behavior. The core of this policy by-pass is the change to this function: 205 static bool is_statbuf_secure(struct stat *statp, int mode) { 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { 207 // The path represents a link or some non-directory file type, 208 // which is not what we expected. Declare it insecure. 209 // 210 return false; 211 } 212 // If the directory is going to be opened readonly, we consider this as secure operation 213 // we do not need to do any more checks. 214 // 215 if ((mode & O_ACCMODE) == O_RDONLY) { 216 return true; 217 } 218 // We have an existing directory, check if the permissions are safe. 219 // 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { 221 // The directory is open for writing and could be subjected 222 // to a symlink or a hard link attack. Declare it insecure. 223 // 224 return false; 225 } 226 // See if the uid of the directory matches the effective uid of the process. 227 // 228 if (statp->st_uid != geteuid()) { 229 // The directory was not created by this user, declare it insecure. 230 // 231 return false; 232 } 233 return true; 234 } Lines 212-217 are added which allows a caller that passes in O_RDONLY to by-pass the security checks on lines 220-225 and 228-232. This implementation is using an attribute of _how_ the data is accessed to override security policy instead of an attribute of _who_ is accessing the data. Here are the code paths that access the modified policy code: is_statbuf_secure() is called by: - is_directory_secure() - is_dirfd_secure() is_directory_secure() is called by: - get_user_name_slow() with O_RDONLY - make_user_tmp_dir() with O_RDWR - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) is_dirfd_secure() is called by: - open_directory_secure() with a mode parameter open_directory_secure() is called by: - open_directory_secure_cwd() with O_RDWR - get_user_name_slow() with O_RDONLY Only the code paths that specify O_RDWR make use of the new policy by-pass code so it looks like only - get_user_name_slow() with O_RDONLY - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) are interesting. The new security policy by-pass will allow get_user_name_slow(): - to process directory entries in a directory that is writable which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that the calling user does not own; the intent of the policy by-pass is to allow this for the 'root' user, but this implementation allows the by-pass for any user. It looks like the get_user_name_slow() code is written safely enough such that any symlink or hard link attack should not cause any issues. The new policy by-pass will allow any user to determine the user name associated with VMs owned by another user. This is a broader policy by-pass than was intended. The new security policy by-pass will allow mmap_attach_shared(): - to process directory entries in a directory that is writable which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that the calling user does not own; the intent of the policy by-pass is to allow this for the 'root' user, but this implementation allows the by-pass for any user. The mmap_attach_shared() code protects itself from a symlink attack by including the 'O_NOFOLLOW' flag when opening the PerfData file and it protects itself from a hardlink attack by checking the hard link count after opening the file. It does not protect itself against being handed a corrupted file or even a very large file that would cause an OOM when the VM tries to map what is supposed to be a PerfData file. The new policy by-pass will allow any user to access the PerfData file associated with VMs owned by another user. This is a broader policy by-pass than was intended. Summary: This implementation of the new security policy by-pass is using an attribute of _how_ the data is accessed to override security policy instead of an attribute of _who_ is accessing the data. This allows the VM to be exposed to some of the attacks that the following fix was designed to prevent: JDK-8050807 Better performing performance data handling Dmitry's response to the code review provides a solution that is based on who is accessing the data and that solution or one like it should be considered. Again, sorry for the lengthy review. Dan From dmitry.samersoff at oracle.com Thu May 14 17:41:53 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 14 May 2015 20:41:53 +0300 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555453BA.2070205@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> Message-ID: <5554DE61.5010403@oracle.com> Peter, Could we just bail out on r == r.next? It gives us less accurate result, but I suspect that If we restart from head we need to flush all counters. As far I understand queue poller removes items one by one from a queue end so if we overtaken by queue poller we can safely assume that we are at the end of the queue. Is it correct? -Dmitry On 2015-05-14 10:50, Peter Levart wrote: > Hi Derek, > > On 05/13/2015 10:34 PM, Derek White wrote: >> Hi Peter, >> >> I don't have smoking gun evidence that your change introduces a bug, >> but I have some concerns... > > I did have a concern too, ... > >> >> On 5/12/15 6:05 PM, Peter Levart wrote: >>> Hi Dmitry, >>> >>> You iterate the queue then, not the unfinalized list. That's more >>> logical. >>> >>> Holding the queue's lock may pause reference handler and finalizer >>> threads for the entire time of iteration. This can blow up the >>> application. Suppose one wants to diagnose the application because he >>> suspects that finalizer thread hardly keeps up with production of >>> finalizable instances. This can happen if the allocation rate of >>> finalizable objects is very high and/or finalize() methods are not >>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>> gradually and is already holding several million objects. Now you >>> initiate the diagnostic command to print the queue. The iteration >>> over and grouping/counting of several millions of Finalizer(s) takes >>> some time. This blocks finalizer thread and reference handler thread. >>> But does not block the threads that allocate finalizable objects. By >>> the time the iteration is over, the JVM blows up and application >>> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >>> >>> It is possible to iterate the elements of the queue for diagnostic >>> purposes without holding a lock. The modification required to do that >>> is the following (havent tested this, but I think it should work): >>> >>> >>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>> >> One issue to watch out for is the garbage collectors inspect the >> Reference.next field from C code directly (to distinguish active vs. >> pending, iterate over oops, etc.). You can search hotspot for >> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >> >> Your change makes "inactive" References superficially look like >> "enqueued" References. The GC code is rather subtle and I haven't yet >> seen a case where it would get confused by this change, but there >> could easily be something like that lurking in the GC code. > > ...but then I thought that GC can in no way treat a Reference > differently whether it is still enqueued in a ReferenceQueue or already > dequeued (inactive) - responsibility is already on the Java side. > Currently the definition of Reference.next is this: > > /* When active: NULL > * pending: this > * Enqueued: next reference in queue (or this if last) > * Inactive: this > */ > @SuppressWarnings("rawtypes") > Reference next; > > We see that, unless GC inspects all ReferenceQueue instances and scans > their lists too, the state of a Reference that is enqueued as last in > list is indistinguishable from the state of inactive Reference. So I > deduced that this distinction (enqueued/inactive) can't be established > solely on the value of .next field ( == this or != this)... > > But I should inspect the GC code too to build a better understanding of > that part of the story... > > Anyway. It turns out that there is already enough state in Reference to > destinguish between it being enqueued as last in list and already > dequeued (inactive) - the additional state is Reference.queue that > transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in > ReferenceQueue.reallyPoll. We need to just make sure the two fields > (r.next and r.queue) are assigned and read in correct order. This can be > achieved either by making Reference.next a volatile field or by a couple > of explicit fences: > > > http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ > > The assignment of r.queue to ReferenceQueue.ENQUEUED already happens > before linking the reference into the queue's head in > ReferenceQueue.enqueue(): > > r.queue = ENQUEUED; > r.next = (head == null) ? r : head; > head = r; > > Both stores are volatile. > >> >>> I also suggest the addition to the ReferenceQueue to be contained >>> (package-private) and as generic as possible. That's why I suggest >>> forEach iteration method with no concrete logic. >>> >>> It would be possible to encapsulate the entire logic into a special >>> package-private class (say java.lang.ref.DiagnosticCommands) with >>> static method(s) (printFinalizationQueue)... You could just expose a >>> package-private forEach static method from Finalizer and code the >>> rest in DiagnosticCommands. >> That's good for encapsulation. But I'm concerned that if "forEach" got >> exposed beyond careful use in DiagnosticCommands, and the Referents >> were leaked back into the heap, then we could get unexpected object >> resurrection after finalization. This isn't a bug on it's own - any >> finalizer might resurrect its object if not careful, but ordinarily >> /only/ the finalizer could resurrect the object. This could invalidate >> application invariants? > > Well, it all stays in the confines of package-private API - internal to > JDK. Any future additional use should be reviewed carefully. Comments on > the forEach() method should warn about that. > >> >> I agree that using a lock over the ReferenceQueue iteration opens up >> /another/ case of diagnostics affecting application behavior. And >> there are pathological scenarios where this gets severe. This is >> unfortunate but not uncommon. There is enough complication here that >> you should be sure that the fix for diagnostics performance doesn't >> introduce subtle bugs to the system in general. A change in this area >> should get a thorough review from both the runtime and GC sides. > > Is the webrev.02 proposed above more acceptable in that respect? It does > not introduce any logical changes to existing code. > >> >> Better yet, the reference handling code in GC and runtime should >> probably be thrown out and re-written, but that's another issue :-) > > I may have some proposals in that direction. Stay tuned. > > Regards, Peter > >> >> - Derek >> >>> Regards, Peter >>> >>> >>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>> Everybody, >>>> >>>> Updated version: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>> >>>> Now it iterates over queue and output result sorted by number of instances. >>>> >>>> -Dmitry >>>> >>>> On 2015-05-07 00:51, Derek White wrote: >>>>> Hi Dmitry, Staffan, >>>>> >>>>> Lots of good comments here. >>>>> >>>>> On the topic of what list should be printed out, I think we should focus >>>>> on objects waiting to be finalized - e.g. the contents of the >>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>> could add a summerizeQueue(TreeMap) method, or a >>>>> general iterator and lambda. >>>>> >>>>> A histogram of objects with finalize methods might also be interesting, >>>>> but you can get most of the same information from a heap histogram >>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>> >>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>> that runs after GC that sorts found References objects that need >>>>> processing into their respective ReferenceQueues). But locking the >>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>> >>>>> The sorting suggestion is a nice touch. >>>>> >>>>> - Derek White, GC team >>>>> >>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>> Hi Dmitry, Staffan, >>>>>> >>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>> Dmitry, >>>>>>> >>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>> well considering the changes to Finalizer. >>>>>>> >>>>>>> I?m a little worried about the potentially very large String that is >>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>> That would allow for outputting one line at a time. The output would >>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>> running out of memory. >>>>>> If the output is just a histogram of # of instances per class name, >>>>>> then it should not be too large, as there are not many different >>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>> finalize() method in the whole jdk/src tree). >>>>>> >>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>> acceptable for diagnostic output to impact performance and/or >>>>>> interfere with synchronization? >>>>>> >>>>>> It might be possible to scan the list without holding a lock for >>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>> Finalizer.next pointer to point back to itself: >>>>>> >>>>>> private void remove() { >>>>>> synchronized (lock) { >>>>>> if (unfinalized == this) { >>>>>> if (this.next != null) { >>>>>> unfinalized = this.next; >>>>>> } else { >>>>>> unfinalized = this.prev; >>>>>> } >>>>>> } >>>>>> if (this.next != null) { >>>>>> this.next.prev = this.prev; >>>>>> } >>>>>> if (this.prev != null) { >>>>>> this.prev.next = this.next; >>>>>> } >>>>>> // this.next = this; must not be set so that we can >>>>>> traverse the list unsynchronized >>>>>> this.prev = this; /* Indicates that this has been >>>>>> finalized */ >>>>>> } >>>>>> } >>>>>> >>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>> pointer could be used instead: >>>>>> >>>>>> private boolean hasBeenFinalized() { >>>>>> return (prev == this); >>>>>> } >>>>>> >>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>> >>>>>> private void add() { >>>>>> synchronized (lock) { >>>>>> if (unfinalized != null) { >>>>>> this.next = unfinalized; >>>>>> unfinalized.prev = this; >>>>>> } >>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>> // instance fully initialized >>>>>> // (in particular the 'next' pointer) >>>>>> U.storeFence(); >>>>>> unfinalized = this; >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> By doing these modifications, I think you can remove >>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>> >>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>> out? >>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>> instances pending processing by finalizer thread because their >>>>>> referents are eligible for finalization (they are not reachable any >>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>> instances for which their referents have not been finalized yet >>>>>> (including those that are still reachable and alive). The later serves >>>>>> two purposes: >>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>> - it is a source of unfinalized instances for >>>>>> running-finalizers-on-exit if requested by >>>>>> System.runFinalizersOnExit(true); >>>>>> >>>>>> So it really depends on what one would like to see. Showing the queue >>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>> list may be interesting if one wants to know how many live + >>>>>> finalization pending instances are there on the heap that override >>>>>> finalize() method. >>>>>> >>>>>> Regards, Peter >>>>>> >>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>> (see the old code again) would also make it easier to digest. >>>>>>> >>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>> permission() and limit access: >>>>>>> >>>>>>> static const JavaPermission permission() { >>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>> "monitor", NULL}; >>>>>>> return p; >>>>>>> } >>>>>>> >>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>> the finalizer queue? Hmm. >>>>>>> >>>>>>> Thanks, >>>>>>> /Staffan >>>>>>> >>>>>>> >>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>> wrote: >>>>>>>> >>>>>>>> Everyone, >>>>>>>> >>>>>>>> Please review the fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>> >>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>> >>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>> with count >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> -- >>>>>>>> Dmitry Samersoff >>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>> * I would love to change the world, but they won't give me the sources. >>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From peter.levart at gmail.com Thu May 14 20:11:27 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 14 May 2015 22:11:27 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5554DE61.5010403@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <5554DE61.5010403@oracle.com> Message-ID: <5555016F.2050802@gmail.com> Hi Dmitry, ReferenceQueue is not really a queue, but a LIFO stack. Scanner is walking the stack from top (the 'head') to bottom (the last element pointing to itself). When poller(s) overtake the scanner, it actually means that the top of the stack has been eaten under scanner's feet while trying to grab it. Restarting from 'head' actually means 'catching-up' wit poller(s) and trying to keep up with them. We don't get the 'eaten' elements, but might be lucky to get some more food until it's all eaten. Usually we will get all of the elements, since poller(s) must synchronize and do additional work, so they are slower and there's also enqueuer(s) that push elements on the top of the stack so poller(s) must eat those last pushed elements before they can continue chasing the scanner... When scanner is overtaken by poller, then there is a chance the scanner will get less elements than there were present in the stack if a snapshot was taken, so catching-up by restarting from 'head' tries to at least recover some of the rest of the elements of that snapshot before they are gone. Does this make more sense now? Regards, Peter On 05/14/2015 07:41 PM, Dmitry Samersoff wrote: > Peter, > > Could we just bail out on r == r.next? > > It gives us less accurate result, but I suspect that If we restart from > head we need to flush all counters. > > As far I understand queue poller removes items one by one from a queue > end so if we overtaken by queue poller we can safely assume that > we are at the end of the queue. > > Is it correct? > > -Dmitry > > On 2015-05-14 10:50, Peter Levart wrote: >> Hi Derek, >> >> On 05/13/2015 10:34 PM, Derek White wrote: >>> Hi Peter, >>> >>> I don't have smoking gun evidence that your change introduces a bug, >>> but I have some concerns... >> I did have a concern too, ... >> >>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>> Hi Dmitry, >>>> >>>> You iterate the queue then, not the unfinalized list. That's more >>>> logical. >>>> >>>> Holding the queue's lock may pause reference handler and finalizer >>>> threads for the entire time of iteration. This can blow up the >>>> application. Suppose one wants to diagnose the application because he >>>> suspects that finalizer thread hardly keeps up with production of >>>> finalizable instances. This can happen if the allocation rate of >>>> finalizable objects is very high and/or finalize() methods are not >>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>>> gradually and is already holding several million objects. Now you >>>> initiate the diagnostic command to print the queue. The iteration >>>> over and grouping/counting of several millions of Finalizer(s) takes >>>> some time. This blocks finalizer thread and reference handler thread. >>>> But does not block the threads that allocate finalizable objects. By >>>> the time the iteration is over, the JVM blows up and application >>>> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >>>> >>>> It is possible to iterate the elements of the queue for diagnostic >>>> purposes without holding a lock. The modification required to do that >>>> is the following (havent tested this, but I think it should work): >>>> >>>> >>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>> >>> One issue to watch out for is the garbage collectors inspect the >>> Reference.next field from C code directly (to distinguish active vs. >>> pending, iterate over oops, etc.). You can search hotspot for >>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>> >>> Your change makes "inactive" References superficially look like >>> "enqueued" References. The GC code is rather subtle and I haven't yet >>> seen a case where it would get confused by this change, but there >>> could easily be something like that lurking in the GC code. >> ...but then I thought that GC can in no way treat a Reference >> differently whether it is still enqueued in a ReferenceQueue or already >> dequeued (inactive) - responsibility is already on the Java side. >> Currently the definition of Reference.next is this: >> >> /* When active: NULL >> * pending: this >> * Enqueued: next reference in queue (or this if last) >> * Inactive: this >> */ >> @SuppressWarnings("rawtypes") >> Reference next; >> >> We see that, unless GC inspects all ReferenceQueue instances and scans >> their lists too, the state of a Reference that is enqueued as last in >> list is indistinguishable from the state of inactive Reference. So I >> deduced that this distinction (enqueued/inactive) can't be established >> solely on the value of .next field ( == this or != this)... >> >> But I should inspect the GC code too to build a better understanding of >> that part of the story... >> >> Anyway. It turns out that there is already enough state in Reference to >> destinguish between it being enqueued as last in list and already >> dequeued (inactive) - the additional state is Reference.queue that >> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >> ReferenceQueue.reallyPoll. We need to just make sure the two fields >> (r.next and r.queue) are assigned and read in correct order. This can be >> achieved either by making Reference.next a volatile field or by a couple >> of explicit fences: >> >> >> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >> >> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >> before linking the reference into the queue's head in >> ReferenceQueue.enqueue(): >> >> r.queue = ENQUEUED; >> r.next = (head == null) ? r : head; >> head = r; >> >> Both stores are volatile. >> >>>> I also suggest the addition to the ReferenceQueue to be contained >>>> (package-private) and as generic as possible. That's why I suggest >>>> forEach iteration method with no concrete logic. >>>> >>>> It would be possible to encapsulate the entire logic into a special >>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>> static method(s) (printFinalizationQueue)... You could just expose a >>>> package-private forEach static method from Finalizer and code the >>>> rest in DiagnosticCommands. >>> That's good for encapsulation. But I'm concerned that if "forEach" got >>> exposed beyond careful use in DiagnosticCommands, and the Referents >>> were leaked back into the heap, then we could get unexpected object >>> resurrection after finalization. This isn't a bug on it's own - any >>> finalizer might resurrect its object if not careful, but ordinarily >>> /only/ the finalizer could resurrect the object. This could invalidate >>> application invariants? >> Well, it all stays in the confines of package-private API - internal to >> JDK. Any future additional use should be reviewed carefully. Comments on >> the forEach() method should warn about that. >> >>> I agree that using a lock over the ReferenceQueue iteration opens up >>> /another/ case of diagnostics affecting application behavior. And >>> there are pathological scenarios where this gets severe. This is >>> unfortunate but not uncommon. There is enough complication here that >>> you should be sure that the fix for diagnostics performance doesn't >>> introduce subtle bugs to the system in general. A change in this area >>> should get a thorough review from both the runtime and GC sides. >> Is the webrev.02 proposed above more acceptable in that respect? It does >> not introduce any logical changes to existing code. >> >>> Better yet, the reference handling code in GC and runtime should >>> probably be thrown out and re-written, but that's another issue :-) >> I may have some proposals in that direction. Stay tuned. >> >> Regards, Peter >> >>> - Derek >>> >>>> Regards, Peter >>>> >>>> >>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>> Everybody, >>>>> >>>>> Updated version: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>> >>>>> Now it iterates over queue and output result sorted by number of instances. >>>>> >>>>> -Dmitry >>>>> >>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>> Hi Dmitry, Staffan, >>>>>> >>>>>> Lots of good comments here. >>>>>> >>>>>> On the topic of what list should be printed out, I think we should focus >>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>> general iterator and lambda. >>>>>> >>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>> but you can get most of the same information from a heap histogram >>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>> >>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>> that runs after GC that sorts found References objects that need >>>>>> processing into their respective ReferenceQueues). But locking the >>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>> >>>>>> The sorting suggestion is a nice touch. >>>>>> >>>>>> - Derek White, GC team >>>>>> >>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>> Hi Dmitry, Staffan, >>>>>>> >>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>> Dmitry, >>>>>>>> >>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>> well considering the changes to Finalizer. >>>>>>>> >>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>> running out of memory. >>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>> then it should not be too large, as there are not many different >>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>> finalize() method in the whole jdk/src tree). >>>>>>> >>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>> interfere with synchronization? >>>>>>> >>>>>>> It might be possible to scan the list without holding a lock for >>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>> Finalizer.next pointer to point back to itself: >>>>>>> >>>>>>> private void remove() { >>>>>>> synchronized (lock) { >>>>>>> if (unfinalized == this) { >>>>>>> if (this.next != null) { >>>>>>> unfinalized = this.next; >>>>>>> } else { >>>>>>> unfinalized = this.prev; >>>>>>> } >>>>>>> } >>>>>>> if (this.next != null) { >>>>>>> this.next.prev = this.prev; >>>>>>> } >>>>>>> if (this.prev != null) { >>>>>>> this.prev.next = this.next; >>>>>>> } >>>>>>> // this.next = this; must not be set so that we can >>>>>>> traverse the list unsynchronized >>>>>>> this.prev = this; /* Indicates that this has been >>>>>>> finalized */ >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>> pointer could be used instead: >>>>>>> >>>>>>> private boolean hasBeenFinalized() { >>>>>>> return (prev == this); >>>>>>> } >>>>>>> >>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>> >>>>>>> private void add() { >>>>>>> synchronized (lock) { >>>>>>> if (unfinalized != null) { >>>>>>> this.next = unfinalized; >>>>>>> unfinalized.prev = this; >>>>>>> } >>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>> // instance fully initialized >>>>>>> // (in particular the 'next' pointer) >>>>>>> U.storeFence(); >>>>>>> unfinalized = this; >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> By doing these modifications, I think you can remove >>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>> >>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>> out? >>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>> instances pending processing by finalizer thread because their >>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>> instances for which their referents have not been finalized yet >>>>>>> (including those that are still reachable and alive). The later serves >>>>>>> two purposes: >>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>> - it is a source of unfinalized instances for >>>>>>> running-finalizers-on-exit if requested by >>>>>>> System.runFinalizersOnExit(true); >>>>>>> >>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>> list may be interesting if one wants to know how many live + >>>>>>> finalization pending instances are there on the heap that override >>>>>>> finalize() method. >>>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>> >>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>> permission() and limit access: >>>>>>>> >>>>>>>> static const JavaPermission permission() { >>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>> "monitor", NULL}; >>>>>>>> return p; >>>>>>>> } >>>>>>>> >>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>> the finalizer queue? Hmm. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> /Staffan >>>>>>>> >>>>>>>> >>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Everyone, >>>>>>>>> >>>>>>>>> Please review the fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>> >>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>> >>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>> with count >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dmitry Samersoff >>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>> * I would love to change the world, but they won't give me the sources. -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Thu May 14 20:35:36 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 14 May 2015 22:35:36 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5555016F.2050802@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <5554DE61.5010403@oracle.com> <5555016F.2050802@gmail.com> Message-ID: <55550718.1000507@gmail.com> On 05/14/2015 10:11 PM, Peter Levart wrote: > Hi Dmitry, > > ReferenceQueue is not really a queue, but a LIFO stack. Scanner is > walking the stack from top (the 'head') to bottom (the last element > pointing to itself). When poller(s) overtake the scanner, it actually > means that the top of the stack has been eaten under scanner's feet > while trying to grab it. Restarting from 'head' actually means > 'catching-up' wit poller(s) and trying to keep up with them. We don't > get the 'eaten' elements, but might be lucky to get some more food > until it's all eaten. Usually we will get all of the elements, since > poller(s) must synchronize and do additional work, so they are slower > and there's also enqueuer(s) that push elements on the top of the > stack so poller(s) must eat those last pushed elements before they can > continue chasing the scanner... > > When scanner is overtaken by poller, then there is a chance the > scanner will get less elements than there were present in the stack if > a snapshot was taken, so catching-up by restarting from 'head' tries > to at least recover some of the rest of the elements of that snapshot > before they are gone. Also, the chance that the scanner is overtaken by poller is greater at the start of race. The scanner and poller start from the same spot and as we know threads are "jumpy" so it will happen quite frequently that a poller jumps before scanner. So just giving-up when overtaken might return 0 or just a few elements when there are millions there in the queue. When scanner finally gets a head start, it will usually lead the race to the end. Peter > > Does this make more sense now? > > Regards, Peter > > On 05/14/2015 07:41 PM, Dmitry Samersoff wrote: >> Peter, >> >> Could we just bail out on r == r.next? >> >> It gives us less accurate result, but I suspect that If we restart from >> head we need to flush all counters. >> >> As far I understand queue poller removes items one by one from a queue >> end so if we overtaken by queue poller we can safely assume that >> we are at the end of the queue. >> >> Is it correct? >> >> -Dmitry >> >> On 2015-05-14 10:50, Peter Levart wrote: >>> Hi Derek, >>> >>> On 05/13/2015 10:34 PM, Derek White wrote: >>>> Hi Peter, >>>> >>>> I don't have smoking gun evidence that your change introduces a bug, >>>> but I have some concerns... >>> I did have a concern too, ... >>> >>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>> Hi Dmitry, >>>>> >>>>> You iterate the queue then, not the unfinalized list. That's more >>>>> logical. >>>>> >>>>> Holding the queue's lock may pause reference handler and finalizer >>>>> threads for the entire time of iteration. This can blow up the >>>>> application. Suppose one wants to diagnose the application because he >>>>> suspects that finalizer thread hardly keeps up with production of >>>>> finalizable instances. This can happen if the allocation rate of >>>>> finalizable objects is very high and/or finalize() methods are not >>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>>>> gradually and is already holding several million objects. Now you >>>>> initiate the diagnostic command to print the queue. The iteration >>>>> over and grouping/counting of several millions of Finalizer(s) takes >>>>> some time. This blocks finalizer thread and reference handler thread. >>>>> But does not block the threads that allocate finalizable objects. By >>>>> the time the iteration is over, the JVM blows up and application >>>>> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >>>>> >>>>> It is possible to iterate the elements of the queue for diagnostic >>>>> purposes without holding a lock. The modification required to do that >>>>> is the following (havent tested this, but I think it should work): >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>> >>>> One issue to watch out for is the garbage collectors inspect the >>>> Reference.next field from C code directly (to distinguish active vs. >>>> pending, iterate over oops, etc.). You can search hotspot for >>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>> >>>> Your change makes "inactive" References superficially look like >>>> "enqueued" References. The GC code is rather subtle and I haven't yet >>>> seen a case where it would get confused by this change, but there >>>> could easily be something like that lurking in the GC code. >>> ...but then I thought that GC can in no way treat a Reference >>> differently whether it is still enqueued in a ReferenceQueue or already >>> dequeued (inactive) - responsibility is already on the Java side. >>> Currently the definition of Reference.next is this: >>> >>> /* When active: NULL >>> * pending: this >>> * Enqueued: next reference in queue (or this if last) >>> * Inactive: this >>> */ >>> @SuppressWarnings("rawtypes") >>> Reference next; >>> >>> We see that, unless GC inspects all ReferenceQueue instances and scans >>> their lists too, the state of a Reference that is enqueued as last in >>> list is indistinguishable from the state of inactive Reference. So I >>> deduced that this distinction (enqueued/inactive) can't be established >>> solely on the value of .next field ( == this or != this)... >>> >>> But I should inspect the GC code too to build a better understanding of >>> that part of the story... >>> >>> Anyway. It turns out that there is already enough state in Reference to >>> destinguish between it being enqueued as last in list and already >>> dequeued (inactive) - the additional state is Reference.queue that >>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>> (r.next and r.queue) are assigned and read in correct order. This can be >>> achieved either by making Reference.next a volatile field or by a couple >>> of explicit fences: >>> >>> >>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>> >>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>> before linking the reference into the queue's head in >>> ReferenceQueue.enqueue(): >>> >>> r.queue = ENQUEUED; >>> r.next = (head == null) ? r : head; >>> head = r; >>> >>> Both stores are volatile. >>> >>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>> (package-private) and as generic as possible. That's why I suggest >>>>> forEach iteration method with no concrete logic. >>>>> >>>>> It would be possible to encapsulate the entire logic into a special >>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>> static method(s) (printFinalizationQueue)... You could just expose a >>>>> package-private forEach static method from Finalizer and code the >>>>> rest in DiagnosticCommands. >>>> That's good for encapsulation. But I'm concerned that if "forEach" got >>>> exposed beyond careful use in DiagnosticCommands, and the Referents >>>> were leaked back into the heap, then we could get unexpected object >>>> resurrection after finalization. This isn't a bug on it's own - any >>>> finalizer might resurrect its object if not careful, but ordinarily >>>> /only/ the finalizer could resurrect the object. This could invalidate >>>> application invariants? >>> Well, it all stays in the confines of package-private API - internal to >>> JDK. Any future additional use should be reviewed carefully. Comments on >>> the forEach() method should warn about that. >>> >>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>> /another/ case of diagnostics affecting application behavior. And >>>> there are pathological scenarios where this gets severe. This is >>>> unfortunate but not uncommon. There is enough complication here that >>>> you should be sure that the fix for diagnostics performance doesn't >>>> introduce subtle bugs to the system in general. A change in this area >>>> should get a thorough review from both the runtime and GC sides. >>> Is the webrev.02 proposed above more acceptable in that respect? It does >>> not introduce any logical changes to existing code. >>> >>>> Better yet, the reference handling code in GC and runtime should >>>> probably be thrown out and re-written, but that's another issue :-) >>> I may have some proposals in that direction. Stay tuned. >>> >>> Regards, Peter >>> >>>> - Derek >>>> >>>>> Regards, Peter >>>>> >>>>> >>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>> Everybody, >>>>>> >>>>>> Updated version: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>> >>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>> Hi Dmitry, Staffan, >>>>>>> >>>>>>> Lots of good comments here. >>>>>>> >>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>> general iterator and lambda. >>>>>>> >>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>> but you can get most of the same information from a heap histogram >>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>> >>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>> that runs after GC that sorts found References objects that need >>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>> >>>>>>> The sorting suggestion is a nice touch. >>>>>>> >>>>>>> - Derek White, GC team >>>>>>> >>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>> Hi Dmitry, Staffan, >>>>>>>> >>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>> Dmitry, >>>>>>>>> >>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>> well considering the changes to Finalizer. >>>>>>>>> >>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>> running out of memory. >>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>> then it should not be too large, as there are not many different >>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>> >>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>> interfere with synchronization? >>>>>>>> >>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>> >>>>>>>> private void remove() { >>>>>>>> synchronized (lock) { >>>>>>>> if (unfinalized == this) { >>>>>>>> if (this.next != null) { >>>>>>>> unfinalized = this.next; >>>>>>>> } else { >>>>>>>> unfinalized = this.prev; >>>>>>>> } >>>>>>>> } >>>>>>>> if (this.next != null) { >>>>>>>> this.next.prev = this.prev; >>>>>>>> } >>>>>>>> if (this.prev != null) { >>>>>>>> this.prev.next = this.next; >>>>>>>> } >>>>>>>> // this.next = this; must not be set so that we can >>>>>>>> traverse the list unsynchronized >>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>> finalized */ >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>> pointer could be used instead: >>>>>>>> >>>>>>>> private boolean hasBeenFinalized() { >>>>>>>> return (prev == this); >>>>>>>> } >>>>>>>> >>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>> >>>>>>>> private void add() { >>>>>>>> synchronized (lock) { >>>>>>>> if (unfinalized != null) { >>>>>>>> this.next = unfinalized; >>>>>>>> unfinalized.prev = this; >>>>>>>> } >>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>> // instance fully initialized >>>>>>>> // (in particular the 'next' pointer) >>>>>>>> U.storeFence(); >>>>>>>> unfinalized = this; >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> By doing these modifications, I think you can remove >>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>> >>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>> out? >>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>> instances pending processing by finalizer thread because their >>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>> instances for which their referents have not been finalized yet >>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>> two purposes: >>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>> - it is a source of unfinalized instances for >>>>>>>> running-finalizers-on-exit if requested by >>>>>>>> System.runFinalizersOnExit(true); >>>>>>>> >>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>> finalization pending instances are there on the heap that override >>>>>>>> finalize() method. >>>>>>>> >>>>>>>> Regards, Peter >>>>>>>> >>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>> >>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>> permission() and limit access: >>>>>>>>> >>>>>>>>> static const JavaPermission permission() { >>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>> "monitor", NULL}; >>>>>>>>> return p; >>>>>>>>> } >>>>>>>>> >>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>> the finalizer queue? Hmm. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> /Staffan >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Everyone, >>>>>>>>>> >>>>>>>>>> Please review the fix: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>> >>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>> >>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>> with count >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Dmitry Samersoff >>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>> * I would love to change the world, but they won't give me the sources. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri May 15 00:15:35 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 14 May 2015 17:15:35 -0700 Subject: RFR (XS) 8079644: memory stomping error with ResourceManagement and TestAgentStress.java Message-ID: <55553AA7.6000901@oracle.com> Please, review the jdk 9 fix for: https://bugs.openjdk.java.net/browse/JDK-8079644 9 hotspot webrev: http://cr.openjdk.java.net/~sspitsyn/webrevs/2015/hotspot/8079644-JVMTI-memstomp.1 Summary: The cached class file structure must be deallocated instead of the cached class file bytes. It is because the cached class file bytes array is a part of the cached class file structure. Testing in progress: In progress: nsk.redefine.testlist, JTREG com/sun/jdi tests, ad-hog closed/serviceability/resource/TestAgentStress.java test from the bug report Thanks, Serguei From dmitry.samersoff at oracle.com Fri May 15 05:09:14 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Fri, 15 May 2015 08:09:14 +0300 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55550718.1000507@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <5554DE61.5010403@oracle.com> <5555016F.2050802@gmail.com> <55550718.1000507@gmail.com> Message-ID: <55557F7A.4050508@oracle.com> Peter, Thank you for the explanation! -Dmitry On 2015-05-14 23:35, Peter Levart wrote: > > > On 05/14/2015 10:11 PM, Peter Levart wrote: >> Hi Dmitry, >> >> ReferenceQueue is not really a queue, but a LIFO stack. Scanner is >> walking the stack from top (the 'head') to bottom (the last element >> pointing to itself). When poller(s) overtake the scanner, it actually >> means that the top of the stack has been eaten under scanner's feet >> while trying to grab it. Restarting from 'head' actually means >> 'catching-up' wit poller(s) and trying to keep up with them. We don't >> get the 'eaten' elements, but might be lucky to get some more food >> until it's all eaten. Usually we will get all of the elements, since >> poller(s) must synchronize and do additional work, so they are slower >> and there's also enqueuer(s) that push elements on the top of the >> stack so poller(s) must eat those last pushed elements before they can >> continue chasing the scanner... >> >> When scanner is overtaken by poller, then there is a chance the >> scanner will get less elements than there were present in the stack if >> a snapshot was taken, so catching-up by restarting from 'head' tries >> to at least recover some of the rest of the elements of that snapshot >> before they are gone. > > Also, the chance that the scanner is overtaken by poller is greater at > the start of race. The scanner and poller start from the same spot and > as we know threads are "jumpy" so it will happen quite frequently that a > poller jumps before scanner. So just giving-up when overtaken might > return 0 or just a few elements when there are millions there in the > queue. When scanner finally gets a head start, it will usually lead the > race to the end. > > Peter > >> >> Does this make more sense now? >> >> Regards, Peter >> >> On 05/14/2015 07:41 PM, Dmitry Samersoff wrote: >>> Peter, >>> >>> Could we just bail out on r == r.next? >>> >>> It gives us less accurate result, but I suspect that If we restart from >>> head we need to flush all counters. >>> >>> As far I understand queue poller removes items one by one from a queue >>> end so if we overtaken by queue poller we can safely assume that >>> we are at the end of the queue. >>> >>> Is it correct? >>> >>> -Dmitry >>> >>> On 2015-05-14 10:50, Peter Levart wrote: >>>> Hi Derek, >>>> >>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>> Hi Peter, >>>>> >>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>> but I have some concerns... >>>> I did have a concern too, ... >>>> >>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>> logical. >>>>>> >>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>> threads for the entire time of iteration. This can blow up the >>>>>> application. Suppose one wants to diagnose the application because he >>>>>> suspects that finalizer thread hardly keeps up with production of >>>>>> finalizable instances. This can happen if the allocation rate of >>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>>>>> gradually and is already holding several million objects. Now you >>>>>> initiate the diagnostic command to print the queue. The iteration >>>>>> over and grouping/counting of several millions of Finalizer(s) takes >>>>>> some time. This blocks finalizer thread and reference handler thread. >>>>>> But does not block the threads that allocate finalizable objects. By >>>>>> the time the iteration is over, the JVM blows up and application >>>>>> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >>>>>> >>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>> purposes without holding a lock. The modification required to do that >>>>>> is the following (havent tested this, but I think it should work): >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>> >>>>> One issue to watch out for is the garbage collectors inspect the >>>>> Reference.next field from C code directly (to distinguish active vs. >>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>> >>>>> Your change makes "inactive" References superficially look like >>>>> "enqueued" References. The GC code is rather subtle and I haven't yet >>>>> seen a case where it would get confused by this change, but there >>>>> could easily be something like that lurking in the GC code. >>>> ...but then I thought that GC can in no way treat a Reference >>>> differently whether it is still enqueued in a ReferenceQueue or already >>>> dequeued (inactive) - responsibility is already on the Java side. >>>> Currently the definition of Reference.next is this: >>>> >>>> /* When active: NULL >>>> * pending: this >>>> * Enqueued: next reference in queue (or this if last) >>>> * Inactive: this >>>> */ >>>> @SuppressWarnings("rawtypes") >>>> Reference next; >>>> >>>> We see that, unless GC inspects all ReferenceQueue instances and scans >>>> their lists too, the state of a Reference that is enqueued as last in >>>> list is indistinguishable from the state of inactive Reference. So I >>>> deduced that this distinction (enqueued/inactive) can't be established >>>> solely on the value of .next field ( == this or != this)... >>>> >>>> But I should inspect the GC code too to build a better understanding of >>>> that part of the story... >>>> >>>> Anyway. It turns out that there is already enough state in Reference to >>>> destinguish between it being enqueued as last in list and already >>>> dequeued (inactive) - the additional state is Reference.queue that >>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>> (r.next and r.queue) are assigned and read in correct order. This can be >>>> achieved either by making Reference.next a volatile field or by a couple >>>> of explicit fences: >>>> >>>> >>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>> >>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>> before linking the reference into the queue's head in >>>> ReferenceQueue.enqueue(): >>>> >>>> r.queue = ENQUEUED; >>>> r.next = (head == null) ? r : head; >>>> head = r; >>>> >>>> Both stores are volatile. >>>> >>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>> forEach iteration method with no concrete logic. >>>>>> >>>>>> It would be possible to encapsulate the entire logic into a special >>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>> static method(s) (printFinalizationQueue)... You could just expose a >>>>>> package-private forEach static method from Finalizer and code the >>>>>> rest in DiagnosticCommands. >>>>> That's good for encapsulation. But I'm concerned that if "forEach" got >>>>> exposed beyond careful use in DiagnosticCommands, and the Referents >>>>> were leaked back into the heap, then we could get unexpected object >>>>> resurrection after finalization. This isn't a bug on it's own - any >>>>> finalizer might resurrect its object if not careful, but ordinarily >>>>> /only/ the finalizer could resurrect the object. This could invalidate >>>>> application invariants? >>>> Well, it all stays in the confines of package-private API - internal to >>>> JDK. Any future additional use should be reviewed carefully. Comments on >>>> the forEach() method should warn about that. >>>> >>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>> /another/ case of diagnostics affecting application behavior. And >>>>> there are pathological scenarios where this gets severe. This is >>>>> unfortunate but not uncommon. There is enough complication here that >>>>> you should be sure that the fix for diagnostics performance doesn't >>>>> introduce subtle bugs to the system in general. A change in this area >>>>> should get a thorough review from both the runtime and GC sides. >>>> Is the webrev.02 proposed above more acceptable in that respect? It does >>>> not introduce any logical changes to existing code. >>>> >>>>> Better yet, the reference handling code in GC and runtime should >>>>> probably be thrown out and re-written, but that's another issue :-) >>>> I may have some proposals in that direction. Stay tuned. >>>> >>>> Regards, Peter >>>> >>>>> - Derek >>>>> >>>>>> Regards, Peter >>>>>> >>>>>> >>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>> Everybody, >>>>>>> >>>>>>> Updated version: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>> >>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>> Hi Dmitry, Staffan, >>>>>>>> >>>>>>>> Lots of good comments here. >>>>>>>> >>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>> general iterator and lambda. >>>>>>>> >>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>> >>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>> >>>>>>>> The sorting suggestion is a nice touch. >>>>>>>> >>>>>>>> - Derek White, GC team >>>>>>>> >>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>> >>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>> Dmitry, >>>>>>>>>> >>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>> >>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>> running out of memory. >>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>> >>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>> interfere with synchronization? >>>>>>>>> >>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>> >>>>>>>>> private void remove() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized == this) { >>>>>>>>> if (this.next != null) { >>>>>>>>> unfinalized = this.next; >>>>>>>>> } else { >>>>>>>>> unfinalized = this.prev; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> if (this.next != null) { >>>>>>>>> this.next.prev = this.prev; >>>>>>>>> } >>>>>>>>> if (this.prev != null) { >>>>>>>>> this.prev.next = this.next; >>>>>>>>> } >>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>> traverse the list unsynchronized >>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>> finalized */ >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>> pointer could be used instead: >>>>>>>>> >>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>> return (prev == this); >>>>>>>>> } >>>>>>>>> >>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>> >>>>>>>>> private void add() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized != null) { >>>>>>>>> this.next = unfinalized; >>>>>>>>> unfinalized.prev = this; >>>>>>>>> } >>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>> // instance fully initialized >>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>> U.storeFence(); >>>>>>>>> unfinalized = this; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>> >>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>> out? >>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>> two purposes: >>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>> >>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>> finalize() method. >>>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>> >>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>> permission() and limit access: >>>>>>>>>> >>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>> "monitor", NULL}; >>>>>>>>>> return p; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> /Staffan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Everyone, >>>>>>>>>>> >>>>>>>>>>> Please review the fix: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>> >>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>> with count >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the source code. From cheleswer.sahu at oracle.com Fri May 15 06:56:28 2015 From: cheleswer.sahu at oracle.com (cheleswer sahu) Date: Fri, 15 May 2015 12:26:28 +0530 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> References: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> Message-ID: <5555989C.9090803@oracle.com> Dear Dan & Dmitry, Thanks for pointing out the security vulnerability in the fix and for your precise review. I am also agree with Dmitry's fix. I will fix the code and resend the review request. Regards, Cheleswer On 5/14/2015 9:46 PM, Gerald Thornbrugh wrote: > Hi Dan, > > When Cheleswer and I discussed this fix my interpretation had a slightly different goal: > > Prior to the initial security fix any user could execute "jps" and get the user names associated > with other user's perf data (i.e. the call to get_user_name_slow() would succeed.). My initial > thought was that this was a regression for all users not just "root" and this goal led to this fix. > At the time I did not see this as a security vulnerability, your review has changed my mind. > > I agree that Dmitry's fix is a more secure fix for the issue and I think we should use it. > > Let me know if you have any questions. > > Thanks! > > Jerry > >> Hi, >> Please review the code changes for >> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >> tested JDK9 with fix successfully. As I do not have an account for >> OpenJDK, >> David Buck will push the fix into jdk9/hs-rt/. >> >> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >> >> Regards, >> Cheleswer > Cheleswer, > > Sorry for the lengthy review, but since this is security related, > I have to be complete. > > The goal: Add a policy by-pass for the 'root' user in order to > fix the regression in jps(1) behavior. > > The core of this policy by-pass is the change to this function: > > 205 static bool is_statbuf_secure(struct stat *statp, int mode) { > 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { > 207 // The path represents a link or some non-directory file type, > 208 // which is not what we expected. Declare it insecure. > 209 // > 210 return false; > 211 } > 212 // If the directory is going to be opened readonly, we consider > this as secure operation > 213 // we do not need to do any more checks. > 214 // > 215 if ((mode & O_ACCMODE) == O_RDONLY) { > 216 return true; > 217 } > 218 // We have an existing directory, check if the permissions are safe. > 219 // > 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { > 221 // The directory is open for writing and could be subjected > 222 // to a symlink or a hard link attack. Declare it insecure. > 223 // > 224 return false; > 225 } > 226 // See if the uid of the directory matches the effective uid of > the process. > 227 // > 228 if (statp->st_uid != geteuid()) { > 229 // The directory was not created by this user, declare it > insecure. > 230 // > 231 return false; > 232 } > 233 return true; > 234 } > > Lines 212-217 are added which allows a caller that passes in O_RDONLY > to by-pass the security checks on lines 220-225 and 228-232. This > implementation is using an attribute of _how_ the data is accessed > to override security policy instead of an attribute of _who_ is > accessing the data. > > Here are the code paths that access the modified policy code: > > is_statbuf_secure() is called by: > > - is_directory_secure() > - is_dirfd_secure() > > is_directory_secure() is called by: > > - get_user_name_slow() with O_RDONLY > - make_user_tmp_dir() with O_RDWR > - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) > > is_dirfd_secure() is called by: > > - open_directory_secure() with a mode parameter > > open_directory_secure() is called by: > > - open_directory_secure_cwd() with O_RDWR > - get_user_name_slow() with O_RDONLY > > Only the code paths that specify O_RDWR make use of > the new policy by-pass code so it looks like only > > - get_user_name_slow() with O_RDONLY > - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) > > are interesting. > > The new security policy by-pass will allow get_user_name_slow(): > > - to process directory entries in a directory that is writable > which makes this use subject to a symlink or hard link attack. > - to process directory entries in a directory that the calling > user does not own; the intent of the policy by-pass is to > allow this for the 'root' user, but this implementation > allows the by-pass for any user. > > It looks like the get_user_name_slow() code is written safely > enough such that any symlink or hard link attack should not > cause any issues. > > The new policy by-pass will allow any user to determine the > user name associated with VMs owned by another user. This is > a broader policy by-pass than was intended. > > > The new security policy by-pass will allow mmap_attach_shared(): > > - to process directory entries in a directory that is writable > which makes this use subject to a symlink or hard link attack. > - to process directory entries in a directory that the calling > user does not own; the intent of the policy by-pass is to > allow this for the 'root' user, but this implementation allows > the by-pass for any user. > > The mmap_attach_shared() code protects itself from a symlink > attack by including the 'O_NOFOLLOW' flag when opening the > PerfData file and it protects itself from a hardlink attack by > checking the hard link count after opening the file. It does > not protect itself against being handed a corrupted file or > even a very large file that would cause an OOM when the VM > tries to map what is supposed to be a PerfData file. > > The new policy by-pass will allow any user to access the > PerfData file associated with VMs owned by another user. This > is a broader policy by-pass than was intended. > > > Summary: > > This implementation of the new security policy by-pass is using > an attribute of _how_ the data is accessed to override security > policy instead of an attribute of _who_ is accessing the data. > This allows the VM to be exposed to some of the attacks that > the following fix was designed to prevent: > > JDK-8050807 Better performing performance data handling > > Dmitry's response to the code review provides a solution that > is based on who is accessing the data and that solution or > one like it should be considered. > > Again, sorry for the lengthy review. > > Dan From dmitry.samersoff at oracle.com Fri May 15 20:23:45 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Fri, 15 May 2015 23:23:45 +0300 Subject: RFR(M,v6): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55550718.1000507@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <5554DE61.5010403@oracle.com> <5555016F.2050802@gmail.com> <55550718.1000507@gmail.com> Message-ID: <555655D1.90208@oracle.com> Everybody, Please review updated version. http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.06/ JDK part provided by Peter Levart, so all credentials belongs to him. -Dmitry On 2015-05-14 23:35, Peter Levart wrote: > > > On 05/14/2015 10:11 PM, Peter Levart wrote: >> Hi Dmitry, >> >> ReferenceQueue is not really a queue, but a LIFO stack. Scanner is >> walking the stack from top (the 'head') to bottom (the last element >> pointing to itself). When poller(s) overtake the scanner, it actually >> means that the top of the stack has been eaten under scanner's feet >> while trying to grab it. Restarting from 'head' actually means >> 'catching-up' wit poller(s) and trying to keep up with them. We don't >> get the 'eaten' elements, but might be lucky to get some more food >> until it's all eaten. Usually we will get all of the elements, since >> poller(s) must synchronize and do additional work, so they are slower >> and there's also enqueuer(s) that push elements on the top of the >> stack so poller(s) must eat those last pushed elements before they can >> continue chasing the scanner... >> >> When scanner is overtaken by poller, then there is a chance the >> scanner will get less elements than there were present in the stack if >> a snapshot was taken, so catching-up by restarting from 'head' tries >> to at least recover some of the rest of the elements of that snapshot >> before they are gone. > > Also, the chance that the scanner is overtaken by poller is greater at > the start of race. The scanner and poller start from the same spot and > as we know threads are "jumpy" so it will happen quite frequently that a > poller jumps before scanner. So just giving-up when overtaken might > return 0 or just a few elements when there are millions there in the > queue. When scanner finally gets a head start, it will usually lead the > race to the end. > > Peter > >> >> Does this make more sense now? >> >> Regards, Peter >> >> On 05/14/2015 07:41 PM, Dmitry Samersoff wrote: >>> Peter, >>> >>> Could we just bail out on r == r.next? >>> >>> It gives us less accurate result, but I suspect that If we restart from >>> head we need to flush all counters. >>> >>> As far I understand queue poller removes items one by one from a queue >>> end so if we overtaken by queue poller we can safely assume that >>> we are at the end of the queue. >>> >>> Is it correct? >>> >>> -Dmitry >>> >>> On 2015-05-14 10:50, Peter Levart wrote: >>>> Hi Derek, >>>> >>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>> Hi Peter, >>>>> >>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>> but I have some concerns... >>>> I did have a concern too, ... >>>> >>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>> logical. >>>>>> >>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>> threads for the entire time of iteration. This can blow up the >>>>>> application. Suppose one wants to diagnose the application because he >>>>>> suspects that finalizer thread hardly keeps up with production of >>>>>> finalizable instances. This can happen if the allocation rate of >>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>>>>> gradually and is already holding several million objects. Now you >>>>>> initiate the diagnostic command to print the queue. The iteration >>>>>> over and grouping/counting of several millions of Finalizer(s) takes >>>>>> some time. This blocks finalizer thread and reference handler thread. >>>>>> But does not block the threads that allocate finalizable objects. By >>>>>> the time the iteration is over, the JVM blows up and application >>>>>> slows down to a halt doing GCs most of the time, getting OOMEs, etc... >>>>>> >>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>> purposes without holding a lock. The modification required to do that >>>>>> is the following (havent tested this, but I think it should work): >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>> >>>>> One issue to watch out for is the garbage collectors inspect the >>>>> Reference.next field from C code directly (to distinguish active vs. >>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>> >>>>> Your change makes "inactive" References superficially look like >>>>> "enqueued" References. The GC code is rather subtle and I haven't yet >>>>> seen a case where it would get confused by this change, but there >>>>> could easily be something like that lurking in the GC code. >>>> ...but then I thought that GC can in no way treat a Reference >>>> differently whether it is still enqueued in a ReferenceQueue or already >>>> dequeued (inactive) - responsibility is already on the Java side. >>>> Currently the definition of Reference.next is this: >>>> >>>> /* When active: NULL >>>> * pending: this >>>> * Enqueued: next reference in queue (or this if last) >>>> * Inactive: this >>>> */ >>>> @SuppressWarnings("rawtypes") >>>> Reference next; >>>> >>>> We see that, unless GC inspects all ReferenceQueue instances and scans >>>> their lists too, the state of a Reference that is enqueued as last in >>>> list is indistinguishable from the state of inactive Reference. So I >>>> deduced that this distinction (enqueued/inactive) can't be established >>>> solely on the value of .next field ( == this or != this)... >>>> >>>> But I should inspect the GC code too to build a better understanding of >>>> that part of the story... >>>> >>>> Anyway. It turns out that there is already enough state in Reference to >>>> destinguish between it being enqueued as last in list and already >>>> dequeued (inactive) - the additional state is Reference.queue that >>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>> (r.next and r.queue) are assigned and read in correct order. This can be >>>> achieved either by making Reference.next a volatile field or by a couple >>>> of explicit fences: >>>> >>>> >>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>> >>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>> before linking the reference into the queue's head in >>>> ReferenceQueue.enqueue(): >>>> >>>> r.queue = ENQUEUED; >>>> r.next = (head == null) ? r : head; >>>> head = r; >>>> >>>> Both stores are volatile. >>>> >>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>> forEach iteration method with no concrete logic. >>>>>> >>>>>> It would be possible to encapsulate the entire logic into a special >>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>> static method(s) (printFinalizationQueue)... You could just expose a >>>>>> package-private forEach static method from Finalizer and code the >>>>>> rest in DiagnosticCommands. >>>>> That's good for encapsulation. But I'm concerned that if "forEach" got >>>>> exposed beyond careful use in DiagnosticCommands, and the Referents >>>>> were leaked back into the heap, then we could get unexpected object >>>>> resurrection after finalization. This isn't a bug on it's own - any >>>>> finalizer might resurrect its object if not careful, but ordinarily >>>>> /only/ the finalizer could resurrect the object. This could invalidate >>>>> application invariants? >>>> Well, it all stays in the confines of package-private API - internal to >>>> JDK. Any future additional use should be reviewed carefully. Comments on >>>> the forEach() method should warn about that. >>>> >>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>> /another/ case of diagnostics affecting application behavior. And >>>>> there are pathological scenarios where this gets severe. This is >>>>> unfortunate but not uncommon. There is enough complication here that >>>>> you should be sure that the fix for diagnostics performance doesn't >>>>> introduce subtle bugs to the system in general. A change in this area >>>>> should get a thorough review from both the runtime and GC sides. >>>> Is the webrev.02 proposed above more acceptable in that respect? It does >>>> not introduce any logical changes to existing code. >>>> >>>>> Better yet, the reference handling code in GC and runtime should >>>>> probably be thrown out and re-written, but that's another issue :-) >>>> I may have some proposals in that direction. Stay tuned. >>>> >>>> Regards, Peter >>>> >>>>> - Derek >>>>> >>>>>> Regards, Peter >>>>>> >>>>>> >>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>> Everybody, >>>>>>> >>>>>>> Updated version: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>> >>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>> Hi Dmitry, Staffan, >>>>>>>> >>>>>>>> Lots of good comments here. >>>>>>>> >>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>> general iterator and lambda. >>>>>>>> >>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>> >>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>> >>>>>>>> The sorting suggestion is a nice touch. >>>>>>>> >>>>>>>> - Derek White, GC team >>>>>>>> >>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>> >>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>> Dmitry, >>>>>>>>>> >>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>> >>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>> running out of memory. >>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>> >>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>> interfere with synchronization? >>>>>>>>> >>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>> >>>>>>>>> private void remove() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized == this) { >>>>>>>>> if (this.next != null) { >>>>>>>>> unfinalized = this.next; >>>>>>>>> } else { >>>>>>>>> unfinalized = this.prev; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> if (this.next != null) { >>>>>>>>> this.next.prev = this.prev; >>>>>>>>> } >>>>>>>>> if (this.prev != null) { >>>>>>>>> this.prev.next = this.next; >>>>>>>>> } >>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>> traverse the list unsynchronized >>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>> finalized */ >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>> pointer could be used instead: >>>>>>>>> >>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>> return (prev == this); >>>>>>>>> } >>>>>>>>> >>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>> >>>>>>>>> private void add() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized != null) { >>>>>>>>> this.next = unfinalized; >>>>>>>>> unfinalized.prev = this; >>>>>>>>> } >>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>> // instance fully initialized >>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>> U.storeFence(); >>>>>>>>> unfinalized = this; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>> >>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>> out? >>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>> two purposes: >>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>> >>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>> finalize() method. >>>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>> >>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>> permission() and limit access: >>>>>>>>>> >>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>> "monitor", NULL}; >>>>>>>>>> return p; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> /Staffan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Everyone, >>>>>>>>>>> >>>>>>>>>>> Please review the fix: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>> >>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>> with count >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From martinrb at google.com Fri May 15 20:59:35 2015 From: martinrb at google.com (Martin Buchholz) Date: Fri, 15 May 2015 13:59:35 -0700 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <555474C0.30108@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <555392EB.8020506@oracle.com> <555474C0.30108@oracle.com> Message-ID: You may submit, but: --- the @test is _NOT_ a javadoc comment - /* is more standard. http://openjdk.java.net/jtreg/tag-spec.html#EXAMPLES -/* +/** * @test * @bug 4530538 * @summary Basic unit test of ThreadMXBean.getAllThreadIds() @@ -32,9 +32,30 @@ */ On Thu, May 14, 2015 at 3:11 AM, Jaroslav Bachorik < jaroslav.bachorik at oracle.com> wrote: > On 13.5.2015 21:14, Martin Buchholz wrote: > >> >> >> It's common for users to introduce parallelism in classloaders or >> jar-loaders (we do this at google) which may cause arbitrary thread >> fluctuations. That makes this particular API rather difficult to test >> robustly, especially if you are only testing against the spec. >> > > Well, in such an environment this test makes absolutely no sense - there > is nothing deterministic in the thread counters we could assert against. > Consider creating a private thread group just for the test and restricting the count assertions to just threads in that thread group - that should be reliable (even if other tests are running concurrently in the same VM). -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek.white at oracle.com Fri May 15 22:46:32 2015 From: derek.white at oracle.com (Derek White) Date: Fri, 15 May 2015 18:46:32 -0400 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555453BA.2070205@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> Message-ID: <55567748.6020304@oracle.com> Hi Peter, Some comments below... On 5/14/15 3:50 AM, Peter Levart wrote: > Hi Derek, > > On 05/13/2015 10:34 PM, Derek White wrote: >> Hi Peter, >> >> I don't have smoking gun evidence that your change introduces a bug, >> but I have some concerns... > > I did have a concern too, ... > >> >> On 5/12/15 6:05 PM, Peter Levart wrote: >>> Hi Dmitry, >>> >>> You iterate the queue then, not the unfinalized list. That's more >>> logical. >>> >>> Holding the queue's lock may pause reference handler and finalizer >>> threads for the entire time of iteration. This can blow up the >>> application. Suppose one wants to diagnose the application because >>> he suspects that finalizer thread hardly keeps up with production of >>> finalizable instances. This can happen if the allocation rate of >>> finalizable objects is very high and/or finalize() methods are not >>> coded to be fast enough. Suppose the queue of Finalizer(s) builds up >>> gradually and is already holding several million objects. Now you >>> initiate the diagnostic command to print the queue. The iteration >>> over and grouping/counting of several millions of Finalizer(s) takes >>> some time. This blocks finalizer thread and reference handler >>> thread. But does not block the threads that allocate finalizable >>> objects. By the time the iteration is over, the JVM blows up and >>> application slows down to a halt doing GCs most of the time, getting >>> OOMEs, etc... >>> >>> It is possible to iterate the elements of the queue for diagnostic >>> purposes without holding a lock. The modification required to do >>> that is the following (havent tested this, but I think it should work): >>> >>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>> >> One issue to watch out for is the garbage collectors inspect the >> Reference.next field from C code directly (to distinguish active vs. >> pending, iterate over oops, etc.). You can search hotspot for >> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >> >> Your change makes "inactive" References superficially look like >> "enqueued" References. The GC code is rather subtle and I haven't yet >> seen a case where it would get confused by this change, but there >> could easily be something like that lurking in the GC code. > > ...but then I thought that GC can in no way treat a Reference > differently whether it is still enqueued in a ReferenceQueue or > already dequeued (inactive) - responsibility is already on the Java > side. Currently the definition of Reference.next is this: > > /* When active: NULL > * pending: this > * Enqueued: next reference in queue (or this if last) > * Inactive: this > */ > @SuppressWarnings("rawtypes") > Reference next; > > We see that, unless GC inspects all ReferenceQueue instances and scans > their lists too, the state of a Reference that is enqueued as last in > list is indistinguishable from the state of inactive Reference. So I > deduced that this distinction (enqueued/inactive) can't be established > solely on the value of .next field ( == this or != this)... > > But I should inspect the GC code too to build a better understanding > of that part of the story... > > Anyway. It turns out that there is already enough state in Reference > to destinguish between it being enqueued as last in list and already > dequeued (inactive) - the additional state is Reference.queue that > transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in > ReferenceQueue.reallyPoll. We need to just make sure the two fields > (r.next and r.queue) are assigned and read in correct order. This can > be achieved either by making Reference.next a volatile field or by a > couple of explicit fences: > > http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ > > The assignment of r.queue to ReferenceQueue.ENQUEUED already happens > before linking the reference into the queue's head in > ReferenceQueue.enqueue(): > > r.queue = ENQUEUED; > r.next = (head == null) ? r : head; > head = r; > > Both stores are volatile. This sounds a bit better. I don't have a good handle on the pros and cons of a volatile field over explicit fences via Unsafe in this case. Kim might jump in soon. I'd like to suggest an entirely less-clever solution. Since the problem is that forEach() might see inconsistent values for the queue and next fields of a Reference, but we don't want to grab a lock walking the whole queue, we could instead grab the lock as we look at each Reference (and do the "back-up" trick if we were interfered with). Of course this is slower than going lock-free, but it's not any more overhead than the ReferenceHandler thread or FinalizerThreads incur. The only benefit of this solution over the others is that it is less clever, and I'm not sure how much cleverness this problem deserves. Given that, and ranking the solutions as "lock" < (clever) "volatile" < "fence", I'd be happier with the "volatile" or "lock" solution if that is sufficient. - Derek >> >>> I also suggest the addition to the ReferenceQueue to be contained >>> (package-private) and as generic as possible. That's why I suggest >>> forEach iteration method with no concrete logic. >>> >>> It would be possible to encapsulate the entire logic into a special >>> package-private class (say java.lang.ref.DiagnosticCommands) with >>> static method(s) (printFinalizationQueue)... You could just expose a >>> package-private forEach static method from Finalizer and code the >>> rest in DiagnosticCommands. >> That's good for encapsulation. But I'm concerned that if "forEach" >> got exposed beyond careful use in DiagnosticCommands, and the >> Referents were leaked back into the heap, then we could get >> unexpected object resurrection after finalization. This isn't a bug >> on it's own - any finalizer might resurrect its object if not >> careful, but ordinarily /only/ the finalizer could resurrect the >> object. This could invalidate application invariants? > > Well, it all stays in the confines of package-private API - internal > to JDK. Any future additional use should be reviewed carefully. > Comments on the forEach() method should warn about that. > >> >> I agree that using a lock over the ReferenceQueue iteration opens up >> /another/ case of diagnostics affecting application behavior. And >> there are pathological scenarios where this gets severe. This is >> unfortunate but not uncommon. There is enough complication here that >> you should be sure that the fix for diagnostics performance doesn't >> introduce subtle bugs to the system in general. A change in this area >> should get a thorough review from both the runtime and GC sides. > > Is the webrev.02 proposed above more acceptable in that respect? It > does not introduce any logical changes to existing code. > >> >> Better yet, the reference handling code in GC and runtime should >> probably be thrown out and re-written, but that's another issue :-) > > I may have some proposals in that direction. Stay tuned. > > Regards, Peter > >> >> - Derek >> >>> Regards, Peter >>> >>> >>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>> Everybody, >>>> >>>> Updated version: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>> >>>> Now it iterates over queue and output result sorted by number of instances. >>>> >>>> -Dmitry >>>> >>>> On 2015-05-07 00:51, Derek White wrote: >>>>> Hi Dmitry, Staffan, >>>>> >>>>> Lots of good comments here. >>>>> >>>>> On the topic of what list should be printed out, I think we should focus >>>>> on objects waiting to be finalized - e.g. the contents of the >>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>> could add a summerizeQueue(TreeMap) method, or a >>>>> general iterator and lambda. >>>>> >>>>> A histogram of objects with finalize methods might also be interesting, >>>>> but you can get most of the same information from a heap histogram >>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>> >>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>> that runs after GC that sorts found References objects that need >>>>> processing into their respective ReferenceQueues). But locking the >>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>> >>>>> The sorting suggestion is a nice touch. >>>>> >>>>> - Derek White, GC team >>>>> >>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>> Hi Dmitry, Staffan, >>>>>> >>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>> Dmitry, >>>>>>> >>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>> well considering the changes to Finalizer. >>>>>>> >>>>>>> I?m a little worried about the potentially very large String that is >>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>> That would allow for outputting one line at a time. The output would >>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>> running out of memory. >>>>>> If the output is just a histogram of # of instances per class name, >>>>>> then it should not be too large, as there are not many different >>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>> finalize() method in the whole jdk/src tree). >>>>>> >>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>> acceptable for diagnostic output to impact performance and/or >>>>>> interfere with synchronization? >>>>>> >>>>>> It might be possible to scan the list without holding a lock for >>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>> Finalizer.next pointer to point back to itself: >>>>>> >>>>>> private void remove() { >>>>>> synchronized (lock) { >>>>>> if (unfinalized == this) { >>>>>> if (this.next != null) { >>>>>> unfinalized = this.next; >>>>>> } else { >>>>>> unfinalized = this.prev; >>>>>> } >>>>>> } >>>>>> if (this.next != null) { >>>>>> this.next.prev = this.prev; >>>>>> } >>>>>> if (this.prev != null) { >>>>>> this.prev.next = this.next; >>>>>> } >>>>>> // this.next = this; must not be set so that we can >>>>>> traverse the list unsynchronized >>>>>> this.prev = this; /* Indicates that this has been >>>>>> finalized */ >>>>>> } >>>>>> } >>>>>> >>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>> pointer could be used instead: >>>>>> >>>>>> private boolean hasBeenFinalized() { >>>>>> return (prev == this); >>>>>> } >>>>>> >>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>> >>>>>> private void add() { >>>>>> synchronized (lock) { >>>>>> if (unfinalized != null) { >>>>>> this.next = unfinalized; >>>>>> unfinalized.prev = this; >>>>>> } >>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>> // instance fully initialized >>>>>> // (in particular the 'next' pointer) >>>>>> U.storeFence(); >>>>>> unfinalized = this; >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> By doing these modifications, I think you can remove >>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>> >>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>> out? >>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>> instances pending processing by finalizer thread because their >>>>>> referents are eligible for finalization (they are not reachable any >>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>> instances for which their referents have not been finalized yet >>>>>> (including those that are still reachable and alive). The later serves >>>>>> two purposes: >>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>> - it is a source of unfinalized instances for >>>>>> running-finalizers-on-exit if requested by >>>>>> System.runFinalizersOnExit(true); >>>>>> >>>>>> So it really depends on what one would like to see. Showing the queue >>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>> list may be interesting if one wants to know how many live + >>>>>> finalization pending instances are there on the heap that override >>>>>> finalize() method. >>>>>> >>>>>> Regards, Peter >>>>>> >>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>> (see the old code again) would also make it easier to digest. >>>>>>> >>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>> permission() and limit access: >>>>>>> >>>>>>> static const JavaPermission permission() { >>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>> "monitor", NULL}; >>>>>>> return p; >>>>>>> } >>>>>>> >>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>> the finalizer queue? Hmm. >>>>>>> >>>>>>> Thanks, >>>>>>> /Staffan >>>>>>> >>>>>>> >>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>> wrote: >>>>>>>> >>>>>>>> Everyone, >>>>>>>> >>>>>>>> Please review the fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>> >>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>> >>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>> with count >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> -- >>>>>>>> Dmitry Samersoff >>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>> * I would love to change the world, but they won't give me the sources. >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek.white at oracle.com Fri May 15 22:59:52 2015 From: derek.white at oracle.com (Derek White) Date: Fri, 15 May 2015 18:59:52 -0400 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55567748.6020304@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> Message-ID: <55567A68.9020802@oracle.com> Hi Dmitry, Peter, I should read my email more frequently - I missed Dmitry's last webrev email. My comments on a preference for volatile over Unsafe are not a strong objection to using Unsafe fences. I just don't have enough experience with Unsafe fences yet to agree that this use is correct. While looking over this Reference code I found 3-6 really simple things that need cleaning up that have been there for years, so I'm not a big cheerleader for more complicated things here :-) - Derek On 5/15/15 6:46 PM, Derek White wrote: > Hi Peter, > > Some comments below... > > On 5/14/15 3:50 AM, Peter Levart wrote: >> Hi Derek, >> >> On 05/13/2015 10:34 PM, Derek White wrote: >>> Hi Peter, >>> >>> I don't have smoking gun evidence that your change introduces a bug, >>> but I have some concerns... >> >> I did have a concern too, ... >> >>> >>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>> Hi Dmitry, >>>> >>>> You iterate the queue then, not the unfinalized list. That's more >>>> logical. >>>> >>>> Holding the queue's lock may pause reference handler and finalizer >>>> threads for the entire time of iteration. This can blow up the >>>> application. Suppose one wants to diagnose the application because >>>> he suspects that finalizer thread hardly keeps up with production >>>> of finalizable instances. This can happen if the allocation rate of >>>> finalizable objects is very high and/or finalize() methods are not >>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>> up gradually and is already holding several million objects. Now >>>> you initiate the diagnostic command to print the queue. The >>>> iteration over and grouping/counting of several millions of >>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>> reference handler thread. But does not block the threads that >>>> allocate finalizable objects. By the time the iteration is over, >>>> the JVM blows up and application slows down to a halt doing GCs >>>> most of the time, getting OOMEs, etc... >>>> >>>> It is possible to iterate the elements of the queue for diagnostic >>>> purposes without holding a lock. The modification required to do >>>> that is the following (havent tested this, but I think it should work): >>>> >>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>> >>> One issue to watch out for is the garbage collectors inspect the >>> Reference.next field from C code directly (to distinguish active vs. >>> pending, iterate over oops, etc.). You can search hotspot for >>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>> >>> Your change makes "inactive" References superficially look like >>> "enqueued" References. The GC code is rather subtle and I haven't >>> yet seen a case where it would get confused by this change, but >>> there could easily be something like that lurking in the GC code. >> >> ...but then I thought that GC can in no way treat a Reference >> differently whether it is still enqueued in a ReferenceQueue or >> already dequeued (inactive) - responsibility is already on the Java >> side. Currently the definition of Reference.next is this: >> >> /* When active: NULL >> * pending: this >> * Enqueued: next reference in queue (or this if last) >> * Inactive: this >> */ >> @SuppressWarnings("rawtypes") >> Reference next; >> >> We see that, unless GC inspects all ReferenceQueue instances and >> scans their lists too, the state of a Reference that is enqueued as >> last in list is indistinguishable from the state of inactive >> Reference. So I deduced that this distinction (enqueued/inactive) >> can't be established solely on the value of .next field ( == this or >> != this)... >> >> But I should inspect the GC code too to build a better understanding >> of that part of the story... >> >> Anyway. It turns out that there is already enough state in Reference >> to destinguish between it being enqueued as last in list and already >> dequeued (inactive) - the additional state is Reference.queue that >> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >> ReferenceQueue.reallyPoll. We need to just make sure the two fields >> (r.next and r.queue) are assigned and read in correct order. This can >> be achieved either by making Reference.next a volatile field or by a >> couple of explicit fences: >> >> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >> >> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >> before linking the reference into the queue's head in >> ReferenceQueue.enqueue(): >> >> r.queue = ENQUEUED; >> r.next = (head == null) ? r : head; >> head = r; >> >> Both stores are volatile. > > This sounds a bit better. I don't have a good handle on the pros and > cons of a volatile field over explicit fences via Unsafe in this case. > Kim might jump in soon. > > I'd like to suggest an entirely less-clever solution. Since the > problem is that forEach() might see inconsistent values for the queue > and next fields of a Reference, but we don't want to grab a lock > walking the whole queue, we could instead grab the lock as we look at > each Reference (and do the "back-up" trick if we were interfered > with). Of course this is slower than going lock-free, but it's not any > more overhead than the ReferenceHandler thread or FinalizerThreads incur. > > The only benefit of this solution over the others is that it is less > clever, and I'm not sure how much cleverness this problem deserves. > Given that, and ranking the solutions as "lock" < (clever) "volatile" > < "fence", I'd be happier with the "volatile" or "lock" solution if > that is sufficient. > > - Derek >>> >>>> I also suggest the addition to the ReferenceQueue to be contained >>>> (package-private) and as generic as possible. That's why I suggest >>>> forEach iteration method with no concrete logic. >>>> >>>> It would be possible to encapsulate the entire logic into a special >>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>> static method(s) (printFinalizationQueue)... You could just expose >>>> a package-private forEach static method from Finalizer and code the >>>> rest in DiagnosticCommands. >>> That's good for encapsulation. But I'm concerned that if "forEach" >>> got exposed beyond careful use in DiagnosticCommands, and the >>> Referents were leaked back into the heap, then we could get >>> unexpected object resurrection after finalization. This isn't a bug >>> on it's own - any finalizer might resurrect its object if not >>> careful, but ordinarily /only/ the finalizer could resurrect the >>> object. This could invalidate application invariants? >> >> Well, it all stays in the confines of package-private API - internal >> to JDK. Any future additional use should be reviewed carefully. >> Comments on the forEach() method should warn about that. >> >>> >>> I agree that using a lock over the ReferenceQueue iteration opens up >>> /another/ case of diagnostics affecting application behavior. And >>> there are pathological scenarios where this gets severe. This is >>> unfortunate but not uncommon. There is enough complication here that >>> you should be sure that the fix for diagnostics performance doesn't >>> introduce subtle bugs to the system in general. A change in this >>> area should get a thorough review from both the runtime and GC sides. >> >> Is the webrev.02 proposed above more acceptable in that respect? It >> does not introduce any logical changes to existing code. >> >>> >>> Better yet, the reference handling code in GC and runtime should >>> probably be thrown out and re-written, but that's another issue :-) >> >> I may have some proposals in that direction. Stay tuned. >> >> Regards, Peter >> >>> >>> - Derek >>> >>>> Regards, Peter >>>> >>>> >>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>> Everybody, >>>>> >>>>> Updated version: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>> >>>>> Now it iterates over queue and output result sorted by number of instances. >>>>> >>>>> -Dmitry >>>>> >>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>> Hi Dmitry, Staffan, >>>>>> >>>>>> Lots of good comments here. >>>>>> >>>>>> On the topic of what list should be printed out, I think we should focus >>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>> general iterator and lambda. >>>>>> >>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>> but you can get most of the same information from a heap histogram >>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>> >>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>> that runs after GC that sorts found References objects that need >>>>>> processing into their respective ReferenceQueues). But locking the >>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>> >>>>>> The sorting suggestion is a nice touch. >>>>>> >>>>>> - Derek White, GC team >>>>>> >>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>> Hi Dmitry, Staffan, >>>>>>> >>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>> Dmitry, >>>>>>>> >>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>> well considering the changes to Finalizer. >>>>>>>> >>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>> running out of memory. >>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>> then it should not be too large, as there are not many different >>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>> finalize() method in the whole jdk/src tree). >>>>>>> >>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>> interfere with synchronization? >>>>>>> >>>>>>> It might be possible to scan the list without holding a lock for >>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>> Finalizer.next pointer to point back to itself: >>>>>>> >>>>>>> private void remove() { >>>>>>> synchronized (lock) { >>>>>>> if (unfinalized == this) { >>>>>>> if (this.next != null) { >>>>>>> unfinalized = this.next; >>>>>>> } else { >>>>>>> unfinalized = this.prev; >>>>>>> } >>>>>>> } >>>>>>> if (this.next != null) { >>>>>>> this.next.prev = this.prev; >>>>>>> } >>>>>>> if (this.prev != null) { >>>>>>> this.prev.next = this.next; >>>>>>> } >>>>>>> // this.next = this; must not be set so that we can >>>>>>> traverse the list unsynchronized >>>>>>> this.prev = this; /* Indicates that this has been >>>>>>> finalized */ >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>> pointer could be used instead: >>>>>>> >>>>>>> private boolean hasBeenFinalized() { >>>>>>> return (prev == this); >>>>>>> } >>>>>>> >>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>> >>>>>>> private void add() { >>>>>>> synchronized (lock) { >>>>>>> if (unfinalized != null) { >>>>>>> this.next = unfinalized; >>>>>>> unfinalized.prev = this; >>>>>>> } >>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>> // instance fully initialized >>>>>>> // (in particular the 'next' pointer) >>>>>>> U.storeFence(); >>>>>>> unfinalized = this; >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> By doing these modifications, I think you can remove >>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>> >>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>> out? >>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>> instances pending processing by finalizer thread because their >>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>> instances for which their referents have not been finalized yet >>>>>>> (including those that are still reachable and alive). The later serves >>>>>>> two purposes: >>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>> - it is a source of unfinalized instances for >>>>>>> running-finalizers-on-exit if requested by >>>>>>> System.runFinalizersOnExit(true); >>>>>>> >>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>> list may be interesting if one wants to know how many live + >>>>>>> finalization pending instances are there on the heap that override >>>>>>> finalize() method. >>>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>> >>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>> permission() and limit access: >>>>>>>> >>>>>>>> static const JavaPermission permission() { >>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>> "monitor", NULL}; >>>>>>>> return p; >>>>>>>> } >>>>>>>> >>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>> the finalizer queue? Hmm. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> /Staffan >>>>>>>> >>>>>>>> >>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Everyone, >>>>>>>>> >>>>>>>>> Please review the fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>> >>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>> >>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>> with count >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dmitry Samersoff >>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>> * I would love to change the world, but they won't give me the sources. >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri May 15 23:12:31 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Fri, 15 May 2015 16:12:31 -0700 Subject: RFR: 8080538 hprof does not work well with multiple agents on non-Solaris platforms Message-ID: Looking for a sponsor... The issue: hprof finds its location by looking for the first library containing an Agent_OnLoad symbol. This may not be libhprof.so. There is workaround logic for this on Solaris, but not on other platforms. Note the comment in src/jdk.hprof.agent/unix/native/libhprof/hprof_md.c:296: /* Just using &Agent_OnLoad will get the first external symbol with * this name in the first .so, which may not be libhprof.so. * On Solaris we can actually ask for the address of our Agent_OnLoad. */ addr = dlsym(RTLD_SELF, "Agent_OnLoad"); /* Just in case the above didn't work (missing linker patch?). */ if ( addr == NULL ) { addr = (void*)&Agent_OnLoad; } Instead of looking for Agent_OnLoad as a symbol, I suggest we just look for the symbol of the current method. Patch: http://cr.openjdk.java.net/~jmanson/8080538/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8080538 Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Sat May 16 07:35:39 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Sat, 16 May 2015 10:35:39 +0300 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55567A68.9020802@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> Message-ID: <5556F34B.5050701@oracle.com> Derek, Personally, I'm for volatile over explicit fence too. So I'll change the code if Peter don't have objections. -Dmitry On 2015-05-16 01:59, Derek White wrote: > Hi Dmitry, Peter, > > I should read my email more frequently - I missed Dmitry's last webrev > email. > > My comments on a preference for volatile over Unsafe are not a strong > objection to using Unsafe fences. I just don't have enough experience > with Unsafe fences yet to agree that this use is correct. > > While looking over this Reference code I found 3-6 really simple things > that need cleaning up that have been there for years, so I'm not a big > cheerleader for more complicated things here :-) > > - Derek > > On 5/15/15 6:46 PM, Derek White wrote: >> Hi Peter, >> >> Some comments below... >> >> On 5/14/15 3:50 AM, Peter Levart wrote: >>> Hi Derek, >>> >>> On 05/13/2015 10:34 PM, Derek White wrote: >>>> Hi Peter, >>>> >>>> I don't have smoking gun evidence that your change introduces a bug, >>>> but I have some concerns... >>> >>> I did have a concern too, ... >>> >>>> >>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>> Hi Dmitry, >>>>> >>>>> You iterate the queue then, not the unfinalized list. That's more >>>>> logical. >>>>> >>>>> Holding the queue's lock may pause reference handler and finalizer >>>>> threads for the entire time of iteration. This can blow up the >>>>> application. Suppose one wants to diagnose the application because >>>>> he suspects that finalizer thread hardly keeps up with production >>>>> of finalizable instances. This can happen if the allocation rate of >>>>> finalizable objects is very high and/or finalize() methods are not >>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>> up gradually and is already holding several million objects. Now >>>>> you initiate the diagnostic command to print the queue. The >>>>> iteration over and grouping/counting of several millions of >>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>> reference handler thread. But does not block the threads that >>>>> allocate finalizable objects. By the time the iteration is over, >>>>> the JVM blows up and application slows down to a halt doing GCs >>>>> most of the time, getting OOMEs, etc... >>>>> >>>>> It is possible to iterate the elements of the queue for diagnostic >>>>> purposes without holding a lock. The modification required to do >>>>> that is the following (havent tested this, but I think it should work): >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>> >>>> One issue to watch out for is the garbage collectors inspect the >>>> Reference.next field from C code directly (to distinguish active vs. >>>> pending, iterate over oops, etc.). You can search hotspot for >>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>> >>>> Your change makes "inactive" References superficially look like >>>> "enqueued" References. The GC code is rather subtle and I haven't >>>> yet seen a case where it would get confused by this change, but >>>> there could easily be something like that lurking in the GC code. >>> >>> ...but then I thought that GC can in no way treat a Reference >>> differently whether it is still enqueued in a ReferenceQueue or >>> already dequeued (inactive) - responsibility is already on the Java >>> side. Currently the definition of Reference.next is this: >>> >>> /* When active: NULL >>> * pending: this >>> * Enqueued: next reference in queue (or this if last) >>> * Inactive: this >>> */ >>> @SuppressWarnings("rawtypes") >>> Reference next; >>> >>> We see that, unless GC inspects all ReferenceQueue instances and >>> scans their lists too, the state of a Reference that is enqueued as >>> last in list is indistinguishable from the state of inactive >>> Reference. So I deduced that this distinction (enqueued/inactive) >>> can't be established solely on the value of .next field ( == this or >>> != this)... >>> >>> But I should inspect the GC code too to build a better understanding >>> of that part of the story... >>> >>> Anyway. It turns out that there is already enough state in Reference >>> to destinguish between it being enqueued as last in list and already >>> dequeued (inactive) - the additional state is Reference.queue that >>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>> (r.next and r.queue) are assigned and read in correct order. This can >>> be achieved either by making Reference.next a volatile field or by a >>> couple of explicit fences: >>> >>> >>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>> >>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>> before linking the reference into the queue's head in >>> ReferenceQueue.enqueue(): >>> >>> r.queue = ENQUEUED; >>> r.next = (head == null) ? r : head; >>> head = r; >>> >>> Both stores are volatile. >> >> This sounds a bit better. I don't have a good handle on the pros and >> cons of a volatile field over explicit fences via Unsafe in this case. >> Kim might jump in soon. >> >> I'd like to suggest an entirely less-clever solution. Since the >> problem is that forEach() might see inconsistent values for the queue >> and next fields of a Reference, but we don't want to grab a lock >> walking the whole queue, we could instead grab the lock as we look at >> each Reference (and do the "back-up" trick if we were interfered >> with). Of course this is slower than going lock-free, but it's not any >> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >> >> The only benefit of this solution over the others is that it is less >> clever, and I'm not sure how much cleverness this problem deserves. >> Given that, and ranking the solutions as "lock" < (clever) "volatile" >> < "fence", I'd be happier with the "volatile" or "lock" solution if >> that is sufficient. >> >> - Derek >>>> >>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>> (package-private) and as generic as possible. That's why I suggest >>>>> forEach iteration method with no concrete logic. >>>>> >>>>> It would be possible to encapsulate the entire logic into a special >>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>> a package-private forEach static method from Finalizer and code the >>>>> rest in DiagnosticCommands. >>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>> got exposed beyond careful use in DiagnosticCommands, and the >>>> Referents were leaked back into the heap, then we could get >>>> unexpected object resurrection after finalization. This isn't a bug >>>> on it's own - any finalizer might resurrect its object if not >>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>> object. This could invalidate application invariants? >>> >>> Well, it all stays in the confines of package-private API - internal >>> to JDK. Any future additional use should be reviewed carefully. >>> Comments on the forEach() method should warn about that. >>> >>>> >>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>> /another/ case of diagnostics affecting application behavior. And >>>> there are pathological scenarios where this gets severe. This is >>>> unfortunate but not uncommon. There is enough complication here that >>>> you should be sure that the fix for diagnostics performance doesn't >>>> introduce subtle bugs to the system in general. A change in this >>>> area should get a thorough review from both the runtime and GC sides. >>> >>> Is the webrev.02 proposed above more acceptable in that respect? It >>> does not introduce any logical changes to existing code. >>> >>>> >>>> Better yet, the reference handling code in GC and runtime should >>>> probably be thrown out and re-written, but that's another issue :-) >>> >>> I may have some proposals in that direction. Stay tuned. >>> >>> Regards, Peter >>> >>>> >>>> - Derek >>>> >>>>> Regards, Peter >>>>> >>>>> >>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>> Everybody, >>>>>> >>>>>> Updated version: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>> >>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>> Hi Dmitry, Staffan, >>>>>>> >>>>>>> Lots of good comments here. >>>>>>> >>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>> general iterator and lambda. >>>>>>> >>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>> but you can get most of the same information from a heap histogram >>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>> >>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>> that runs after GC that sorts found References objects that need >>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>> >>>>>>> The sorting suggestion is a nice touch. >>>>>>> >>>>>>> - Derek White, GC team >>>>>>> >>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>> Hi Dmitry, Staffan, >>>>>>>> >>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>> Dmitry, >>>>>>>>> >>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>> well considering the changes to Finalizer. >>>>>>>>> >>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>> running out of memory. >>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>> then it should not be too large, as there are not many different >>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>> >>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>> interfere with synchronization? >>>>>>>> >>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>> >>>>>>>> private void remove() { >>>>>>>> synchronized (lock) { >>>>>>>> if (unfinalized == this) { >>>>>>>> if (this.next != null) { >>>>>>>> unfinalized = this.next; >>>>>>>> } else { >>>>>>>> unfinalized = this.prev; >>>>>>>> } >>>>>>>> } >>>>>>>> if (this.next != null) { >>>>>>>> this.next.prev = this.prev; >>>>>>>> } >>>>>>>> if (this.prev != null) { >>>>>>>> this.prev.next = this.next; >>>>>>>> } >>>>>>>> // this.next = this; must not be set so that we can >>>>>>>> traverse the list unsynchronized >>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>> finalized */ >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>> pointer could be used instead: >>>>>>>> >>>>>>>> private boolean hasBeenFinalized() { >>>>>>>> return (prev == this); >>>>>>>> } >>>>>>>> >>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>> >>>>>>>> private void add() { >>>>>>>> synchronized (lock) { >>>>>>>> if (unfinalized != null) { >>>>>>>> this.next = unfinalized; >>>>>>>> unfinalized.prev = this; >>>>>>>> } >>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>> // instance fully initialized >>>>>>>> // (in particular the 'next' pointer) >>>>>>>> U.storeFence(); >>>>>>>> unfinalized = this; >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> By doing these modifications, I think you can remove >>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>> >>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>> out? >>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>> instances pending processing by finalizer thread because their >>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>> instances for which their referents have not been finalized yet >>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>> two purposes: >>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>> - it is a source of unfinalized instances for >>>>>>>> running-finalizers-on-exit if requested by >>>>>>>> System.runFinalizersOnExit(true); >>>>>>>> >>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>> finalization pending instances are there on the heap that override >>>>>>>> finalize() method. >>>>>>>> >>>>>>>> Regards, Peter >>>>>>>> >>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>> >>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>> permission() and limit access: >>>>>>>>> >>>>>>>>> static const JavaPermission permission() { >>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>> "monitor", NULL}; >>>>>>>>> return p; >>>>>>>>> } >>>>>>>>> >>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>> the finalizer queue? Hmm. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> /Staffan >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Everyone, >>>>>>>>>> >>>>>>>>>> Please review the fix: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>> >>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>> >>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>> with count >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Dmitry Samersoff >>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>> * I would love to change the world, but they won't give me the sources. >>>>> >>>> >>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jaroslav.bachorik at oracle.com Sat May 16 09:18:16 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Sat, 16 May 2015 11:18:16 +0200 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <555392EB.8020506@oracle.com> <555474C0.30108@oracle.com> Message-ID: <55570B58.7090403@oracle.com> On 15.5.2015 22:59, Martin Buchholz wrote: > You may submit, but: > > --- > > the @test is _NOT_ a javadoc comment - /* is more standard. > http://openjdk.java.net/jtreg/tag-spec.html#EXAMPLES > > -/* > +/** > * @test > * @bug 4530538 > * @summary Basic unit test of ThreadMXBean.getAllThreadIds() > @@ -32,9 +32,30 @@ > */ Will do. > > > > On Thu, May 14, 2015 at 3:11 AM, Jaroslav Bachorik > > wrote: > > On 13.5.2015 21:14, Martin Buchholz wrote: > > > > It's common for users to introduce parallelism in classloaders or > jar-loaders (we do this at google) which may cause arbitrary thread > fluctuations. That makes this particular API rather difficult > to test > robustly, especially if you are only testing against the spec. > > > Well, in such an environment this test makes absolutely no sense - > there is nothing deterministic in the thread counters we could > assert against. > > > Consider creating a private thread group just for the test and > restricting the count assertions to just threads in that thread group - > that should be reliable (even if other tests are running concurrently in > the same VM). I don't know how would you want to make this work. The ThreadMXBean counters are global per VM. Per thread group statistics are not supported. -JB- From peter.levart at gmail.com Sat May 16 12:38:09 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 16 May 2015 14:38:09 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5556F34B.5050701@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> Message-ID: <55573A31.7050306@gmail.com> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: > Derek, > > Personally, I'm for volatile over explicit fence too. > > So I'll change the code if Peter don't have objections. There are no objections. There's one unneeded volatile store to .next field then in enqueue(), but it is followed by another volatile store, so there shouldn't be overhead on Intel and SPARC at least. Regards, Peter > -Dmitry > > On 2015-05-16 01:59, Derek White wrote: >> Hi Dmitry, Peter, >> >> I should read my email more frequently - I missed Dmitry's last webrev >> email. >> >> My comments on a preference for volatile over Unsafe are not a strong >> objection to using Unsafe fences. I just don't have enough experience >> with Unsafe fences yet to agree that this use is correct. >> >> While looking over this Reference code I found 3-6 really simple things >> that need cleaning up that have been there for years, so I'm not a big >> cheerleader for more complicated things here :-) >> >> - Derek >> >> On 5/15/15 6:46 PM, Derek White wrote: >>> Hi Peter, >>> >>> Some comments below... >>> >>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>> Hi Derek, >>>> >>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>> Hi Peter, >>>>> >>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>> but I have some concerns... >>>> I did have a concern too, ... >>>> >>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>> logical. >>>>>> >>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>> threads for the entire time of iteration. This can blow up the >>>>>> application. Suppose one wants to diagnose the application because >>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>> up gradually and is already holding several million objects. Now >>>>>> you initiate the diagnostic command to print the queue. The >>>>>> iteration over and grouping/counting of several millions of >>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>> reference handler thread. But does not block the threads that >>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>> most of the time, getting OOMEs, etc... >>>>>> >>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>> purposes without holding a lock. The modification required to do >>>>>> that is the following (havent tested this, but I think it should work): >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>> >>>>> One issue to watch out for is the garbage collectors inspect the >>>>> Reference.next field from C code directly (to distinguish active vs. >>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>> >>>>> Your change makes "inactive" References superficially look like >>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>> yet seen a case where it would get confused by this change, but >>>>> there could easily be something like that lurking in the GC code. >>>> ...but then I thought that GC can in no way treat a Reference >>>> differently whether it is still enqueued in a ReferenceQueue or >>>> already dequeued (inactive) - responsibility is already on the Java >>>> side. Currently the definition of Reference.next is this: >>>> >>>> /* When active: NULL >>>> * pending: this >>>> * Enqueued: next reference in queue (or this if last) >>>> * Inactive: this >>>> */ >>>> @SuppressWarnings("rawtypes") >>>> Reference next; >>>> >>>> We see that, unless GC inspects all ReferenceQueue instances and >>>> scans their lists too, the state of a Reference that is enqueued as >>>> last in list is indistinguishable from the state of inactive >>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>> can't be established solely on the value of .next field ( == this or >>>> != this)... >>>> >>>> But I should inspect the GC code too to build a better understanding >>>> of that part of the story... >>>> >>>> Anyway. It turns out that there is already enough state in Reference >>>> to destinguish between it being enqueued as last in list and already >>>> dequeued (inactive) - the additional state is Reference.queue that >>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>> (r.next and r.queue) are assigned and read in correct order. This can >>>> be achieved either by making Reference.next a volatile field or by a >>>> couple of explicit fences: >>>> >>>> >>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>> >>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>> before linking the reference into the queue's head in >>>> ReferenceQueue.enqueue(): >>>> >>>> r.queue = ENQUEUED; >>>> r.next = (head == null) ? r : head; >>>> head = r; >>>> >>>> Both stores are volatile. >>> This sounds a bit better. I don't have a good handle on the pros and >>> cons of a volatile field over explicit fences via Unsafe in this case. >>> Kim might jump in soon. >>> >>> I'd like to suggest an entirely less-clever solution. Since the >>> problem is that forEach() might see inconsistent values for the queue >>> and next fields of a Reference, but we don't want to grab a lock >>> walking the whole queue, we could instead grab the lock as we look at >>> each Reference (and do the "back-up" trick if we were interfered >>> with). Of course this is slower than going lock-free, but it's not any >>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>> >>> The only benefit of this solution over the others is that it is less >>> clever, and I'm not sure how much cleverness this problem deserves. >>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>> that is sufficient. >>> >>> - Derek >>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>> forEach iteration method with no concrete logic. >>>>>> >>>>>> It would be possible to encapsulate the entire logic into a special >>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>> a package-private forEach static method from Finalizer and code the >>>>>> rest in DiagnosticCommands. >>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>> Referents were leaked back into the heap, then we could get >>>>> unexpected object resurrection after finalization. This isn't a bug >>>>> on it's own - any finalizer might resurrect its object if not >>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>> object. This could invalidate application invariants? >>>> Well, it all stays in the confines of package-private API - internal >>>> to JDK. Any future additional use should be reviewed carefully. >>>> Comments on the forEach() method should warn about that. >>>> >>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>> /another/ case of diagnostics affecting application behavior. And >>>>> there are pathological scenarios where this gets severe. This is >>>>> unfortunate but not uncommon. There is enough complication here that >>>>> you should be sure that the fix for diagnostics performance doesn't >>>>> introduce subtle bugs to the system in general. A change in this >>>>> area should get a thorough review from both the runtime and GC sides. >>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>> does not introduce any logical changes to existing code. >>>> >>>>> Better yet, the reference handling code in GC and runtime should >>>>> probably be thrown out and re-written, but that's another issue :-) >>>> I may have some proposals in that direction. Stay tuned. >>>> >>>> Regards, Peter >>>> >>>>> - Derek >>>>> >>>>>> Regards, Peter >>>>>> >>>>>> >>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>> Everybody, >>>>>>> >>>>>>> Updated version: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>> >>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>> Hi Dmitry, Staffan, >>>>>>>> >>>>>>>> Lots of good comments here. >>>>>>>> >>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>> general iterator and lambda. >>>>>>>> >>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>> >>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>> >>>>>>>> The sorting suggestion is a nice touch. >>>>>>>> >>>>>>>> - Derek White, GC team >>>>>>>> >>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>> >>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>> Dmitry, >>>>>>>>>> >>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>> >>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>> running out of memory. >>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>> >>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>> interfere with synchronization? >>>>>>>>> >>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>> >>>>>>>>> private void remove() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized == this) { >>>>>>>>> if (this.next != null) { >>>>>>>>> unfinalized = this.next; >>>>>>>>> } else { >>>>>>>>> unfinalized = this.prev; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> if (this.next != null) { >>>>>>>>> this.next.prev = this.prev; >>>>>>>>> } >>>>>>>>> if (this.prev != null) { >>>>>>>>> this.prev.next = this.next; >>>>>>>>> } >>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>> traverse the list unsynchronized >>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>> finalized */ >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>> pointer could be used instead: >>>>>>>>> >>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>> return (prev == this); >>>>>>>>> } >>>>>>>>> >>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>> >>>>>>>>> private void add() { >>>>>>>>> synchronized (lock) { >>>>>>>>> if (unfinalized != null) { >>>>>>>>> this.next = unfinalized; >>>>>>>>> unfinalized.prev = this; >>>>>>>>> } >>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>> // instance fully initialized >>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>> U.storeFence(); >>>>>>>>> unfinalized = this; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>> >>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>> out? >>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>> two purposes: >>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>> >>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>> finalize() method. >>>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>> >>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>> permission() and limit access: >>>>>>>>>> >>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>> "monitor", NULL}; >>>>>>>>>> return p; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> /Staffan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Everyone, >>>>>>>>>>> >>>>>>>>>>> Please review the fix: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>> >>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>> with count >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>> * I would love to change the world, but they won't give me the sources. -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Sat May 16 12:48:06 2015 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 16 May 2015 14:48:06 +0200 Subject: RFR(M,v3): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55573A31.7050306@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> Message-ID: <55573C86.8030807@gmail.com> On 05/16/2015 02:38 PM, Peter Levart wrote: > > > On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >> Derek, >> >> Personally, I'm for volatile over explicit fence too. >> >> So I'll change the code if Peter don't have objections. > > There are no objections. There's one unneeded volatile store to .next > field then in enqueue(), but it is followed by another volatile store, > so there shouldn't be overhead on Intel and SPARC at least. > > Regards, Peter If you make .next field volatile, then perhaps you could also cache it's value in reallyPoll() into a local variable, so that it is not read twice. Like this: private Reference reallyPoll() { /* Must hold lock */ Reference r = head; if (r != null) { Reference rn = r.next; // HERE !!! head = (rn == r) ? null : rn; // Unchecked due to the next field having a raw type in Reference r.queue = NULL; r.next = r; queueLength--; if (r instanceof FinalReference) { sun.misc.VM.addFinalRefCount(-1); } return r; } return null; } Peter > >> -Dmitry >> >> On 2015-05-16 01:59, Derek White wrote: >>> Hi Dmitry, Peter, >>> >>> I should read my email more frequently - I missed Dmitry's last webrev >>> email. >>> >>> My comments on a preference for volatile over Unsafe are not a strong >>> objection to using Unsafe fences. I just don't have enough experience >>> with Unsafe fences yet to agree that this use is correct. >>> >>> While looking over this Reference code I found 3-6 really simple things >>> that need cleaning up that have been there for years, so I'm not a big >>> cheerleader for more complicated things here :-) >>> >>> - Derek >>> >>> On 5/15/15 6:46 PM, Derek White wrote: >>>> Hi Peter, >>>> >>>> Some comments below... >>>> >>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>> Hi Derek, >>>>> >>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>> Hi Peter, >>>>>> >>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>> but I have some concerns... >>>>> I did have a concern too, ... >>>>> >>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>> Hi Dmitry, >>>>>>> >>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>> logical. >>>>>>> >>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>> application. Suppose one wants to diagnose the application because >>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>> up gradually and is already holding several million objects. Now >>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>> iteration over and grouping/counting of several millions of >>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>> reference handler thread. But does not block the threads that >>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>> most of the time, getting OOMEs, etc... >>>>>>> >>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>> purposes without holding a lock. The modification required to do >>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>> >>>>>>> >>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>> >>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>> >>>>>> Your change makes "inactive" References superficially look like >>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>> yet seen a case where it would get confused by this change, but >>>>>> there could easily be something like that lurking in the GC code. >>>>> ...but then I thought that GC can in no way treat a Reference >>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>> already dequeued (inactive) - responsibility is already on the Java >>>>> side. Currently the definition of Reference.next is this: >>>>> >>>>> /* When active: NULL >>>>> * pending: this >>>>> * Enqueued: next reference in queue (or this if last) >>>>> * Inactive: this >>>>> */ >>>>> @SuppressWarnings("rawtypes") >>>>> Reference next; >>>>> >>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>> scans their lists too, the state of a Reference that is enqueued as >>>>> last in list is indistinguishable from the state of inactive >>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>> can't be established solely on the value of .next field ( == this or >>>>> != this)... >>>>> >>>>> But I should inspect the GC code too to build a better understanding >>>>> of that part of the story... >>>>> >>>>> Anyway. It turns out that there is already enough state in Reference >>>>> to destinguish between it being enqueued as last in list and already >>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>> be achieved either by making Reference.next a volatile field or by a >>>>> couple of explicit fences: >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>> >>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>> before linking the reference into the queue's head in >>>>> ReferenceQueue.enqueue(): >>>>> >>>>> r.queue = ENQUEUED; >>>>> r.next = (head == null) ? r : head; >>>>> head = r; >>>>> >>>>> Both stores are volatile. >>>> This sounds a bit better. I don't have a good handle on the pros and >>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>> Kim might jump in soon. >>>> >>>> I'd like to suggest an entirely less-clever solution. Since the >>>> problem is that forEach() might see inconsistent values for the queue >>>> and next fields of a Reference, but we don't want to grab a lock >>>> walking the whole queue, we could instead grab the lock as we look at >>>> each Reference (and do the "back-up" trick if we were interfered >>>> with). Of course this is slower than going lock-free, but it's not any >>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>> >>>> The only benefit of this solution over the others is that it is less >>>> clever, and I'm not sure how much cleverness this problem deserves. >>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>> that is sufficient. >>>> >>>> - Derek >>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>> forEach iteration method with no concrete logic. >>>>>>> >>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>> rest in DiagnosticCommands. >>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>> Referents were leaked back into the heap, then we could get >>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>> on it's own - any finalizer might resurrect its object if not >>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>> object. This could invalidate application invariants? >>>>> Well, it all stays in the confines of package-private API - internal >>>>> to JDK. Any future additional use should be reviewed carefully. >>>>> Comments on the forEach() method should warn about that. >>>>> >>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>> there are pathological scenarios where this gets severe. This is >>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>> introduce subtle bugs to the system in general. A change in this >>>>>> area should get a thorough review from both the runtime and GC sides. >>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>> does not introduce any logical changes to existing code. >>>>> >>>>>> Better yet, the reference handling code in GC and runtime should >>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>> I may have some proposals in that direction. Stay tuned. >>>>> >>>>> Regards, Peter >>>>> >>>>>> - Derek >>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>> >>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>> Everybody, >>>>>>>> >>>>>>>> Updated version: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>> >>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>> >>>>>>>>> Lots of good comments here. >>>>>>>>> >>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>> general iterator and lambda. >>>>>>>>> >>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>> >>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>> >>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>> >>>>>>>>> - Derek White, GC team >>>>>>>>> >>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>> >>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>> Dmitry, >>>>>>>>>>> >>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>> >>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>> running out of memory. >>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>> >>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>> interfere with synchronization? >>>>>>>>>> >>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>> >>>>>>>>>> private void remove() { >>>>>>>>>> synchronized (lock) { >>>>>>>>>> if (unfinalized == this) { >>>>>>>>>> if (this.next != null) { >>>>>>>>>> unfinalized = this.next; >>>>>>>>>> } else { >>>>>>>>>> unfinalized = this.prev; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> if (this.next != null) { >>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>> } >>>>>>>>>> if (this.prev != null) { >>>>>>>>>> this.prev.next = this.next; >>>>>>>>>> } >>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>> traverse the list unsynchronized >>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>> finalized */ >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>> pointer could be used instead: >>>>>>>>>> >>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>> return (prev == this); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>> >>>>>>>>>> private void add() { >>>>>>>>>> synchronized (lock) { >>>>>>>>>> if (unfinalized != null) { >>>>>>>>>> this.next = unfinalized; >>>>>>>>>> unfinalized.prev = this; >>>>>>>>>> } >>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>> // instance fully initialized >>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>> U.storeFence(); >>>>>>>>>> unfinalized = this; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>> >>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>> out? >>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>> two purposes: >>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>> >>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>> finalize() method. >>>>>>>>>> >>>>>>>>>> Regards, Peter >>>>>>>>>> >>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>> >>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>> permission() and limit access: >>>>>>>>>>> >>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>> return p; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> /Staffan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Everyone, >>>>>>>>>>>> >>>>>>>>>>>> Please review the fix: >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>> >>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>> >>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>> with count >>>>>>>>>>>> >>>>>>>>>>>> -Dmitry >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Sat May 16 17:03:21 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Sat, 16 May 2015 20:03:21 +0300 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <555232C8.4000704@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> Message-ID: <55577859.5080406@oracle.com> Andrew, On 2015-05-12 20:05, Andrew Haley wrote: > http://cr.openjdk.java.net/~aph/8078521-4/ HSDB.java: 988: If you removed else part, you don't need to check CPU here at all, just leave a comment. LinuxAARCH64CFrame.java: 65 - it's better to add brackets around 2 * ADDRESS_SIZE 69,73 - please, remove space after bracket. -Dmitry -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From staffan.larsen at oracle.com Mon May 18 07:24:48 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 18 May 2015 09:24:48 +0200 Subject: RFR (XS) 8079644: memory stomping error with ResourceManagement and TestAgentStress.java In-Reply-To: <55553AA7.6000901@oracle.com> References: <55553AA7.6000901@oracle.com> Message-ID: <464DD67C-11FD-46CA-A5EA-568A4601EA46@oracle.com> Looks good! Thanks, /Staffan > On 15 maj 2015, at 02:15, serguei.spitsyn at oracle.com wrote: > > Please, review the jdk 9 fix for: > https://bugs.openjdk.java.net/browse/JDK-8079644 > > > 9 hotspot webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2015/hotspot/8079644-JVMTI-memstomp.1 > > > Summary: > > The cached class file structure must be deallocated instead of the cached class file bytes. > It is because the cached class file bytes array is a part of the cached class file structure. > > > Testing in progress: > In progress: nsk.redefine.testlist, JTREG com/sun/jdi tests, > ad-hog closed/serviceability/resource/TestAgentStress.java test from the bug report > > > Thanks, > Serguei From staffan.larsen at oracle.com Mon May 18 08:13:03 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 18 May 2015 10:13:03 +0200 Subject: RFR: 8080538 hprof does not work well with multiple agents on non-Solaris platforms In-Reply-To: References: Message-ID: This looks good, but given that hprof is going away [1] I?m not sure it?s worth the time to fix it. Anyway, I?m ok with the fix. Thanks, /Staffan [1] https://bugs.openjdk.java.net/browse/JDK-8046661 > On 16 maj 2015, at 01:12, Jeremy Manson wrote: > > Looking for a sponsor... > > The issue: hprof finds its location by looking for the first library containing an Agent_OnLoad symbol. This may not be libhprof.so. There is workaround logic for this on Solaris, but not on other platforms. > > Note the comment in src/jdk.hprof.agent/unix/native/libhprof/hprof_md.c:296: > > /* Just using &Agent_OnLoad will get the first external symbol with > * this name in the first .so, which may not be libhprof.so. > * On Solaris we can actually ask for the address of our Agent_OnLoad. > */ > addr = dlsym(RTLD_SELF, "Agent_OnLoad"); > /* Just in case the above didn't work (missing linker patch?). */ > if ( addr == NULL ) { > addr = (void*)&Agent_OnLoad; > } > > Instead of looking for Agent_OnLoad as a symbol, I suggest we just look for the symbol of the current method. > > Patch: > http://cr.openjdk.java.net/~jmanson/8080538/webrev.00/ > Bug: > https://bugs.openjdk.java.net/browse/JDK-8080538 > > Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.larsen at oracle.com Mon May 18 08:21:04 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 18 May 2015 10:21:04 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <5553702E.8050507@oracle.com> References: <5553702E.8050507@oracle.com> Message-ID: <26294D10-2A52-4061-8295-884336725373@oracle.com> Looks good, but why is the reverseBytes needed? 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); Thanks, /Staffan > On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: > > Please, review the following change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 > Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 > > The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. > > The perf counters show stale data and mislead the users. > > Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. > > In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. > > The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. > > Thanks, > > -JB- From staffan.larsen at oracle.com Mon May 18 08:22:38 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 18 May 2015 10:22:38 +0200 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: References: Message-ID: Looks like a good patch to me. /Staffan > On 14 maj 2015, at 18:12, Srinivas Ramakrishna wrote: > > https://bugs.openjdk.java.net/browse/JDK-8080345 > > > > On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna > wrote: > > With perm gen going away (and being replaced by metaspace) in JDK 8, it makes sense that the counter > sun.gc.policy.generations should be "2", rather than "3". However, in JDK 8 that counter still says 3. > As I understand, the intention was that this counter would allow you to (for example) know the range of > the sun.gc.generation.$num.* counters describing each of $num < sun.gc.policy.generations in the heap. > Recall that the erstwhile perm gen in JDK 7 used to be synonymous with sun.gc.generation.2, but the > JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. > > The fix is simple, and I can submit a patch. Is there an existing bug for this? > > thanks! > -- ramki > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Mon May 18 08:25:57 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 18 May 2015 10:25:57 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <26294D10-2A52-4061-8295-884336725373@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> Message-ID: <5559A215.5040608@oracle.com> On 18.5.2015 10:21, Staffan Larsen wrote: > Looks good, but why is the reverseBytes needed? > > 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. -JB- > > Thanks, > /Staffan > >> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >> >> Please, review the following change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >> >> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >> >> The perf counters show stale data and mislead the users. >> >> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >> >> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >> >> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >> >> Thanks, >> >> -JB- > From staffan.larsen at oracle.com Mon May 18 08:41:10 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 18 May 2015 10:41:10 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <5559A215.5040608@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> Message-ID: <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> > On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: > > On 18.5.2015 10:21, Staffan Larsen wrote: >> Looks good, but why is the reverseBytes needed? >> >> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); > > For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: private PerfCounter(String name, int type) { this.name = name; ByteBuffer bb = perf.createLong(name, type, U_None, 0L); bb.order(ByteOrder.nativeOrder()); this.lb = bb.asLongBuffer(); } /Staffan > > -JB- > >> >> Thanks, >> /Staffan >> >>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>> >>> Please, review the following change >>> >>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>> >>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>> >>> The perf counters show stale data and mislead the users. >>> >>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>> >>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>> >>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>> >>> Thanks, >>> >>> -JB- >> > From jaroslav.bachorik at oracle.com Mon May 18 08:53:13 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 18 May 2015 10:53:13 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> Message-ID: <5559A879.1000301@oracle.com> On 18.5.2015 10:41, Staffan Larsen wrote: > >> On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: >> >> On 18.5.2015 10:21, Staffan Larsen wrote: >>> Looks good, but why is the reverseBytes needed? >>> >>> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); >> >> For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. > > I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: > > private PerfCounter(String name, int type) { > this.name = name; > ByteBuffer bb = perf.createLong(name, type, U_None, 0L); > bb.order(ByteOrder.nativeOrder()); > this.lb = bb.asLongBuffer(); > } Aha. Ok. I will try it with this and get back to this review. -JB- > > /Staffan > >> >> -JB- >> >>> >>> Thanks, >>> /Staffan >>> >>>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>>> >>>> Please, review the following change >>>> >>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>>> >>>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>>> >>>> The perf counters show stale data and mislead the users. >>>> >>>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>>> >>>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>>> >>>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>>> >>>> Thanks, >>>> >>>> -JB- >>> >> > From cheleswer.sahu at oracle.com Mon May 18 09:59:54 2015 From: cheleswer.sahu at oracle.com (cheleswer sahu) Date: Mon, 18 May 2015 15:29:54 +0530 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <5555989C.9090803@oracle.com> References: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> <5555989C.9090803@oracle.com> Message-ID: <5559B81A.7000102@oracle.com> Hi, I have fixed the code and tested. It's working fine. Please review the changes. Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.02/ Regards, Cheleswer On 5/15/2015 12:26 PM, cheleswer sahu wrote: > Dear Dan & Dmitry, > Thanks for pointing out the security vulnerability in the fix and for > your precise review. I am also agree with Dmitry's fix. I will fix the > code and resend the review request. > > Regards, > Cheleswer > > On 5/14/2015 9:46 PM, Gerald Thornbrugh wrote: >> Hi Dan, >> >> When Cheleswer and I discussed this fix my interpretation had a >> slightly different goal: >> >> Prior to the initial security fix any user could execute "jps" and >> get the user names associated >> with other user's perf data (i.e. the call to get_user_name_slow() >> would succeed.). My initial >> thought was that this was a regression for all users not just "root" >> and this goal led to this fix. >> At the time I did not see this as a security vulnerability, your >> review has changed my mind. >> >> I agree that Dmitry's fix is a more secure fix for the issue and I >> think we should use it. >> >> Let me know if you have any questions. >> >> Thanks! >> >> Jerry >> >>> Hi, >>> Please review the code changes for >>> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >>> tested JDK9 with fix successfully. As I do not have an account for >>> OpenJDK, >>> David Buck will push the fix into jdk9/hs-rt/. >>> >>> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >>> >>> Regards, >>> Cheleswer >> Cheleswer, >> >> Sorry for the lengthy review, but since this is security related, >> I have to be complete. >> >> The goal: Add a policy by-pass for the 'root' user in order to >> fix the regression in jps(1) behavior. >> >> The core of this policy by-pass is the change to this function: >> >> 205 static bool is_statbuf_secure(struct stat *statp, int mode) { >> 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { >> 207 // The path represents a link or some non-directory file >> type, >> 208 // which is not what we expected. Declare it insecure. >> 209 // >> 210 return false; >> 211 } >> 212 // If the directory is going to be opened readonly, we consider >> this as secure operation >> 213 // we do not need to do any more checks. >> 214 // >> 215 if ((mode & O_ACCMODE) == O_RDONLY) { >> 216 return true; >> 217 } >> 218 // We have an existing directory, check if the permissions >> are safe. >> 219 // >> 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { >> 221 // The directory is open for writing and could be subjected >> 222 // to a symlink or a hard link attack. Declare it insecure. >> 223 // >> 224 return false; >> 225 } >> 226 // See if the uid of the directory matches the effective uid of >> the process. >> 227 // >> 228 if (statp->st_uid != geteuid()) { >> 229 // The directory was not created by this user, declare it >> insecure. >> 230 // >> 231 return false; >> 232 } >> 233 return true; >> 234 } >> >> Lines 212-217 are added which allows a caller that passes in O_RDONLY >> to by-pass the security checks on lines 220-225 and 228-232. This >> implementation is using an attribute of _how_ the data is accessed >> to override security policy instead of an attribute of _who_ is >> accessing the data. >> >> Here are the code paths that access the modified policy code: >> >> is_statbuf_secure() is called by: >> >> - is_directory_secure() >> - is_dirfd_secure() >> >> is_directory_secure() is called by: >> >> - get_user_name_slow() with O_RDONLY >> - make_user_tmp_dir() with O_RDWR >> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >> >> is_dirfd_secure() is called by: >> >> - open_directory_secure() with a mode parameter >> >> open_directory_secure() is called by: >> >> - open_directory_secure_cwd() with O_RDWR >> - get_user_name_slow() with O_RDONLY >> >> Only the code paths that specify O_RDWR make use of >> the new policy by-pass code so it looks like only >> >> - get_user_name_slow() with O_RDONLY >> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >> >> are interesting. >> >> The new security policy by-pass will allow get_user_name_slow(): >> >> - to process directory entries in a directory that is writable >> which makes this use subject to a symlink or hard link attack. >> - to process directory entries in a directory that the calling >> user does not own; the intent of the policy by-pass is to >> allow this for the 'root' user, but this implementation >> allows the by-pass for any user. >> >> It looks like the get_user_name_slow() code is written safely >> enough such that any symlink or hard link attack should not >> cause any issues. >> >> The new policy by-pass will allow any user to determine the >> user name associated with VMs owned by another user. This is >> a broader policy by-pass than was intended. >> >> >> The new security policy by-pass will allow mmap_attach_shared(): >> >> - to process directory entries in a directory that is writable >> which makes this use subject to a symlink or hard link attack. >> - to process directory entries in a directory that the calling >> user does not own; the intent of the policy by-pass is to >> allow this for the 'root' user, but this implementation allows >> the by-pass for any user. >> >> The mmap_attach_shared() code protects itself from a symlink >> attack by including the 'O_NOFOLLOW' flag when opening the >> PerfData file and it protects itself from a hardlink attack by >> checking the hard link count after opening the file. It does >> not protect itself against being handed a corrupted file or >> even a very large file that would cause an OOM when the VM >> tries to map what is supposed to be a PerfData file. >> >> The new policy by-pass will allow any user to access the >> PerfData file associated with VMs owned by another user. This >> is a broader policy by-pass than was intended. >> >> >> Summary: >> >> This implementation of the new security policy by-pass is using >> an attribute of _how_ the data is accessed to override security >> policy instead of an attribute of _who_ is accessing the data. >> This allows the VM to be exposed to some of the attacks that >> the following fix was designed to prevent: >> >> JDK-8050807 Better performing performance data handling >> >> Dmitry's response to the code review provides a solution that >> is based on who is accessing the data and that solution or >> one like it should be considered. >> >> Again, sorry for the lengthy review. >> >> Dan > From dmitry.samersoff at oracle.com Mon May 18 11:39:21 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Mon, 18 May 2015 14:39:21 +0300 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <5559B81A.7000102@oracle.com> References: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> <5555989C.9090803@oracle.com> <5559B81A.7000102@oracle.com> Message-ID: <5559CF69.9020804@oracle.com> Cheleswer, 1. perfMemory_linux.cpp:222 missed space in !=0) 2. We have couple of other perfMemory_*.cpp files that have to be updated as well. -Dmitry On 2015-05-18 12:59, cheleswer sahu wrote: > Hi, > I have fixed the code and tested. It's working fine. Please review the > changes. > Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.02/ > > Regards, > Cheleswer > > On 5/15/2015 12:26 PM, cheleswer sahu wrote: >> Dear Dan & Dmitry, >> Thanks for pointing out the security vulnerability in the fix and for >> your precise review. I am also agree with Dmitry's fix. I will fix the >> code and resend the review request. >> >> Regards, >> Cheleswer >> >> On 5/14/2015 9:46 PM, Gerald Thornbrugh wrote: >>> Hi Dan, >>> >>> When Cheleswer and I discussed this fix my interpretation had a >>> slightly different goal: >>> >>> Prior to the initial security fix any user could execute "jps" and >>> get the user names associated >>> with other user's perf data (i.e. the call to get_user_name_slow() >>> would succeed.). My initial >>> thought was that this was a regression for all users not just "root" >>> and this goal led to this fix. >>> At the time I did not see this as a security vulnerability, your >>> review has changed my mind. >>> >>> I agree that Dmitry's fix is a more secure fix for the issue and I >>> think we should use it. >>> >>> Let me know if you have any questions. >>> >>> Thanks! >>> >>> Jerry >>> >>>> Hi, >>>> Please review the code changes for >>>> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >>>> tested JDK9 with fix successfully. As I do not have an account for >>>> OpenJDK, >>>> David Buck will push the fix into jdk9/hs-rt/. >>>> >>>> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >>>> >>>> Regards, >>>> Cheleswer >>> Cheleswer, >>> >>> Sorry for the lengthy review, but since this is security related, >>> I have to be complete. >>> >>> The goal: Add a policy by-pass for the 'root' user in order to >>> fix the regression in jps(1) behavior. >>> >>> The core of this policy by-pass is the change to this function: >>> >>> 205 static bool is_statbuf_secure(struct stat *statp, int mode) { >>> 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { >>> 207 // The path represents a link or some non-directory file >>> type, >>> 208 // which is not what we expected. Declare it insecure. >>> 209 // >>> 210 return false; >>> 211 } >>> 212 // If the directory is going to be opened readonly, we consider >>> this as secure operation >>> 213 // we do not need to do any more checks. >>> 214 // >>> 215 if ((mode & O_ACCMODE) == O_RDONLY) { >>> 216 return true; >>> 217 } >>> 218 // We have an existing directory, check if the permissions >>> are safe. >>> 219 // >>> 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { >>> 221 // The directory is open for writing and could be subjected >>> 222 // to a symlink or a hard link attack. Declare it insecure. >>> 223 // >>> 224 return false; >>> 225 } >>> 226 // See if the uid of the directory matches the effective uid of >>> the process. >>> 227 // >>> 228 if (statp->st_uid != geteuid()) { >>> 229 // The directory was not created by this user, declare it >>> insecure. >>> 230 // >>> 231 return false; >>> 232 } >>> 233 return true; >>> 234 } >>> >>> Lines 212-217 are added which allows a caller that passes in O_RDONLY >>> to by-pass the security checks on lines 220-225 and 228-232. This >>> implementation is using an attribute of _how_ the data is accessed >>> to override security policy instead of an attribute of _who_ is >>> accessing the data. >>> >>> Here are the code paths that access the modified policy code: >>> >>> is_statbuf_secure() is called by: >>> >>> - is_directory_secure() >>> - is_dirfd_secure() >>> >>> is_directory_secure() is called by: >>> >>> - get_user_name_slow() with O_RDONLY >>> - make_user_tmp_dir() with O_RDWR >>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>> >>> is_dirfd_secure() is called by: >>> >>> - open_directory_secure() with a mode parameter >>> >>> open_directory_secure() is called by: >>> >>> - open_directory_secure_cwd() with O_RDWR >>> - get_user_name_slow() with O_RDONLY >>> >>> Only the code paths that specify O_RDWR make use of >>> the new policy by-pass code so it looks like only >>> >>> - get_user_name_slow() with O_RDONLY >>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>> >>> are interesting. >>> >>> The new security policy by-pass will allow get_user_name_slow(): >>> >>> - to process directory entries in a directory that is writable >>> which makes this use subject to a symlink or hard link attack. >>> - to process directory entries in a directory that the calling >>> user does not own; the intent of the policy by-pass is to >>> allow this for the 'root' user, but this implementation >>> allows the by-pass for any user. >>> >>> It looks like the get_user_name_slow() code is written safely >>> enough such that any symlink or hard link attack should not >>> cause any issues. >>> >>> The new policy by-pass will allow any user to determine the >>> user name associated with VMs owned by another user. This is >>> a broader policy by-pass than was intended. >>> >>> >>> The new security policy by-pass will allow mmap_attach_shared(): >>> >>> - to process directory entries in a directory that is writable >>> which makes this use subject to a symlink or hard link attack. >>> - to process directory entries in a directory that the calling >>> user does not own; the intent of the policy by-pass is to >>> allow this for the 'root' user, but this implementation allows >>> the by-pass for any user. >>> >>> The mmap_attach_shared() code protects itself from a symlink >>> attack by including the 'O_NOFOLLOW' flag when opening the >>> PerfData file and it protects itself from a hardlink attack by >>> checking the hard link count after opening the file. It does >>> not protect itself against being handed a corrupted file or >>> even a very large file that would cause an OOM when the VM >>> tries to map what is supposed to be a PerfData file. >>> >>> The new policy by-pass will allow any user to access the >>> PerfData file associated with VMs owned by another user. This >>> is a broader policy by-pass than was intended. >>> >>> >>> Summary: >>> >>> This implementation of the new security policy by-pass is using >>> an attribute of _how_ the data is accessed to override security >>> policy instead of an attribute of _who_ is accessing the data. >>> This allows the VM to be exposed to some of the attacks that >>> the following fix was designed to prevent: >>> >>> JDK-8050807 Better performing performance data handling >>> >>> Dmitry's response to the code review provides a solution that >>> is based on who is accessing the data and that solution or >>> one like it should be considered. >>> >>> Again, sorry for the lengthy review. >>> >>> Dan >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Mon May 18 12:17:31 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Mon, 18 May 2015 15:17:31 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55573C86.8030807@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> Message-ID: <5559D85B.2050500@oracle.com> Everyone, Please review updated version of the fix: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ Most important part of the fix provided by Peter Levart, so all credentials belongs to him. -Dmitry On 2015-05-16 15:48, Peter Levart wrote: > > > On 05/16/2015 02:38 PM, Peter Levart wrote: >> >> >> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >>> Derek, >>> >>> Personally, I'm for volatile over explicit fence too. >>> >>> So I'll change the code if Peter don't have objections. >> >> There are no objections. There's one unneeded volatile store to .next >> field then in enqueue(), but it is followed by another volatile store, >> so there shouldn't be overhead on Intel and SPARC at least. >> >> Regards, Peter > > If you make .next field volatile, then perhaps you could also cache it's > value in reallyPoll() into a local variable, so that it is not read > twice. Like this: > > private Reference reallyPoll() { /* Must hold lock */ > Reference r = head; > if (r != null) { > Reference rn = r.next; // HERE !!! > head = (rn == r) ? > null : > rn; // Unchecked due to the next field having a raw type > in Reference > r.queue = NULL; > r.next = r; > queueLength--; > if (r instanceof FinalReference) { > sun.misc.VM.addFinalRefCount(-1); > } > return r; > } > return null; > } > > > > Peter > > >> >>> -Dmitry >>> >>> On 2015-05-16 01:59, Derek White wrote: >>>> Hi Dmitry, Peter, >>>> >>>> I should read my email more frequently - I missed Dmitry's last webrev >>>> email. >>>> >>>> My comments on a preference for volatile over Unsafe are not a strong >>>> objection to using Unsafe fences. I just don't have enough experience >>>> with Unsafe fences yet to agree that this use is correct. >>>> >>>> While looking over this Reference code I found 3-6 really simple things >>>> that need cleaning up that have been there for years, so I'm not a big >>>> cheerleader for more complicated things here :-) >>>> >>>> - Derek >>>> >>>> On 5/15/15 6:46 PM, Derek White wrote: >>>>> Hi Peter, >>>>> >>>>> Some comments below... >>>>> >>>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>>> Hi Derek, >>>>>> >>>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>>> Hi Peter, >>>>>>> >>>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>>> but I have some concerns... >>>>>> I did have a concern too, ... >>>>>> >>>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>>> logical. >>>>>>>> >>>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>>> application. Suppose one wants to diagnose the application because >>>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>>> up gradually and is already holding several million objects. Now >>>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>>> iteration over and grouping/counting of several millions of >>>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>>> reference handler thread. But does not block the threads that >>>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>>> most of the time, getting OOMEs, etc... >>>>>>>> >>>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>>> purposes without holding a lock. The modification required to do >>>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>>> >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>>> >>>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>>> >>>>>>> Your change makes "inactive" References superficially look like >>>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>>> yet seen a case where it would get confused by this change, but >>>>>>> there could easily be something like that lurking in the GC code. >>>>>> ...but then I thought that GC can in no way treat a Reference >>>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>>> already dequeued (inactive) - responsibility is already on the Java >>>>>> side. Currently the definition of Reference.next is this: >>>>>> >>>>>> /* When active: NULL >>>>>> * pending: this >>>>>> * Enqueued: next reference in queue (or this if last) >>>>>> * Inactive: this >>>>>> */ >>>>>> @SuppressWarnings("rawtypes") >>>>>> Reference next; >>>>>> >>>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>>> scans their lists too, the state of a Reference that is enqueued as >>>>>> last in list is indistinguishable from the state of inactive >>>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>>> can't be established solely on the value of .next field ( == this or >>>>>> != this)... >>>>>> >>>>>> But I should inspect the GC code too to build a better understanding >>>>>> of that part of the story... >>>>>> >>>>>> Anyway. It turns out that there is already enough state in Reference >>>>>> to destinguish between it being enqueued as last in list and already >>>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>>> be achieved either by making Reference.next a volatile field or by a >>>>>> couple of explicit fences: >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>>> >>>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>>> before linking the reference into the queue's head in >>>>>> ReferenceQueue.enqueue(): >>>>>> >>>>>> r.queue = ENQUEUED; >>>>>> r.next = (head == null) ? r : head; >>>>>> head = r; >>>>>> >>>>>> Both stores are volatile. >>>>> This sounds a bit better. I don't have a good handle on the pros and >>>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>>> Kim might jump in soon. >>>>> >>>>> I'd like to suggest an entirely less-clever solution. Since the >>>>> problem is that forEach() might see inconsistent values for the queue >>>>> and next fields of a Reference, but we don't want to grab a lock >>>>> walking the whole queue, we could instead grab the lock as we look at >>>>> each Reference (and do the "back-up" trick if we were interfered >>>>> with). Of course this is slower than going lock-free, but it's not any >>>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>>> >>>>> The only benefit of this solution over the others is that it is less >>>>> clever, and I'm not sure how much cleverness this problem deserves. >>>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>>> that is sufficient. >>>>> >>>>> - Derek >>>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>>> forEach iteration method with no concrete logic. >>>>>>>> >>>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>>> rest in DiagnosticCommands. >>>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>>> Referents were leaked back into the heap, then we could get >>>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>>> on it's own - any finalizer might resurrect its object if not >>>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>>> object. This could invalidate application invariants? >>>>>> Well, it all stays in the confines of package-private API - internal >>>>>> to JDK. Any future additional use should be reviewed carefully. >>>>>> Comments on the forEach() method should warn about that. >>>>>> >>>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>>> there are pathological scenarios where this gets severe. This is >>>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>>> introduce subtle bugs to the system in general. A change in this >>>>>>> area should get a thorough review from both the runtime and GC sides. >>>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>>> does not introduce any logical changes to existing code. >>>>>> >>>>>>> Better yet, the reference handling code in GC and runtime should >>>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>>> I may have some proposals in that direction. Stay tuned. >>>>>> >>>>>> Regards, Peter >>>>>> >>>>>>> - Derek >>>>>>> >>>>>>>> Regards, Peter >>>>>>>> >>>>>>>> >>>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>>> Everybody, >>>>>>>>> >>>>>>>>> Updated version: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>>> >>>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>> >>>>>>>>>> Lots of good comments here. >>>>>>>>>> >>>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>>> general iterator and lambda. >>>>>>>>>> >>>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>>> >>>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>>> >>>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>>> >>>>>>>>>> - Derek White, GC team >>>>>>>>>> >>>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>> >>>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>>> Dmitry, >>>>>>>>>>>> >>>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>>> >>>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>>> running out of memory. >>>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>>> >>>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>>> interfere with synchronization? >>>>>>>>>>> >>>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>>> >>>>>>>>>>> private void remove() { >>>>>>>>>>> synchronized (lock) { >>>>>>>>>>> if (unfinalized == this) { >>>>>>>>>>> if (this.next != null) { >>>>>>>>>>> unfinalized = this.next; >>>>>>>>>>> } else { >>>>>>>>>>> unfinalized = this.prev; >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> if (this.next != null) { >>>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>>> } >>>>>>>>>>> if (this.prev != null) { >>>>>>>>>>> this.prev.next = this.next; >>>>>>>>>>> } >>>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>>> traverse the list unsynchronized >>>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>>> finalized */ >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>>> pointer could be used instead: >>>>>>>>>>> >>>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>>> return (prev == this); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>>> >>>>>>>>>>> private void add() { >>>>>>>>>>> synchronized (lock) { >>>>>>>>>>> if (unfinalized != null) { >>>>>>>>>>> this.next = unfinalized; >>>>>>>>>>> unfinalized.prev = this; >>>>>>>>>>> } >>>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>>> // instance fully initialized >>>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>>> U.storeFence(); >>>>>>>>>>> unfinalized = this; >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>>> >>>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>>> out? >>>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>>> two purposes: >>>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>>> >>>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>>> finalize() method. >>>>>>>>>>> >>>>>>>>>>> Regards, Peter >>>>>>>>>>> >>>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>>> >>>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>>> permission() and limit access: >>>>>>>>>>>> >>>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>>> return p; >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> /Staffan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> Please review the fix: >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>>> >>>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>>> >>>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>>> with count >>>>>>>>>>>>> >>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From serguei.spitsyn at oracle.com Mon May 18 16:59:08 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 18 May 2015 09:59:08 -0700 Subject: RFR (XS) 8079644: memory stomping error with ResourceManagement and TestAgentStress.java In-Reply-To: <464DD67C-11FD-46CA-A5EA-568A4601EA46@oracle.com> References: <55553AA7.6000901@oracle.com> <464DD67C-11FD-46CA-A5EA-568A4601EA46@oracle.com> Message-ID: <555A1A5C.1020005@oracle.com> Thanks a lot, Staffan! Serguei On 5/18/15 12:24 AM, Staffan Larsen wrote: > Looks good! > > Thanks, > /Staffan > >> On 15 maj 2015, at 02:15, serguei.spitsyn at oracle.com wrote: >> >> Please, review the jdk 9 fix for: >> https://bugs.openjdk.java.net/browse/JDK-8079644 >> >> >> 9 hotspot webrev: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2015/hotspot/8079644-JVMTI-memstomp.1 >> >> >> Summary: >> >> The cached class file structure must be deallocated instead of the cached class file bytes. >> It is because the cached class file bytes array is a part of the cached class file structure. >> >> >> Testing in progress: >> In progress: nsk.redefine.testlist, JTREG com/sun/jdi tests, >> ad-hog closed/serviceability/resource/TestAgentStress.java test from the bug report >> >> >> Thanks, >> Serguei From jaroslav.bachorik at oracle.com Mon May 18 17:23:16 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 18 May 2015 19:23:16 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> Message-ID: <555A2004.1030702@oracle.com> On 18.5.2015 10:41, Staffan Larsen wrote: > >> On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: >> >> On 18.5.2015 10:21, Staffan Larsen wrote: >>> Looks good, but why is the reverseBytes needed? >>> >>> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); >> >> For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. > > I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: > > private PerfCounter(String name, int type) { > this.name = name; > ByteBuffer bb = perf.createLong(name, type, U_None, 0L); > bb.order(ByteOrder.nativeOrder()); > this.lb = bb.asLongBuffer(); > } Done. Please, see http://cr.openjdk.java.net/~jbachorik/8075926/webrev.01 for the updated webrev. -JB- > > /Staffan > >> >> -JB- >> >>> >>> Thanks, >>> /Staffan >>> >>>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>>> >>>> Please, review the following change >>>> >>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>>> >>>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>>> >>>> The perf counters show stale data and mislead the users. >>>> >>>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>>> >>>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>>> >>>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>>> >>>> Thanks, >>>> >>>> -JB- >>> >> > From martinrb at google.com Mon May 18 20:57:57 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 18 May 2015 13:57:57 -0700 Subject: RFR: 8080538 hprof does not work well with multiple agents on non-Solaris platforms In-Reply-To: References: Message-ID: Pushed to jdk9 dev. On Mon, May 18, 2015 at 1:13 AM, Staffan Larsen wrote: > This looks good, but given that hprof is going away [1] I?m not sure it?s > worth the time to fix it. Anyway, I?m ok with the fix. > > Thanks, > /Staffan > > [1] https://bugs.openjdk.java.net/browse/JDK-8046661 > > > On 16 maj 2015, at 01:12, Jeremy Manson wrote: > > Looking for a sponsor... > > The issue: hprof finds its location by looking for the first library > containing an Agent_OnLoad symbol. This may not be libhprof.so. There is > workaround logic for this on Solaris, but not on other platforms. > > Note the comment in > src/jdk.hprof.agent/unix/native/libhprof/hprof_md.c:296: > > /* Just using &Agent_OnLoad will get the first external symbol with > * this name in the first .so, which may not be libhprof.so. > * On Solaris we can actually ask for the address of our Agent_OnLoad. > */ > addr = dlsym(RTLD_SELF, "Agent_OnLoad"); > /* Just in case the above didn't work (missing linker patch?). */ > if ( addr == NULL ) { > addr = (void*)&Agent_OnLoad; > } > > Instead of looking for Agent_OnLoad as a symbol, I suggest we just look > for the symbol of the current method. > > Patch: > http://cr.openjdk.java.net/~jmanson/8080538/webrev.00/ > Bug: > https://bugs.openjdk.java.net/browse/JDK-8080538 > > Jeremy > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinrb at google.com Mon May 18 21:04:09 2015 From: martinrb at google.com (Martin Buchholz) Date: Mon, 18 May 2015 14:04:09 -0700 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <55570B58.7090403@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <555392EB.8020506@oracle.com> <555474C0.30108@oracle.com> <55570B58.7090403@oracle.com> Message-ID: On Sat, May 16, 2015 at 2:18 AM, Jaroslav Bachorik < jaroslav.bachorik at oracle.com> wrote: > On 15.5.2015 22:59, Martin Buchholz wrote: > >> >> Consider creating a private thread group just for the test and >> restricting the count assertions to just threads in that thread group - >> that should be reliable (even if other tests are running concurrently in >> the same VM). >> > > I don't know how would you want to make this work. The ThreadMXBean > counters are global per VM. Per thread group statistics are not supported. Well, yes, this API is inherently hard to test, but if you filtered the returned data to restrict to just one thread group, where you have control over all threads in the thread group, then you should be able to make 100% reliable assertions about the completeness. You will not be able to prove that the returned information from e.g. getThreadCount is accurate, but it should at least be >= the count of live threads that you can control. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Mon May 18 22:54:28 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 18 May 2015 16:54:28 -0600 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <5554C68B.5090702@oracle.com> References: <55546A48.6050108@oracle.com> <55546C24.6050701@oracle.com> <5554C68B.5090702@oracle.com> Message-ID: <555A6DA4.9050803@oracle.com> Just for the record... Fixing a critical typo below... On 5/14/15 10:00 AM, Daniel D. Daugherty wrote: > On 5/14/15 3:34 AM, cheleswer sahu wrote: >> Hi, >> Please review the code changes for >> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >> tested JDK9 with fix successfully. As I do not have an account for >> OpenJDK, >> David Buck will push the fix into jdk9/hs-rt/. >> >> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >> >> Regards, >> Cheleswer > > Cheleswer, > > Sorry for the lengthy review, but since this is security related, > I have to be complete. > > The goal: Add a policy by-pass for the 'root' user in order to > fix the regression in jps(1) behavior. > > The core of this policy by-pass is the change to this function: > > 205 static bool is_statbuf_secure(struct stat *statp, int mode) { > 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { > 207 // The path represents a link or some non-directory file type, > 208 // which is not what we expected. Declare it insecure. > 209 // > 210 return false; > 211 } > 212 // If the directory is going to be opened readonly, we consider > this as secure operation > 213 // we do not need to do any more checks. > 214 // > 215 if ((mode & O_ACCMODE) == O_RDONLY) { > 216 return true; > 217 } > 218 // We have an existing directory, check if the permissions are > safe. > 219 // > 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { > 221 // The directory is open for writing and could be subjected > 222 // to a symlink or a hard link attack. Declare it insecure. > 223 // > 224 return false; > 225 } > 226 // See if the uid of the directory matches the effective uid of > the process. > 227 // > 228 if (statp->st_uid != geteuid()) { > 229 // The directory was not created by this user, declare it > insecure. > 230 // > 231 return false; > 232 } > 233 return true; > 234 } > > Lines 212-217 are added which allows a caller that passes in O_RDONLY > to by-pass the security checks on lines 220-225 and 228-232. This > implementation is using an attribute of _how_ the data is accessed > to override security policy instead of an attribute of _who_ is > accessing the data. > > Here are the code paths that access the modified policy code: > > is_statbuf_secure() is called by: > > - is_directory_secure() > - is_dirfd_secure() > > is_directory_secure() is called by: > > - get_user_name_slow() with O_RDONLY > - make_user_tmp_dir() with O_RDWR > - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) > > is_dirfd_secure() is called by: > > - open_directory_secure() with a mode parameter > > open_directory_secure() is called by: > > - open_directory_secure_cwd() with O_RDWR > - get_user_name_slow() with O_RDONLY > > Only the code paths that specify O_RDWR make use of The above line should be: Only the code paths that specify O_RDONLY make use of Dan > the new policy by-pass code so it looks like only > > - get_user_name_slow() with O_RDONLY > - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) > > are interesting. > > The new security policy by-pass will allow get_user_name_slow(): > > - to process directory entries in a directory that is writable > which makes this use subject to a symlink or hard link attack. > - to process directory entries in a directory that the calling > user does not own; the intent of the policy by-pass is to > allow this for the 'root' user, but this implementation > allows the by-pass for any user. > > It looks like the get_user_name_slow() code is written safely > enough such that any symlink or hard link attack should not > cause any issues. > > The new policy by-pass will allow any user to determine the > user name associated with VMs owned by another user. This is > a broader policy by-pass than was intended. > > > The new security policy by-pass will allow mmap_attach_shared(): > > - to process directory entries in a directory that is writable > which makes this use subject to a symlink or hard link attack. > - to process directory entries in a directory that the calling > user does not own; the intent of the policy by-pass is to > allow this for the 'root' user, but this implementation allows > the by-pass for any user. > > The mmap_attach_shared() code protects itself from a symlink > attack by including the 'O_NOFOLLOW' flag when opening the > PerfData file and it protects itself from a hardlink attack by > checking the hard link count after opening the file. It does > not protect itself against being handed a corrupted file or > even a very large file that would cause an OOM when the VM > tries to map what is supposed to be a PerfData file. > > The new policy by-pass will allow any user to access the > PerfData file associated with VMs owned by another user. This > is a broader policy by-pass than was intended. > > > Summary: > > This implementation of the new security policy by-pass is using > an attribute of _how_ the data is accessed to override security > policy instead of an attribute of _who_ is accessing the data. > This allows the VM to be exposed to some of the attacks that > the following fix was designed to prevent: > > JDK-8050807 Better performing performance data handling > > Dmitry's response to the code review provides a solution that > is based on who is accessing the data and that solution or > one like it should be considered. > > Again, sorry for the lengthy review. > > Dan > > > From daniel.daugherty at oracle.com Mon May 18 22:59:53 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 18 May 2015 16:59:53 -0600 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <5559CF69.9020804@oracle.com> References: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> <5555989C.9090803@oracle.com> <5559B81A.7000102@oracle.com> <5559CF69.9020804@oracle.com> Message-ID: <555A6EE9.1000503@oracle.com> Cheleswer, This review is going to require another detailed code path analysis to see what the root user would be exposed to with this policy by-pass. Short version of what I'm worried about: The root user is potentially accessing the perf-data directory and/or perf-data files that it does not own. The root user may be exposed to attacks by the owner(s) of the perf-data directory and/or perf-data files. I'll need to redo a wider version of the code path analysis in order to see if there is exposure here. Dan On 5/18/15 5:39 AM, Dmitry Samersoff wrote: > Cheleswer, > > 1. perfMemory_linux.cpp:222 missed space in !=0) > > 2. We have couple of other perfMemory_*.cpp files that have to be > updated as well. > > -Dmitry > > On 2015-05-18 12:59, cheleswer sahu wrote: >> Hi, >> I have fixed the code and tested. It's working fine. Please review the >> changes. >> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.02/ >> >> Regards, >> Cheleswer >> >> On 5/15/2015 12:26 PM, cheleswer sahu wrote: >>> Dear Dan & Dmitry, >>> Thanks for pointing out the security vulnerability in the fix and for >>> your precise review. I am also agree with Dmitry's fix. I will fix the >>> code and resend the review request. >>> >>> Regards, >>> Cheleswer >>> >>> On 5/14/2015 9:46 PM, Gerald Thornbrugh wrote: >>>> Hi Dan, >>>> >>>> When Cheleswer and I discussed this fix my interpretation had a >>>> slightly different goal: >>>> >>>> Prior to the initial security fix any user could execute "jps" and >>>> get the user names associated >>>> with other user's perf data (i.e. the call to get_user_name_slow() >>>> would succeed.). My initial >>>> thought was that this was a regression for all users not just "root" >>>> and this goal led to this fix. >>>> At the time I did not see this as a security vulnerability, your >>>> review has changed my mind. >>>> >>>> I agree that Dmitry's fix is a more secure fix for the issue and I >>>> think we should use it. >>>> >>>> Let me know if you have any questions. >>>> >>>> Thanks! >>>> >>>> Jerry >>>> >>>>> Hi, >>>>> Please review the code changes for >>>>> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >>>>> tested JDK9 with fix successfully. As I do not have an account for >>>>> OpenJDK, >>>>> David Buck will push the fix into jdk9/hs-rt/. >>>>> >>>>> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >>>>> >>>>> Regards, >>>>> Cheleswer >>>> Cheleswer, >>>> >>>> Sorry for the lengthy review, but since this is security related, >>>> I have to be complete. >>>> >>>> The goal: Add a policy by-pass for the 'root' user in order to >>>> fix the regression in jps(1) behavior. >>>> >>>> The core of this policy by-pass is the change to this function: >>>> >>>> 205 static bool is_statbuf_secure(struct stat *statp, int mode) { >>>> 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { >>>> 207 // The path represents a link or some non-directory file >>>> type, >>>> 208 // which is not what we expected. Declare it insecure. >>>> 209 // >>>> 210 return false; >>>> 211 } >>>> 212 // If the directory is going to be opened readonly, we consider >>>> this as secure operation >>>> 213 // we do not need to do any more checks. >>>> 214 // >>>> 215 if ((mode & O_ACCMODE) == O_RDONLY) { >>>> 216 return true; >>>> 217 } >>>> 218 // We have an existing directory, check if the permissions >>>> are safe. >>>> 219 // >>>> 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { >>>> 221 // The directory is open for writing and could be subjected >>>> 222 // to a symlink or a hard link attack. Declare it insecure. >>>> 223 // >>>> 224 return false; >>>> 225 } >>>> 226 // See if the uid of the directory matches the effective uid of >>>> the process. >>>> 227 // >>>> 228 if (statp->st_uid != geteuid()) { >>>> 229 // The directory was not created by this user, declare it >>>> insecure. >>>> 230 // >>>> 231 return false; >>>> 232 } >>>> 233 return true; >>>> 234 } >>>> >>>> Lines 212-217 are added which allows a caller that passes in O_RDONLY >>>> to by-pass the security checks on lines 220-225 and 228-232. This >>>> implementation is using an attribute of _how_ the data is accessed >>>> to override security policy instead of an attribute of _who_ is >>>> accessing the data. >>>> >>>> Here are the code paths that access the modified policy code: >>>> >>>> is_statbuf_secure() is called by: >>>> >>>> - is_directory_secure() >>>> - is_dirfd_secure() >>>> >>>> is_directory_secure() is called by: >>>> >>>> - get_user_name_slow() with O_RDONLY >>>> - make_user_tmp_dir() with O_RDWR >>>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>>> >>>> is_dirfd_secure() is called by: >>>> >>>> - open_directory_secure() with a mode parameter >>>> >>>> open_directory_secure() is called by: >>>> >>>> - open_directory_secure_cwd() with O_RDWR >>>> - get_user_name_slow() with O_RDONLY >>>> >>>> Only the code paths that specify O_RDWR make use of >>>> the new policy by-pass code so it looks like only >>>> >>>> - get_user_name_slow() with O_RDONLY >>>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>>> >>>> are interesting. >>>> >>>> The new security policy by-pass will allow get_user_name_slow(): >>>> >>>> - to process directory entries in a directory that is writable >>>> which makes this use subject to a symlink or hard link attack. >>>> - to process directory entries in a directory that the calling >>>> user does not own; the intent of the policy by-pass is to >>>> allow this for the 'root' user, but this implementation >>>> allows the by-pass for any user. >>>> >>>> It looks like the get_user_name_slow() code is written safely >>>> enough such that any symlink or hard link attack should not >>>> cause any issues. >>>> >>>> The new policy by-pass will allow any user to determine the >>>> user name associated with VMs owned by another user. This is >>>> a broader policy by-pass than was intended. >>>> >>>> >>>> The new security policy by-pass will allow mmap_attach_shared(): >>>> >>>> - to process directory entries in a directory that is writable >>>> which makes this use subject to a symlink or hard link attack. >>>> - to process directory entries in a directory that the calling >>>> user does not own; the intent of the policy by-pass is to >>>> allow this for the 'root' user, but this implementation allows >>>> the by-pass for any user. >>>> >>>> The mmap_attach_shared() code protects itself from a symlink >>>> attack by including the 'O_NOFOLLOW' flag when opening the >>>> PerfData file and it protects itself from a hardlink attack by >>>> checking the hard link count after opening the file. It does >>>> not protect itself against being handed a corrupted file or >>>> even a very large file that would cause an OOM when the VM >>>> tries to map what is supposed to be a PerfData file. >>>> >>>> The new policy by-pass will allow any user to access the >>>> PerfData file associated with VMs owned by another user. This >>>> is a broader policy by-pass than was intended. >>>> >>>> >>>> Summary: >>>> >>>> This implementation of the new security policy by-pass is using >>>> an attribute of _how_ the data is accessed to override security >>>> policy instead of an attribute of _who_ is accessing the data. >>>> This allows the VM to be exposed to some of the attacks that >>>> the following fix was designed to prevent: >>>> >>>> JDK-8050807 Better performing performance data handling >>>> >>>> Dmitry's response to the code review provides a solution that >>>> is based on who is accessing the data and that solution or >>>> one like it should be considered. >>>> >>>> Again, sorry for the lengthy review. >>>> >>>> Dan > From staffan.larsen at oracle.com Tue May 19 06:35:33 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 19 May 2015 08:35:33 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <555A2004.1030702@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> <555A2004.1030702@oracle.com> Message-ID: <539AF958-B6BF-4541-B2A5-23E656C4990F@oracle.com> Looks good! Thanks, /Staffan > On 18 maj 2015, at 19:23, Jaroslav Bachorik wrote: > > On 18.5.2015 10:41, Staffan Larsen wrote: >> >>> On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: >>> >>> On 18.5.2015 10:21, Staffan Larsen wrote: >>>> Looks good, but why is the reverseBytes needed? >>>> >>>> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); >>> >>> For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. >> >> I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: >> >> private PerfCounter(String name, int type) { >> this.name = name; >> ByteBuffer bb = perf.createLong(name, type, U_None, 0L); >> bb.order(ByteOrder.nativeOrder()); >> this.lb = bb.asLongBuffer(); >> } > > Done. Please, see http://cr.openjdk.java.net/~jbachorik/8075926/webrev.01 for the updated webrev. > > -JB- > >> >> /Staffan >> >>> >>> -JB- >>> >>>> >>>> Thanks, >>>> /Staffan >>>> >>>>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>>>> >>>>> Please, review the following change >>>>> >>>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>>>> >>>>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>>>> >>>>> The perf counters show stale data and mislead the users. >>>>> >>>>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>>>> >>>>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>>>> >>>>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>>>> >>>>> Thanks, >>>>> >>>>> -JB- >>>> >>> >> > From chris.plummer at oracle.com Tue May 19 06:37:54 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 18 May 2015 23:37:54 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled Message-ID: <555ADA42.9070603@oracle.com> An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Tue May 19 07:44:56 2015 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 May 2015 17:44:56 +1000 Subject: RFR 8078143: java/lang/management/ThreadMXBean/AllThreadIds.java fails intermittently In-Reply-To: <5554741B.3040000@oracle.com> References: <55423B75.6000703@oracle.com> <5542740B.6080507@oracle.com> <55533981.70204@oracle.com> <55540110.3050008@oracle.com> <5554741B.3040000@oracle.com> Message-ID: <555AE9F8.2030804@oracle.com> On 14/05/2015 8:08 PM, Jaroslav Bachorik wrote: > On 14.5.2015 03:57, David Holmes wrote: >> On 13/05/2015 9:46 PM, Jaroslav Bachorik wrote: >>> On 1.5.2015 21:55, Martin Buchholz wrote: >>>> >>>> >>>> On Thu, Apr 30, 2015 at 11:27 AM, Jaroslav Bachorik >>>> > >>>> wrote: >>>> >>>> On 30.4.2015 19:18, Martin Buchholz wrote: >>>> >>>> Tests that sleep can almost always be better written some >>>> other way. >>>> In this case, I would prefer busy-waiting with timeout until >>>> the >>>> expected condition becomes true. >>>> >>>> >>>> The thing is that in case of a real issue with the thread >>>> counters we >>>> a/ would be busy-waiting till the test times out (using an >>>> arbitrary >>>> delay is also problematic due to different performance of different >>>> machines running with different configurations) >>>> >>>> >>>> Far less problematic (performance-wise and reliability-wise) than the >>>> fixed sleep. >>>> >>>> b/ would get a rather confusing message about the test timing >>>> out at >>>> the end >>>> >>>> >>>> You can easily improve the error message. >>> >>> Well, not that easily. It is not possible to get a notification when >>> JTREG decides to timeout the test. So you will get the standard JTREG >>> message and that's all. >>> >>> I was able to modify the test to wait for a given condition and provide >>> useful messages in case of mismatch and retry. For the price of an >>> increased complexity. On the other hand, the test should be much more >>> resilient to timing errors caused by slow setups. >>> >>> Updated webrev: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.01 >> >> Certainly increased complexity - took me quite a while to figure out >> what it all meant. :) waitForCondition should be waitTillEqual as that >> is the only condition checked (unless you'd like to make it more complex >> and pass in a Predicate ;-) ). > > I started with Predicate and then realized it brought more problems than > benefits ... and didn't rename the method afterwords. 'waitTillEquals' > sounds good. > >> >> I think the Thread.yield would be better as a short sleep > > You mean not to use Thread.yield() and do Thread.sleep(1) instead? > > I've moved the logical blocks into separate methods to make the test > more explicit about the steps it takes. Incidentally, this made the > webrev also more readable ;) > > Update: http://cr.openjdk.java.net/~jbachorik/8078143/webrev.02 This seems fine to me. Thanks, David > -JB- > >> >> Thanks, >> David >> ----- >> >> >>> Thanks, >>> >>> -JB- > From peter.levart at gmail.com Tue May 19 09:52:38 2015 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 19 May 2015 11:52:38 +0200 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5559D85B.2050500@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: <555B07E6.4040500@gmail.com> Hi Dmitry, This looks good. Just alignment of new code in Finalizer seems a little wobbly. Regards, Peter On 05/18/2015 02:17 PM, Dmitry Samersoff wrote: > Everyone, > > Please review updated version of the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ > > Most important part of the fix provided by Peter Levart, so all > credentials belongs to him. > > -Dmitry > > On 2015-05-16 15:48, Peter Levart wrote: >> >> On 05/16/2015 02:38 PM, Peter Levart wrote: >>> >>> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >>>> Derek, >>>> >>>> Personally, I'm for volatile over explicit fence too. >>>> >>>> So I'll change the code if Peter don't have objections. >>> There are no objections. There's one unneeded volatile store to .next >>> field then in enqueue(), but it is followed by another volatile store, >>> so there shouldn't be overhead on Intel and SPARC at least. >>> >>> Regards, Peter >> If you make .next field volatile, then perhaps you could also cache it's >> value in reallyPoll() into a local variable, so that it is not read >> twice. Like this: >> >> private Reference reallyPoll() { /* Must hold lock */ >> Reference r = head; >> if (r != null) { >> Reference rn = r.next; // HERE !!! >> head = (rn == r) ? >> null : >> rn; // Unchecked due to the next field having a raw type >> in Reference >> r.queue = NULL; >> r.next = r; >> queueLength--; >> if (r instanceof FinalReference) { >> sun.misc.VM.addFinalRefCount(-1); >> } >> return r; >> } >> return null; >> } >> >> >> >> Peter >> >> >>>> -Dmitry >>>> >>>> On 2015-05-16 01:59, Derek White wrote: >>>>> Hi Dmitry, Peter, >>>>> >>>>> I should read my email more frequently - I missed Dmitry's last webrev >>>>> email. >>>>> >>>>> My comments on a preference for volatile over Unsafe are not a strong >>>>> objection to using Unsafe fences. I just don't have enough experience >>>>> with Unsafe fences yet to agree that this use is correct. >>>>> >>>>> While looking over this Reference code I found 3-6 really simple things >>>>> that need cleaning up that have been there for years, so I'm not a big >>>>> cheerleader for more complicated things here :-) >>>>> >>>>> - Derek >>>>> >>>>> On 5/15/15 6:46 PM, Derek White wrote: >>>>>> Hi Peter, >>>>>> >>>>>> Some comments below... >>>>>> >>>>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>>>> Hi Derek, >>>>>>> >>>>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>>>> but I have some concerns... >>>>>>> I did have a concern too, ... >>>>>>> >>>>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, >>>>>>>>> >>>>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>>>> logical. >>>>>>>>> >>>>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>>>> application. Suppose one wants to diagnose the application because >>>>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>>>> up gradually and is already holding several million objects. Now >>>>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>>>> iteration over and grouping/counting of several millions of >>>>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>>>> reference handler thread. But does not block the threads that >>>>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>>>> most of the time, getting OOMEs, etc... >>>>>>>>> >>>>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>>>> purposes without holding a lock. The modification required to do >>>>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>>>> >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>>>> >>>>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>>>> >>>>>>>> Your change makes "inactive" References superficially look like >>>>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>>>> yet seen a case where it would get confused by this change, but >>>>>>>> there could easily be something like that lurking in the GC code. >>>>>>> ...but then I thought that GC can in no way treat a Reference >>>>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>>>> already dequeued (inactive) - responsibility is already on the Java >>>>>>> side. Currently the definition of Reference.next is this: >>>>>>> >>>>>>> /* When active: NULL >>>>>>> * pending: this >>>>>>> * Enqueued: next reference in queue (or this if last) >>>>>>> * Inactive: this >>>>>>> */ >>>>>>> @SuppressWarnings("rawtypes") >>>>>>> Reference next; >>>>>>> >>>>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>>>> scans their lists too, the state of a Reference that is enqueued as >>>>>>> last in list is indistinguishable from the state of inactive >>>>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>>>> can't be established solely on the value of .next field ( == this or >>>>>>> != this)... >>>>>>> >>>>>>> But I should inspect the GC code too to build a better understanding >>>>>>> of that part of the story... >>>>>>> >>>>>>> Anyway. It turns out that there is already enough state in Reference >>>>>>> to destinguish between it being enqueued as last in list and already >>>>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>>>> be achieved either by making Reference.next a volatile field or by a >>>>>>> couple of explicit fences: >>>>>>> >>>>>>> >>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>>>> >>>>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>>>> before linking the reference into the queue's head in >>>>>>> ReferenceQueue.enqueue(): >>>>>>> >>>>>>> r.queue = ENQUEUED; >>>>>>> r.next = (head == null) ? r : head; >>>>>>> head = r; >>>>>>> >>>>>>> Both stores are volatile. >>>>>> This sounds a bit better. I don't have a good handle on the pros and >>>>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>>>> Kim might jump in soon. >>>>>> >>>>>> I'd like to suggest an entirely less-clever solution. Since the >>>>>> problem is that forEach() might see inconsistent values for the queue >>>>>> and next fields of a Reference, but we don't want to grab a lock >>>>>> walking the whole queue, we could instead grab the lock as we look at >>>>>> each Reference (and do the "back-up" trick if we were interfered >>>>>> with). Of course this is slower than going lock-free, but it's not any >>>>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>>>> >>>>>> The only benefit of this solution over the others is that it is less >>>>>> clever, and I'm not sure how much cleverness this problem deserves. >>>>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>>>> that is sufficient. >>>>>> >>>>>> - Derek >>>>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>>>> forEach iteration method with no concrete logic. >>>>>>>>> >>>>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>>>> rest in DiagnosticCommands. >>>>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>>>> Referents were leaked back into the heap, then we could get >>>>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>>>> on it's own - any finalizer might resurrect its object if not >>>>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>>>> object. This could invalidate application invariants? >>>>>>> Well, it all stays in the confines of package-private API - internal >>>>>>> to JDK. Any future additional use should be reviewed carefully. >>>>>>> Comments on the forEach() method should warn about that. >>>>>>> >>>>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>>>> there are pathological scenarios where this gets severe. This is >>>>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>>>> introduce subtle bugs to the system in general. A change in this >>>>>>>> area should get a thorough review from both the runtime and GC sides. >>>>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>>>> does not introduce any logical changes to existing code. >>>>>>> >>>>>>>> Better yet, the reference handling code in GC and runtime should >>>>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>>>> I may have some proposals in that direction. Stay tuned. >>>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>>> - Derek >>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>> >>>>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>>>> Everybody, >>>>>>>>>> >>>>>>>>>> Updated version: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>>>> >>>>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>> >>>>>>>>>>> Lots of good comments here. >>>>>>>>>>> >>>>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>>>> general iterator and lambda. >>>>>>>>>>> >>>>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>>>> >>>>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>>>> >>>>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>>>> >>>>>>>>>>> - Derek White, GC team >>>>>>>>>>> >>>>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>>> >>>>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>>>> Dmitry, >>>>>>>>>>>>> >>>>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>>>> >>>>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>>>> running out of memory. >>>>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>>>> >>>>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>>>> interfere with synchronization? >>>>>>>>>>>> >>>>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>>>> >>>>>>>>>>>> private void remove() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized == this) { >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> unfinalized = this.next; >>>>>>>>>>>> } else { >>>>>>>>>>>> unfinalized = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> if (this.prev != null) { >>>>>>>>>>>> this.prev.next = this.next; >>>>>>>>>>>> } >>>>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>>>> traverse the list unsynchronized >>>>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>>>> finalized */ >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>>>> pointer could be used instead: >>>>>>>>>>>> >>>>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>>>> return (prev == this); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>>>> >>>>>>>>>>>> private void add() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized != null) { >>>>>>>>>>>> this.next = unfinalized; >>>>>>>>>>>> unfinalized.prev = this; >>>>>>>>>>>> } >>>>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>>>> // instance fully initialized >>>>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>>>> U.storeFence(); >>>>>>>>>>>> unfinalized = this; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>>>> >>>>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>>>> out? >>>>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>>>> two purposes: >>>>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>>>> >>>>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>>>> finalize() method. >>>>>>>>>>>> >>>>>>>>>>>> Regards, Peter >>>>>>>>>>>> >>>>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>>>> >>>>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>>>> permission() and limit access: >>>>>>>>>>>>> >>>>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>>>> return p; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> /Staffan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please review the fix: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>>>> >>>>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>>>> with count >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. > From jaroslav.bachorik at oracle.com Tue May 19 13:35:38 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 19 May 2015 15:35:38 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <539AF958-B6BF-4541-B2A5-23E656C4990F@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> <555A2004.1030702@oracle.com> <539AF958-B6BF-4541-B2A5-23E656C4990F@oracle.com> Message-ID: <555B3C2A.80604@oracle.com> On 19.5.2015 08:35, Staffan Larsen wrote: > Looks good! Sorry Staffan, I forgot to include test/sun/management/jmxremote/bootstrap/JvmstatCountersTest.java in the webrev :( http://cr.openjdk.java.net/~jbachorik/8075926/webrev.02 -JB- > > Thanks, > /Staffan > >> On 18 maj 2015, at 19:23, Jaroslav Bachorik wrote: >> >> On 18.5.2015 10:41, Staffan Larsen wrote: >>> >>>> On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: >>>> >>>> On 18.5.2015 10:21, Staffan Larsen wrote: >>>>> Looks good, but why is the reverseBytes needed? >>>>> >>>>> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); >>>> >>>> For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. >>> >>> I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: >>> >>> private PerfCounter(String name, int type) { >>> this.name = name; >>> ByteBuffer bb = perf.createLong(name, type, U_None, 0L); >>> bb.order(ByteOrder.nativeOrder()); >>> this.lb = bb.asLongBuffer(); >>> } >> >> Done. Please, see http://cr.openjdk.java.net/~jbachorik/8075926/webrev.01 for the updated webrev. >> >> -JB- >> >>> >>> /Staffan >>> >>>> >>>> -JB- >>>> >>>>> >>>>> Thanks, >>>>> /Staffan >>>>> >>>>>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>>>>> >>>>>> Please, review the following change >>>>>> >>>>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>>>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>>>>> >>>>>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>>>>> >>>>>> The perf counters show stale data and mislead the users. >>>>>> >>>>>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>>>>> >>>>>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>>>>> >>>>>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -JB- >>>>> >>>> >>> >> > From jaroslav.bachorik at oracle.com Tue May 19 13:39:12 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 19 May 2015 15:39:12 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties Message-ID: <555B3D00.3080204@oracle.com> Please, review the following change Issue : https://bugs.openjdk.java.net/browse/JDK-8080663 Webrev: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 The title says it all. This enhancement is about replacing the arbitrary reflection based code with a cleaner approach based on sun.misc.SharedSecrets class. Thanks, -JB- From daniel.fuchs at oracle.com Tue May 19 14:11:16 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 19 May 2015 16:11:16 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <555B3D00.3080204@oracle.com> References: <555B3D00.3080204@oracle.com> Message-ID: <555B4484.3080906@oracle.com> On 19/05/15 15:39, Jaroslav Bachorik wrote: > Please, review the following change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8080663 > Webrev: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 > > The title says it all. This enhancement is about replacing the arbitrary > reflection based code with a cleaner approach based on > sun.misc.SharedSecrets class. Hi Jaroslav, The logic looks good - but I notice the webrev has no test. I would be surprised if there weren't a test somewhere that tests the MXBean OpenType mapping with @ConstructorProperties - so I'd suggest at least finding that test and adding @bug 8080663 to it (and then also adding the noreg-existing keyword to 8080663) best regards, -- daniel > > Thanks, > > -JB- From chris.plummer at oracle.com Tue May 19 14:25:46 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 19 May 2015 07:25:46 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled Message-ID: <555B47EA.3000900@oracle.com> Hi, Please review the following changes for allowing java debugging when CDS is enabled. Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 The VM changes are simple. I removed the check that prevents debugging with CDS enabled, and added logic that will map the CDS archive RW when debugging is enabled. The tests are a bit more complex. There are a bunch of existing JDI tests for testing debugging support. Rather than start from scratch or clone them, I instead just wrote wrapper tests that put the relevant JDI test classes in the archive, and then invoke the JDI test. I did this for 3 of the JDI tests. If you feel there are others that would be good candidates, I'd be happy to add them. I'm looking for ones that would result in modification of the RO class metadata, such as setting a breakpoint (for which I already added two tests). Testing done: -Using JPRT to run the new jtreg tests on all platforms. -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. -Regular JPRT "-testset hotspot" run -Putting the JCK JVMTI tests in the archive and then running them. -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and then running them. -Putting a simple test class in the archive and then setting a breakpoint on it using jdb Some of the above testing resulted in the discovery of bugs that still need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. I also verified that without the change to map the archive RW, the above testing resulted in a SEGV, which is what you would expect (and actually want to see to prove that the testing is effective). thanks, Chris From aph at redhat.com Tue May 19 15:20:14 2015 From: aph at redhat.com (Andrew Haley) Date: Tue, 19 May 2015 16:20:14 +0100 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <55577859.5080406@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> Message-ID: <555B54AE.9050100@redhat.com> On 05/16/2015 06:03 PM, Dmitry Samersoff wrote: > Andrew, > > On 2015-05-12 20:05, Andrew Haley wrote: >> http://cr.openjdk.java.net/~aph/8078521-4/ > > HSDB.java: > > 988: If you removed else part, you don't need to check CPU here at all, > just leave a comment. This will now enable the code for PPC; I guess that's OK. > LinuxAARCH64CFrame.java: > > 65 - it's better to add brackets around 2 * ADDRESS_SIZE > 69,73 - please, remove space after bracket. Done. http://cr.openjdk.java.net/~aph/8078521-5/ Thanks, Andrew. From dmitry.samersoff at oracle.com Tue May 19 16:49:07 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 19 May 2015 19:49:07 +0300 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <555B54AE.9050100@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> Message-ID: <555B6983.30000@oracle.com> Andrew, Looks good for me. (not a Reviewer) > This will now enable the code for PPC; I guess that's OK. I think so too. -Dmitry On 2015-05-19 18:20, Andrew Haley wrote: > On 05/16/2015 06:03 PM, Dmitry Samersoff wrote: >> Andrew, >> >> On 2015-05-12 20:05, Andrew Haley wrote: >>> http://cr.openjdk.java.net/~aph/8078521-4/ >> >> HSDB.java: >> >> 988: If you removed else part, you don't need to check CPU here at all, >> just leave a comment. > > This will now enable the code for PPC; I guess that's OK. > >> LinuxAARCH64CFrame.java: >> >> 65 - it's better to add brackets around 2 * ADDRESS_SIZE >> 69,73 - please, remove space after bracket. > > Done. > > http://cr.openjdk.java.net/~aph/8078521-5/ > > Thanks, > Andrew. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From staffan.larsen at oracle.com Tue May 19 17:46:17 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 19 May 2015 19:46:17 +0200 Subject: RFR 8075926: Add a sun.management.JMXConnectorServer perf counter to track its state In-Reply-To: <555B3C2A.80604@oracle.com> References: <5553702E.8050507@oracle.com> <26294D10-2A52-4061-8295-884336725373@oracle.com> <5559A215.5040608@oracle.com> <16C104CB-0EC7-4275-8940-24B9D14BA7BF@oracle.com> <555A2004.1030702@oracle.com> <539AF958-B6BF-4541-B2A5-23E656C4990F@oracle.com> <555B3C2A.80604@oracle.com> Message-ID: <71F4EF1C-2298-4FA9-8ED5-BC94312F6FDF@oracle.com> Still good! > On 19 maj 2015, at 15:35, Jaroslav Bachorik wrote: > > On 19.5.2015 08:35, Staffan Larsen wrote: >> Looks good! > > Sorry Staffan, I forgot to include test/sun/management/jmxremote/bootstrap/JvmstatCountersTest.java in the webrev :( > > http://cr.openjdk.java.net/~jbachorik/8075926/webrev.02 > > -JB- > > >> >> Thanks, >> /Staffan >> >>> On 18 maj 2015, at 19:23, Jaroslav Bachorik wrote: >>> >>> On 18.5.2015 10:41, Staffan Larsen wrote: >>>> >>>>> On 18 maj 2015, at 10:25, Jaroslav Bachorik wrote: >>>>> >>>>> On 18.5.2015 10:21, Staffan Larsen wrote: >>>>>> Looks good, but why is the reverseBytes needed? >>>>>> >>>>>> 60 this.bb.asLongBuffer().put(Long.reverseBytes(l)); >>>>> >>>>> For some reason Perf.createLong(...) will create bytebuffer with the HILO byte order reversed when compared to how long is usually represented. >>>> >>>> I think you should set the order() of the ByteBuffer before you create the LongBuffer. See the code in sun.misc.PerfCounter: >>>> >>>> private PerfCounter(String name, int type) { >>>> this.name = name; >>>> ByteBuffer bb = perf.createLong(name, type, U_None, 0L); >>>> bb.order(ByteOrder.nativeOrder()); >>>> this.lb = bb.asLongBuffer(); >>>> } >>> >>> Done. Please, see http://cr.openjdk.java.net/~jbachorik/8075926/webrev.01 for the updated webrev. >>> >>> -JB- >>> >>>> >>>> /Staffan >>>> >>>>> >>>>> -JB- >>>>> >>>>>> >>>>>> Thanks, >>>>>> /Staffan >>>>>> >>>>>>> On 13 maj 2015, at 17:39, Jaroslav Bachorik wrote: >>>>>>> >>>>>>> Please, review the following change >>>>>>> >>>>>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8075926 >>>>>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8075926/webrev.00 >>>>>>> >>>>>>> The sun.management.JMXConnectorServer.. perf counters are not updated when the remote management agent is stopped. >>>>>>> >>>>>>> The perf counters show stale data and mislead the users. >>>>>>> >>>>>>> Since it is not possible to 'un-export' perf counters we need an additional counter tracking the version of the related perf counters in use. >>>>>>> >>>>>>> In the current implementation each start of the remote management agent will export a new set of the related perf counters (sun.management.JMXConnectorServer..) with the part increased by one. The first remote management agent start will use 0 as its version. >>>>>>> >>>>>>> The new counter 'sun.management.JMXConnectorServer.remote' will have value of -1 if the remote management is stopped and non-negative number corresponding to the version of the related perf counters in use. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -JB- >>>>>> >>>>> >>>> >>> >> > From kim.barrett at oracle.com Tue May 19 20:41:17 2015 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 19 May 2015 16:41:17 -0400 Subject: RFR(M, v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5559D85B.2050500@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: <349D924B-DDA8-4F18-8E8F-A0272ED39B4F@oracle.com> On May 18, 2015, at 8:17 AM, Dmitry Samersoff wrote: > > Everyone, > > Please review updated version of the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ > > Most important part of the fix provided by Peter Levart, so all > credentials belongs to him. > > -Dmitry Looks good, except a bit in Finalizer.printFinalizerQueue, specifically: 122 /* Clear stack slot containing this variable, to decrease 123 the chances of false retention with a conservative GC */ 124 referent = null; Peter already noted these lines are mis-indented. However I?m not convinced that assignment is actually useful, since I think it can be optimized away. From derek.white at oracle.com Tue May 19 21:10:42 2015 From: derek.white at oracle.com (Derek White) Date: Tue, 19 May 2015 17:10:42 -0400 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5559D85B.2050500@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: <555BA6D2.2050600@oracle.com> On 5/18/15 8:17 AM, Dmitry Samersoff wrote: > Everyone, > > Please review updated version of the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ > > Most important part of the fix provided by Peter Levart, so all > credentials belongs to him. > > -Dmitry Looks good to me. I also pinged Mandy to check it from a Library perspective. It sounds like she has a review in progress... - Derek > > On 2015-05-16 15:48, Peter Levart wrote: >> >> On 05/16/2015 02:38 PM, Peter Levart wrote: >>> >>> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >>>> Derek, >>>> >>>> Personally, I'm for volatile over explicit fence too. >>>> >>>> So I'll change the code if Peter don't have objections. >>> There are no objections. There's one unneeded volatile store to .next >>> field then in enqueue(), but it is followed by another volatile store, >>> so there shouldn't be overhead on Intel and SPARC at least. >>> >>> Regards, Peter >> If you make .next field volatile, then perhaps you could also cache it's >> value in reallyPoll() into a local variable, so that it is not read >> twice. Like this: >> >> private Reference reallyPoll() { /* Must hold lock */ >> Reference r = head; >> if (r != null) { >> Reference rn = r.next; // HERE !!! >> head = (rn == r) ? >> null : >> rn; // Unchecked due to the next field having a raw type >> in Reference >> r.queue = NULL; >> r.next = r; >> queueLength--; >> if (r instanceof FinalReference) { >> sun.misc.VM.addFinalRefCount(-1); >> } >> return r; >> } >> return null; >> } >> >> >> >> Peter >> >> >>>> -Dmitry >>>> >>>> On 2015-05-16 01:59, Derek White wrote: >>>>> Hi Dmitry, Peter, >>>>> >>>>> I should read my email more frequently - I missed Dmitry's last webrev >>>>> email. >>>>> >>>>> My comments on a preference for volatile over Unsafe are not a strong >>>>> objection to using Unsafe fences. I just don't have enough experience >>>>> with Unsafe fences yet to agree that this use is correct. >>>>> >>>>> While looking over this Reference code I found 3-6 really simple things >>>>> that need cleaning up that have been there for years, so I'm not a big >>>>> cheerleader for more complicated things here :-) >>>>> >>>>> - Derek >>>>> >>>>> On 5/15/15 6:46 PM, Derek White wrote: >>>>>> Hi Peter, >>>>>> >>>>>> Some comments below... >>>>>> >>>>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>>>> Hi Derek, >>>>>>> >>>>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>>>> but I have some concerns... >>>>>>> I did have a concern too, ... >>>>>>> >>>>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, >>>>>>>>> >>>>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>>>> logical. >>>>>>>>> >>>>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>>>> application. Suppose one wants to diagnose the application because >>>>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>>>> up gradually and is already holding several million objects. Now >>>>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>>>> iteration over and grouping/counting of several millions of >>>>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>>>> reference handler thread. But does not block the threads that >>>>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>>>> most of the time, getting OOMEs, etc... >>>>>>>>> >>>>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>>>> purposes without holding a lock. The modification required to do >>>>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>>>> >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>>>> >>>>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>>>> >>>>>>>> Your change makes "inactive" References superficially look like >>>>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>>>> yet seen a case where it would get confused by this change, but >>>>>>>> there could easily be something like that lurking in the GC code. >>>>>>> ...but then I thought that GC can in no way treat a Reference >>>>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>>>> already dequeued (inactive) - responsibility is already on the Java >>>>>>> side. Currently the definition of Reference.next is this: >>>>>>> >>>>>>> /* When active: NULL >>>>>>> * pending: this >>>>>>> * Enqueued: next reference in queue (or this if last) >>>>>>> * Inactive: this >>>>>>> */ >>>>>>> @SuppressWarnings("rawtypes") >>>>>>> Reference next; >>>>>>> >>>>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>>>> scans their lists too, the state of a Reference that is enqueued as >>>>>>> last in list is indistinguishable from the state of inactive >>>>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>>>> can't be established solely on the value of .next field ( == this or >>>>>>> != this)... >>>>>>> >>>>>>> But I should inspect the GC code too to build a better understanding >>>>>>> of that part of the story... >>>>>>> >>>>>>> Anyway. It turns out that there is already enough state in Reference >>>>>>> to destinguish between it being enqueued as last in list and already >>>>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>>>> be achieved either by making Reference.next a volatile field or by a >>>>>>> couple of explicit fences: >>>>>>> >>>>>>> >>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>>>> >>>>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>>>> before linking the reference into the queue's head in >>>>>>> ReferenceQueue.enqueue(): >>>>>>> >>>>>>> r.queue = ENQUEUED; >>>>>>> r.next = (head == null) ? r : head; >>>>>>> head = r; >>>>>>> >>>>>>> Both stores are volatile. >>>>>> This sounds a bit better. I don't have a good handle on the pros and >>>>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>>>> Kim might jump in soon. >>>>>> >>>>>> I'd like to suggest an entirely less-clever solution. Since the >>>>>> problem is that forEach() might see inconsistent values for the queue >>>>>> and next fields of a Reference, but we don't want to grab a lock >>>>>> walking the whole queue, we could instead grab the lock as we look at >>>>>> each Reference (and do the "back-up" trick if we were interfered >>>>>> with). Of course this is slower than going lock-free, but it's not any >>>>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>>>> >>>>>> The only benefit of this solution over the others is that it is less >>>>>> clever, and I'm not sure how much cleverness this problem deserves. >>>>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>>>> that is sufficient. >>>>>> >>>>>> - Derek >>>>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>>>> forEach iteration method with no concrete logic. >>>>>>>>> >>>>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>>>> rest in DiagnosticCommands. >>>>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>>>> Referents were leaked back into the heap, then we could get >>>>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>>>> on it's own - any finalizer might resurrect its object if not >>>>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>>>> object. This could invalidate application invariants? >>>>>>> Well, it all stays in the confines of package-private API - internal >>>>>>> to JDK. Any future additional use should be reviewed carefully. >>>>>>> Comments on the forEach() method should warn about that. >>>>>>> >>>>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>>>> there are pathological scenarios where this gets severe. This is >>>>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>>>> introduce subtle bugs to the system in general. A change in this >>>>>>>> area should get a thorough review from both the runtime and GC sides. >>>>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>>>> does not introduce any logical changes to existing code. >>>>>>> >>>>>>>> Better yet, the reference handling code in GC and runtime should >>>>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>>>> I may have some proposals in that direction. Stay tuned. >>>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>>> - Derek >>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>> >>>>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>>>> Everybody, >>>>>>>>>> >>>>>>>>>> Updated version: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>>>> >>>>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>> >>>>>>>>>>> Lots of good comments here. >>>>>>>>>>> >>>>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>>>> general iterator and lambda. >>>>>>>>>>> >>>>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>>>> >>>>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>>>> >>>>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>>>> >>>>>>>>>>> - Derek White, GC team >>>>>>>>>>> >>>>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>>> >>>>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>>>> Dmitry, >>>>>>>>>>>>> >>>>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>>>> >>>>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>>>> running out of memory. >>>>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>>>> >>>>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>>>> interfere with synchronization? >>>>>>>>>>>> >>>>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>>>> >>>>>>>>>>>> private void remove() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized == this) { >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> unfinalized = this.next; >>>>>>>>>>>> } else { >>>>>>>>>>>> unfinalized = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> if (this.prev != null) { >>>>>>>>>>>> this.prev.next = this.next; >>>>>>>>>>>> } >>>>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>>>> traverse the list unsynchronized >>>>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>>>> finalized */ >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>>>> pointer could be used instead: >>>>>>>>>>>> >>>>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>>>> return (prev == this); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>>>> >>>>>>>>>>>> private void add() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized != null) { >>>>>>>>>>>> this.next = unfinalized; >>>>>>>>>>>> unfinalized.prev = this; >>>>>>>>>>>> } >>>>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>>>> // instance fully initialized >>>>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>>>> U.storeFence(); >>>>>>>>>>>> unfinalized = this; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>>>> >>>>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>>>> out? >>>>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>>>> two purposes: >>>>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>>>> >>>>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>>>> finalize() method. >>>>>>>>>>>> >>>>>>>>>>>> Regards, Peter >>>>>>>>>>>> >>>>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>>>> >>>>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>>>> permission() and limit access: >>>>>>>>>>>>> >>>>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>>>> return p; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> /Staffan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please review the fix: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>>>> >>>>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>>>> with count >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. > From ivan.gerasimov at oracle.com Tue May 19 23:54:06 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 20 May 2015 02:54:06 +0300 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... Message-ID: <555BCD1E.9010406@oracle.com> Hello! Since the beginning of the year a warning about an expired timeout has been seen a few times. It is most likely to observe this under a heavy load, when many threads are exiting. The propose is to increase the timeout. The rationale is that if an application needs more time for the threads to exit, let's give it that time. BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ Sincerely yours, Ivan From mandy.chung at oracle.com Wed May 20 02:54:11 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 19 May 2015 19:54:11 -0700 Subject: RFR(M, v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5559D85B.2050500@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> > On May 18, 2015, at 5:17 AM, Dmitry Samersoff wrote: > > Please review updated version of the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ > > Most important part of the fix provided by Peter Levart, so all > credentials belongs to him. My apology for being late to the review. The subject line doesn?t catch my attention early enough :) I have to do further detail review tomorrow or so to follow the discussion and closely inspect the reference implementation. Just to give you a quick comment. I?m okay to add ReferenceQueue.forEach method at the first glance. However I have trouble for Finalizer.printFinalizationQueue method that doesn?t belong there. What are the other alternatives you have explored? Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Wed May 20 04:26:55 2015 From: david.holmes at oracle.com (David Holmes) Date: Wed, 20 May 2015 14:26:55 +1000 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <555B47EA.3000900@oracle.com> References: <555B47EA.3000900@oracle.com> Message-ID: <555C0D0F.6090108@oracle.com> Hi Chris, On 20/05/2015 12:25 AM, Chris Plummer wrote: > Hi, > > Please review the following changes for allowing java debugging when CDS > is enabled. > > Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ > Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 > > The VM changes are simple. I removed the check that prevents debugging > with CDS enabled, and added logic that will map the CDS archive RW when > debugging is enabled. > > The tests are a bit more complex. There are a bunch of existing JDI > tests for testing debugging support. Rather than start from scratch or > clone them, I instead just wrote wrapper tests that put the relevant JDI > test classes in the archive, and then invoke the JDI test. I did this I'm not sure that the jdk tests will always be available in the same test bundle as the hotspot tests. I know they will be in the full forest but some testing uses test bundles not full repos. David ----- > for 3 of the JDI tests. If you feel there are others that would be good > candidates, I'd be happy to add them. I'm looking for ones that would > result in modification of the RO class metadata, such as setting a > breakpoint (for which I already added two tests). > > Testing done: > -Using JPRT to run the new jtreg tests on all platforms. > -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. > -Regular JPRT "-testset hotspot" run > -Putting the JCK JVMTI tests in the archive and then running them. > -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and then > running them. > -Putting a simple test class in the archive and then setting a > breakpoint on it using jdb > > Some of the above testing resulted in the discovery of bugs that still > need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. > > I also verified that without the change to map the archive RW, the above > testing resulted in a SEGV, which is what you would expect (and actually > want to see to prove that the testing is effective). > > thanks, > > Chris > From dmitry.samersoff at oracle.com Wed May 20 06:51:23 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 20 May 2015 09:51:23 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> Message-ID: <555C2EEB.1050505@oracle.com> Mandy, > However I have trouble for > Finalizer.printFinalizationQueue method that doesn?t belong there. > What are the other alternatives you have explored? Other alternatives could be to do all hashing/sorting/printing on native layer i.e. implement printFinalizationQueue inside VM. Both options has pros and cons - Java based solution requires less JNI calls and better readable but takes more memory. It might be better to return an array of Map.Entry objects to VM rather than one huge string. -Dmitry On 2015-05-20 05:54, Mandy Chung wrote: > >> On May 18, 2015, at 5:17 AM, Dmitry Samersoff >> > wrote: >> >> Please review updated version of the fix: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >> >> Most important part of the fix provided by Peter Levart, so all >> credentials belongs to him. > > > My apology for being late to the review. The subject line doesn?t catch > my attention early enough :) > > I have to do further detail review tomorrow or so to follow the > discussion and closely inspect the reference implementation. Just to > give you a quick comment. I?m okay to add ReferenceQueue.forEach method > at the first glance. However I have trouble for > Finalizer.printFinalizationQueue method that doesn?t belong there. What > are the other alternatives you have explored? > > Mandy > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From david.holmes at oracle.com Wed May 20 07:04:52 2015 From: david.holmes at oracle.com (David Holmes) Date: Wed, 20 May 2015 17:04:52 +1000 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... In-Reply-To: <555BCD1E.9010406@oracle.com> References: <555BCD1E.9010406@oracle.com> Message-ID: <555C3214.50004@oracle.com> Hi Ivan, On 20/05/2015 9:54 AM, Ivan Gerasimov wrote: > Hello! > > Since the beginning of the year a warning about an expired timeout has > been seen a few times. > It is most likely to observe this under a heavy load, when many threads > are exiting. > > The propose is to increase the timeout. > The rationale is that if an application needs more time for the threads > to exit, let's give it that time. Rather than just bumping the timeout a little I would suggest bumping it a lot - make it a few minutes. If that fails then it suggests we have a real problem not just a slow or loaded machine. Otherwise if we just keep bumping the timeout it serves no purpose. Cheers, David > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 > WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ > > Sincerely yours, > Ivan From peter.levart at gmail.com Wed May 20 08:22:28 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 20 May 2015 10:22:28 +0200 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555C2EEB.1050505@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> Message-ID: <555C4444.6050008@gmail.com> On 05/20/2015 08:51 AM, Dmitry Samersoff wrote: > Mandy, > >> However I have trouble for >> Finalizer.printFinalizationQueue method that doesn?t belong there. >> What are the other alternatives you have explored? > Other alternatives could be to do all hashing/sorting/printing on native > layer i.e. implement printFinalizationQueue inside VM. > > Both options has pros and cons - Java based solution requires less JNI > calls and better readable but takes more memory. > > It might be better to return an array of Map.Entry > objects to VM rather than one huge string. > > -Dmitry Hi Dmitry, What about creating a special package-private java.lang.ref.DiagnosticCommands class which is then used as the home of static printFinalizationQueue method (and possible future diagnostic commands methods in the package). You could then expose a static package-private method from Finalizer: static void forEachEnqueued(Consumer> action) { queue.forEach(action); } ...and use it to implement the printFinalizationQueue. Regards, Peter > > > > On 2015-05-20 05:54, Mandy Chung wrote: >>> On May 18, 2015, at 5:17 AM, Dmitry Samersoff >>> > wrote: >>> >>> Please review updated version of the fix: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >>> >>> Most important part of the fix provided by Peter Levart, so all >>> credentials belongs to him. >> >> My apology for being late to the review. The subject line doesn?t catch >> my attention early enough :) >> >> I have to do further detail review tomorrow or so to follow the >> discussion and closely inspect the reference implementation. Just to >> give you a quick comment. I?m okay to add ReferenceQueue.forEach method >> at the first glance. However I have trouble for >> Finalizer.printFinalizationQueue method that doesn?t belong there. What >> are the other alternatives you have explored? >> >> Mandy >> > From dmitry.samersoff at oracle.com Wed May 20 08:42:04 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 20 May 2015 11:42:04 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555C4444.6050008@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <555C4444.6050008@gmail.com> Message-ID: <555C48DC.7000005@oracle.com> Peter, > What about creating a special package-private > java.lang.ref.DiagnosticCommands class I'm not quite happy with current printFinalizationQueue method - love to have a way to print directly to DCMD pipe from Java rather than return a huge string to VM. But lang.ref.Finalizer is cached by VM (see SystemDictionary::Finalizer_klass()) and is known as special i.e. check-when-modify-hotspot class. We don't plan to expand this DCMD so I'm reluctant to create a separate class for one simple static method - it adds extra cost of compiling/loading/care. -Dmitry On 2015-05-20 11:22, Peter Levart wrote: > > > On 05/20/2015 08:51 AM, Dmitry Samersoff wrote: >> Mandy, >> >>> However I have trouble for >>> Finalizer.printFinalizationQueue method that doesn?t belong there. >>> What are the other alternatives you have explored? >> Other alternatives could be to do all hashing/sorting/printing on native >> layer i.e. implement printFinalizationQueue inside VM. >> >> Both options has pros and cons - Java based solution requires less JNI >> calls and better readable but takes more memory. >> >> It might be better to return an array of Map.Entry >> objects to VM rather than one huge string. >> >> -Dmitry > > Hi Dmitry, > > What about creating a special package-private > java.lang.ref.DiagnosticCommands class which is then used as the home of > static printFinalizationQueue method (and possible future diagnostic > commands methods in the package). You could then expose a static > package-private method from Finalizer: > > static void forEachEnqueued(Consumer> action) { > queue.forEach(action); > } > > ...and use it to implement the printFinalizationQueue. > > Regards, Peter > > >> >> >> >> On 2015-05-20 05:54, Mandy Chung wrote: >>>> On May 18, 2015, at 5:17 AM, Dmitry Samersoff >>>> > >>>> wrote: >>>> >>>> Please review updated version of the fix: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >>>> >>>> Most important part of the fix provided by Peter Levart, so all >>>> credentials belongs to him. >>> >>> My apology for being late to the review. The subject line doesn?t catch >>> my attention early enough :) >>> >>> I have to do further detail review tomorrow or so to follow the >>> discussion and closely inspect the reference implementation. Just to >>> give you a quick comment. I?m okay to add ReferenceQueue.forEach method >>> at the first glance. However I have trouble for >>> Finalizer.printFinalizationQueue method that doesn?t belong there. What >>> are the other alternatives you have explored? >>> >>> Mandy >>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mattis.castegren at oracle.com Wed May 20 10:27:24 2015 From: mattis.castegren at oracle.com (Mattis Castegren) Date: Wed, 20 May 2015 03:27:24 -0700 (PDT) Subject: Internal - RE: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: References: Message-ID: Hi How do we usually handle these bugs where we have an external patch to take in, should someone from Dev take ownership of the bug? And should we fix this in 8u60? Kind Regards /Mattis From: Staffan Larsen Sent: den 18 maj 2015 10:23 To: Srinivas Ramakrishna Cc: serviceability-dev at openjdk.java.net serviceability-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net Subject: Re: PerfData counter: sun.gc.policy.generations in JDK 8 Looks like a good patch to me. /Staffan On 14 maj 2015, at 18:12, Srinivas Ramakrishna wrote: https://bugs.openjdk.java.net/browse/JDK-8080345 On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna wrote: With perm gen going away (and being replaced by metaspace) in JDK 8, it makes sense that the counter sun.gc.policy.generations should be "2", rather than "3". However, in JDK 8 that counter still says 3. As I understand, the intention was that this counter would allow you to (for example) know the range of the sun.gc.generation.$num.* counters describing each of $num < sun.gc.policy.generations in the heap. Recall that the erstwhile perm gen in JDK 7 used to be synonymous with sun.gc.generation.2, but the JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. The fix is simple, and I can submit a patch. Is there an existing bug for this? thanks! -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.larsen at oracle.com Wed May 20 11:19:58 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Wed, 20 May 2015 13:19:58 +0200 Subject: RFR(M, v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5559D85B.2050500@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: Dmitry, I?ve look at the changes on the hotspot side. vm/services/diagnosticCommand.hpp: 270 static const char* impact() { 271 return "Low"; 272 } I wonder if the impact should be ?Medium? instead. There aren?t any good guidelines for what impact means, but finalizerinfo does have non-negible impact. test/serviceability/dcmd/gc/FinalizerInfoTest.java: 69 while(wasInitialized != objectsCount); I don?t get the point of this loop. wasInitialized will always be equal to objectsCount at this point. 72 while(wasTrapped < 1); Perhaps the System.gc() call should be inside this loop? test/serviceability/dcmd/gc/HeapInfoTest.java: Can you add some output verification here? /Staffan > On 18 maj 2015, at 14:17, Dmitry Samersoff wrote: > > Everyone, > > Please review updated version of the fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ > > Most important part of the fix provided by Peter Levart, so all > credentials belongs to him. > > -Dmitry > > On 2015-05-16 15:48, Peter Levart wrote: >> >> >> On 05/16/2015 02:38 PM, Peter Levart wrote: >>> >>> >>> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >>>> Derek, >>>> >>>> Personally, I'm for volatile over explicit fence too. >>>> >>>> So I'll change the code if Peter don't have objections. >>> >>> There are no objections. There's one unneeded volatile store to .next >>> field then in enqueue(), but it is followed by another volatile store, >>> so there shouldn't be overhead on Intel and SPARC at least. >>> >>> Regards, Peter >> >> If you make .next field volatile, then perhaps you could also cache it's >> value in reallyPoll() into a local variable, so that it is not read >> twice. Like this: >> >> private Reference reallyPoll() { /* Must hold lock */ >> Reference r = head; >> if (r != null) { >> Reference rn = r.next; // HERE !!! >> head = (rn == r) ? >> null : >> rn; // Unchecked due to the next field having a raw type >> in Reference >> r.queue = NULL; >> r.next = r; >> queueLength--; >> if (r instanceof FinalReference) { >> sun.misc.VM.addFinalRefCount(-1); >> } >> return r; >> } >> return null; >> } >> >> >> >> Peter >> >> >>> >>>> -Dmitry >>>> >>>> On 2015-05-16 01:59, Derek White wrote: >>>>> Hi Dmitry, Peter, >>>>> >>>>> I should read my email more frequently - I missed Dmitry's last webrev >>>>> email. >>>>> >>>>> My comments on a preference for volatile over Unsafe are not a strong >>>>> objection to using Unsafe fences. I just don't have enough experience >>>>> with Unsafe fences yet to agree that this use is correct. >>>>> >>>>> While looking over this Reference code I found 3-6 really simple things >>>>> that need cleaning up that have been there for years, so I'm not a big >>>>> cheerleader for more complicated things here :-) >>>>> >>>>> - Derek >>>>> >>>>> On 5/15/15 6:46 PM, Derek White wrote: >>>>>> Hi Peter, >>>>>> >>>>>> Some comments below... >>>>>> >>>>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>>>> Hi Derek, >>>>>>> >>>>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>>>> but I have some concerns... >>>>>>> I did have a concern too, ... >>>>>>> >>>>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>>>> Hi Dmitry, >>>>>>>>> >>>>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>>>> logical. >>>>>>>>> >>>>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>>>> application. Suppose one wants to diagnose the application because >>>>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>>>> up gradually and is already holding several million objects. Now >>>>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>>>> iteration over and grouping/counting of several millions of >>>>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>>>> reference handler thread. But does not block the threads that >>>>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>>>> most of the time, getting OOMEs, etc... >>>>>>>>> >>>>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>>>> purposes without holding a lock. The modification required to do >>>>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>>>> >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>>>> >>>>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>>>> >>>>>>>> Your change makes "inactive" References superficially look like >>>>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>>>> yet seen a case where it would get confused by this change, but >>>>>>>> there could easily be something like that lurking in the GC code. >>>>>>> ...but then I thought that GC can in no way treat a Reference >>>>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>>>> already dequeued (inactive) - responsibility is already on the Java >>>>>>> side. Currently the definition of Reference.next is this: >>>>>>> >>>>>>> /* When active: NULL >>>>>>> * pending: this >>>>>>> * Enqueued: next reference in queue (or this if last) >>>>>>> * Inactive: this >>>>>>> */ >>>>>>> @SuppressWarnings("rawtypes") >>>>>>> Reference next; >>>>>>> >>>>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>>>> scans their lists too, the state of a Reference that is enqueued as >>>>>>> last in list is indistinguishable from the state of inactive >>>>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>>>> can't be established solely on the value of .next field ( == this or >>>>>>> != this)... >>>>>>> >>>>>>> But I should inspect the GC code too to build a better understanding >>>>>>> of that part of the story... >>>>>>> >>>>>>> Anyway. It turns out that there is already enough state in Reference >>>>>>> to destinguish between it being enqueued as last in list and already >>>>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>>>> be achieved either by making Reference.next a volatile field or by a >>>>>>> couple of explicit fences: >>>>>>> >>>>>>> >>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>>>> >>>>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>>>> before linking the reference into the queue's head in >>>>>>> ReferenceQueue.enqueue(): >>>>>>> >>>>>>> r.queue = ENQUEUED; >>>>>>> r.next = (head == null) ? r : head; >>>>>>> head = r; >>>>>>> >>>>>>> Both stores are volatile. >>>>>> This sounds a bit better. I don't have a good handle on the pros and >>>>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>>>> Kim might jump in soon. >>>>>> >>>>>> I'd like to suggest an entirely less-clever solution. Since the >>>>>> problem is that forEach() might see inconsistent values for the queue >>>>>> and next fields of a Reference, but we don't want to grab a lock >>>>>> walking the whole queue, we could instead grab the lock as we look at >>>>>> each Reference (and do the "back-up" trick if we were interfered >>>>>> with). Of course this is slower than going lock-free, but it's not any >>>>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>>>> >>>>>> The only benefit of this solution over the others is that it is less >>>>>> clever, and I'm not sure how much cleverness this problem deserves. >>>>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>>>> that is sufficient. >>>>>> >>>>>> - Derek >>>>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>>>> forEach iteration method with no concrete logic. >>>>>>>>> >>>>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>>>> rest in DiagnosticCommands. >>>>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>>>> Referents were leaked back into the heap, then we could get >>>>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>>>> on it's own - any finalizer might resurrect its object if not >>>>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>>>> object. This could invalidate application invariants? >>>>>>> Well, it all stays in the confines of package-private API - internal >>>>>>> to JDK. Any future additional use should be reviewed carefully. >>>>>>> Comments on the forEach() method should warn about that. >>>>>>> >>>>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>>>> there are pathological scenarios where this gets severe. This is >>>>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>>>> introduce subtle bugs to the system in general. A change in this >>>>>>>> area should get a thorough review from both the runtime and GC sides. >>>>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>>>> does not introduce any logical changes to existing code. >>>>>>> >>>>>>>> Better yet, the reference handling code in GC and runtime should >>>>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>>>> I may have some proposals in that direction. Stay tuned. >>>>>>> >>>>>>> Regards, Peter >>>>>>> >>>>>>>> - Derek >>>>>>>> >>>>>>>>> Regards, Peter >>>>>>>>> >>>>>>>>> >>>>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>>>> Everybody, >>>>>>>>>> >>>>>>>>>> Updated version: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>>>> >>>>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>> >>>>>>>>>>> Lots of good comments here. >>>>>>>>>>> >>>>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>>>> general iterator and lambda. >>>>>>>>>>> >>>>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>>>> >>>>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>>>> >>>>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>>>> >>>>>>>>>>> - Derek White, GC team >>>>>>>>>>> >>>>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>>> >>>>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>>>> Dmitry, >>>>>>>>>>>>> >>>>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>>>> >>>>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>>>> running out of memory. >>>>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>>>> >>>>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>>>> interfere with synchronization? >>>>>>>>>>>> >>>>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>>>> >>>>>>>>>>>> private void remove() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized == this) { >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> unfinalized = this.next; >>>>>>>>>>>> } else { >>>>>>>>>>>> unfinalized = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>>>> } >>>>>>>>>>>> if (this.prev != null) { >>>>>>>>>>>> this.prev.next = this.next; >>>>>>>>>>>> } >>>>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>>>> traverse the list unsynchronized >>>>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>>>> finalized */ >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>>>> pointer could be used instead: >>>>>>>>>>>> >>>>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>>>> return (prev == this); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>>>> >>>>>>>>>>>> private void add() { >>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>> if (unfinalized != null) { >>>>>>>>>>>> this.next = unfinalized; >>>>>>>>>>>> unfinalized.prev = this; >>>>>>>>>>>> } >>>>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>>>> // instance fully initialized >>>>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>>>> U.storeFence(); >>>>>>>>>>>> unfinalized = this; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>>>> >>>>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>>>> out? >>>>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>>>> two purposes: >>>>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>>>> >>>>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>>>> finalize() method. >>>>>>>>>>>> >>>>>>>>>>>> Regards, Peter >>>>>>>>>>>> >>>>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>>>> >>>>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>>>> permission() and limit access: >>>>>>>>>>>>> >>>>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>>>> return p; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> /Staffan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please review the fix: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>>>> >>>>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>>>> with count >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. >>> >> > > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From peter.levart at gmail.com Wed May 20 11:44:46 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 20 May 2015 13:44:46 +0200 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555C48DC.7000005@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <555C4444.6050008@gmail.com> <555C48DC.7000005@oracle.com> Message-ID: <555C73AE.8050601@gmail.com> On 05/20/2015 10:42 AM, Dmitry Samersoff wrote: > Peter, > >> What about creating a special package-private >> java.lang.ref.DiagnosticCommands class > I'm not quite happy with current printFinalizationQueue method - love to > have a way to print directly to DCMD pipe from Java rather than return a > huge string to VM. > > But lang.ref.Finalizer is cached by VM (see > SystemDictionary::Finalizer_klass()) and is known as special > i.e. check-when-modify-hotspot class. > > We don't plan to expand this DCMD so I'm reluctant to create a separate > class for one simple static method - it adds extra cost of > compiling/loading/care. > > -Dmitry Ok then. I see diagnostic commands mostly format their output in native code. But for some of them (like this finalizer histogram) it would be nice to have a Java wrapper for hotspot's outputStream. It wouldn't be difficult to create such a class with JNI bindings, but where should one put it? Into which module? Otherwise, the simplest unformated data structure to return from such a method is an Object[] where you have String/Integer objects intermingled in pairs. Returning a Map.Entry[] is already more complicated to iterate and navigate in native code. Map even more so. Regards, Peter > > On 2015-05-20 11:22, Peter Levart wrote: >> >> On 05/20/2015 08:51 AM, Dmitry Samersoff wrote: >>> Mandy, >>> >>>> However I have trouble for >>>> Finalizer.printFinalizationQueue method that doesn?t belong there. >>>> What are the other alternatives you have explored? >>> Other alternatives could be to do all hashing/sorting/printing on native >>> layer i.e. implement printFinalizationQueue inside VM. >>> >>> Both options has pros and cons - Java based solution requires less JNI >>> calls and better readable but takes more memory. >>> >>> It might be better to return an array of Map.Entry >>> objects to VM rather than one huge string. >>> >>> -Dmitry >> Hi Dmitry, >> >> What about creating a special package-private >> java.lang.ref.DiagnosticCommands class which is then used as the home of >> static printFinalizationQueue method (and possible future diagnostic >> commands methods in the package). You could then expose a static >> package-private method from Finalizer: >> >> static void forEachEnqueued(Consumer> action) { >> queue.forEach(action); >> } >> >> ...and use it to implement the printFinalizationQueue. >> >> Regards, Peter >> >> >>> >>> >>> On 2015-05-20 05:54, Mandy Chung wrote: >>>>> On May 18, 2015, at 5:17 AM, Dmitry Samersoff >>>>> > >>>>> wrote: >>>>> >>>>> Please review updated version of the fix: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >>>>> >>>>> Most important part of the fix provided by Peter Levart, so all >>>>> credentials belongs to him. >>>> My apology for being late to the review. The subject line doesn?t catch >>>> my attention early enough :) >>>> >>>> I have to do further detail review tomorrow or so to follow the >>>> discussion and closely inspect the reference implementation. Just to >>>> give you a quick comment. I?m okay to add ReferenceQueue.forEach method >>>> at the first glance. However I have trouble for >>>> Finalizer.printFinalizationQueue method that doesn?t belong there. What >>>> are the other alternatives you have explored? >>>> >>>> Mandy >>>> > From dmitry.samersoff at oracle.com Wed May 20 12:11:57 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 20 May 2015 15:11:57 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555C73AE.8050601@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <555C4444.6050008@gmail.com> <555C48DC.7000005@oracle.com> <555C73AE.8050601@gmail.com> Message-ID: <555C7A0D.3010803@oracle.com> Peter, > I see diagnostic commands mostly format their output in native code. > But for some of them (like this finalizer histogram) it would be nice > to have a Java wrapper for hotspot's outputStream. I love to have an ability to write pure-java DCMD's without touching of hotspot code but it's a long term goal, out of scope of this fix. Just a wrapper around hotspot output stream has a couple of underwear complications around memory management and error handling, so it should be a separate project. -Dmitry On 2015-05-20 14:44, Peter Levart wrote: > > > On 05/20/2015 10:42 AM, Dmitry Samersoff wrote: >> Peter, >> >>> What about creating a special package-private >>> java.lang.ref.DiagnosticCommands class >> I'm not quite happy with current printFinalizationQueue method - love to >> have a way to print directly to DCMD pipe from Java rather than return a >> huge string to VM. >> >> But lang.ref.Finalizer is cached by VM (see >> SystemDictionary::Finalizer_klass()) and is known as special >> i.e. check-when-modify-hotspot class. >> >> We don't plan to expand this DCMD so I'm reluctant to create a separate >> class for one simple static method - it adds extra cost of >> compiling/loading/care. >> >> -Dmitry > > Ok then. > > I see diagnostic commands mostly format their output in native code. But > for some of them (like this finalizer histogram) it would be nice to > have a Java wrapper for hotspot's outputStream. It wouldn't be difficult > to create such a class with JNI bindings, but where should one put it? > Into which module? > > Otherwise, the simplest unformated data structure to return from such a > method is an Object[] where you have String/Integer objects intermingled > in pairs. Returning a Map.Entry[] is already more > complicated to iterate and navigate in native code. Map > even more so. > > Regards, Peter > >> >> On 2015-05-20 11:22, Peter Levart wrote: >>> >>> On 05/20/2015 08:51 AM, Dmitry Samersoff wrote: >>>> Mandy, >>>> >>>>> However I have trouble for >>>>> Finalizer.printFinalizationQueue method that doesn?t belong there. >>>>> What are the other alternatives you have explored? >>>> Other alternatives could be to do all hashing/sorting/printing on >>>> native >>>> layer i.e. implement printFinalizationQueue inside VM. >>>> >>>> Both options has pros and cons - Java based solution requires less JNI >>>> calls and better readable but takes more memory. >>>> >>>> It might be better to return an array of Map.Entry >>>> objects to VM rather than one huge string. >>>> >>>> -Dmitry >>> Hi Dmitry, >>> >>> What about creating a special package-private >>> java.lang.ref.DiagnosticCommands class which is then used as the home of >>> static printFinalizationQueue method (and possible future diagnostic >>> commands methods in the package). You could then expose a static >>> package-private method from Finalizer: >>> >>> static void forEachEnqueued(Consumer> action) { >>> queue.forEach(action); >>> } >>> >>> ...and use it to implement the printFinalizationQueue. >>> >>> Regards, Peter >>> >>> >>>> >>>> >>>> On 2015-05-20 05:54, Mandy Chung wrote: >>>>>> On May 18, 2015, at 5:17 AM, Dmitry Samersoff >>>>>> > >>>>>> wrote: >>>>>> >>>>>> Please review updated version of the fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >>>>>> >>>>>> Most important part of the fix provided by Peter Levart, so all >>>>>> credentials belongs to him. >>>>> My apology for being late to the review. The subject line doesn?t >>>>> catch >>>>> my attention early enough :) >>>>> >>>>> I have to do further detail review tomorrow or so to follow the >>>>> discussion and closely inspect the reference implementation. Just to >>>>> give you a quick comment. I?m okay to add ReferenceQueue.forEach >>>>> method >>>>> at the first glance. However I have trouble for >>>>> Finalizer.printFinalizationQueue method that doesn?t belong there. >>>>> What >>>>> are the other alternatives you have explored? >>>>> >>>>> Mandy >>>>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Wed May 20 12:28:01 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 20 May 2015 15:28:01 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> Message-ID: <555C7DD1.7090600@oracle.com> Staffan, On 2015-05-20 14:19, Staffan Larsen wrote: > Dmitry, > > I?ve look at the changes on the hotspot side. > > vm/services/diagnosticCommand.hpp: > > 270 static const char* impact() { > 271 return "Low"; > 272 } > > I wonder if the impact should be ?Medium? instead. There aren?t any good guidelines for what impact means, but finalizerinfo does have non-negible impact. OK. I'll change it. > > > test/serviceability/dcmd/gc/FinalizerInfoTest.java: > > 69 while(wasInitialized != objectsCount); > > I don?t get the point of this loop. wasInitialized will always be equal to objectsCount at this point. Agree. It left from one of previous experiments. > > 72 while(wasTrapped < 1); > > Perhaps the System.gc() call should be inside this loop? System.gc() instruct hotspot to start GC thread, but this thread may not start immediately. We need this loop to wait for GC thread. > test/serviceability/dcmd/gc/HeapInfoTest.java: > > Can you add some output verification here? This DCMD calls Universe::heap()->print_on() and output of this command should be covered by GC tests, more complicated than this one, because actual output depends to GC and heap parameters. I can just check for presence of "Metaspace" world in the DCMD output. -Dmitry > > > /Staffan > > >> On 18 maj 2015, at 14:17, Dmitry Samersoff wrote: >> >> Everyone, >> >> Please review updated version of the fix: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.07/ >> >> Most important part of the fix provided by Peter Levart, so all >> credentials belongs to him. >> >> -Dmitry >> >> On 2015-05-16 15:48, Peter Levart wrote: >>> >>> >>> On 05/16/2015 02:38 PM, Peter Levart wrote: >>>> >>>> >>>> On 05/16/2015 09:35 AM, Dmitry Samersoff wrote: >>>>> Derek, >>>>> >>>>> Personally, I'm for volatile over explicit fence too. >>>>> >>>>> So I'll change the code if Peter don't have objections. >>>> >>>> There are no objections. There's one unneeded volatile store to .next >>>> field then in enqueue(), but it is followed by another volatile store, >>>> so there shouldn't be overhead on Intel and SPARC at least. >>>> >>>> Regards, Peter >>> >>> If you make .next field volatile, then perhaps you could also cache it's >>> value in reallyPoll() into a local variable, so that it is not read >>> twice. Like this: >>> >>> private Reference reallyPoll() { /* Must hold lock */ >>> Reference r = head; >>> if (r != null) { >>> Reference rn = r.next; // HERE !!! >>> head = (rn == r) ? >>> null : >>> rn; // Unchecked due to the next field having a raw type >>> in Reference >>> r.queue = NULL; >>> r.next = r; >>> queueLength--; >>> if (r instanceof FinalReference) { >>> sun.misc.VM.addFinalRefCount(-1); >>> } >>> return r; >>> } >>> return null; >>> } >>> >>> >>> >>> Peter >>> >>> >>>> >>>>> -Dmitry >>>>> >>>>> On 2015-05-16 01:59, Derek White wrote: >>>>>> Hi Dmitry, Peter, >>>>>> >>>>>> I should read my email more frequently - I missed Dmitry's last webrev >>>>>> email. >>>>>> >>>>>> My comments on a preference for volatile over Unsafe are not a strong >>>>>> objection to using Unsafe fences. I just don't have enough experience >>>>>> with Unsafe fences yet to agree that this use is correct. >>>>>> >>>>>> While looking over this Reference code I found 3-6 really simple things >>>>>> that need cleaning up that have been there for years, so I'm not a big >>>>>> cheerleader for more complicated things here :-) >>>>>> >>>>>> - Derek >>>>>> >>>>>> On 5/15/15 6:46 PM, Derek White wrote: >>>>>>> Hi Peter, >>>>>>> >>>>>>> Some comments below... >>>>>>> >>>>>>> On 5/14/15 3:50 AM, Peter Levart wrote: >>>>>>>> Hi Derek, >>>>>>>> >>>>>>>> On 05/13/2015 10:34 PM, Derek White wrote: >>>>>>>>> Hi Peter, >>>>>>>>> >>>>>>>>> I don't have smoking gun evidence that your change introduces a bug, >>>>>>>>> but I have some concerns... >>>>>>>> I did have a concern too, ... >>>>>>>> >>>>>>>>> On 5/12/15 6:05 PM, Peter Levart wrote: >>>>>>>>>> Hi Dmitry, >>>>>>>>>> >>>>>>>>>> You iterate the queue then, not the unfinalized list. That's more >>>>>>>>>> logical. >>>>>>>>>> >>>>>>>>>> Holding the queue's lock may pause reference handler and finalizer >>>>>>>>>> threads for the entire time of iteration. This can blow up the >>>>>>>>>> application. Suppose one wants to diagnose the application because >>>>>>>>>> he suspects that finalizer thread hardly keeps up with production >>>>>>>>>> of finalizable instances. This can happen if the allocation rate of >>>>>>>>>> finalizable objects is very high and/or finalize() methods are not >>>>>>>>>> coded to be fast enough. Suppose the queue of Finalizer(s) builds >>>>>>>>>> up gradually and is already holding several million objects. Now >>>>>>>>>> you initiate the diagnostic command to print the queue. The >>>>>>>>>> iteration over and grouping/counting of several millions of >>>>>>>>>> Finalizer(s) takes some time. This blocks finalizer thread and >>>>>>>>>> reference handler thread. But does not block the threads that >>>>>>>>>> allocate finalizable objects. By the time the iteration is over, >>>>>>>>>> the JVM blows up and application slows down to a halt doing GCs >>>>>>>>>> most of the time, getting OOMEs, etc... >>>>>>>>>> >>>>>>>>>> It is possible to iterate the elements of the queue for diagnostic >>>>>>>>>> purposes without holding a lock. The modification required to do >>>>>>>>>> that is the following (havent tested this, but I think it should work): >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.01/ >>>>>>>>>> >>>>>>>>> One issue to watch out for is the garbage collectors inspect the >>>>>>>>> Reference.next field from C code directly (to distinguish active vs. >>>>>>>>> pending, iterate over oops, etc.). You can search hotspot for >>>>>>>>> java_lang_ref_Reference::next*, java_lang_ref_Reference::set_next*. >>>>>>>>> >>>>>>>>> Your change makes "inactive" References superficially look like >>>>>>>>> "enqueued" References. The GC code is rather subtle and I haven't >>>>>>>>> yet seen a case where it would get confused by this change, but >>>>>>>>> there could easily be something like that lurking in the GC code. >>>>>>>> ...but then I thought that GC can in no way treat a Reference >>>>>>>> differently whether it is still enqueued in a ReferenceQueue or >>>>>>>> already dequeued (inactive) - responsibility is already on the Java >>>>>>>> side. Currently the definition of Reference.next is this: >>>>>>>> >>>>>>>> /* When active: NULL >>>>>>>> * pending: this >>>>>>>> * Enqueued: next reference in queue (or this if last) >>>>>>>> * Inactive: this >>>>>>>> */ >>>>>>>> @SuppressWarnings("rawtypes") >>>>>>>> Reference next; >>>>>>>> >>>>>>>> We see that, unless GC inspects all ReferenceQueue instances and >>>>>>>> scans their lists too, the state of a Reference that is enqueued as >>>>>>>> last in list is indistinguishable from the state of inactive >>>>>>>> Reference. So I deduced that this distinction (enqueued/inactive) >>>>>>>> can't be established solely on the value of .next field ( == this or >>>>>>>> != this)... >>>>>>>> >>>>>>>> But I should inspect the GC code too to build a better understanding >>>>>>>> of that part of the story... >>>>>>>> >>>>>>>> Anyway. It turns out that there is already enough state in Reference >>>>>>>> to destinguish between it being enqueued as last in list and already >>>>>>>> dequeued (inactive) - the additional state is Reference.queue that >>>>>>>> transitions from ReferenceQueue.ENQUEUED -> ReferenceQueue.NULL in >>>>>>>> ReferenceQueue.reallyPoll. We need to just make sure the two fields >>>>>>>> (r.next and r.queue) are assigned and read in correct order. This can >>>>>>>> be achieved either by making Reference.next a volatile field or by a >>>>>>>> couple of explicit fences: >>>>>>>> >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.02/ >>>>>>>> >>>>>>>> The assignment of r.queue to ReferenceQueue.ENQUEUED already happens >>>>>>>> before linking the reference into the queue's head in >>>>>>>> ReferenceQueue.enqueue(): >>>>>>>> >>>>>>>> r.queue = ENQUEUED; >>>>>>>> r.next = (head == null) ? r : head; >>>>>>>> head = r; >>>>>>>> >>>>>>>> Both stores are volatile. >>>>>>> This sounds a bit better. I don't have a good handle on the pros and >>>>>>> cons of a volatile field over explicit fences via Unsafe in this case. >>>>>>> Kim might jump in soon. >>>>>>> >>>>>>> I'd like to suggest an entirely less-clever solution. Since the >>>>>>> problem is that forEach() might see inconsistent values for the queue >>>>>>> and next fields of a Reference, but we don't want to grab a lock >>>>>>> walking the whole queue, we could instead grab the lock as we look at >>>>>>> each Reference (and do the "back-up" trick if we were interfered >>>>>>> with). Of course this is slower than going lock-free, but it's not any >>>>>>> more overhead than the ReferenceHandler thread or FinalizerThreads incur. >>>>>>> >>>>>>> The only benefit of this solution over the others is that it is less >>>>>>> clever, and I'm not sure how much cleverness this problem deserves. >>>>>>> Given that, and ranking the solutions as "lock" < (clever) "volatile" >>>>>>> < "fence", I'd be happier with the "volatile" or "lock" solution if >>>>>>> that is sufficient. >>>>>>> >>>>>>> - Derek >>>>>>>>>> I also suggest the addition to the ReferenceQueue to be contained >>>>>>>>>> (package-private) and as generic as possible. That's why I suggest >>>>>>>>>> forEach iteration method with no concrete logic. >>>>>>>>>> >>>>>>>>>> It would be possible to encapsulate the entire logic into a special >>>>>>>>>> package-private class (say java.lang.ref.DiagnosticCommands) with >>>>>>>>>> static method(s) (printFinalizationQueue)... You could just expose >>>>>>>>>> a package-private forEach static method from Finalizer and code the >>>>>>>>>> rest in DiagnosticCommands. >>>>>>>>> That's good for encapsulation. But I'm concerned that if "forEach" >>>>>>>>> got exposed beyond careful use in DiagnosticCommands, and the >>>>>>>>> Referents were leaked back into the heap, then we could get >>>>>>>>> unexpected object resurrection after finalization. This isn't a bug >>>>>>>>> on it's own - any finalizer might resurrect its object if not >>>>>>>>> careful, but ordinarily /only/ the finalizer could resurrect the >>>>>>>>> object. This could invalidate application invariants? >>>>>>>> Well, it all stays in the confines of package-private API - internal >>>>>>>> to JDK. Any future additional use should be reviewed carefully. >>>>>>>> Comments on the forEach() method should warn about that. >>>>>>>> >>>>>>>>> I agree that using a lock over the ReferenceQueue iteration opens up >>>>>>>>> /another/ case of diagnostics affecting application behavior. And >>>>>>>>> there are pathological scenarios where this gets severe. This is >>>>>>>>> unfortunate but not uncommon. There is enough complication here that >>>>>>>>> you should be sure that the fix for diagnostics performance doesn't >>>>>>>>> introduce subtle bugs to the system in general. A change in this >>>>>>>>> area should get a thorough review from both the runtime and GC sides. >>>>>>>> Is the webrev.02 proposed above more acceptable in that respect? It >>>>>>>> does not introduce any logical changes to existing code. >>>>>>>> >>>>>>>>> Better yet, the reference handling code in GC and runtime should >>>>>>>>> probably be thrown out and re-written, but that's another issue :-) >>>>>>>> I may have some proposals in that direction. Stay tuned. >>>>>>>> >>>>>>>> Regards, Peter >>>>>>>> >>>>>>>>> - Derek >>>>>>>>> >>>>>>>>>> Regards, Peter >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 05/12/2015 07:10 PM, Dmitry Samersoff wrote: >>>>>>>>>>> Everybody, >>>>>>>>>>> >>>>>>>>>>> Updated version: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.03/ >>>>>>>>>>> >>>>>>>>>>> Now it iterates over queue and output result sorted by number of instances. >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> On 2015-05-07 00:51, Derek White wrote: >>>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>>> >>>>>>>>>>>> Lots of good comments here. >>>>>>>>>>>> >>>>>>>>>>>> On the topic of what list should be printed out, I think we should focus >>>>>>>>>>>> on objects waiting to be finalized - e.g. the contents of the >>>>>>>>>>>> ReferenceQueue. It's more of a pain to walk the ReferenceQueue, but you >>>>>>>>>>>> could add a summerizeQueue(TreeMap) method, or a >>>>>>>>>>>> general iterator and lambda. >>>>>>>>>>>> >>>>>>>>>>>> A histogram of objects with finalize methods might also be interesting, >>>>>>>>>>>> but you can get most of the same information from a heap histogram >>>>>>>>>>>> (ClassHistogramDCmd, and search for count of Finalizer instances). >>>>>>>>>>>> >>>>>>>>>>>> BTW, by only locking the ReferenceQueue, we should only be blocking the >>>>>>>>>>>> FinalizerThread from making progress (and I think there's a GC thread >>>>>>>>>>>> that runs after GC that sorts found References objects that need >>>>>>>>>>>> processing into their respective ReferenceQueues). But locking the >>>>>>>>>>>> "unfinalized" list blocks initializing any object with a finalize() method. >>>>>>>>>>>> >>>>>>>>>>>> The sorting suggestion is a nice touch. >>>>>>>>>>>> >>>>>>>>>>>> - Derek White, GC team >>>>>>>>>>>> >>>>>>>>>>>> On 5/5/15 10:40 AM, Peter Levart wrote: >>>>>>>>>>>>> Hi Dmitry, Staffan, >>>>>>>>>>>>> >>>>>>>>>>>>> On 05/05/2015 12:38 PM, Staffan Larsen wrote: >>>>>>>>>>>>>> Dmitry, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think this should be reviewed on hotspot-gc and core-libs-dev as >>>>>>>>>>>>>> well considering the changes to Finalizer. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I?m a little worried about the potentially very large String that is >>>>>>>>>>>>>> returned from printFinalizationQueue(). A possible different approach >>>>>>>>>>>>>> would be to write printFinalizationQueue() in C++ inside Hotspot. >>>>>>>>>>>>>> That would allow for outputting one line at a time. The output would >>>>>>>>>>>>>> still be saved in memory (since the stream is buffered), but at least >>>>>>>>>>>>>> the data is only saved once in memory, then. It would make the code a >>>>>>>>>>>>>> bit harder to write, so its a question of how scared we are of >>>>>>>>>>>>>> running out of memory. >>>>>>>>>>>>> If the output is just a histogram of # of instances per class name, >>>>>>>>>>>>> then it should not be too large, as there are not many different >>>>>>>>>>>>> classes overriding finalize() method (I counted 72 overriddings of >>>>>>>>>>>>> finalize() method in the whole jdk/src tree). >>>>>>>>>>>>> >>>>>>>>>>>>> I'm more concerned about the fact that while traversing the list, a >>>>>>>>>>>>> lock is held, which might impact prompt finalization(). Is it >>>>>>>>>>>>> acceptable for diagnostic output to impact performance and/or >>>>>>>>>>>>> interfere with synchronization? >>>>>>>>>>>>> >>>>>>>>>>>>> It might be possible to scan the list without holding a lock for >>>>>>>>>>>>> diagnostic purposes if Finalizer.remove() didn't set the removed >>>>>>>>>>>>> Finalizer.next pointer to point back to itself: >>>>>>>>>>>>> >>>>>>>>>>>>> private void remove() { >>>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>>> if (unfinalized == this) { >>>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>>> unfinalized = this.next; >>>>>>>>>>>>> } else { >>>>>>>>>>>>> unfinalized = this.prev; >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> if (this.next != null) { >>>>>>>>>>>>> this.next.prev = this.prev; >>>>>>>>>>>>> } >>>>>>>>>>>>> if (this.prev != null) { >>>>>>>>>>>>> this.prev.next = this.next; >>>>>>>>>>>>> } >>>>>>>>>>>>> // this.next = this; must not be set so that we can >>>>>>>>>>>>> traverse the list unsynchronized >>>>>>>>>>>>> this.prev = this; /* Indicates that this has been >>>>>>>>>>>>> finalized */ >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> For detecting whether a Finalizer is already processed, the 'prev' >>>>>>>>>>>>> pointer could be used instead: >>>>>>>>>>>>> >>>>>>>>>>>>> private boolean hasBeenFinalized() { >>>>>>>>>>>>> return (prev == this); >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> Also, to make sure a data race dereferencing 'unfinalized' in >>>>>>>>>>>>> unsynchronized printFinalizationQueue() would get you a fully >>>>>>>>>>>>> initialized Finalizer instance (in particular the next pointer), you >>>>>>>>>>>>> would have to make the 'unfinalized' field volatile or insert an >>>>>>>>>>>>> Unsafe.storeFence() before assigning to 'unfinalized': >>>>>>>>>>>>> >>>>>>>>>>>>> private void add() { >>>>>>>>>>>>> synchronized (lock) { >>>>>>>>>>>>> if (unfinalized != null) { >>>>>>>>>>>>> this.next = unfinalized; >>>>>>>>>>>>> unfinalized.prev = this; >>>>>>>>>>>>> } >>>>>>>>>>>>> // make sure a data race dereferencing 'unfinalized' >>>>>>>>>>>>> // in printFinalizationQueue() does see the Finalizer >>>>>>>>>>>>> // instance fully initialized >>>>>>>>>>>>> // (in particular the 'next' pointer) >>>>>>>>>>>>> U.storeFence(); >>>>>>>>>>>>> unfinalized = this; >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> By doing these modifications, I think you can remove >>>>>>>>>>>>> synchronized(lock) {} from printFinalizationQueue(). >>>>>>>>>>>>> >>>>>>>>>>>>>> I see you are traversing the ?unfinalized? list in Finalizer, whereas >>>>>>>>>>>>>> the old SA code for ?-finalizerinfo' traverses the ?queue? list. I am >>>>>>>>>>>>>> not sure of the difference, perhaps someone from the GC team can help >>>>>>>>>>>>>> out? >>>>>>>>>>>>> The 'queue' is a ReferenceQueue of Finalizer (FinalReference) >>>>>>>>>>>>> instances pending processing by finalizer thread because their >>>>>>>>>>>>> referents are eligible for finalization (they are not reachable any >>>>>>>>>>>>> more). The 'unfinalized' is a doubly-linked list of all Finalizer >>>>>>>>>>>>> instances for which their referents have not been finalized yet >>>>>>>>>>>>> (including those that are still reachable and alive). The later serves >>>>>>>>>>>>> two purposes: >>>>>>>>>>>>> - it keeps Finalizer instances reachable until they are processed >>>>>>>>>>>>> - it is a source of unfinalized instances for >>>>>>>>>>>>> running-finalizers-on-exit if requested by >>>>>>>>>>>>> System.runFinalizersOnExit(true); >>>>>>>>>>>>> >>>>>>>>>>>>> So it really depends on what one would like to see. Showing the queue >>>>>>>>>>>>> may be interesting if one wants to see how the finalizer thread is >>>>>>>>>>>>> coping with processing the finalize() invocations. Showing unfinalized >>>>>>>>>>>>> list may be interesting if one wants to know how many live + >>>>>>>>>>>>> finalization pending instances are there on the heap that override >>>>>>>>>>>>> finalize() method. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Peter >>>>>>>>>>>>> >>>>>>>>>>>>>> For the output, it would be a nice touch to sort it on the number of >>>>>>>>>>>>>> references for each type. Perhaps outputting it more like a table >>>>>>>>>>>>>> (see the old code again) would also make it easier to digest. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In diagnosticCommand.hpp, the new commands should override >>>>>>>>>>>>>> permission() and limit access: >>>>>>>>>>>>>> >>>>>>>>>>>>>> static const JavaPermission permission() { >>>>>>>>>>>>>> JavaPermission p = {"java.lang.management.ManagementPermission", >>>>>>>>>>>>>> "monitor", NULL}; >>>>>>>>>>>>>> return p; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> The two tests don?t validate the output in any way. Would it be >>>>>>>>>>>>>> possible to add validation? Perhaps hard to make sure an object is on >>>>>>>>>>>>>> the finalizer queue? Hmm. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> /Staffan >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 5 maj 2015, at 12:01, Dmitry Samersoff >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Everyone, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please review the fix: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> heap dcmd outputs the same information as SIGBREAK >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> finalizerinfo dcmd outputs a list of all classes in finalization queue >>>>>>>>>>>>>>> with count >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dmitry Samersoff >>>>>>>>>>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>>>>>>>>>> * I would love to change the world, but they won't give me the sources. >>>> >>> >> >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From ivan.gerasimov at oracle.com Wed May 20 13:57:08 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 20 May 2015 16:57:08 +0300 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... In-Reply-To: <555C3214.50004@oracle.com> References: <555BCD1E.9010406@oracle.com> <555C3214.50004@oracle.com> Message-ID: <555C92B4.2010507@oracle.com> On 20.05.2015 10:04, David Holmes wrote: > Hi Ivan, > > On 20/05/2015 9:54 AM, Ivan Gerasimov wrote: >> Hello! >> >> Since the beginning of the year a warning about an expired timeout has >> been seen a few times. >> It is most likely to observe this under a heavy load, when many threads >> are exiting. >> >> The propose is to increase the timeout. >> The rationale is that if an application needs more time for the threads >> to exit, let's give it that time. > > Rather than just bumping the timeout a little I would suggest bumping > it a lot - make it a few minutes. If that fails then it suggests we > have a real problem not just a slow or loaded machine. Otherwise if we > just keep bumping the timeout it serves no purpose. > Alright. Then let's make it 5 minutes for both product and debug bits: WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/01/webrev/ Sincerely yours, Ivn > Cheers, > David > > >> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 >> WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ >> >> Sincerely yours, >> Ivan > > From mandy.chung at oracle.com Wed May 20 14:38:19 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 20 May 2015 07:38:19 -0700 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <555B3D00.3080204@oracle.com> References: <555B3D00.3080204@oracle.com> Message-ID: On 05/19/2015 06:39 AM, Jaroslav Bachorik wrote: > Please, review the following change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8080663 > Webrev: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 > > The title says it all. This enhancement is about replacing the arbitrary reflection based code with a cleaner approach based on sun.misc.SharedSecrets class. > Using the shared secret approach is okay. Some suggestion: Since both sun.misc.JavaBeansIntrospectorAccess and ConstructorPropertiesAccess are for JMX to access java.beans types, it could be simplified to have one single interface (e.g. sun.misc.JavaBeansAccess) to define Method getReadMethod(Class, String); String[] getConstructorProperties(Constructor); AnnotationHelper and BeansIntrospector can then be merged into the new JavaBeansAccessor class. Can you add a regression for this, if not exist, as Daniel points out? Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed May 20 14:56:38 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 20 May 2015 15:56:38 +0100 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: References: <555B3D00.3080204@oracle.com> Message-ID: <555CA0A6.6040405@oracle.com> On 20/05/2015 15:38, Mandy Chung wrote: > > Using the shared secret approach is okay. Some suggestion: > > Since both sun.misc.JavaBeansIntrospectorAccess and > ConstructorPropertiesAccess are for JMX to access java.beans types, > it could be simplified to have one single interface (e.g. > sun.misc.JavaBeansAccess) to define > Method getReadMethod(Class, String); > String[] getConstructorProperties(Constructor); > > AnnotationHelper and BeansIntrospector can then be merged into the new > JavaBeansAccessor class. > I agree and it make it easier to get to other beans classes if needed. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Wed May 20 15:19:47 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 20 May 2015 09:19:47 -0600 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... In-Reply-To: <555C92B4.2010507@oracle.com> References: <555BCD1E.9010406@oracle.com> <555C3214.50004@oracle.com> <555C92B4.2010507@oracle.com> Message-ID: <555CA613.9010007@oracle.com> On 5/20/15 7:57 AM, Ivan Gerasimov wrote: > > > On 20.05.2015 10:04, David Holmes wrote: >> Hi Ivan, >> >> On 20/05/2015 9:54 AM, Ivan Gerasimov wrote: >>> Hello! >>> >>> Since the beginning of the year a warning about an expired timeout has >>> been seen a few times. >>> It is most likely to observe this under a heavy load, when many threads >>> are exiting. >>> >>> The propose is to increase the timeout. >>> The rationale is that if an application needs more time for the threads >>> to exit, let's give it that time. >> >> Rather than just bumping the timeout a little I would suggest bumping >> it a lot - make it a few minutes. If that fails then it suggests we >> have a real problem not just a slow or loaded machine. Otherwise if >> we just keep bumping the timeout it serves no purpose. >> > > Alright. Then let's make it 5 minutes for both product and debug bits: > > WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/01/webrev/ Thumbs up. If I remember correctly, we'll only hit the 5 minute timeout if we end up with the bottleneck where too many threads are queued up to leave... and none of the threads that are leaving make progress... Dan > > Sincerely yours, > Ivn > > >> Cheers, >> David >> >> >>> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 >>> WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ >>> >>> Sincerely yours, >>> Ivan >> >> > From ivan.gerasimov at oracle.com Wed May 20 16:00:54 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 20 May 2015 19:00:54 +0300 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... In-Reply-To: <555CA613.9010007@oracle.com> References: <555BCD1E.9010406@oracle.com> <555C3214.50004@oracle.com> <555C92B4.2010507@oracle.com> <555CA613.9010007@oracle.com> Message-ID: <555CAFB6.1040802@oracle.com> On 20.05.2015 18:19, Daniel D. Daugherty wrote: > On 5/20/15 7:57 AM, Ivan Gerasimov wrote: >> >> >> On 20.05.2015 10:04, David Holmes wrote: >>> Hi Ivan, >>> >>> On 20/05/2015 9:54 AM, Ivan Gerasimov wrote: >>>> Hello! >>>> >>>> Since the beginning of the year a warning about an expired timeout has >>>> been seen a few times. >>>> It is most likely to observe this under a heavy load, when many >>>> threads >>>> are exiting. >>>> >>>> The propose is to increase the timeout. >>>> The rationale is that if an application needs more time for the >>>> threads >>>> to exit, let's give it that time. >>> >>> Rather than just bumping the timeout a little I would suggest >>> bumping it a lot - make it a few minutes. If that fails then it >>> suggests we have a real problem not just a slow or loaded machine. >>> Otherwise if we just keep bumping the timeout it serves no purpose. >>> >> >> Alright. Then let's make it 5 minutes for both product and debug bits: >> >> WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/01/webrev/ > > Thumbs up. If I remember correctly, we'll only hit the > 5 minute timeout if we end up with the bottleneck where > too many threads are queued up to leave... and none of > the threads that are leaving make progress... > Yes. That's correct. Thank you Dan. Sincerely yours, Ivan > Dan > > >> >> Sincerely yours, >> Ivn >> >> >>> Cheers, >>> David >>> >>> >>>> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 >>>> WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ >>>> >>>> Sincerely yours, >>>> Ivan >>> >>> >> > > > From chris.plummer at oracle.com Wed May 20 18:18:56 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 20 May 2015 11:18:56 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <555C0D0F.6090108@oracle.com> References: <555B47EA.3000900@oracle.com> <555C0D0F.6090108@oracle.com> Message-ID: <555CD010.9080309@oracle.com> On 5/19/15 9:26 PM, David Holmes wrote: > Hi Chris, > > On 20/05/2015 12:25 AM, Chris Plummer wrote: >> Hi, >> >> Please review the following changes for allowing java debugging when CDS >> is enabled. >> >> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >> >> The VM changes are simple. I removed the check that prevents debugging >> with CDS enabled, and added logic that will map the CDS archive RW when >> debugging is enabled. >> >> The tests are a bit more complex. There are a bunch of existing JDI >> tests for testing debugging support. Rather than start from scratch or >> clone them, I instead just wrote wrapper tests that put the relevant JDI >> test classes in the archive, and then invoke the JDI test. I did this > > I'm not sure that the jdk tests will always be available in the same > test bundle as the hotspot tests. I know they will be in the full > forest but some testing uses test bundles not full repos. I'm not familiar with test bundles. Who creates them and who uses them? Chris > > David > ----- > > >> for 3 of the JDI tests. If you feel there are others that would be good >> candidates, I'd be happy to add them. I'm looking for ones that would >> result in modification of the RO class metadata, such as setting a >> breakpoint (for which I already added two tests). >> >> Testing done: >> -Using JPRT to run the new jtreg tests on all platforms. >> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >> -Regular JPRT "-testset hotspot" run >> -Putting the JCK JVMTI tests in the archive and then running them. >> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and then >> running them. >> -Putting a simple test class in the archive and then setting a >> breakpoint on it using jdb >> >> Some of the above testing resulted in the discovery of bugs that still >> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >> >> I also verified that without the change to map the archive RW, the above >> testing resulted in a SEGV, which is what you would expect (and actually >> want to see to prove that the testing is effective). >> >> thanks, >> >> Chris >> From mandy.chung at oracle.com Wed May 20 23:07:21 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 20 May 2015 16:07:21 -0700 Subject: RFR(M, v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <555C2EEB.1050505@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> Message-ID: <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> > On May 19, 2015, at 11:51 PM, Dmitry Samersoff wrote: > > Other alternatives could be to do all hashing/sorting/printing on native > layer i.e. implement printFinalizationQueue inside VM. > > Both options has pros and cons - Java based solution requires less JNI > calls and better readable but takes more memory. > > It might be better to return an array of Map.Entry > objects to VM rather than one huge string. The output and formatting should be done by jcmd. What you really need to get a peek on the finalizer queue and print the histogram. The VM has the heap histogram implementation. Have you considered leveraging that? 5: 1012 40480 java.lang.ref.Finalizer You can find the registered Finalizer instances. The downside is that icmd -finalizerinfo stops the world. I think it?s not unreasonable for this diagnostic command to be expensive like -heap command. Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Thu May 21 01:24:59 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 20 May 2015 19:24:59 -0600 Subject: RFR[ 9u-dev] JDK-8075773 - jps running as root fails with 7u75, works fine with 7u72 In-Reply-To: <5559B81A.7000102@oracle.com> References: <3ce36420-bd10-443f-b4e2-5f69735dda17@default> <5555989C.9090803@oracle.com> <5559B81A.7000102@oracle.com> Message-ID: <555D33EB.9070808@oracle.com> On 5/18/15 3:59 AM, cheleswer sahu wrote: > Hi, > I have fixed the code and tested. It's working fine. Please review the > changes. > Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.02/ Cheleswer, Sorry for another lengthy review, but since this is security related, I have to be complete. The goal: Add a policy by-pass for the 'root' user in order to fix the regression in jps(1) behavior. The core of this policy by-pass is the change to this function: 200 // Check if the given statbuf is considered a secure directory for 201 // the backing store files. Returns true if the directory is considered 202 // a secure location. Returns false if the statbuf is a symbolic link or 203 // if an error occurred. 204 // 205 static bool is_statbuf_secure(struct stat *statp) { 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { 207 // The path represents a link or some non-directory file type, 208 // which is not what we expected. Declare it insecure. 209 // 210 return false; 211 } 212 // We have an existing directory, check if the permissions are safe. 213 // 214 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { 215 // The directory is open for writing and could be subjected 216 // to a symlink or a hard link attack. Declare it insecure. 217 // 218 return false; 219 } 220 //If user is not root then see if the uid of the directory matches the effective uid of the process. 221 uid_t euid = geteuid(); 222 if ((euid !=0) && (statp->st_uid != euid)) { 223 // The directory was not created by this user, declare it insecure. 224 // 225 return false; 226 } 227 return true; 228 } The udiffs make it easier to see what was changed: - // See if the uid of the directory matches the effective uid of the process. - // - if (statp->st_uid != geteuid()) { + //If user is not root then see if the uid of the directory matches the effective uid of the process. + uid_t euid = geteuid(); + if ((euid !=0) && (statp->st_uid != euid)) { Line 222 allows the root user to by-pass the security check (an ownership check) on the remainder of the line. This is a very narrow and focused policy by-pass. Here are the code paths that access the modified policy code: is_statbuf_secure() is called by: - is_directory_secure() - is_dirfd_secure() is_directory_secure() is called by: - get_user_name_slow() - make_user_tmp_dir() - mmap_attach_shared() is_dirfd_secure() is called by: - open_directory_secure() open_directory_secure() is called by: - open_directory_secure_cwd() - get_user_name_slow() get_user_name_slow() analysis ----------------------------- The new security policy by-pass will allow get_user_name_slow() for the 'root' user: - to process directory entries in a directory that is writable by the owner which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that is owned by another user. It looks like the get_user_name_slow() code is written safely enough such that any symlink or hard link attack on the 'root' user by the directory owner should not cause any issues. The 'root' user will be able to determine the user name associated with VMs owned by another user which is exactly the intent of the policy by-pass. make_user_tmp_dir() analysis ---------------------------- The new security policy by-pass will allow make_user_tmp_dir() for the 'root' user: - to use an existing directory that was created by another user as the 'root' user's specific temporary directory - the directory is writable by the owner which makes this use subject to a symlink or hard link attack. The caller of make_user_tmp_dir() is create_sharedmem_resources() and it has to be analyzed to see if properly guards against a symlink and/or hardlink attack. create_sharedmem_resources() analysis ------------------------------------- The new security policy by-pass will allow create_sharedmem_resources() for the 'root' user: - to accept use of an existing directory that was created by another user as the 'root' user's specific temporary directory (via a call to make_user_tmp_dir() and a call to open_directory_secure_cwd()) - to accept use of an existing file that was created by another user as the 'root' user's specific shared memory file create_sharedmem_resources() guards against symlink attack by using the O_NOFOLLOW option when it opens the shared memory file. It guards against hard-link attack by checking for a link count greater than '1' after opening the file, but before it truncates the file. mmap_attach_shared() analysis() ------------------------------- The new security policy by-pass will allow mmap_attach_shared() for the 'root' user: - to use an existing directory that was created by another user as the 'root' user's specific temporary directory - the directory is writable by the owner which makes this use subject to a symlink or hard link attack. - to accept use of an existing file that was created by another user as the 'root' user's specific shared memory file - to process directory entries in a directory that is writable by the owner which makes this use subject to a symlink or hard link attack. - to process directory entries in a directory that is owned by another user. mmap_attach_shared() guards against symlink attack by using the O_NOFOLLOW option when it opens the shared memory file. It guards against hard-link attack by checking for a link count greater than '1' after opening the file. It does not protect itself against being handed a corrupted file or even a very large file that would cause an OOM when the VM tries to map what is supposed to be a PerfData file. Summary: This implementation has a much narrower and focused policy by-pass. The by-pass allows the 'root' user to access and use the shared memory files (PerfData) for other users. It also allows the 'root' user to use a shared memory file for itself that was created by another user. By using PerfData files for other users and by using a PerfData file for itself that can be modified by another user, the 'root' user is exposed to corrupted data and/or OOM attacks. It looks like a denial-of-service attack could be mounted here. The attack prevents the 'root' user from using the 'jps' cmd to query the various VMs on the system by a user hand crafting a corrupted PerfData file that will cause 'jps' to OOM crash. Again, sorry for the lengthy review. Dan > > Regards, > Cheleswer > > On 5/15/2015 12:26 PM, cheleswer sahu wrote: >> Dear Dan & Dmitry, >> Thanks for pointing out the security vulnerability in the fix and for >> your precise review. I am also agree with Dmitry's fix. I will fix >> the code and resend the review request. >> >> Regards, >> Cheleswer >> >> On 5/14/2015 9:46 PM, Gerald Thornbrugh wrote: >>> Hi Dan, >>> >>> When Cheleswer and I discussed this fix my interpretation had a >>> slightly different goal: >>> >>> Prior to the initial security fix any user could execute "jps" and >>> get the user names associated >>> with other user's perf data (i.e. the call to get_user_name_slow() >>> would succeed.). My initial >>> thought was that this was a regression for all users not just "root" >>> and this goal led to this fix. >>> At the time I did not see this as a security vulnerability, your >>> review has changed my mind. >>> >>> I agree that Dmitry's fix is a more secure fix for the issue and I >>> think we should use it. >>> >>> Let me know if you have any questions. >>> >>> Thanks! >>> >>> Jerry >>> >>>> Hi, >>>> Please review the code changes for >>>> https://bugs.openjdk.java.net/browse/JDK-8075773. I have built and >>>> tested JDK9 with fix successfully. As I do not have an account for >>>> OpenJDK, >>>> David Buck will push the fix into jdk9/hs-rt/. >>>> >>>> Web review link: http://cr.openjdk.java.net/~dbuck/8075773/webrev.01/ >>>> >>>> Regards, >>>> Cheleswer >>> Cheleswer, >>> >>> Sorry for the lengthy review, but since this is security related, >>> I have to be complete. >>> >>> The goal: Add a policy by-pass for the 'root' user in order to >>> fix the regression in jps(1) behavior. >>> >>> The core of this policy by-pass is the change to this function: >>> >>> 205 static bool is_statbuf_secure(struct stat *statp, int mode) { >>> 206 if (S_ISLNK(statp->st_mode) || !S_ISDIR(statp->st_mode)) { >>> 207 // The path represents a link or some non-directory file >>> type, >>> 208 // which is not what we expected. Declare it insecure. >>> 209 // >>> 210 return false; >>> 211 } >>> 212 // If the directory is going to be opened readonly, we >>> consider >>> this as secure operation >>> 213 // we do not need to do any more checks. >>> 214 // >>> 215 if ((mode & O_ACCMODE) == O_RDONLY) { >>> 216 return true; >>> 217 } >>> 218 // We have an existing directory, check if the permissions >>> are safe. >>> 219 // >>> 220 if ((statp->st_mode & (S_IWGRP|S_IWOTH)) != 0) { >>> 221 // The directory is open for writing and could be subjected >>> 222 // to a symlink or a hard link attack. Declare it insecure. >>> 223 // >>> 224 return false; >>> 225 } >>> 226 // See if the uid of the directory matches the effective >>> uid of >>> the process. >>> 227 // >>> 228 if (statp->st_uid != geteuid()) { >>> 229 // The directory was not created by this user, declare it >>> insecure. >>> 230 // >>> 231 return false; >>> 232 } >>> 233 return true; >>> 234 } >>> >>> Lines 212-217 are added which allows a caller that passes in O_RDONLY >>> to by-pass the security checks on lines 220-225 and 228-232. This >>> implementation is using an attribute of _how_ the data is accessed >>> to override security policy instead of an attribute of _who_ is >>> accessing the data. >>> >>> Here are the code paths that access the modified policy code: >>> >>> is_statbuf_secure() is called by: >>> >>> - is_directory_secure() >>> - is_dirfd_secure() >>> >>> is_directory_secure() is called by: >>> >>> - get_user_name_slow() with O_RDONLY >>> - make_user_tmp_dir() with O_RDWR >>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>> >>> is_dirfd_secure() is called by: >>> >>> - open_directory_secure() with a mode parameter >>> >>> open_directory_secure() is called by: >>> >>> - open_directory_secure_cwd() with O_RDWR >>> - get_user_name_slow() with O_RDONLY >>> >>> Only the code paths that specify O_RDWR make use of >>> the new policy by-pass code so it looks like only >>> >>> - get_user_name_slow() with O_RDONLY >>> - mmap_attach_shared() with (O_RDONLY | O_NOFOLLOW) >>> >>> are interesting. >>> >>> The new security policy by-pass will allow get_user_name_slow(): >>> >>> - to process directory entries in a directory that is writable >>> which makes this use subject to a symlink or hard link attack. >>> - to process directory entries in a directory that the calling >>> user does not own; the intent of the policy by-pass is to >>> allow this for the 'root' user, but this implementation >>> allows the by-pass for any user. >>> >>> It looks like the get_user_name_slow() code is written safely >>> enough such that any symlink or hard link attack should not >>> cause any issues. >>> >>> The new policy by-pass will allow any user to determine the >>> user name associated with VMs owned by another user. This is >>> a broader policy by-pass than was intended. >>> >>> >>> The new security policy by-pass will allow mmap_attach_shared(): >>> >>> - to process directory entries in a directory that is writable >>> which makes this use subject to a symlink or hard link attack. >>> - to process directory entries in a directory that the calling >>> user does not own; the intent of the policy by-pass is to >>> allow this for the 'root' user, but this implementation allows >>> the by-pass for any user. >>> >>> The mmap_attach_shared() code protects itself from a symlink >>> attack by including the 'O_NOFOLLOW' flag when opening the >>> PerfData file and it protects itself from a hardlink attack by >>> checking the hard link count after opening the file. It does >>> not protect itself against being handed a corrupted file or >>> even a very large file that would cause an OOM when the VM >>> tries to map what is supposed to be a PerfData file. >>> >>> The new policy by-pass will allow any user to access the >>> PerfData file associated with VMs owned by another user. This >>> is a broader policy by-pass than was intended. >>> >>> >>> Summary: >>> >>> This implementation of the new security policy by-pass is using >>> an attribute of _how_ the data is accessed to override security >>> policy instead of an attribute of _who_ is accessing the data. >>> This allows the VM to be exposed to some of the attacks that >>> the following fix was designed to prevent: >>> >>> JDK-8050807 Better performing performance data handling >>> >>> Dmitry's response to the code review provides a solution that >>> is based on who is accessing the data and that solution or >>> one like it should be considered. >>> >>> Again, sorry for the lengthy review. >>> >>> Dan >> > > From david.holmes at oracle.com Thu May 21 06:07:17 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 21 May 2015 16:07:17 +1000 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <555CD010.9080309@oracle.com> References: <555B47EA.3000900@oracle.com> <555C0D0F.6090108@oracle.com> <555CD010.9080309@oracle.com> Message-ID: <555D7615.5060108@oracle.com> Hi Chris, On 21/05/2015 4:18 AM, Chris Plummer wrote: > On 5/19/15 9:26 PM, David Holmes wrote: >> Hi Chris, >> >> On 20/05/2015 12:25 AM, Chris Plummer wrote: >>> Hi, >>> >>> Please review the following changes for allowing java debugging when CDS >>> is enabled. >>> >>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> The VM changes are simple. I removed the check that prevents debugging >>> with CDS enabled, and added logic that will map the CDS archive RW when >>> debugging is enabled. >>> >>> The tests are a bit more complex. There are a bunch of existing JDI >>> tests for testing debugging support. Rather than start from scratch or >>> clone them, I instead just wrote wrapper tests that put the relevant JDI >>> test classes in the archive, and then invoke the JDI test. I did this >> >> I'm not sure that the jdk tests will always be available in the same >> test bundle as the hotspot tests. I know they will be in the full >> forest but some testing uses test bundles not full repos. > I'm not familiar with test bundles. Who creates them and who uses them? I was thinking of the test-image target in the build, which is then used to create a test-bundle for JPRT. But that seems to be only native code that must be built as part of the binary build process. So I think the actual test execution will be using the source repository - for JPRT anyway. Not sure how nightly testing handles things though - if it is executing the hotspot jtreg tests only does it use a full repo or just the hotspot/test directory tree? David ----- > > Chris >> >> David >> ----- >> >> >>> for 3 of the JDI tests. If you feel there are others that would be good >>> candidates, I'd be happy to add them. I'm looking for ones that would >>> result in modification of the RO class metadata, such as setting a >>> breakpoint (for which I already added two tests). >>> >>> Testing done: >>> -Using JPRT to run the new jtreg tests on all platforms. >>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>> -Regular JPRT "-testset hotspot" run >>> -Putting the JCK JVMTI tests in the archive and then running them. >>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and then >>> running them. >>> -Putting a simple test class in the archive and then setting a >>> breakpoint on it using jdb >>> >>> Some of the above testing resulted in the discovery of bugs that still >>> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>> >>> I also verified that without the change to map the archive RW, the above >>> testing resulted in a SEGV, which is what you would expect (and actually >>> want to see to prove that the testing is effective). >>> >>> thanks, >>> >>> Chris >>> > From david.holmes at oracle.com Thu May 21 06:12:56 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 21 May 2015 16:12:56 +1000 Subject: RFR (XXS) 8069068: VM warning: WaitForMultipleObjects timed out (0) ... In-Reply-To: <555C92B4.2010507@oracle.com> References: <555BCD1E.9010406@oracle.com> <555C3214.50004@oracle.com> <555C92B4.2010507@oracle.com> Message-ID: <555D7768.5030804@oracle.com> Looks good! Thanks, David On 20/05/2015 11:57 PM, Ivan Gerasimov wrote: > > > On 20.05.2015 10:04, David Holmes wrote: >> Hi Ivan, >> >> On 20/05/2015 9:54 AM, Ivan Gerasimov wrote: >>> Hello! >>> >>> Since the beginning of the year a warning about an expired timeout has >>> been seen a few times. >>> It is most likely to observe this under a heavy load, when many threads >>> are exiting. >>> >>> The propose is to increase the timeout. >>> The rationale is that if an application needs more time for the threads >>> to exit, let's give it that time. >> >> Rather than just bumping the timeout a little I would suggest bumping >> it a lot - make it a few minutes. If that fails then it suggests we >> have a real problem not just a slow or loaded machine. Otherwise if we >> just keep bumping the timeout it serves no purpose. >> > > Alright. Then let's make it 5 minutes for both product and debug bits: > > WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/01/webrev/ > > Sincerely yours, > Ivn > > >> Cheers, >> David >> >> >>> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8069068 >>> WEBREV: http://cr.openjdk.java.net/~igerasim/8069068/00/webrev/ >>> >>> Sincerely yours, >>> Ivan >> >> > From yekaterina.kantserova at oracle.com Thu May 21 07:33:42 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 21 May 2015 09:33:42 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 Message-ID: <555D8A56.3050001@oracle.com> Hi, Could I please have a review of this new test. bug: https://bugs.openjdk.java.net/browse/JDK-8080828 webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. Thanks, Katja From staffan.larsen at oracle.com Thu May 21 08:13:04 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 21 May 2015 10:13:04 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: <555D8A56.3050001@oracle.com> References: <555D8A56.3050001@oracle.com> Message-ID: Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. > On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this new test. > > bug: https://bugs.openjdk.java.net/browse/JDK-8080828 > webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 > > This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. > > Thanks, > Katja From staffan.larsen at oracle.com Thu May 21 08:15:29 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 21 May 2015 10:15:29 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: References: <555D8A56.3050001@oracle.com> Message-ID: I just realize that you also need to add a check if SA can attach on the current platform. See the test in sa/jmap-hashcode: if (!Platform.shouldSAAttach()) { System.out.println("SA attach not expected to work - test skipped."); return; } Thanks, /Staffan > On 21 maj 2015, at 10:13, Staffan Larsen wrote: > > Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. > >> On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: >> >> Hi, >> >> Could I please have a review of this new test. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8080828 >> webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 >> >> This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. >> >> Thanks, >> Katja > From yekaterina.kantserova at oracle.com Thu May 21 08:58:10 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 21 May 2015 10:58:10 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: References: <555D8A56.3050001@oracle.com> Message-ID: <555D9E22.6090601@oracle.com> Staffan, thanks fro looking at it! I'll fix according the test your comments. // Katja On 05/21/2015 10:15 AM, Staffan Larsen wrote: > I just realize that you also need to add a check if SA can attach on the current platform. See the test in sa/jmap-hashcode: > > if (!Platform.shouldSAAttach()) { > System.out.println("SA attach not expected to work - test skipped."); > return; > } > > Thanks, > /Staffan > >> On 21 maj 2015, at 10:13, Staffan Larsen wrote: >> >> Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. >> >>> On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: >>> >>> Hi, >>> >>> Could I please have a review of this new test. >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8080828 >>> webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 >>> >>> This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. >>> >>> Thanks, >>> Katja From yekaterina.kantserova at oracle.com Thu May 21 10:55:06 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 21 May 2015 12:55:06 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: References: <555D8A56.3050001@oracle.com> Message-ID: <555DB98A.7050401@oracle.com> The new webrev can be found here: http://cr.openjdk.java.net/~ykantser/8080828/webrev.01 Thanks, Katja On 05/21/2015 10:15 AM, Staffan Larsen wrote: > I just realize that you also need to add a check if SA can attach on the current platform. See the test in sa/jmap-hashcode: > > if (!Platform.shouldSAAttach()) { > System.out.println("SA attach not expected to work - test skipped."); > return; > } > > Thanks, > /Staffan > >> On 21 maj 2015, at 10:13, Staffan Larsen wrote: >> >> Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. >> >>> On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: >>> >>> Hi, >>> >>> Could I please have a review of this new test. >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8080828 >>> webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 >>> >>> This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. >>> >>> Thanks, >>> Katja From staffan.larsen at oracle.com Thu May 21 11:04:10 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 21 May 2015 13:04:10 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: <555DB98A.7050401@oracle.com> References: <555D8A56.3050001@oracle.com> <555DB98A.7050401@oracle.com> Message-ID: <5C6FDB60-85C3-4130-ABF0-52368963E0DB@oracle.com> Looks good! Thanks, /Staffan > On 21 maj 2015, at 12:55, Yekaterina Kantserova wrote: > > The new webrev can be found here: http://cr.openjdk.java.net/~ykantser/8080828/webrev.01 > > Thanks, > Katja > > > > On 05/21/2015 10:15 AM, Staffan Larsen wrote: >> I just realize that you also need to add a check if SA can attach on the current platform. See the test in sa/jmap-hashcode: >> >> if (!Platform.shouldSAAttach()) { >> System.out.println("SA attach not expected to work - test skipped."); >> return; >> } >> >> Thanks, >> /Staffan >> >>> On 21 maj 2015, at 10:13, Staffan Larsen wrote: >>> >>> Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. >>> >>>> On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: >>>> >>>> Hi, >>>> >>>> Could I please have a review of this new test. >>>> >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8080828 >>>> webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 >>>> >>>> This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. >>>> >>>> Thanks, >>>> Katja > From yekaterina.kantserova at oracle.com Fri May 22 06:40:45 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Fri, 22 May 2015 08:40:45 +0200 Subject: RFR(XS): 8080828: Create sanity test for JDK-8080155 In-Reply-To: <5C6FDB60-85C3-4130-ABF0-52368963E0DB@oracle.com> References: <555D8A56.3050001@oracle.com> <555DB98A.7050401@oracle.com> <5C6FDB60-85C3-4130-ABF0-52368963E0DB@oracle.com> Message-ID: <555ECF6D.2030003@oracle.com> Staffan, thanks for the review! On 05/21/2015 01:04 PM, Staffan Larsen wrote: > Looks good! > > Thanks, > /Staffan > >> On 21 maj 2015, at 12:55, Yekaterina Kantserova wrote: >> >> The new webrev can be found here: http://cr.openjdk.java.net/~ykantser/8080828/webrev.01 >> >> Thanks, >> Katja >> >> >> >> On 05/21/2015 10:15 AM, Staffan Larsen wrote: >>> I just realize that you also need to add a check if SA can attach on the current platform. See the test in sa/jmap-hashcode: >>> >>> if (!Platform.shouldSAAttach()) { >>> System.out.println("SA attach not expected to work - test skipped."); >>> return; >>> } >>> >>> Thanks, >>> /Staffan >>> >>>> On 21 maj 2015, at 10:13, Staffan Larsen wrote: >>>> >>>> Can we add some kind of validation of the output? Something simple, just to verify that we actually ran what we think we ran. >>>> >>>>> On 21 maj 2015, at 09:33, Yekaterina Kantserova wrote: >>>>> >>>>> Hi, >>>>> >>>>> Could I please have a review of this new test. >>>>> >>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8080828 >>>>> webrev: http://cr.openjdk.java.net/~ykantser/8080828/webrev.00 >>>>> >>>>> This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080155 if running with -Xcomp. >>>>> >>>>> Thanks, >>>>> Katja From yekaterina.kantserova at oracle.com Fri May 22 07:17:33 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Fri, 22 May 2015 09:17:33 +0200 Subject: 8080855: Create sanity test for JDK-8080692 Message-ID: <555ED80D.5000806@oracle.com> Hi, Could I please have a review of this new test. bug: https://bugs.openjdk.java.net/browse/JDK-8080855 webrev: http://cr.openjdk.java.net/~ykantser/8080855/webrev.00/ This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080692 if running with -Xcomp. Thanks, Katja From staffan.larsen at oracle.com Fri May 22 07:20:45 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 22 May 2015 09:20:45 +0200 Subject: 8080855: Create sanity test for JDK-8080692 In-Reply-To: <555ED80D.5000806@oracle.com> References: <555ED80D.5000806@oracle.com> Message-ID: <2EB33CDA-5402-428E-8E70-0C64581D697A@oracle.com> Looks good! Thanks, /Staffan > On 22 maj 2015, at 09:17, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this new test. > > bug: https://bugs.openjdk.java.net/browse/JDK-8080855 > webrev: http://cr.openjdk.java.net/~ykantser/8080855/webrev.00/ > > This test will trigger the exception reported in https://bugs.openjdk.java.net/browse/JDK-8080692 if running with -Xcomp. > > Thanks, > Katja From per.liden at oracle.com Fri May 22 11:23:08 2015 From: per.liden at oracle.com (Per Liden) Date: Fri, 22 May 2015 13:23:08 +0200 Subject: RFR(xs): 8080930: SA changes broke bootcycle-images builds Message-ID: <555F119C.9000202@oracle.com> JDK-8080581 introduced two new packages in the SA, but missed to add those to make/sa.files. This problem went unnoticed since it's only exposed when doing bootcycle-images builds where JDK9's javac no longer emits classes for implicit compilations. This patch makes sure the two new packages are compiled explicitly. Webrev: http://cr.openjdk.java.net/~pliden/8080930/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8080930 Testing: make bootcycle-images now works cheers, /Per From staffan.larsen at oracle.com Fri May 22 11:32:14 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 22 May 2015 13:32:14 +0200 Subject: RFR(xs): 8080930: SA changes broke bootcycle-images builds In-Reply-To: <555F119C.9000202@oracle.com> References: <555F119C.9000202@oracle.com> Message-ID: Looks good! Thanks, /Staffan > On 22 maj 2015, at 13:23, Per Liden wrote: > > JDK-8080581 introduced two new packages in the SA, but missed to add those to make/sa.files. This problem went unnoticed since it's only exposed when doing bootcycle-images builds where JDK9's javac no longer emits classes for implicit compilations. > > This patch makes sure the two new packages are compiled explicitly. > > Webrev: http://cr.openjdk.java.net/~pliden/8080930/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8080930 > > Testing: make bootcycle-images now works > > cheers, > /Per > From per.liden at oracle.com Fri May 22 11:33:00 2015 From: per.liden at oracle.com (Per Liden) Date: Fri, 22 May 2015 13:33:00 +0200 Subject: RFR(xs): 8080930: SA changes broke bootcycle-images builds In-Reply-To: References: <555F119C.9000202@oracle.com> Message-ID: <555F13EC.6020100@oracle.com> Thanks Staffan! /Per On 2015-05-22 13:32, Staffan Larsen wrote: > Looks good! > > Thanks, > /Staffan > >> On 22 maj 2015, at 13:23, Per Liden wrote: >> >> JDK-8080581 introduced two new packages in the SA, but missed to add those to make/sa.files. This problem went unnoticed since it's only exposed when doing bootcycle-images builds where JDK9's javac no longer emits classes for implicit compilations. >> >> This patch makes sure the two new packages are compiled explicitly. >> >> Webrev: http://cr.openjdk.java.net/~pliden/8080930/webrev.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8080930 >> >> Testing: make bootcycle-images now works >> >> cheers, >> /Per >> > From bengt.rutisson at oracle.com Mon May 25 06:55:11 2015 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 25 May 2015 08:55:11 +0200 Subject: RFR(xs): 8080930: SA changes broke bootcycle-images builds In-Reply-To: <555F119C.9000202@oracle.com> References: <555F119C.9000202@oracle.com> Message-ID: <5562C74F.70709@oracle.com> Hi Per, On 2015-05-22 13:23, Per Liden wrote: > JDK-8080581 introduced two new packages in the SA, but missed to add > those to make/sa.files. This problem went unnoticed since it's only > exposed when doing bootcycle-images builds where JDK9's javac no > longer emits classes for implicit compilations. > > This patch makes sure the two new packages are compiled explicitly. > > Webrev: http://cr.openjdk.java.net/~pliden/8080930/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8080930 Looks good as far as I can tell. Bengt > > Testing: make bootcycle-images now works > > cheers, > /Per > From per.liden at oracle.com Mon May 25 06:59:42 2015 From: per.liden at oracle.com (Per Liden) Date: Mon, 25 May 2015 08:59:42 +0200 Subject: RFR(xs): 8080930: SA changes broke bootcycle-images builds In-Reply-To: <5562C74F.70709@oracle.com> References: <555F119C.9000202@oracle.com> <5562C74F.70709@oracle.com> Message-ID: <5562C85E.4080504@oracle.com> On 2015-05-25 08:55, Bengt Rutisson wrote: > > Hi Per, > > On 2015-05-22 13:23, Per Liden wrote: >> JDK-8080581 introduced two new packages in the SA, but missed to add >> those to make/sa.files. This problem went unnoticed since it's only >> exposed when doing bootcycle-images builds where JDK9's javac no >> longer emits classes for implicit compilations. >> >> This patch makes sure the two new packages are compiled explicitly. >> >> Webrev: http://cr.openjdk.java.net/~pliden/8080930/webrev.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8080930 > > Looks good as far as I can tell. > > Bengt Thanks Bengt. /Per > >> >> Testing: make bootcycle-images now works >> >> cheers, >> /Per >> > From jaroslav.bachorik at oracle.com Mon May 25 09:34:40 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 25 May 2015 11:34:40 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: References: <555B3D00.3080204@oracle.com> Message-ID: <5562ECB0.3070402@oracle.com> On 20.5.2015 16:38, Mandy Chung wrote: > On 05/19/2015 06:39 AM, Jaroslav Bachorik wrote: >> Please, review the following change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8080663 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 >> >> The title says it all. This enhancement is about replacing the >> arbitrary reflection based code with a cleaner approach based on >> sun.misc.SharedSecrets class. >> > > Using the shared secret approach is okay. Some suggestion: > > Since both sun.misc.JavaBeansIntrospectorAccess and > ConstructorPropertiesAccess are for JMX to access java.beans types, it > could be simplified to have one single interface (e.g. > sun.misc.JavaBeansAccess) to define > Method getReadMethod(Class, String); > String[] getConstructorProperties(Constructor); > > AnnotationHelper and BeansIntrospector can then be merged into the new > JavaBeansAccessor class. Webrev with the merged accessor interface: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 > > Can you add a regression for this, if not exist, as Daniel points out? There is a bunch of tests exercising the proper handling of @ConstructorAnnotationProperties - I added this issue number to the list of bug ids. Doesn't really seem to be worth adding a separate regtest. -JB- > > Mandy From Alan.Bateman at oracle.com Mon May 25 11:50:12 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 25 May 2015 12:50:12 +0100 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <5562ECB0.3070402@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> Message-ID: <55630C74.6090309@oracle.com> On 25/05/2015 10:34, Jaroslav Bachorik wrote: > > Webrev with the merged accessor interface: > http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 I think you mean: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.01/ I agree this shouldn't require a new test because this code is well exercised by existing tests. Best to add a noreg-* label to the JIRA issue rather than needlessly adding the JIRA issue number to many tests. Is there a guarantee that Introspector will be initialized to ensure that the secret is set before JavaBeansAccessor methods are invoked? In the original code then BeanIntrospector attempts to load java.beans.Introspector but I can't see the equivalent in the updated code. A minor comment on JavaBeansAccessor is that it would be good to keep the line length consistent for the area. Otherwise looks okay to me. -Alan. From jaroslav.bachorik at oracle.com Mon May 25 11:56:13 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 25 May 2015 13:56:13 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <5562ECB0.3070402@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> Message-ID: <55630DDD.5070605@oracle.com> On 25.5.2015 11:34, Jaroslav Bachorik wrote: > On 20.5.2015 16:38, Mandy Chung wrote: >> On 05/19/2015 06:39 AM, Jaroslav Bachorik wrote: >>> Please, review the following change >>> >>> Issue : https://bugs.openjdk.java.net/browse/JDK-8080663 >>> Webrev: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 >>> >>> The title says it all. This enhancement is about replacing the >>> arbitrary reflection based code with a cleaner approach based on >>> sun.misc.SharedSecrets class. >>> >> >> Using the shared secret approach is okay. Some suggestion: >> >> Since both sun.misc.JavaBeansIntrospectorAccess and >> ConstructorPropertiesAccess are for JMX to access java.beans types, it >> could be simplified to have one single interface (e.g. >> sun.misc.JavaBeansAccess) to define >> Method getReadMethod(Class, String); >> String[] getConstructorProperties(Constructor); >> >> AnnotationHelper and BeansIntrospector can then be merged into the new >> JavaBeansAccessor class. > > Webrev with the merged accessor interface: > http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 Sorry, wrong URL. Should be > http://cr.openjdk.java.net/~jbachorik/8080663/webrev.01 -JB- > > >> >> Can you add a regression for this, if not exist, as Daniel points out? > > There is a bunch of tests exercising the proper handling of > @ConstructorAnnotationProperties - I added this issue number to the list > of bug ids. Doesn't really seem to be worth adding a separate regtest. > > -JB- > >> >> Mandy > From jaroslav.bachorik at oracle.com Mon May 25 12:06:03 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 25 May 2015 14:06:03 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <55630C74.6090309@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> <55630C74.6090309@oracle.com> Message-ID: <5563102B.8050404@oracle.com> On 25.5.2015 13:50, Alan Bateman wrote: > > > On 25/05/2015 10:34, Jaroslav Bachorik wrote: >> >> Webrev with the merged accessor interface: >> http://cr.openjdk.java.net/~jbachorik/8080663/webrev.00 > I think you mean: > http://cr.openjdk.java.net/~jbachorik/8080663/webrev.01/ > > I agree this shouldn't require a new test because this code is well > exercised by existing tests. Best to add a noreg-* label to the JIRA > issue rather than needlessly adding the JIRA issue number to many tests. NP. 'noreg-other' should do. > > Is there a guarantee that Introspector will be initialized to ensure > that the secret is set before JavaBeansAccessor methods are invoked? In > the original code then BeanIntrospector attempts to load > java.beans.Introspector but I can't see the equivalent in the updated code. JavaBeansAccessor L39-46 - in the static initializer of this class 'Class.forName("java.beans.Introspector", true, JavaBeansAccessor.class.getClassLoader());' is called. > > A minor comment on JavaBeansAccessor is that it would be good to keep > the line length consistent for the area. Ok. > > Otherwise looks okay to me. Thanks. -JB- > > -Alan. From Alan.Bateman at oracle.com Mon May 25 12:15:50 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 25 May 2015 13:15:50 +0100 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <5563102B.8050404@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> <55630C74.6090309@oracle.com> <5563102B.8050404@oracle.com> Message-ID: <55631276.6050103@oracle.com> On 25/05/2015 13:06, Jaroslav Bachorik wrote: > > JavaBeansAccessor L39-46 - in the static initializer of this class > 'Class.forName("java.beans.Introspector", true, > JavaBeansAccessor.class.getClassLoader());' is called. Thanks, I didn't spot that for some reason. In that case I think the approach looks okay. -Alan From mandy.chung at oracle.com Mon May 25 18:27:59 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 25 May 2015 11:27:59 -0700 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <5562ECB0.3070402@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> Message-ID: > On May 25, 2015, at 2:34 AM, Jaroslav Bachorik wrote: > > > Webrev with the merged accessor interface: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.01 > > >> >> Can you add a regression for this, if not exist, as Daniel points out? > > There is a bunch of tests exercising the proper handling of @ConstructorAnnotationProperties - I added this issue number to the list of bug ids. Doesn't really seem to be worth adding a separate regtest. Since there are existing regression tests exercising @CP and Introspector, this is well covered then (it wasn?t clear to me where these existing tests are and hence my comment). webrev.01 looks good to me. Nit: JavaBeansAccessor and JavaBeansAccess - in the javadoc of getConstructorPropertiesValue method, value and null may be more appropriate to use {@code ?} Mandy From jesper.wilhelmsson at oracle.com Mon May 25 20:49:12 2015 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Mon, 25 May 2015 22:49:12 +0200 Subject: G1 young STW time in MBean In-Reply-To: References: <556341CF.9000705@oracle.com> <55634D94.1080301@oracle.com> Message-ID: <55638AC8.4080401@oracle.com> Including the serviceability team. There might be plans for G1 monitoring in the future but I don't know too much about what and when. I know there are commercial products that does a pretty good job at G1 monitoring but I don't know if that is an option for you. Best, /Jesper Jakub Kubrynski skrev den 25/5/15 22:42: > Maybe we could propose some JSR about that? Using G1 without proper monitoring > is like living on the edge :) > > Cheers, > Jakub > > 2015-05-25 18:28 GMT+02:00 Jesper Wilhelmsson >: > > As far as I know there hasn't been much development done to get G1 to play > well with the MBeans. And I don't think there is anything planned in this > area. Maybe the serviceability team has other plans for the MBeans. > /Jesper > > > Jakub Kubrynski skrev den 25/5/15 17:43: > > Do you know if there is any development planned in this area? Also there > is no > avgPauseTime, promoted and survived mbean information for G1. The only > available > substitution I see for now is to get safepoint timings. > > Best, > Jakub Kubrynski > > 25 maj 2015 17:37 "Jesper Wilhelmsson" > >> napisa?(a): > > Hi Jakub, > > The times reported in the MBeans are total GC times. There is > currently no > way to get STW times through the MBeans. > > Best regards, > /Jesper > > > Jakub Kubrynski skrev den 25/5/15 10:56: > > Hi, > > is there any possibility to get STW time for G1 young > collection through > MBean? > The duration reported here is IMHO total GC time (concurrent + > stw). > > -- > Best regards, > Jakub Kubrynski > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -- > Best regards, > Jakub Kubrynski From staffan.larsen at oracle.com Tue May 26 06:17:39 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 26 May 2015 08:17:39 +0200 Subject: G1 young STW time in MBean In-Reply-To: <55638AC8.4080401@oracle.com> References: <556341CF.9000705@oracle.com> <55634D94.1080301@oracle.com> <55638AC8.4080401@oracle.com> Message-ID: Try out Java Flight Recorder - it has a lot more data in it. /Staffan > On 25 maj 2015, at 22:49, Jesper Wilhelmsson wrote: > > Including the serviceability team. > > There might be plans for G1 monitoring in the future but I don't know too much about what and when. I know there are commercial products that does a pretty good job at G1 monitoring but I don't know if that is an option for you. > > Best, > /Jesper > > > Jakub Kubrynski skrev den 25/5/15 22:42: >> Maybe we could propose some JSR about that? Using G1 without proper monitoring >> is like living on the edge :) >> >> Cheers, >> Jakub >> >> 2015-05-25 18:28 GMT+02:00 Jesper Wilhelmsson > >: >> >> As far as I know there hasn't been much development done to get G1 to play >> well with the MBeans. And I don't think there is anything planned in this >> area. Maybe the serviceability team has other plans for the MBeans. >> /Jesper >> >> >> Jakub Kubrynski skrev den 25/5/15 17:43: >> >> Do you know if there is any development planned in this area? Also there >> is no >> avgPauseTime, promoted and survived mbean information for G1. The only >> available >> substitution I see for now is to get safepoint timings. >> >> Best, >> Jakub Kubrynski >> >> 25 maj 2015 17:37 "Jesper Wilhelmsson" > >> > >> napisa?(a): >> >> Hi Jakub, >> >> The times reported in the MBeans are total GC times. There is >> currently no >> way to get STW times through the MBeans. >> >> Best regards, >> /Jesper >> >> >> Jakub Kubrynski skrev den 25/5/15 10:56: >> >> Hi, >> >> is there any possibility to get STW time for G1 young >> collection through >> MBean? >> The duration reported here is IMHO total GC time (concurrent + >> stw). >> >> -- >> Best regards, >> Jakub Kubrynski >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> > > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> -- >> Best regards, >> Jakub Kubrynski From staffan.larsen at oracle.com Tue May 26 06:26:31 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Tue, 26 May 2015 08:26:31 +0200 Subject: G1 young STW time in MBean In-Reply-To: References: <556341CF.9000705@oracle.com> <55634D94.1080301@oracle.com> <55638AC8.4080401@oracle.com> Message-ID: > On 26 maj 2015, at 08:22, Jakub Kubrynski wrote: > > Jesper already pointed me about JMC. The reason we're not using it is that it cannot be integrated with our production monitoring. It's more problem solving tool than continuous APM. So the question is if Oracle is going to implement some MBeans for G1, and if not maybe we could propose JSR about it? > I don?t think there are any open issues or plans for this, so contributions are always welcome! /Staffan > Cheers, > Jakub > > 26 maj 2015 08:17 "Staffan Larsen" > napisa?(a): > Try out Java Flight Recorder - it has a lot more data in it. > > /Staffan > > > On 25 maj 2015, at 22:49, Jesper Wilhelmsson > wrote: > > > > Including the serviceability team. > > > > There might be plans for G1 monitoring in the future but I don't know too much about what and when. I know there are commercial products that does a pretty good job at G1 monitoring but I don't know if that is an option for you. > > > > Best, > > /Jesper > > > > > > Jakub Kubrynski skrev den 25/5/15 22:42: > >> Maybe we could propose some JSR about that? Using G1 without proper monitoring > >> is like living on the edge :) > >> > >> Cheers, > >> Jakub > >> > >> 2015-05-25 18:28 GMT+02:00 Jesper Wilhelmsson > >> >>: > >> > >> As far as I know there hasn't been much development done to get G1 to play > >> well with the MBeans. And I don't think there is anything planned in this > >> area. Maybe the serviceability team has other plans for the MBeans. > >> /Jesper > >> > >> > >> Jakub Kubrynski skrev den 25/5/15 17:43: > >> > >> Do you know if there is any development planned in this area? Also there > >> is no > >> avgPauseTime, promoted and survived mbean information for G1. The only > >> available > >> substitution I see for now is to get safepoint timings. > >> > >> Best, > >> Jakub Kubrynski > >> > >> 25 maj 2015 17:37 "Jesper Wilhelmsson" > >> > > >> > >> >>> napisa?(a): > >> > >> Hi Jakub, > >> > >> The times reported in the MBeans are total GC times. There is > >> currently no > >> way to get STW times through the MBeans. > >> > >> Best regards, > >> /Jesper > >> > >> > >> Jakub Kubrynski skrev den 25/5/15 10:56: > >> > >> Hi, > >> > >> is there any possibility to get STW time for G1 young > >> collection through > >> MBean? > >> The duration reported here is IMHO total GC time (concurrent + > >> stw). > >> > >> -- > >> Best regards, > >> Jakub Kubrynski > >> > >> > >> _______________________________________________ > >> hotspot-gc-use mailing list > >> hotspot-gc-use at openjdk.java.net > > >> > >> >> > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > >> > >> > >> > >> > >> -- > >> Best regards, > >> Jakub Kubrynski > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Tue May 26 06:36:52 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 26 May 2015 08:36:52 +0200 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> Message-ID: <55641484.3030804@oracle.com> On 25.5.2015 20:27, Mandy Chung wrote: > >> On May 25, 2015, at 2:34 AM, Jaroslav Bachorik wrote: >> >> >> Webrev with the merged accessor interface: http://cr.openjdk.java.net/~jbachorik/8080663/webrev.01 >> >> >>> >>> Can you add a regression for this, if not exist, as Daniel points out? >> >> There is a bunch of tests exercising the proper handling of @ConstructorAnnotationProperties - I added this issue number to the list of bug ids. Doesn't really seem to be worth adding a separate regtest. > > Since there are existing regression tests exercising @CP and Introspector, this is well covered then (it wasn?t clear to me where these existing tests are and hence my comment). > > webrev.01 looks good to me. Nit: JavaBeansAccessor and JavaBeansAccess - in the javadoc of getConstructorPropertiesValue method, value and null may be more appropriate to use {@code ?} Nits addressed - http://cr.openjdk.java.net/~jbachorik/8080663/webrev.02 I removed the bug ids from the tests and used 'noreg-other' with an explanation in the issue comment (noreg-existing doesn't seem to exist any more). I also slightly reformatted the JavaBeansAccessor class to better fit 80 chars line length limit. -JB- > > Mandy > From Alan.Bateman at oracle.com Tue May 26 07:44:20 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 26 May 2015 08:44:20 +0100 Subject: RFR 8080663: Use sun.misc.SharedSecrets to allow access from java.management to @ConstructorProperties In-Reply-To: <55641484.3030804@oracle.com> References: <555B3D00.3080204@oracle.com> <5562ECB0.3070402@oracle.com> <55641484.3030804@oracle.com> Message-ID: <55642454.7020603@oracle.com> On 26/05/2015 07:36, Jaroslav Bachorik wrote: > > Nits addressed - http://cr.openjdk.java.net/~jbachorik/8080663/webrev.02 > > I removed the bug ids from the tests and used 'noreg-other' with an > explanation in the issue comment (noreg-existing doesn't seem to exist > any more). > > I also slightly reformatted the JavaBeansAccessor class to better fit > 80 chars line length limit. > I think this looks okay now. -Alan. From daniel.daugherty at oracle.com Tue May 26 13:42:13 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 26 May 2015 07:42:13 -0600 Subject: Issues with dtrace enabled in OpenJdk 9 In-Reply-To: <816652759.861398.1432569179316.JavaMail.yahoo@mail.yahoo.com> References: <816652759.861398.1432569179316.JavaMail.yahoo@mail.yahoo.com> Message-ID: <55647835.2050401@oracle.com> Adding the Serviceability alias. Dan On 5/25/15 9:52 AM, Mahesh Pujari wrote: > Hi, > > I am trying to build OpenJDK 9 with dtrace enabled on my Ubuntu machine (Linux 3.13.0-45-generic #74-Ubuntu), I have asked this question on build-dev at openjdk.java.net (http://mail.openjdk.java.net/pipermail/build-dev/2015-May/014969.html) and I was directed to this mailing list (including distro-pkg mailing list, but had no luck there, so trying out here). > > If SDT headers are found then dtrace is enabled by default, this is what I understood. Now when I build, I end-up with below errors > ... > ... > vmThread.o: In function `VMOperationQueue::add(VM_Operation*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:156: undefined reference to `__dtrace_hotspot___vmops__request' > vmThread.o: In function `VMThread::evaluate_operation(VM_Operation*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:354: undefined reference to `__dtrace_hotspot___vmops__begin' > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:374: undefined reference to `__dtrace_hotspot___vmops__end' > ... > .. > classLoadingService.o: In function `ClassLoadingService::notify_class_unloaded(InstanceKlass*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/services/classLoadingService.cpp:119: undefined reference to `__dtrace_hotspot___class__unloaded' > classLoadingService.o: In function `ClassLoadingService::notify_class_loaded(InstanceKlass*, bool)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/services/classLoadingService.cpp:144: undefined reference to `__dtrace_hotspot___class__loaded' > compileBroker.o: In function `CompileBroker::invoke_compiler_on_method(CompileTask*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/compiler/compileBroker.cpp:1927: undefined reference to `__dtrace_hotspot___method__compile__begin' > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/compiler/compileBroker.cpp:2028: undefined reference to `__dtrace_hotspot___method__compile__end' > ... > ... > > Compilation is success but during linkage things fail. Can someone help me with this, any directions to what I am missing. > > thanks and regards, > Mahesh Pujari > From dmitry.samersoff at oracle.com Tue May 26 14:14:24 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 26 May 2015 17:14:24 +0300 Subject: RFR(M,v7): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> Message-ID: <55647FC0.7050901@oracle.com> On 2015-05-21 02:07, Mandy Chung wrote: > >> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >> > wrote: >> >> Other alternatives could be to do all hashing/sorting/printing on native >> layer i.e. implement printFinalizationQueue inside VM. >> >> Both options has pros and cons - Java based solution requires less JNI >> calls and better readable but takes more memory. >> >> It might be better to return an array of Map.Entry >> objects to VM rather than one huge string. > > The output and formatting should be done by jcmd. done. > What you really need > to get a peek on the finalizer queue and print the histogram. The VM > has the heap histogram implementation. Have you considered leveraging > that? > > 5: 1012 40480 java.lang.ref.Finalizer One of previous versions count total a number of instances registered for finalization but then we decided that number of unreachable instances awaiting finalization has more value for customer. > You can find the registered Finalizer instances. The downside is that > icmd -finalizerinfo stops the world. I think it?s not unreasonable for > this diagnostic command to be expensive like -heap command. Current implementation is lock-free and don't stop the world, we decided to make it less expensive at the cost of less accurate results. -Dmitry -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Tue May 26 14:16:10 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 26 May 2015 17:16:10 +0300 Subject: RFR(M,v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> Message-ID: <5564802A.2010403@oracle.com> Hi Everybody, http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.09/ Please review updated webrev - printFinalizationQueue now returns and array of Map.Entry >> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >> > wrote: >> >> Other alternatives could be to do all hashing/sorting/printing on native >> layer i.e. implement printFinalizationQueue inside VM. >> >> Both options has pros and cons - Java based solution requires less JNI >> calls and better readable but takes more memory. >> >> It might be better to return an array of Map.Entry >> objects to VM rather than one huge string. > > The output and formatting should be done by jcmd. What you really need > to get a peek on the finalizer queue and print the histogram. The VM > has the heap histogram implementation. Have you considered leveraging > that? > > 5: 1012 40480 java.lang.ref.Finalizer > > You can find the registered Finalizer instances. The downside is that > icmd -finalizerinfo stops the world. I think it?s not unreasonable for > this diagnostic command to be expensive like -heap command. > > Mandy -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jk at codearte.io Tue May 26 06:22:25 2015 From: jk at codearte.io (Jakub Kubrynski) Date: Tue, 26 May 2015 08:22:25 +0200 Subject: G1 young STW time in MBean In-Reply-To: References: <556341CF.9000705@oracle.com> <55634D94.1080301@oracle.com> <55638AC8.4080401@oracle.com> Message-ID: Jesper already pointed me about JMC. The reason we're not using it is that it cannot be integrated with our production monitoring. It's more problem solving tool than continuous APM. So the question is if Oracle is going to implement some MBeans for G1, and if not maybe we could propose JSR about it? Cheers, Jakub 26 maj 2015 08:17 "Staffan Larsen" napisa?(a): > Try out Java Flight Recorder - it has a lot more data in it. > > /Staffan > > > On 25 maj 2015, at 22:49, Jesper Wilhelmsson < > jesper.wilhelmsson at oracle.com> wrote: > > > > Including the serviceability team. > > > > There might be plans for G1 monitoring in the future but I don't know > too much about what and when. I know there are commercial products that > does a pretty good job at G1 monitoring but I don't know if that is an > option for you. > > > > Best, > > /Jesper > > > > > > Jakub Kubrynski skrev den 25/5/15 22:42: > >> Maybe we could propose some JSR about that? Using G1 without proper > monitoring > >> is like living on the edge :) > >> > >> Cheers, > >> Jakub > >> > >> 2015-05-25 18:28 GMT+02:00 Jesper Wilhelmsson < > jesper.wilhelmsson at oracle.com > >> >: > >> > >> As far as I know there hasn't been much development done to get G1 > to play > >> well with the MBeans. And I don't think there is anything planned in > this > >> area. Maybe the serviceability team has other plans for the MBeans. > >> /Jesper > >> > >> > >> Jakub Kubrynski skrev den 25/5/15 17:43: > >> > >> Do you know if there is any development planned in this area? > Also there > >> is no > >> avgPauseTime, promoted and survived mbean information for G1. > The only > >> available > >> substitution I see for now is to get safepoint timings. > >> > >> Best, > >> Jakub Kubrynski > >> > >> 25 maj 2015 17:37 "Jesper Wilhelmsson" < > jesper.wilhelmsson at oracle.com > >> > >> >> >> napisa?(a): > >> > >> Hi Jakub, > >> > >> The times reported in the MBeans are total GC times. There > is > >> currently no > >> way to get STW times through the MBeans. > >> > >> Best regards, > >> /Jesper > >> > >> > >> Jakub Kubrynski skrev den 25/5/15 10:56: > >> > >> Hi, > >> > >> is there any possibility to get STW time for G1 young > >> collection through > >> MBean? > >> The duration reported here is IMHO total GC time > (concurrent + > >> stw). > >> > >> -- > >> Best regards, > >> Jakub Kubrynski > >> > >> > >> _______________________________________________ > >> hotspot-gc-use mailing list > >> hotspot-gc-use at openjdk.java.net hotspot-gc-use at openjdk.java.net> > >> >> > > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > >> > >> > >> > >> > >> -- > >> Best regards, > >> Jakub Kubrynski > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek.white at oracle.com Tue May 26 18:43:46 2015 From: derek.white at oracle.com (Derek White) Date: Tue, 26 May 2015 14:43:46 -0400 Subject: RFR(M,v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5564802A.2010403@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> Message-ID: <5564BEE2.7020105@oracle.com> On 5/26/15 10:16 AM, Dmitry Samersoff wrote: > Hi Everybody, > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.09/ > > Please review updated webrev - > > printFinalizationQueue now returns and array of Map.Entry and all formatting is done on VM side. > > -Dmitry Hi Dmitry, I looked at the jdk_webrev. Everything looks great except the spacing looks off in line 124 of Finalizer.java. My JNI is too rusty to provide a quick opinion on the hotspot_webrev portion. - Derek > > On 2015-05-21 02:07, Mandy Chung wrote: >>> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >>> > wrote: >>> >>> Other alternatives could be to do all hashing/sorting/printing on native >>> layer i.e. implement printFinalizationQueue inside VM. >>> >>> Both options has pros and cons - Java based solution requires less JNI >>> calls and better readable but takes more memory. >>> >>> It might be better to return an array of Map.Entry >>> objects to VM rather than one huge string. >> The output and formatting should be done by jcmd. What you really need >> to get a peek on the finalizer queue and print the histogram. The VM >> has the heap histogram implementation. Have you considered leveraging >> that? >> >> 5: 1012 40480 java.lang.ref.Finalizer >> >> You can find the registered Finalizer instances. The downside is that >> icmd -finalizerinfo stops the world. I think it?s not unreasonable for >> this diagnostic command to be expensive like -heap command. >> >> Mandy > From peter.levart at gmail.com Wed May 27 07:34:34 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 27 May 2015 09:34:34 +0200 Subject: RFR(M,v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5564802A.2010403@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> Message-ID: <5565738A.4040109@gmail.com> Hi Dmitry, The jdk part looks OK (no great changes on this side from last webrev). Is there a particular reason why the return type of printFinalizayionQueue() method is Object[] and not Map.Entry[] ? For the hotspot part, I have a few reservations. You expect that the type of array elements will be HashMap.Node and that the key/value fields will be at fixed offsets. Is this even true for all architectures (32bit, 64bit +-UseCompressedOops)? The type of HashMap entry is controlled by code in HashMap which has a long history of changes. Next time the implementation of HashMap changes, your code could break. Would it be possible to only use public API? To invoke methods on Map.Entry interface to obtain the key and value? Regards, Peter On 05/26/2015 04:16 PM, Dmitry Samersoff wrote: > Hi Everybody, > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.09/ > > Please review updated webrev - > > printFinalizationQueue now returns and array of Map.Entry and all formatting is done on VM side. > > -Dmitry > > On 2015-05-21 02:07, Mandy Chung wrote: >>> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >>> > wrote: >>> >>> Other alternatives could be to do all hashing/sorting/printing on native >>> layer i.e. implement printFinalizationQueue inside VM. >>> >>> Both options has pros and cons - Java based solution requires less JNI >>> calls and better readable but takes more memory. >>> >>> It might be better to return an array of Map.Entry >>> objects to VM rather than one huge string. >> The output and formatting should be done by jcmd. What you really need >> to get a peek on the finalizer queue and print the histogram. The VM >> has the heap histogram implementation. Have you considered leveraging >> that? >> >> 5: 1012 40480 java.lang.ref.Finalizer >> >> You can find the registered Finalizer instances. The downside is that >> icmd -finalizerinfo stops the world. I think it?s not unreasonable for >> this diagnostic command to be expensive like -heap command. >> >> Mandy > From vladimir.x.ivanov at oracle.com Wed May 27 10:00:35 2015 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Wed, 27 May 2015 13:00:35 +0300 Subject: [9] RFR (S): 8059340: ConstantPool::_resolved_references is missing in heap dump In-Reply-To: <555A2228.3080908@oracle.com> References: <542971DC.2090803@oracle.com> <24133E27-C024-41D8-80EF-B5707CFF0D0C@oracle.com> <552FFBAA.6050103@oracle.com> <55358D0A.1020407@oracle.com> <21C42551-29AA-44DC-AFE5-6156D681503D@oracle.com> <5536975F.60108@oracle.com> <555A2228.3080908@oracle.com> Message-ID: <556595C3.7080800@oracle.com> Unfortunately, it broke class redefinition (see JDK-8081004 [1]) and I don't see a reasonable way to fix that. The problem is that multiple class versions can be alive at the same moment w.r.t. class redefinition. Every class version should have its dedicated set of resolved_reference slots for invokedynamic/invokehandle instructions. They still have dedicated constant pools (and hence constant pool caches), but resolved_references array is shared now. It's not viable to simply append slots to the array during class redefinition since it causes memory leak (array size grows indefinitely). Compactification strategies are also complex, since slot indices should be either (1) stable (they are recorded in CPCEs) and slots are reference counted (in order to free and reuse them on next class redefinition); or (2) all class versions are patched with new indices on every class redefinition. IMO both approaches complicate things too much compared to keeping dedicated resolved_references per ConstantPool instance. Serguei, Staffan, what do you think? Do you see any viable solutions to the problem? Otherwise, I'm inclined to backout the change. Best regards, Vladimir Ivanov [1] https://bugs.openjdk.java.net/browse/JDK-8081004 On 5/18/15 8:32 PM, Vladimir Ivanov wrote: > Here's updated version: > http://cr.openjdk.java.net/~vlivanov/8059340/webrev.01 > > Moved ConstantPool::_resolved_references to mirror class instance. > > Fixed a couple of issues in CDS and JVMTI (class redefinition) caused by > this change. > > I had to hard code Class::resolved_references offset since it is used in > template interpreter which is generated earlier than j.l.Class is loaded > during VM bootstrap. > > Testing: hotspot/test, vm testbase (in progress) > > Best regards, > Vladimir Ivanov > > On 4/21/15 9:30 PM, Vladimir Ivanov wrote: >> Coleen, Chris, >> >> I'll proceed with moving ConstantPool::_resolved_references to j.l.Class >> instance then. >> >> Thanks for the feedback. >> >> Best regards, >> Vladimir Ivanov >> >> On 4/21/15 3:22 AM, Christian Thalinger wrote: >>> >>>> On Apr 20, 2015, at 4:34 PM, Coleen Phillimore >>>> > >>>> wrote: >>>> >>>> >>>> Vladimir, >>>> >>>> I think that changing the format of the heap dump isn't a good idea >>>> either. >>>> >>>> On 4/16/15, 2:12 PM, Vladimir Ivanov wrote: >>>>> (sorry for really late response; just got enough time to return to >>>>> the bug) >>>> >>>> I'd forgotten about it! >>>>> >>>>> Coleen, Staffan, >>>>> >>>>> Thanks a lot for the feedback! >>>>> >>>>> After thinking about the fix more, I don't think that using reserved >>>>> oop slot in CLASS DUMP for recording _resolved_references is the best >>>>> thing to do. IMO the change causes too much work for the users (heap >>>>> dump analysis tools). >>>>> >>>>> It needs specification update and then heap dump analyzers should be >>>>> updated as well. >>>>> >>>>> I have 2 alternative approaches (hacky and not-so-hacky :-)): >>>>> >>>>> - artificial class static field in the dump ("" >>>>> + optional id to guarantee unique name); >>>>> >>>>> - add j.l.Class::_resolved_references field; >>>>> Not sure how much overhead (mostly reads from bytecode) the move >>>>> from ConstantPool to j.l.Class adds, so I propose just to duplicate >>>>> it for now. >>>> >>>> I really like this second approach, so much so that I had a prototype >>>> for moving resolved_references directly to the j.l.Class object about >>>> a year ago. I couldn't find any benefit other than consolidating oops >>>> so the GC would have less work to do. If the resolved_references are >>>> moved to j.l.C instance, they can not be jobjects and the >>>> ClassLoaderData::_handles area wouldn't have to contain them (but >>>> there are other things that could go there so don't delete the >>>> _handles field yet). >>>> >>>> The change I had was relatively simple. The only annoying part was >>>> that getting to the resolved references has to be in macroAssembler >>>> and do: >>>> >>>> go through method->cpCache->constants->instanceKlass->java_mirror() >>>> rather than >>>> method->cpCache->constants->resolved_references->jmethod indirection >>>> >>>> I think it only affects the interpreter so the extra indirection >>>> wouldn't affect performance, so don't duplicate it! You don't want to >>>> increase space used by j.l.C without taking it out somewhere else! >>> >>> I like this approach. Can we do this? >>> >>>> >>>>> >>>>> What do you think about that? >>>> >>>> Is this bug worth doing this? I don't know but I'd really like it. >>>> >>>> Coleen >>>> >>>>> >>>>> Best regards, >>>>> Vladimir Ivanov >>>>> >>>>> On 10/6/14 11:35 AM, Staffan Larsen wrote: >>>>>> This looks like a good approach. However, there are a couple of more >>>>>> places that need to be updated. >>>>>> >>>>>> The hprof binary format is described in >>>>>> jdk/src/jdk.hprof.agent/share/native/libhprof/manual.html and needs >>>>>> to be updated. It?s also more formally specified in hprof_b_spec.h >>>>>> in the same directory. >>>>>> >>>>>> The hprof JVMTI agent in jdk/src/jdk.hprof.agent code would also >>>>>> need to be updated to show this field. Since this is a JVMTI agent >>>>>> it needs to be possible to find the resolved_refrences array via the >>>>>> JVMTI heap walking API. Perhaps that already works? - I haven?t >>>>>> looked. >>>>>> >>>>>> Finally, the Serviceability Agent implements yet another hprof >>>>>> binary dumper in >>>>>> hotspot/agent//src/share/classes/sun/jvm/hotspot/utilities/HeapHprofBinWriter.java >>>>>> >>>>>> >>>>>> which also needs to write this reference. >>>>>> >>>>>> Thanks, >>>>>> /Staffan >>>>>> >>>>>> On 29 sep 2014, at 16:51, Vladimir Ivanov >>>>>> > >>>>>> wrote: >>>>>> >>>>>>> http://cr.openjdk.java.net/~vlivanov/8059340/webrev.00/ >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8059340 >>>>>>> >>>>>>> VM heap dump doesn't contain ConstantPool::_resolved_references for >>>>>>> classes which have resolved references. >>>>>>> >>>>>>> ConstantPool::_resolved_references points to an Object[] holding >>>>>>> resolved constant pool entries (patches for VM anonymous classes, >>>>>>> linked CallSite & MethodType for invokedynamic instructions). >>>>>>> >>>>>>> I've decided to use reserved slot in HPROF class header format. >>>>>>> It requires an update in jhat to correctly display new info. >>>>>>> >>>>>>> The other approach I tried was to dump the reference as a fake >>>>>>> static field [1], but storing VM internal >>>>>>> ConstantPool::_resolved_references among user defined fields looks >>>>>>> confusing. >>>>>>> >>>>>>> Testing: manual (verified that corresponding arrays are properly >>>>>>> linked in Nashorn heap dump). >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Best regards, >>>>>>> Vladimir Ivanov >>>>>>> >>>>>>> [1] http://cr.openjdk.java.net/~vlivanov/8059340/static >>> From staffan.larsen at oracle.com Wed May 27 10:47:05 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Wed, 27 May 2015 12:47:05 +0200 Subject: RFR: 8081292 sun/tools/jmap/BasicJMapTest.java timed out Message-ID: <95F5D67B-9238-4888-B574-26E7AFB54969@oracle.com> The test sometimes does not complete in the default timeout so I?m doubling it. Thanks, /Staffan diff --git a/test/sun/tools/jmap/BasicJMapTest.java b/test/sun/tools/jmap/BasicJMapTest.java --- a/test/sun/tools/jmap/BasicJMapTest.java +++ b/test/sun/tools/jmap/BasicJMapTest.java @@ -44,7 +44,7 @@ * @build jdk.test.lib.hprof.module.* * @build jdk.test.lib.hprof.parser.* * @build jdk.test.lib.hprof.utils.* - * @run main BasicJMapTest + * @run main/timeout=240 BasicJMapTest */ public class BasicJMapTest { From markus.gronlund at oracle.com Wed May 27 10:44:30 2015 From: markus.gronlund at oracle.com (Markus Gronlund) Date: Wed, 27 May 2015 03:44:30 -0700 (PDT) Subject: RFR: 8081292 sun/tools/jmap/BasicJMapTest.java timed out In-Reply-To: <95F5D67B-9238-4888-B574-26E7AFB54969@oracle.com> References: <95F5D67B-9238-4888-B574-26E7AFB54969@oracle.com> Message-ID: <0788482b-1cf6-4935-bdd8-17fd42b4c14d@default> Looks good. /Markus -----Original Message----- From: Staffan Larsen Sent: den 27 maj 2015 12:47 To: serviceability-dev Subject: RFR: 8081292 sun/tools/jmap/BasicJMapTest.java timed out The test sometimes does not complete in the default timeout so I?m doubling it. Thanks, /Staffan diff --git a/test/sun/tools/jmap/BasicJMapTest.java b/test/sun/tools/jmap/BasicJMapTest.java --- a/test/sun/tools/jmap/BasicJMapTest.java +++ b/test/sun/tools/jmap/BasicJMapTest.java @@ -44,7 +44,7 @@ * @build jdk.test.lib.hprof.module.* * @build jdk.test.lib.hprof.parser.* * @build jdk.test.lib.hprof.utils.* - * @run main BasicJMapTest + * @run main/timeout=240 BasicJMapTest */ public class BasicJMapTest { From yekaterina.kantserova at oracle.com Wed May 27 13:02:00 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Wed, 27 May 2015 15:02:00 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows Message-ID: <5565C048.4050801@oracle.com> Hi, Could I please have a review of this fix. bug: https://bugs.openjdk.java.net/browse/JDK-8081037 webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 webrev hotspot: http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 From the bug: "The problem is most likely that SA will pause the target process while it is running. In this case, the target process is the same as the process that launched SA. That process is also handling the output from SA over a pipe, but when that pipe fills up the process cannot empty it and the SA process is blocked because it cannot write any more output. Deadlock." The solutions is to start a separate target process. Dmitry Samersoff has already created a test application for such cases so I've decided to move it on the top level library instead of duplicating it. The test application will reside under test/lib/share/classes/jdk/test/lib/apps and the test under test/lib-test/jdk/test/lib/apps. Thanks, Katja From jaroslav.bachorik at oracle.com Wed May 27 13:05:09 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Wed, 27 May 2015 15:05:09 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <5565C048.4050801@oracle.com> References: <5565C048.4050801@oracle.com> Message-ID: <5565C105.7060900@oracle.com> Looks fine! -JB- On 27.5.2015 15:02, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8081037 > webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 > webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 > webrev hotspot: > http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 > > From the bug: > "The problem is most likely that SA will pause the target process while > it is running. In this case, the target process is the same as the > process that launched SA. That process is also handling the output from > SA over a pipe, but when that pipe fills up the process cannot empty it > and the SA process is blocked because it cannot write any more output. > Deadlock." > > The solutions is to start a separate target process. Dmitry Samersoff > has already created a test application for such cases so I've decided to > move it on the top level library instead of duplicating it. The test > application will reside under test/lib/share/classes/jdk/test/lib/apps > and the test under test/lib-test/jdk/test/lib/apps. > > Thanks, > Katja From staffan.larsen at oracle.com Wed May 27 13:19:31 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Wed, 27 May 2015 15:19:31 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <5565C048.4050801@oracle.com> References: <5565C048.4050801@oracle.com> Message-ID: <821F5B9A-D18B-4458-81A1-D94B839F9CDC@oracle.com> Looks good! Thanks, /Staffan > On 27 maj 2015, at 15:02, Yekaterina Kantserova wrote: > > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8081037 > webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 > webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 > webrev hotspot: http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 > > From the bug: > "The problem is most likely that SA will pause the target process while it is running. In this case, the target process is the same as the process that launched SA. That process is also handling the output from SA over a pipe, but when that pipe fills up the process cannot empty it and the SA process is blocked because it cannot write any more output. Deadlock." > > The solutions is to start a separate target process. Dmitry Samersoff has already created a test application for such cases so I've decided to move it on the top level library instead of duplicating it. The test application will reside under test/lib/share/classes/jdk/test/lib/apps and the test under test/lib-test/jdk/test/lib/apps. > > Thanks, > Katja From dmitry.samersoff at oracle.com Wed May 27 13:26:34 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 27 May 2015 16:26:34 +0300 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <5565C048.4050801@oracle.com> References: <5565C048.4050801@oracle.com> Message-ID: <5565C60A.9090404@oracle.com> Katja, Looks good for me. Thank you for doing it! -Dmitry On 2015-05-27 16:02, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8081037 > webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 > webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 > webrev hotspot: > http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 > > From the bug: > "The problem is most likely that SA will pause the target process while > it is running. In this case, the target process is the same as the > process that launched SA. That process is also handling the output from > SA over a pipe, but when that pipe fills up the process cannot empty it > and the SA process is blocked because it cannot write any more output. > Deadlock." > > The solutions is to start a separate target process. Dmitry Samersoff > has already created a test application for such cases so I've decided to > move it on the top level library instead of duplicating it. The test > application will reside under test/lib/share/classes/jdk/test/lib/apps > and the test under test/lib-test/jdk/test/lib/apps. > > Thanks, > Katja -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mandy.chung at oracle.com Wed May 27 13:32:38 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 27 May 2015 06:32:38 -0700 Subject: RFR(M, v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5565738A.4040109@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> Message-ID: <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> > On May 27, 2015, at 12:34 AM, Peter Levart wrote: > > Hi Dmitry, > > The jdk part looks OK (no great changes on this side from last webrev). Is there a particular reason why the return type of printFinalizayionQueue() method is Object[] and not Map.Entry[] ? > Taking it further - is it simpler to return String[] of all classnames including the duplicated ones and have the VM do the count? Are you concerned with the size of the String[]? > For the hotspot part, I have a few reservations. You expect that the type of array elements will be HashMap.Node and that the key/value fields will be at fixed offsets. Is this even true for all architectures (32bit, 64bit +-UseCompressedOops)? > > The type of HashMap entry is controlled by code in HashMap which has a long history of changes. Next time the implementation of HashMap changes, your code could break. Would it be possible to only use public API? To invoke methods on Map.Entry interface to obtain the key and value? > Indeed, depending on the HashMap internal implementation is a bad idea. Mandy > Regards, Peter > > On 05/26/2015 04:16 PM, Dmitry Samersoff wrote: >> Hi Everybody, >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.09/ >> >> Please review updated webrev - >> >> printFinalizationQueue now returns and array of Map.Entry> and all formatting is done on VM side. >> >> -Dmitry >> >> On 2015-05-21 02:07, Mandy Chung wrote: >>>> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >>>> > wrote: >>>> >>>> Other alternatives could be to do all hashing/sorting/printing on native >>>> layer i.e. implement printFinalizationQueue inside VM. >>>> >>>> Both options has pros and cons - Java based solution requires less JNI >>>> calls and better readable but takes more memory. >>>> >>>> It might be better to return an array of Map.Entry >>>> objects to VM rather than one huge string. >>> The output and formatting should be done by jcmd. What you really need >>> to get a peek on the finalizer queue and print the histogram. The VM >>> has the heap histogram implementation. Have you considered leveraging >>> that? >>> >>> 5: 1012 40480 java.lang.ref.Finalizer >>> >>> You can find the registered Finalizer instances. The downside is that >>> icmd -finalizerinfo stops the world. I think it?s not unreasonable for >>> this diagnostic command to be expensive like -heap command. >>> >>> Mandy >> > From yekaterina.kantserova at oracle.com Wed May 27 13:38:11 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Wed, 27 May 2015 15:38:11 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <5565C105.7060900@oracle.com> References: <5565C048.4050801@oracle.com> <5565C105.7060900@oracle.com> Message-ID: <5565C8C3.4040901@oracle.com> Jaroslav, Staffan, Dmitry, thank you for the reviews! // Katja On 05/27/2015 03:05 PM, Jaroslav Bachorik wrote: > Looks fine! > > -JB- > > On 27.5.2015 15:02, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8081037 >> webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 >> webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 >> webrev hotspot: >> http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 >> >> From the bug: >> "The problem is most likely that SA will pause the target process while >> it is running. In this case, the target process is the same as the >> process that launched SA. That process is also handling the output from >> SA over a pipe, but when that pipe fills up the process cannot empty it >> and the SA process is blocked because it cannot write any more output. >> Deadlock." >> >> The solutions is to start a separate target process. Dmitry Samersoff >> has already created a test application for such cases so I've decided to >> move it on the top level library instead of duplicating it. The test >> application will reside under test/lib/share/classes/jdk/test/lib/apps >> and the test under test/lib-test/jdk/test/lib/apps. >> >> Thanks, >> Katja > From jesper.wilhelmsson at oracle.com Wed May 27 20:14:19 2015 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Wed, 27 May 2015 22:14:19 +0200 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <553504CF.8080904@gmail.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> Message-ID: <5566259B.8010202@oracle.com> Hi, I like that you removed _jvmti_force_gc from is_user_requested_gc() and used this method throughout. It is cleaner and is_user_requested_gc() makes more sense now. In vmCMSOperations.cpp I think the comment should say GCCause::_dcmd_gc_run. Besides that minor comment, looks good! Thanks, /Jesper Yasumasa Suenaga skrev den 20/4/15 15:53: > Hi all, > > I've uploaded webrev for this enhancement. > Could you review it? > > http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ > > > Thanks, > > Yasumasa > > > On 2015/03/11 22:13, Yasumasa Suenaga wrote: >> Hi all, >> >>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>> _dcmd_gc_run >>> to it. >> >> I've uploaded new webrev, and I've applied it to new patch. >> Could you review it? >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >> >> I also updated jtreg testcase. >> It works fine in my environment. >> >> >> Thanks, >> >> Yasumasa >> >> >> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>> Hi Mikael, >>> >>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>> returns true for some set of GCCause enum values, such as >>>> _java_lang_system_gc and _dcmd_gc_run >>> >>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>> This function is used with GCCause::is_serviceability_requested_gc() . >>> CMSCollector::is_external_interruption() and >>> AdaptiveSizePolicy::check_gc_overhead_limit() >>> >>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>> _jvmti_force_gc >>> is selected. >>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>> _dcmd_gc_run >>> to it. >>> >>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>> be necessary. >>> >>> We can use GCCause::is_user_requested_gc() if the proposal in above is accepted. >>> >>> >>> Thanks >>> >>> Yasumasa >>> >>> >>> >>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>> Hi Yasumasa, >>>> >>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>> Hi all, >>>>> >>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic Command. >>>>> However, it has been backouted because test is failed [1] and it is not >>>>> considered >>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>> >>>>> I've created patch for this enhancement. >>>>> Could you review it? >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>> >>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>> returns true for some set of GCCause enum values, such as >>>> _java_lang_system_gc and _dcmd_gc_run >>>> >>>> Given that the documentation of the GC.run command is: >>>> "GC.run >>>> Call java.lang.System.gc(). >>>> >>>> Impact: Medium: Depends on Java heap size and content. >>>> >>>> Syntax: GC.run" >>>> >>>> I interpret the documentation that the GC is supposed to be (for all intents >>>> and purposes) equivalent to the application invoking System.gc(). >>>> >>>> This would also require updates to other places where we refer to the >>>> _java_lang_system_gc GCCause, such as UseAdaptiveSizePolicyWithSystemGC >>>> >>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>> be necessary. >>>> >>>> /Mikael >>>> >>>>> >>>>> >>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>> So I need a Sponsor. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> [1] >>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>> >>>>> [2] >>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>> >>>>> From yasuenag at gmail.com Wed May 27 23:52:33 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 08:52:33 +0900 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <5566259B.8010202@oracle.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> <5566259B.8010202@oracle.com> Message-ID: <556658C1.4090300@gmail.com> Hi Jesper, Thank you for your comment. I will fix it. Thanks, Yasumasa On 2015/05/28 5:14, Jesper Wilhelmsson wrote: > Hi, > > I like that you removed _jvmti_force_gc from is_user_requested_gc() and used this method throughout. It is cleaner and is_user_requested_gc() makes more sense now. > > In vmCMSOperations.cpp I think the comment should say GCCause::_dcmd_gc_run. > > Besides that minor comment, looks good! > > Thanks, > /Jesper > > > Yasumasa Suenaga skrev den 20/4/15 15:53: >> Hi all, >> >> I've uploaded webrev for this enhancement. >> Could you review it? >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >> >> >> Thanks, >> >> Yasumasa >> >> >> On 2015/03/11 22:13, Yasumasa Suenaga wrote: >>> Hi all, >>> >>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>> _dcmd_gc_run >>>> to it. >>> >>> I've uploaded new webrev, and I've applied it to new patch. >>> Could you review it? >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>> >>> I also updated jtreg testcase. >>> It works fine in my environment. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>>> Hi Mikael, >>>> >>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>> returns true for some set of GCCause enum values, such as >>>>> _java_lang_system_gc and _dcmd_gc_run >>>> >>>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>>> This function is used with GCCause::is_serviceability_requested_gc() . >>>> CMSCollector::is_external_interruption() and >>>> AdaptiveSizePolicy::check_gc_overhead_limit() >>>> >>>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>>> _jvmti_force_gc >>>> is selected. >>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>> _dcmd_gc_run >>>> to it. >>>> >>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>> be necessary. >>>> >>>> We can use GCCause::is_user_requested_gc() if the proposal in above is accepted. >>>> >>>> >>>> Thanks >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>>> Hi Yasumasa, >>>>> >>>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>>> Hi all, >>>>>> >>>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic Command. >>>>>> However, it has been backouted because test is failed [1] and it is not >>>>>> considered >>>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>>> >>>>>> I've created patch for this enhancement. >>>>>> Could you review it? >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>>> >>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>> returns true for some set of GCCause enum values, such as >>>>> _java_lang_system_gc and _dcmd_gc_run >>>>> >>>>> Given that the documentation of the GC.run command is: >>>>> "GC.run >>>>> Call java.lang.System.gc(). >>>>> >>>>> Impact: Medium: Depends on Java heap size and content. >>>>> >>>>> Syntax: GC.run" >>>>> >>>>> I interpret the documentation that the GC is supposed to be (for all intents >>>>> and purposes) equivalent to the application invoking System.gc(). >>>>> >>>>> This would also require updates to other places where we refer to the >>>>> _java_lang_system_gc GCCause, such as UseAdaptiveSizePolicyWithSystemGC >>>>> >>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>> be necessary. >>>>> >>>>> /Mikael >>>>> >>>>>> >>>>>> >>>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>>> So I need a Sponsor. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> [1] >>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>>> >>>>>> [2] >>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>>> >>>>>> From yasuenag at gmail.com Thu May 28 03:19:45 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 12:19:45 +0900 Subject: JDK-8081295: Build failed with GCC 5.1.1 Message-ID: <55668951.9070309@gmail.com> Hi all, I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, however, it was failed. I found several problems: System: Fedora release 22 (Twenty Two) x86_64 - gcc-5.1.1-1.fc22.x86_64 Problems: 1. Array bounds check in GCC - jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c - jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c - jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c - jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c It seems to be bug of GCC: Bug 59124: [4.8/4.9/5/6 Regression] Wrong warnings "array subscript is above array bounds" https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 I think implementations of these files have no problem. So I propose to ignore warning(s) of compiler through pragma option as workaround when we use GCC [1]. 2. Local variables might be clobbered - jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c SplashDecodePng() calls setjmp(3). Some local variables initialize before setjmp() call, and use after it. Their initial values are only used at cleanup code (*done* label) and actual values are stored only after setjmp() call. So I think we can ignore this error through pragma option [1]. 3. Incorrect condition - jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c searchAllSourceNames() returns int value. However, branch condition in eventFilterRestricted_passesFilter() treats it as boolean value. I received a comment from Erik to use --disable-warnings-as-errors, however I could not avoid error in 3. Thanks, Yasumasa [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html From yasuenag at gmail.com Thu May 28 03:27:15 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 12:27:15 +0900 Subject: JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: <55668951.9070309@gmail.com> References: <55668951.9070309@gmail.com> Message-ID: <55668B13.1000303@gmail.com> I've uploaded webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ Please review. Thanks, Yasumasa On 2015/05/28 12:19, Yasumasa Suenaga wrote: > Hi all, > > I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, however, it was failed. > I found several problems: > > > System: > Fedora release 22 (Twenty Two) x86_64 > - gcc-5.1.1-1.fc22.x86_64 > > > Problems: > 1. Array bounds check in GCC > - jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c > - jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c > - jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c > - jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c > > It seems to be bug of GCC: > Bug 59124: [4.8/4.9/5/6 Regression] Wrong warnings "array subscript is above array bounds" > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 > > I think implementations of these files have no problem. > So I propose to ignore warning(s) of compiler through pragma option as > workaround when we use GCC [1]. > > > 2. Local variables might be clobbered > - jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c > > SplashDecodePng() calls setjmp(3). > Some local variables initialize before setjmp() call, and use after it. > Their initial values are only used at cleanup code (*done* label) and > actual values are stored only after setjmp() call. > So I think we can ignore this error through pragma option [1]. > > > 3. Incorrect condition > - jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c > > searchAllSourceNames() returns int value. However, branch condition in > eventFilterRestricted_passesFilter() treats it as boolean value. > > > I received a comment from Erik to use --disable-warnings-as-errors, > however I could not avoid error in 3. > > > Thanks, > > Yasumasa > > > [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html > From yasuenag at gmail.com Thu May 28 03:31:31 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 12:31:31 +0900 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <556658C1.4090300@gmail.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> <5566259B.8010202@oracle.com> <556658C1.4090300@gmail.com> Message-ID: <55668C13.9080508@gmail.com> I've uploaded new webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.02/ I need a Sponsor and more reviewer. Please review it. Thanks, Yasumasa On 2015/05/28 8:52, Yasumasa Suenaga wrote: > Hi Jesper, > > Thank you for your comment. > I will fix it. > > > Thanks, > > Yasumasa > > > On 2015/05/28 5:14, Jesper Wilhelmsson wrote: >> Hi, >> >> I like that you removed _jvmti_force_gc from is_user_requested_gc() and used this method throughout. It is cleaner and is_user_requested_gc() makes more sense now. >> >> In vmCMSOperations.cpp I think the comment should say GCCause::_dcmd_gc_run. >> >> Besides that minor comment, looks good! >> >> Thanks, >> /Jesper >> >> >> Yasumasa Suenaga skrev den 20/4/15 15:53: >>> Hi all, >>> >>> I've uploaded webrev for this enhancement. >>> Could you review it? >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2015/03/11 22:13, Yasumasa Suenaga wrote: >>>> Hi all, >>>> >>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>> _dcmd_gc_run >>>>> to it. >>>> >>>> I've uploaded new webrev, and I've applied it to new patch. >>>> Could you review it? >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>> >>>> I also updated jtreg testcase. >>>> It works fine in my environment. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>>>> Hi Mikael, >>>>> >>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>> returns true for some set of GCCause enum values, such as >>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>> >>>>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>>>> This function is used with GCCause::is_serviceability_requested_gc() . >>>>> CMSCollector::is_external_interruption() and >>>>> AdaptiveSizePolicy::check_gc_overhead_limit() >>>>> >>>>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>>>> _jvmti_force_gc >>>>> is selected. >>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>> _dcmd_gc_run >>>>> to it. >>>>> >>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>> be necessary. >>>>> >>>>> We can use GCCause::is_user_requested_gc() if the proposal in above is accepted. >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>>>> Hi Yasumasa, >>>>>> >>>>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic Command. >>>>>>> However, it has been backouted because test is failed [1] and it is not >>>>>>> considered >>>>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>>>> >>>>>>> I've created patch for this enhancement. >>>>>>> Could you review it? >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>>>> >>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>> returns true for some set of GCCause enum values, such as >>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>> >>>>>> Given that the documentation of the GC.run command is: >>>>>> "GC.run >>>>>> Call java.lang.System.gc(). >>>>>> >>>>>> Impact: Medium: Depends on Java heap size and content. >>>>>> >>>>>> Syntax: GC.run" >>>>>> >>>>>> I interpret the documentation that the GC is supposed to be (for all intents >>>>>> and purposes) equivalent to the application invoking System.gc(). >>>>>> >>>>>> This would also require updates to other places where we refer to the >>>>>> _java_lang_system_gc GCCause, such as UseAdaptiveSizePolicyWithSystemGC >>>>>> >>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>> be necessary. >>>>>> >>>>>> /Mikael >>>>>> >>>>>>> >>>>>>> >>>>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>>>> So I need a Sponsor. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>>>> >>>>>>> [2] >>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>>>> >>>>>>> From serguei.spitsyn at oracle.com Thu May 28 08:34:18 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 28 May 2015 01:34:18 -0700 Subject: RFR: 8081292 sun/tools/jmap/BasicJMapTest.java timed out In-Reply-To: <95F5D67B-9238-4888-B574-26E7AFB54969@oracle.com> References: <95F5D67B-9238-4888-B574-26E7AFB54969@oracle.com> Message-ID: <5566D30A.10101@oracle.com> It looks good. Thanks, Serguei On 5/27/15 3:47 AM, Staffan Larsen wrote: > The test sometimes does not complete in the default timeout so I?m doubling it. > > Thanks, > /Staffan > > > diff --git a/test/sun/tools/jmap/BasicJMapTest.java b/test/sun/tools/jmap/BasicJMapTest.java > --- a/test/sun/tools/jmap/BasicJMapTest.java > +++ b/test/sun/tools/jmap/BasicJMapTest.java > @@ -44,7 +44,7 @@ > * @build jdk.test.lib.hprof.module.* > * @build jdk.test.lib.hprof.parser.* > * @build jdk.test.lib.hprof.utils.* > - * @run main BasicJMapTest > + * @run main/timeout=240 BasicJMapTest > */ > public class BasicJMapTest { > From staffan.larsen at oracle.com Thu May 28 10:38:16 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 28 May 2015 12:38:16 +0200 Subject: RFR: 8080928 Uninitialised variable in hotspot/src/share/vm/prims/jvmtiEnvBase.cpp Message-ID: Please review this simple fix for an uninitialized variable. webrev: http://cr.openjdk.java.net/~sla/8080928/webrev.00/src/share/vm/prims/jvmtiEnvBase.cpp.sdiff.html Thanks, /Staffan -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.gronlund at oracle.com Thu May 28 10:36:29 2015 From: markus.gronlund at oracle.com (Markus Gronlund) Date: Thu, 28 May 2015 03:36:29 -0700 (PDT) Subject: RFR: 8080928 Uninitialised variable in hotspot/src/share/vm/prims/jvmtiEnvBase.cpp In-Reply-To: References: Message-ID: Looks good. /Markus From: Staffan Larsen Sent: den 28 maj 2015 12:38 To: serviceability-dev Subject: RFR: 8080928 Uninitialised variable in hotspot/src/share/vm/prims/jvmtiEnvBase.cpp Please review this simple fix for an uninitialized variable. webrev: http://cr.openjdk.java.net/~sla/8080928/webrev.00/src/share/vm/prims/jvmtiEnvBase.cpp.sdiff.html Thanks, /Staffan -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Thu May 28 10:54:02 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 28 May 2015 03:54:02 -0700 Subject: RFR: 8080928 Uninitialised variable in hotspot/src/share/vm/prims/jvmtiEnvBase.cpp In-Reply-To: References: Message-ID: <5566F3CA.10401@oracle.com> Looks good. Thanks, Serguei On 5/28/15 3:38 AM, Staffan Larsen wrote: > Please review this simple fix for an uninitialized variable. > > webrev: > http://cr.openjdk.java.net/~sla/8080928/webrev.00/src/share/vm/prims/jvmtiEnvBase.cpp.sdiff.html > > > > Thanks, > /Staffan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yasuenag at gmail.com Thu May 28 11:52:49 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 20:52:49 +0900 Subject: [OpenJDK 2D-Dev] JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: References: <55668951.9070309@gmail.com> <55668B13.1000303@gmail.com> <5566DD67.9000301@oracle.com> Message-ID: Hi Alexander, Thank you for your comment. I agree to use DISABLED_WARNINGS_gcc. Should I make a patch after JDK-8080695 committed? Thanks, Yasumasa 2015/05/28 18:19 "Alexander Zvegintsev" : Hello Yasumasa, I prefer to avoid such pragma usage, as for me code is more readable without it. So we can add array-bounds to DISABLED_WARNINGS_gcc in make/lib/Awt2dLibraries.gmk for BUILD_LIBJAVAJPEG and BUILD_LIBMLIB_IMAGE. We can nullify row_pointers and image_data after setjmp call to remove warnings in splashscreen_png.c: diff -r 729dffc8afa0 src/java.desktop/share/native/libsplashscreen/splashscreen_png.c --- a/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c +++ b/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c @@ -77,6 +77,8 @@ SplashDecodePng(Splash * splash, png_rw_ #else if (setjmp(png_jmpbuf(png_ptr))) { #endif + row_pointers = NULL; + image_data = NULL; goto done; } (or declare them as volatile, but I prefer the first option). BTW, there is another issue about this splashscreen_png.c gcc build failure [1]. [1] https://bugs.openjdk.java.net/browse/JDK-8080695 Thanks, Alexander. On 05/28/2015 06:27 AM, Yasumasa Suenaga wrote: > I've uploaded webrev: > http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ > > Please review. > > > Thanks, > > Yasumasa > > > On 2015/05/28 12:19, Yasumasa Suenaga wrote: > >> Hi all, >> >> I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, however, it was >> failed. >> I found several problems: >> >> >> System: >> Fedora release 22 (Twenty Two) x86_64 >> - gcc-5.1.1-1.fc22.x86_64 >> >> >> Problems: >> 1. Array bounds check in GCC >> - jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c >> - jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c >> - >> jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c >> - >> jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c >> >> It seems to be bug of GCC: >> Bug 59124: [4.8/4.9/5/6 Regression] Wrong warnings "array >> subscript is above array bounds" >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 >> >> I think implementations of these files have no problem. >> So I propose to ignore warning(s) of compiler through pragma >> option as >> workaround when we use GCC [1]. >> >> >> 2. Local variables might be clobbered >> - >> jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> >> SplashDecodePng() calls setjmp(3). >> Some local variables initialize before setjmp() call, and use >> after it. >> Their initial values are only used at cleanup code (*done* label) >> and >> actual values are stored only after setjmp() call. >> So I think we can ignore this error through pragma option [1]. >> >> >> 3. Incorrect condition >> - jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c >> >> searchAllSourceNames() returns int value. However, branch >> condition in >> eventFilterRestricted_passesFilter() treats it as boolean value. >> >> >> I received a comment from Erik to use --disable-warnings-as-errors, >> however I could not avoid error in 3. >> >> >> Thanks, >> >> Yasumasa >> >> >> [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.zvegintsev at oracle.com Thu May 28 09:18:31 2015 From: alexander.zvegintsev at oracle.com (Alexander Zvegintsev) Date: Thu, 28 May 2015 12:18:31 +0300 Subject: [OpenJDK 2D-Dev] JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: <55668B13.1000303@gmail.com> References: <55668951.9070309@gmail.com> <55668B13.1000303@gmail.com> Message-ID: <5566DD67.9000301@oracle.com> Hello Yasumasa, I prefer to avoid such pragma usage, as for me code is more readable without it. So we can add array-bounds to DISABLED_WARNINGS_gcc in make/lib/Awt2dLibraries.gmk for BUILD_LIBJAVAJPEG and BUILD_LIBMLIB_IMAGE. We can nullify row_pointers and image_data after setjmp call to remove warnings in splashscreen_png.c: diff -r 729dffc8afa0 src/java.desktop/share/native/libsplashscreen/splashscreen_png.c --- a/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c +++ b/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c @@ -77,6 +77,8 @@ SplashDecodePng(Splash * splash, png_rw_ #else if (setjmp(png_jmpbuf(png_ptr))) { #endif + row_pointers = NULL; + image_data = NULL; goto done; } (or declare them as volatile, but I prefer the first option). BTW, there is another issue about this splashscreen_png.c gcc build failure [1]. [1] https://bugs.openjdk.java.net/browse/JDK-8080695 Thanks, Alexander. On 05/28/2015 06:27 AM, Yasumasa Suenaga wrote: > I've uploaded webrev: > http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ > > Please review. > > > Thanks, > > Yasumasa > > > On 2015/05/28 12:19, Yasumasa Suenaga wrote: >> Hi all, >> >> I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, however, it was failed. >> I found several problems: >> >> >> System: >> Fedora release 22 (Twenty Two) x86_64 >> - gcc-5.1.1-1.fc22.x86_64 >> >> >> Problems: >> 1. Array bounds check in GCC >> - jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c >> - jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c >> - jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c >> - jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c >> >> It seems to be bug of GCC: >> Bug 59124: [4.8/4.9/5/6 Regression] Wrong warnings "array subscript is above array bounds" >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 >> >> I think implementations of these files have no problem. >> So I propose to ignore warning(s) of compiler through pragma option as >> workaround when we use GCC [1]. >> >> >> 2. Local variables might be clobbered >> - jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> >> SplashDecodePng() calls setjmp(3). >> Some local variables initialize before setjmp() call, and use after it. >> Their initial values are only used at cleanup code (*done* label) and >> actual values are stored only after setjmp() call. >> So I think we can ignore this error through pragma option [1]. >> >> >> 3. Incorrect condition >> - jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c >> >> searchAllSourceNames() returns int value. However, branch condition in >> eventFilterRestricted_passesFilter() treats it as boolean value. >> >> >> I received a comment from Erik to use --disable-warnings-as-errors, >> however I could not avoid error in 3. >> >> >> Thanks, >> >> Yasumasa >> >> >> [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html >> From alexander.zvegintsev at oracle.com Thu May 28 13:00:20 2015 From: alexander.zvegintsev at oracle.com (Alexander Zvegintsev) Date: Thu, 28 May 2015 16:00:20 +0300 Subject: [OpenJDK 2D-Dev] JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: References: <55668951.9070309@gmail.com> <55668B13.1000303@gmail.com> <5566DD67.9000301@oracle.com> Message-ID: <55671164.6070309@oracle.com> I think that you can just exclude libsplashscreen part from your fix, since it is covered by JDK-8080695. Thanks, Alexander. On 05/28/2015 02:52 PM, Yasumasa Suenaga wrote: > > Hi Alexander, > > Thank you for your comment. > > I agree to use DISABLED_WARNINGS_gcc. > > Should I make a patch afterJDK-8080695 > committed? > > Thanks, > > Yasumasa > > 2015/05/28 18:19 "Alexander Zvegintsev" > >: > > Hello Yasumasa, > > I prefer to avoid such pragma usage, as for me code is more > readable without it. > So we can add array-bounds to DISABLED_WARNINGS_gcc in > make/lib/Awt2dLibraries.gmk for BUILD_LIBJAVAJPEG and > BUILD_LIBMLIB_IMAGE. > > We can nullify row_pointers and image_data after setjmp call to > remove warnings in splashscreen_png.c: > diff -r 729dffc8afa0 > src/java.desktop/share/native/libsplashscreen/splashscreen_png.c > --- a/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c > +++ b/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c > @@ -77,6 +77,8 @@ SplashDecodePng(Splash * splash, png_rw_ > #else > if (setjmp(png_jmpbuf(png_ptr))) { > #endif > + row_pointers = NULL; > + image_data = NULL; > goto done; > } > (or declare them as volatile, but I prefer the first option). > BTW, there is another issue about this splashscreen_png.c gcc > build failure [1]. > > [1] https://bugs.openjdk.java.net/browse/JDK-8080695 > > Thanks, > > Alexander. > > > On 05/28/2015 06:27 AM, Yasumasa Suenaga wrote: > > I've uploaded webrev: > http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ > > > Please review. > > > Thanks, > > Yasumasa > > > On 2015/05/28 12:19, Yasumasa Suenaga wrote: > > Hi all, > > I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, > however, it was failed. > I found several problems: > > > System: > Fedora release 22 (Twenty Two) x86_64 > - gcc-5.1.1-1.fc22.x86_64 > > > Problems: > 1. Array bounds check in GCC > - > jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c > - > jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c > - > jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c > - > jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c > > It seems to be bug of GCC: > Bug 59124: [4.8/4.9/5/6 Regression] Wrong > warnings "array subscript is above array bounds" > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 > > I think implementations of these files have no problem. > So I propose to ignore warning(s) of compiler > through pragma option as > workaround when we use GCC [1]. > > > 2. Local variables might be clobbered > - > jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c > > SplashDecodePng() calls setjmp(3). > Some local variables initialize before setjmp() > call, and use after it. > Their initial values are only used at cleanup code > (*done* label) and > actual values are stored only after setjmp() call. > So I think we can ignore this error through pragma > option [1]. > > > 3. Incorrect condition > - > jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c > > searchAllSourceNames() returns int value. However, > branch condition in > eventFilterRestricted_passesFilter() treats it as > boolean value. > > > I received a comment from Erik to use > --disable-warnings-as-errors, > however I could not avoid error in 3. > > > Thanks, > > Yasumasa > > > [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.wilhelmsson at oracle.com Thu May 28 13:47:50 2015 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Thu, 28 May 2015 15:47:50 +0200 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <55668C13.9080508@gmail.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> <5566259B.8010202@oracle.com> <556658C1.4090300@gmail.com> <55668C13.9080508@gmail.com> Message-ID: <55671C86.5000109@oracle.com> Looks good! I'll sponsor it. /Jesper Yasumasa Suenaga skrev den 28/5/15 05:31: > I've uploaded new webrev: > http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.02/ > > I need a Sponsor and more reviewer. > Please review it. > > > Thanks, > > Yasumasa > > > On 2015/05/28 8:52, Yasumasa Suenaga wrote: >> Hi Jesper, >> >> Thank you for your comment. >> I will fix it. >> >> >> Thanks, >> >> Yasumasa >> >> >> On 2015/05/28 5:14, Jesper Wilhelmsson wrote: >>> Hi, >>> >>> I like that you removed _jvmti_force_gc from is_user_requested_gc() and used >>> this method throughout. It is cleaner and is_user_requested_gc() makes more >>> sense now. >>> >>> In vmCMSOperations.cpp I think the comment should say GCCause::_dcmd_gc_run. >>> >>> Besides that minor comment, looks good! >>> >>> Thanks, >>> /Jesper >>> >>> >>> Yasumasa Suenaga skrev den 20/4/15 15:53: >>>> Hi all, >>>> >>>> I've uploaded webrev for this enhancement. >>>> Could you review it? >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> On 2015/03/11 22:13, Yasumasa Suenaga wrote: >>>>> Hi all, >>>>> >>>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>>> _dcmd_gc_run >>>>>> to it. >>>>> >>>>> I've uploaded new webrev, and I've applied it to new patch. >>>>> Could you review it? >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>>> >>>>> I also updated jtreg testcase. >>>>> It works fine in my environment. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>>>>> Hi Mikael, >>>>>> >>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>>> returns true for some set of GCCause enum values, such as >>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>> >>>>>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>>>>> This function is used with GCCause::is_serviceability_requested_gc() . >>>>>> CMSCollector::is_external_interruption() and >>>>>> AdaptiveSizePolicy::check_gc_overhead_limit() >>>>>> >>>>>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>>>>> _jvmti_force_gc >>>>>> is selected. >>>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>>> _dcmd_gc_run >>>>>> to it. >>>>>> >>>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>>> be necessary. >>>>>> >>>>>> We can use GCCause::is_user_requested_gc() if the proposal in above is >>>>>> accepted. >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> >>>>>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>>>>> Hi Yasumasa, >>>>>>> >>>>>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic Command. >>>>>>>> However, it has been backouted because test is failed [1] and it is not >>>>>>>> considered >>>>>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>>>>> >>>>>>>> I've created patch for this enhancement. >>>>>>>> Could you review it? >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>>>>> >>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>>> returns true for some set of GCCause enum values, such as >>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>>> >>>>>>> Given that the documentation of the GC.run command is: >>>>>>> "GC.run >>>>>>> Call java.lang.System.gc(). >>>>>>> >>>>>>> Impact: Medium: Depends on Java heap size and content. >>>>>>> >>>>>>> Syntax: GC.run" >>>>>>> >>>>>>> I interpret the documentation that the GC is supposed to be (for all intents >>>>>>> and purposes) equivalent to the application invoking System.gc(). >>>>>>> >>>>>>> This would also require updates to other places where we refer to the >>>>>>> _java_lang_system_gc GCCause, such as UseAdaptiveSizePolicyWithSystemGC >>>>>>> >>>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>>> be necessary. >>>>>>> >>>>>>> /Mikael >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>>>>> So I need a Sponsor. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>>>>> >>>>>>>> >>>>>>>> [2] >>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>>>>> >>>>>>>> >>>>>>>> From yasuenag at gmail.com Thu May 28 14:03:05 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 23:03:05 +0900 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <55671C86.5000109@oracle.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> <5566259B.8010202@oracle.com> <556658C1.4090300@gmail.com> <55668C13.9080508@gmail.com> <55671C86.5000109@oracle.com> Message-ID: <55672019.7070406@gmail.com> > I'll sponsor it. Thank you Jesper! I will send a chengeset after reviewing. Yasumasa On 2015/05/28 22:47, Jesper Wilhelmsson wrote: > Looks good! > I'll sponsor it. > /Jesper > > Yasumasa Suenaga skrev den 28/5/15 05:31: >> I've uploaded new webrev: >> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.02/ >> >> I need a Sponsor and more reviewer. >> Please review it. >> >> >> Thanks, >> >> Yasumasa >> >> >> On 2015/05/28 8:52, Yasumasa Suenaga wrote: >>> Hi Jesper, >>> >>> Thank you for your comment. >>> I will fix it. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2015/05/28 5:14, Jesper Wilhelmsson wrote: >>>> Hi, >>>> >>>> I like that you removed _jvmti_force_gc from is_user_requested_gc() and used >>>> this method throughout. It is cleaner and is_user_requested_gc() makes more >>>> sense now. >>>> >>>> In vmCMSOperations.cpp I think the comment should say GCCause::_dcmd_gc_run. >>>> >>>> Besides that minor comment, looks good! >>>> >>>> Thanks, >>>> /Jesper >>>> >>>> >>>> Yasumasa Suenaga skrev den 20/4/15 15:53: >>>>> Hi all, >>>>> >>>>> I've uploaded webrev for this enhancement. >>>>> Could you review it? >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> On 2015/03/11 22:13, Yasumasa Suenaga wrote: >>>>>> Hi all, >>>>>> >>>>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>>>> _dcmd_gc_run >>>>>>> to it. >>>>>> >>>>>> I've uploaded new webrev, and I've applied it to new patch. >>>>>> Could you review it? >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>>>> >>>>>> I also updated jtreg testcase. >>>>>> It works fine in my environment. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>>>>>> Hi Mikael, >>>>>>> >>>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>>>> returns true for some set of GCCause enum values, such as >>>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>>> >>>>>>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>>>>>> This function is used with GCCause::is_serviceability_requested_gc() . >>>>>>> CMSCollector::is_external_interruption() and >>>>>>> AdaptiveSizePolicy::check_gc_overhead_limit() >>>>>>> >>>>>>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>>>>>> _jvmti_force_gc >>>>>>> is selected. >>>>>>> So I think we can remove _jvmti_force_gc from is_user_requested_gc() and add >>>>>>> _dcmd_gc_run >>>>>>> to it. >>>>>>> >>>>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>>>> be necessary. >>>>>>> >>>>>>> We can use GCCause::is_user_requested_gc() if the proposal in above is >>>>>>> accepted. >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>>>>>> Hi Yasumasa, >>>>>>>> >>>>>>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic Command. >>>>>>>>> However, it has been backouted because test is failed [1] and it is not >>>>>>>>> considered >>>>>>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>>>>>> >>>>>>>>> I've created patch for this enhancement. >>>>>>>>> Could you review it? >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>>>>>> >>>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() which >>>>>>>> returns true for some set of GCCause enum values, such as >>>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>>>> >>>>>>>> Given that the documentation of the GC.run command is: >>>>>>>> "GC.run >>>>>>>> Call java.lang.System.gc(). >>>>>>>> >>>>>>>> Impact: Medium: Depends on Java heap size and content. >>>>>>>> >>>>>>>> Syntax: GC.run" >>>>>>>> >>>>>>>> I interpret the documentation that the GC is supposed to be (for all intents >>>>>>>> and purposes) equivalent to the application invoking System.gc(). >>>>>>>> >>>>>>>> This would also require updates to other places where we refer to the >>>>>>>> _java_lang_system_gc GCCause, such as UseAdaptiveSizePolicyWithSystemGC >>>>>>>> >>>>>>>> A "grep" for _java_lang_system_gc should yield more places where updates may >>>>>>>> be necessary. >>>>>>>> >>>>>>>> /Mikael >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>>>>>> So I need a Sponsor. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>>>>>> >>>>>>>>> >>>>>>>>> [2] >>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>>>>>> >>>>>>>>> >>>>>>>>> From peter.levart at gmail.com Thu May 28 14:35:14 2015 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 28 May 2015 16:35:14 +0200 Subject: RFR(M,v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> Message-ID: <556727A2.9060209@gmail.com> Hi Mandy, On 05/27/2015 03:32 PM, Mandy Chung wrote: >> On May 27, 2015, at 12:34 AM, Peter Levart wrote: >> >> Hi Dmitry, >> >> The jdk part looks OK (no great changes on this side from last webrev). Is there a particular reason why the return type of printFinalizayionQueue() method is Object[] and not Map.Entry[] ? >> > Taking it further - is it simpler to return String[] of all classnames including the duplicated ones and have the VM do the count? Are you concerned with the size of the String[]? Yes, the histogram is much smaller than the list of all instances. There can be millions of instances waiting in finalizer queue, but only a few distinct classes. What could be done in Java to simplify things in native code but still not format the output is to convert the array of Map.Entry(s) into an Object[] array of alternating {String, int[], String, int[], .... } Would this simplify native code for the price of a little extra work in Java? The sizes of old and new arrays are not big (# of distinct classes of finalizable objects in the queue). > >> For the hotspot part, I have a few reservations. You expect that the type of array elements will be HashMap.Node and that the key/value fields will be at fixed offsets. Is this even true for all architectures (32bit, 64bit +-UseCompressedOops)? >> >> The type of HashMap entry is controlled by code in HashMap which has a long history of changes. Next time the implementation of HashMap changes, your code could break. Would it be possible to only use public API? To invoke methods on Map.Entry interface to obtain the key and value? >> > Indeed, depending on the HashMap internal implementation is a bad idea. > > Mandy Regards, Peter > >> Regards, Peter >> >> On 05/26/2015 04:16 PM, Dmitry Samersoff wrote: >>> Hi Everybody, >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.09/ >>> >>> Please review updated webrev - >>> >>> printFinalizationQueue now returns and array of Map.Entry>> and all formatting is done on VM side. >>> >>> -Dmitry >>> >>> On 2015-05-21 02:07, Mandy Chung wrote: >>>>> On May 19, 2015, at 11:51 PM, Dmitry Samersoff >>>>> > wrote: >>>>> >>>>> Other alternatives could be to do all hashing/sorting/printing on native >>>>> layer i.e. implement printFinalizationQueue inside VM. >>>>> >>>>> Both options has pros and cons - Java based solution requires less JNI >>>>> calls and better readable but takes more memory. >>>>> >>>>> It might be better to return an array of Map.Entry >>>>> objects to VM rather than one huge string. >>>> The output and formatting should be done by jcmd. What you really need >>>> to get a peek on the finalizer queue and print the histogram. The VM >>>> has the heap histogram implementation. Have you considered leveraging >>>> that? >>>> >>>> 5: 1012 40480 java.lang.ref.Finalizer >>>> >>>> You can find the registered Finalizer instances. The downside is that >>>> icmd -finalizerinfo stops the world. I think it?s not unreasonable for >>>> this diagnostic command to be expensive like -heap command. >>>> >>>> Mandy From yasuenag at gmail.com Thu May 28 14:41:37 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 28 May 2015 23:41:37 +0900 Subject: [OpenJDK 2D-Dev] JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: <55671164.6070309@oracle.com> References: <55668951.9070309@gmail.com> <55668B13.1000303@gmail.com> <5566DD67.9000301@oracle.com> <55671164.6070309@oracle.com> Message-ID: <55672921.3050508@gmail.com> Hi Alexander, I've uploaded new webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.02/ This patch excludes libsplashscreen, and fix Awt2dLibraries.gmk . Could you review it? Thanks, Yasumasa On 2015/05/28 22:00, Alexander Zvegintsev wrote: > I think that you can just exclude libsplashscreen part from your fix, > since it is covered by JDK-8080695. > > Thanks, > > Alexander. > > On 05/28/2015 02:52 PM, Yasumasa Suenaga wrote: >> >> Hi Alexander, >> >> Thank you for your comment. >> >> I agree to use DISABLED_WARNINGS_gcc. >> >> Should I make a patch afterJDK-8080695 committed? >> >> Thanks, >> >> Yasumasa >> >> 2015/05/28 18:19 "Alexander Zvegintsev" >: >> >> Hello Yasumasa, >> >> I prefer to avoid such pragma usage, as for me code is more readable without it. >> So we can add array-bounds to DISABLED_WARNINGS_gcc in make/lib/Awt2dLibraries.gmk for BUILD_LIBJAVAJPEG and BUILD_LIBMLIB_IMAGE. >> >> We can nullify row_pointers and image_data after setjmp call to remove warnings in splashscreen_png.c: >> diff -r 729dffc8afa0 src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> --- a/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> +++ b/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> @@ -77,6 +77,8 @@ SplashDecodePng(Splash * splash, png_rw_ >> #else >> if (setjmp(png_jmpbuf(png_ptr))) { >> #endif >> + row_pointers = NULL; >> + image_data = NULL; >> goto done; >> } >> (or declare them as volatile, but I prefer the first option). >> BTW, there is another issue about this splashscreen_png.c gcc build failure [1]. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8080695 >> >> Thanks, >> >> Alexander. >> >> >> On 05/28/2015 06:27 AM, Yasumasa Suenaga wrote: >> >> I've uploaded webrev: >> http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ >> >> Please review. >> >> >> Thanks, >> >> Yasumasa >> >> >> On 2015/05/28 12:19, Yasumasa Suenaga wrote: >> >> Hi all, >> >> I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, however, it was failed. >> I found several problems: >> >> >> System: >> Fedora release 22 (Twenty Two) x86_64 >> - gcc-5.1.1-1.fc22.x86_64 >> >> >> Problems: >> 1. Array bounds check in GCC >> - jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c >> - jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c >> - jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c >> - jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c >> >> It seems to be bug of GCC: >> Bug 59124: [4.8/4.9/5/6 Regression] Wrong warnings "array subscript is above array bounds" >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 >> >> I think implementations of these files have no problem. >> So I propose to ignore warning(s) of compiler through pragma option as >> workaround when we use GCC [1]. >> >> >> 2. Local variables might be clobbered >> - jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >> >> SplashDecodePng() calls setjmp(3). >> Some local variables initialize before setjmp() call, and use after it. >> Their initial values are only used at cleanup code (*done* label) and >> actual values are stored only after setjmp() call. >> So I think we can ignore this error through pragma option [1]. >> >> >> 3. Incorrect condition >> - jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c >> >> searchAllSourceNames() returns int value. However, branch condition in >> eventFilterRestricted_passesFilter() treats it as boolean value. >> >> >> I received a comment from Erik to use --disable-warnings-as-errors, >> however I could not avoid error in 3. >> >> >> Thanks, >> >> Yasumasa >> >> >> [1] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html >> >> > From yekaterina.kantserova at oracle.com Thu May 28 14:45:04 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Thu, 28 May 2015 16:45:04 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <5565C048.4050801@oracle.com> References: <5565C048.4050801@oracle.com> Message-ID: <556729F0.70703@oracle.com> Hi, due to https://bugs.openjdk.java.net/browse/JDK-8081381 I wasn't able to push this fix. The problem is LingeredApp.java contains JDK 9 feature java.lang.Process.getPid() but the test library is compiled with JDK 8 today. This issue is not trivial to solve so I suggest a temporary fix to test/lib/Makefile. webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.01 Thanks, Katja On 05/27/2015 03:02 PM, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8081037 > webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 > webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 > webrev hotspot: > http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 > > From the bug: > "The problem is most likely that SA will pause the target process > while it is running. In this case, the target process is the same as > the process that launched SA. That process is also handling the output > from SA over a pipe, but when that pipe fills up the process cannot > empty it and the SA process is blocked because it cannot write any > more output. Deadlock." > > The solutions is to start a separate target process. Dmitry Samersoff > has already created a test application for such cases so I've decided > to move it on the top level library instead of duplicating it. The > test application will reside under > test/lib/share/classes/jdk/test/lib/apps and the test under > test/lib-test/jdk/test/lib/apps. > > Thanks, > Katja From alexander.zvegintsev at oracle.com Thu May 28 15:48:57 2015 From: alexander.zvegintsev at oracle.com (Alexander Zvegintsev) Date: Thu, 28 May 2015 18:48:57 +0300 Subject: [OpenJDK 2D-Dev] JDK-8081295: Build failed with GCC 5.1.1 In-Reply-To: <55672921.3050508@gmail.com> References: <55668951.9070309@gmail.com> <55668B13.1000303@gmail.com> <5566DD67.9000301@oracle.com> <55671164.6070309@oracle.com> <55672921.3050508@gmail.com> Message-ID: <556738E9.1000304@oracle.com> Hi Yasumasa, the fix looks good to me. Thanks, Alexander. On 05/28/2015 05:41 PM, Yasumasa Suenaga wrote: > Hi Alexander, > > I've uploaded new webrev: > http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.02/ > > This patch excludes libsplashscreen, and fix Awt2dLibraries.gmk . > Could you review it? > > > Thanks, > > Yasumasa > > > > On 2015/05/28 22:00, Alexander Zvegintsev wrote: >> I think that you can just exclude libsplashscreen part from your fix, >> since it is covered by JDK-8080695. >> >> Thanks, >> >> Alexander. >> >> On 05/28/2015 02:52 PM, Yasumasa Suenaga wrote: >>> >>> Hi Alexander, >>> >>> Thank you for your comment. >>> >>> I agree to use DISABLED_WARNINGS_gcc. >>> >>> Should I make a patch afterJDK-8080695 >>> committed? >>> >>> Thanks, >>> >>> Yasumasa >>> >>> 2015/05/28 18:19 "Alexander Zvegintsev" >>> >> >: >>> >>> Hello Yasumasa, >>> >>> I prefer to avoid such pragma usage, as for me code is more >>> readable without it. >>> So we can add array-bounds to DISABLED_WARNINGS_gcc in >>> make/lib/Awt2dLibraries.gmk for BUILD_LIBJAVAJPEG and >>> BUILD_LIBMLIB_IMAGE. >>> >>> We can nullify row_pointers and image_data after setjmp call to >>> remove warnings in splashscreen_png.c: >>> diff -r 729dffc8afa0 >>> src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >>> --- >>> a/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >>> +++ >>> b/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >>> @@ -77,6 +77,8 @@ SplashDecodePng(Splash * splash, png_rw_ >>> #else >>> if (setjmp(png_jmpbuf(png_ptr))) { >>> #endif >>> + row_pointers = NULL; >>> + image_data = NULL; >>> goto done; >>> } >>> (or declare them as volatile, but I prefer the first option). >>> BTW, there is another issue about this splashscreen_png.c gcc >>> build failure [1]. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8080695 >>> >>> Thanks, >>> >>> Alexander. >>> >>> >>> On 05/28/2015 06:27 AM, Yasumasa Suenaga wrote: >>> >>> I've uploaded webrev: >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8081295/webrev.01/ >>> >>> >>> Please review. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2015/05/28 12:19, Yasumasa Suenaga wrote: >>> >>> Hi all, >>> >>> I tried to build jdk9/dev on Fedora22 with GCC 5.1.1, >>> however, it was failed. >>> I found several problems: >>> >>> >>> System: >>> Fedora release 22 (Twenty Two) x86_64 >>> - gcc-5.1.1-1.fc22.x86_64 >>> >>> >>> Problems: >>> 1. Array bounds check in GCC >>> - >>> jdk/src/java.desktop/share/native/libjavajpeg/jcmaster.c >>> - >>> jdk/src/java.desktop/share/native/libjavajpeg/jquant1.c >>> - >>> jdk/src/java.desktop/share/native/libmlib_image/mlib_ImageLookUp_64.c >>> - >>> jdk/src/java.desktop/share/native/libmlib_image/mlib_c_ImageLookUp_f.c >>> >>> It seems to be bug of GCC: >>> Bug 59124: [4.8/4.9/5/6 Regression] Wrong >>> warnings "array subscript is above array bounds" >>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59124 >>> >>> I think implementations of these files have no >>> problem. >>> So I propose to ignore warning(s) of compiler >>> through pragma option as >>> workaround when we use GCC [1]. >>> >>> >>> 2. Local variables might be clobbered >>> - >>> jdk/src/java.desktop/share/native/libsplashscreen/splashscreen_png.c >>> >>> SplashDecodePng() calls setjmp(3). >>> Some local variables initialize before setjmp() >>> call, and use after it. >>> Their initial values are only used at cleanup >>> code (*done* label) and >>> actual values are stored only after setjmp() call. >>> So I think we can ignore this error through >>> pragma option [1]. >>> >>> >>> 3. Incorrect condition >>> - >>> jdk/src/jdk.jdwp.agent/share/native/libjdwp/eventFilter.c >>> >>> searchAllSourceNames() returns int value. >>> However, branch condition in >>> eventFilterRestricted_passesFilter() treats it as >>> boolean value. >>> >>> >>> I received a comment from Erik to use >>> --disable-warnings-as-errors, >>> however I could not avoid error in 3. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> [1] >>> https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html >>> >>> >> From mandy.chung at oracle.com Thu May 28 18:06:00 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 28 May 2015 11:06:00 -0700 Subject: RFR(M,v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556727A2.9060209@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> Message-ID: <55675908.6030406@oracle.com> On 05/28/2015 07:35 AM, Peter Levart wrote: > Hi Mandy, > > On 05/27/2015 03:32 PM, Mandy Chung wrote: >> Taking it further - is it simpler to return String[] of all >> classnames including the duplicated ones and have the VM do the >> count? Are you concerned with the size of the String[]? > > Yes, the histogram is much smaller than the list of all instances. > There can be millions of instances waiting in finalizer queue, but > only a few distinct classes. Do you happen to know what libraries are the major contributors to these millions of finalizers? It has been strongly recommended to avoid finalizers (see Effective Java Item 7). I'm not surprised that existing code is still using finalizers while we should really encourage them to migrate away from it. > What could be done in Java to simplify things in native code but still > not format the output is to convert the array of Map.Entry(s) into an > Object[] array of alternating {String, int[], String, int[], .... } > > Would this simplify native code for the price of a little extra work > in Java? The sizes of old and new arrays are not big (# of distinct > classes of finalizable objects in the queue). I also prefer writing in Java and writing less native code (much simplified). It's an interface that we have to agree upon and keep it simple. Having the returned Object[] as alternate String and int[] is a reasonable compromise. ReferenceQueue.java - you can move @SuppressWarning from the method to just the field variable "rn" @SuppressWarnings("unchecked") Reference rn = r.next; Finalizer.java It's better to rename printFinalizationQueue as it inspects the finalizer reference queue (maybe getFinalizerHistogram?). Can you move this method to the end of this class and add the comment saying that this is invoked by VM for jcmd -finalizerinfo support and @return to describe the returned content. I also think you can remove @SuppressWarnings for this method. Mandy From staffan.larsen at oracle.com Thu May 28 18:35:13 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 28 May 2015 20:35:13 +0200 Subject: RFR(M, v9): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55675908.6030406@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> Message-ID: <2FB398BE-12CE-49AC-91F6-BC1C5B2F7F7A@oracle.com> > On 28 maj 2015, at 20:06, Mandy Chung wrote: > > > On 05/28/2015 07:35 AM, Peter Levart wrote: >> Hi Mandy, >> >> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>> Taking it further - is it simpler to return String[] of all classnames including the duplicated ones and have the VM do the count? Are you concerned with the size of the String[]? >> >> Yes, the histogram is much smaller than the list of all instances. There can be millions of instances waiting in finalizer queue, but only a few distinct classes. > > Do you happen to know what libraries are the major contributors to these millions of finalizers? > > It has been strongly recommended to avoid finalizers (see Effective Java Item 7). I'm not surprised that existing code is still using finalizers while we should really encourage them to migrate away from it. Having ways to introspect the finalizer queue is one way to make people aware that they have a problem :-) > >> What could be done in Java to simplify things in native code but still not format the output is to convert the array of Map.Entry(s) into an Object[] array of alternating {String, int[], String, int[], .... } >> >> Would this simplify native code for the price of a little extra work in Java? The sizes of old and new arrays are not big (# of distinct classes of finalizable objects in the queue). > > I also prefer writing in Java and writing less native code (much simplified). It's an interface that we have to agree upon and keep it simple. Having the returned Object[] as alternate String and int[] is a reasonable compromise. > > ReferenceQueue.java - you can move @SuppressWarning from the method to just the field variable "rn" > @SuppressWarnings("unchecked") > Reference rn = r.next; > > Finalizer.java > It's better to rename printFinalizationQueue as it inspects the finalizer reference queue (maybe getFinalizerHistogram?). Can you move this method to the end of this class and add the comment saying that this is invoked by VM for jcmd -finalizerinfo support and @return to describe the returned content. I also think you can remove @SuppressWarnings for this method. > > Mandy From staffan.larsen at oracle.com Fri May 29 08:53:49 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 29 May 2015 10:53:49 +0200 Subject: [urgent] RFR: 8081470 com/sun/jdi tests are failing with "Error. failed to clean up files after test" with jtreg 4.1 b12 Message-ID: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> Can I have a fast review of the following change that is currently blocking hotspot pushes in jprt. jtreg 4.1b12 adds stricter checking of @library tags. Some com/sun/jdi tests have @library clauses that are not needed. I do not intend to wait 24 hours before pushing this... webrev: http://cr.openjdk.java.net/~sla/8081470/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8081470 Thanks, /Staffan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikael.gerdin at oracle.com Fri May 29 08:54:47 2015 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Fri, 29 May 2015 10:54:47 +0200 Subject: [urgent] RFR: 8081470 com/sun/jdi tests are failing with "Error. failed to clean up files after test" with jtreg 4.1 b12 In-Reply-To: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> References: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> Message-ID: <55682957.3020405@oracle.com> Hi Staffan, On 2015-05-29 10:53, Staffan Larsen wrote: > Can I have a fast review of the following change that is currently blocking hotspot pushes in jprt. jtreg 4.1b12 adds stricter checking of @library tags. Some com/sun/jdi tests have @library clauses that are not needed. > > I do not intend to wait 24 hours before pushing this... > > webrev: http://cr.openjdk.java.net/~sla/8081470/webrev.00/ Looks good, Reviewed. /Mikael > bug: https://bugs.openjdk.java.net/browse/JDK-8081470 > > Thanks, > /Staffan > From bengt.rutisson at oracle.com Fri May 29 08:54:48 2015 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Fri, 29 May 2015 10:54:48 +0200 Subject: [urgent] RFR: 8081470 com/sun/jdi tests are failing with "Error. failed to clean up files after test" with jtreg 4.1 b12 In-Reply-To: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> References: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> Message-ID: <55682958.5060703@oracle.com> On 2015-05-29 10:53, Staffan Larsen wrote: > Can I have a fast review of the following change that is currently blocking hotspot pushes in jprt. jtreg 4.1b12 adds stricter checking of @library tags. Some com/sun/jdi tests have @library clauses that are not needed. > > I do not intend to wait 24 hours before pushing this... > > webrev: http://cr.openjdk.java.net/~sla/8081470/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8081470 Looks good to me too. Bengt > > Thanks, > /Staffan From igor.ignatyev at oracle.com Fri May 29 08:57:31 2015 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 29 May 2015 11:57:31 +0300 Subject: [urgent] RFR: 8081470 com/sun/jdi tests are failing with "Error. failed to clean up files after test" with jtreg 4.1 b12 In-Reply-To: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> References: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> Message-ID: <556829FB.2070402@oracle.com> Looks good to me. Igor On 05/29/2015 11:53 AM, Staffan Larsen wrote: > Can I have a fast review of the following change that is currently blocking hotspot pushes in jprt. jtreg 4.1b12 adds stricter checking of @library tags. Some com/sun/jdi tests have @library clauses that are not needed. > > I do not intend to wait 24 hours before pushing this... > > webrev: http://cr.openjdk.java.net/~sla/8081470/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8081470 > > Thanks, > /Staffan > From staffan.larsen at oracle.com Fri May 29 09:07:06 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 29 May 2015 11:07:06 +0200 Subject: [urgent] RFR: 8081470 com/sun/jdi tests are failing with "Error. failed to clean up files after test" with jtreg 4.1 b12 In-Reply-To: <556829FB.2070402@oracle.com> References: <11FEF918-2B72-49D6-926D-A6CD90E4019D@oracle.com> <556829FB.2070402@oracle.com> Message-ID: Thanks all. Fix is in the queue to jdk9/hs/jdk and will then be pulled into the group repos. > On 29 maj 2015, at 10:57, Igor Ignatyev wrote: > > Looks good to me. > > Igor > > On 05/29/2015 11:53 AM, Staffan Larsen wrote: >> Can I have a fast review of the following change that is currently blocking hotspot pushes in jprt. jtreg 4.1b12 adds stricter checking of @library tags. Some com/sun/jdi tests have @library clauses that are not needed. >> >> I do not intend to wait 24 hours before pushing this... >> >> webrev: http://cr.openjdk.java.net/~sla/8081470/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8081470 >> >> Thanks, >> /Staffan >> From yasuenag at gmail.com Fri May 29 09:38:39 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Fri, 29 May 2015 18:38:39 +0900 Subject: RFR: JDK-8081475: SystemTap does not work when JDK is compiled with GCC 5 Message-ID: <5568339F.2070004@gmail.com> Hi all, I tried to build jdk9/hs-rt on Fedora 22 with GCC 5.1.1 . However, libjvm.so did not have .note.stapsdt section. When I ran make command with LOG=info, I found a message as below: --------------- **NOTICE** Dtrace support disabled: gcc version is too old --------------- hotspot/make/linux/makefiles/dtrace.make checks C compiler version. However, its condition cannot check 5.x or later. So I fix the condition. Could you review it? http://cr.openjdk.java.net/~ysuenaga/JDK-8081475/webrev.00/ This patch works fine on my environment. This issue is occurred on 8u-dev. So I also want to fix it for 8u-dev. I'm jdk9 committer, but I'm not employee at Oracle. So I need a Sponsor. Thanks, Yasumasa From david.holmes at oracle.com Fri May 29 10:04:46 2015 From: david.holmes at oracle.com (David Holmes) Date: Fri, 29 May 2015 20:04:46 +1000 Subject: RFR: JDK-8081475: SystemTap does not work when JDK is compiled with GCC 5 In-Reply-To: <5568339F.2070004@gmail.com> References: <5568339F.2070004@gmail.com> Message-ID: <556839BE.902@oracle.com> Hi Yasumasa, Fix looks good to me. I think someone had tried to handle this but simply got it wrong as the check would pass for 4.4+ and 5+.4+ :( I noone else jumps in on a local Friday I can sponsor this on my Monday. Cheers, David On 29/05/2015 7:38 PM, Yasumasa Suenaga wrote: > Hi all, > > I tried to build jdk9/hs-rt on Fedora 22 with GCC 5.1.1 . > However, libjvm.so did not have .note.stapsdt section. > > When I ran make command with LOG=info, I found a message as below: > --------------- > **NOTICE** Dtrace support disabled: gcc version is too old > --------------- > > hotspot/make/linux/makefiles/dtrace.make checks C compiler version. > However, its condition cannot check 5.x or later. > > So I fix the condition. Could you review it? > http://cr.openjdk.java.net/~ysuenaga/JDK-8081475/webrev.00/ > > This patch works fine on my environment. > This issue is occurred on 8u-dev. So I also want to fix it for 8u-dev. > > > I'm jdk9 committer, but I'm not employee at Oracle. > So I need a Sponsor. > > > Thanks, > > Yasumasa > From pujarimahesh_kumar at yahoo.com Fri May 29 10:09:04 2015 From: pujarimahesh_kumar at yahoo.com (Mahesh Pujari) Date: Fri, 29 May 2015 10:09:04 +0000 (UTC) Subject: Issues with dtrace enabled in OpenJdk 9 In-Reply-To: <55647835.2050401@oracle.com> References: <55647835.2050401@oracle.com> Message-ID: <1654770900.500152.1432894144653.JavaMail.yahoo@mail.yahoo.com> Hi, ?I am able to compile OpenJDK9 with dtrace on Linux container with Ubuntu distribution (but still having compilation issues on host machine). Now as I went a step a head, I tried to test "method__entry" marker. Below is the script probe process("JDK_PATH/lib/amd64/server/libjvm.so").mark("thread__start") { ??? printf("Marker thread__start [%s] %d %d %d %d\n", user_string($arg1),$arg2,$arg3,$arg4,$arg5); } Note: Need to replace JDK_PATH with actual path. above script did run and I did got output as below Marker thread__start [Reference Handler] 17 2 21151 1 .... .... But I am not able to use hotspot provider i.e. below script does not run stap -e 'probe hotspot.thread_* { printf("%s: %s%s\n", ctime(gettimeofday_s()), name, thread_name) }' \ -c 'java Hello' And I end up in below erorr semantic error: while resolving probe point: identifier 'hotspot' at :1:7 ??????? source: probe hotspot.thread_* { printf("%s: %s%s\n", ctime(gettimeofday_s()), name, thread_name) } ????????????????????? ^ semantic error: probe point mismatch (similar: oneshot, tcp, init, scsi, stap): identifier 'hotspot' at :1:7 ??????? source: probe hotspot.thread_* { printf("%s: %s%s\n", ctime(gettimeofday_s()), name, thread_name) } ????????????????????? ^ Can some one point me to where I can get hotspot provider or am I missing something. thanks and regards, Mahesh Pujari On Tuesday, May 26, 2015 7:10 PM, Daniel D. Daugherty wrote: Adding the Serviceability alias. Dan On 5/25/15 9:52 AM, Mahesh Pujari wrote: > Hi, > >? I am trying to build OpenJDK 9 with dtrace enabled on my Ubuntu machine (Linux 3.13.0-45-generic #74-Ubuntu), I have asked this question on build-dev at openjdk.java.net (http://mail.openjdk.java.net/pipermail/build-dev/2015-May/014969.html) and I was directed to this mailing list (including distro-pkg mailing list, but had no luck there, so trying out here). > >? If SDT headers are found then dtrace is enabled by default, this is what I understood. Now when I build, I end-up with below errors > ... > ... > vmThread.o: In function `VMOperationQueue::add(VM_Operation*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:156: undefined reference to `__dtrace_hotspot___vmops__request' > vmThread.o: In function `VMThread::evaluate_operation(VM_Operation*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:354: undefined reference to `__dtrace_hotspot___vmops__begin' > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/runtime/vmThread.cpp:374: undefined reference to `__dtrace_hotspot___vmops__end' > ... > .. > classLoadingService.o: In function `ClassLoadingService::notify_class_unloaded(InstanceKlass*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/services/classLoadingService.cpp:119: undefined reference to `__dtrace_hotspot___class__unloaded' > classLoadingService.o: In function `ClassLoadingService::notify_class_loaded(InstanceKlass*, bool)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/services/classLoadingService.cpp:144: undefined reference to `__dtrace_hotspot___class__loaded' > compileBroker.o: In function `CompileBroker::invoke_compiler_on_method(CompileTask*)': > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/compiler/compileBroker.cpp:1927: undefined reference to `__dtrace_hotspot___method__compile__begin' > /mnt/ubuntu/dev/jdk9/hotspot/src/share/vm/compiler/compileBroker.cpp:2028: undefined reference to `__dtrace_hotspot___method__compile__end' > ... > ... > >? Compilation is success but during linkage things fail. Can someone help me with this, any directions to what I am missing. > > thanks and regards, > Mahesh Pujari > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Sun May 31 11:43:09 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Sun, 31 May 2015 14:43:09 +0300 Subject: RFR(M, v10): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <55675908.6030406@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> Message-ID: <556AF3CD.50900@oracle.com> Everyone, Please take a look at new version of the fix. http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ Changes (to previous version) are in Finalizer.java and diagnosticCommand.cpp This version copy data from Map.Entry<> to array of FinalizerHistogramEntry instances then, VM prints content of this array. -Dmitry On 2015-05-28 21:06, Mandy Chung wrote: > > On 05/28/2015 07:35 AM, Peter Levart wrote: >> Hi Mandy, >> >> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>> Taking it further - is it simpler to return String[] of all >>> classnames including the duplicated ones and have the VM do the >>> count? Are you concerned with the size of the String[]? >> >> Yes, the histogram is much smaller than the list of all instances. >> There can be millions of instances waiting in finalizer queue, but >> only a few distinct classes. > > Do you happen to know what libraries are the major contributors to these > millions of finalizers? > > It has been strongly recommended to avoid finalizers (see Effective Java > Item 7). I'm not surprised that existing code is still using > finalizers while we should really encourage them to migrate away from it. > >> What could be done in Java to simplify things in native code but still >> not format the output is to convert the array of Map.Entry(s) into an >> Object[] array of alternating {String, int[], String, int[], .... } >> >> Would this simplify native code for the price of a little extra work >> in Java? The sizes of old and new arrays are not big (# of distinct >> classes of finalizable objects in the queue). > > I also prefer writing in Java and writing less native code (much > simplified). It's an interface that we have to agree upon and keep it > simple. Having the returned Object[] as alternate String and int[] is a > reasonable compromise. > > ReferenceQueue.java - you can move @SuppressWarning from the method to > just the field variable "rn" > @SuppressWarnings("unchecked") > Reference rn = r.next; > > Finalizer.java > It's better to rename printFinalizationQueue as it inspects the > finalizer reference queue (maybe getFinalizerHistogram?). Can you move > this method to the end of this class and add the comment saying that > this is invoked by VM for jcmd -finalizerinfo support and @return to > describe the returned content. I also think you can remove > @SuppressWarnings for this method. > > Mandy -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources.