From martijnverburg at gmail.com Tue Mar 3 10:32:47 2015 From: martijnverburg at gmail.com (Martijn Verburg) Date: Tue, 3 Mar 2015 10:32:47 +0000 Subject: JCov and coverage numbers on OpenJDK Message-ID: Hi All, As some of you know we've been running some experiments in the Cloudbees incubator to see if we can get accurate code coverage numbers using JCov on the jdk9 forest in particular. John Oliver has gone back and reviewed the process and the numbers and we *think* we've gone about it the right way. Before we even think about taking the next step to start producing these numbers regularly in the incubator, we need to make sure that we've used JCov correctly and that the numbers are not misleading. It would be great to have a technical call with John Oliver, Mani, someone from Rory's team (the person who does the internal OpenJDK numbers?) and probably Jonathan Gibbons. Does next Tuesday suit folks? It all depends on timezones (John Oliver, Mani and myself are GMT) Cheers, Martijn -------------- next part -------------- An HTML attachment was scrubbed... URL: From rory.odonnell at oracle.com Tue Mar 3 10:45:46 2015 From: rory.odonnell at oracle.com (Rory O'Donnell) Date: Tue, 03 Mar 2015 10:45:46 +0000 Subject: JCov and coverage numbers on OpenJDK In-Reply-To: References: Message-ID: <54F590DA.7040109@oracle.com> Hi Martijn, Can you send out a link to the lastest results ? Tuesday, will work for me, checking if Balchandra is available. Rgds,Rory On 03/03/2015 10:32, Martijn Verburg wrote: > Hi All, > > As some of you know we've been running some experiments in the Cloudbees > incubator to see if we can get accurate code coverage numbers using JCov on > the jdk9 forest in particular. > > John Oliver has gone back and reviewed the process and the numbers and we > *think* we've gone about it the right way. > > Before we even think about taking the next step to start producing these > numbers regularly in the incubator, we need to make sure that we've used > JCov correctly and that the numbers are not misleading. > > It would be great to have a technical call with John Oliver, Mani, someone > from Rory's team (the person who does the internal OpenJDK numbers?) and > probably Jonathan Gibbons. > > Does next Tuesday suit folks? It all depends on timezones (John Oliver, > Mani and myself are GMT) > > Cheers, > Martijn -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland From martijnverburg at gmail.com Tue Mar 3 11:13:42 2015 From: martijnverburg at gmail.com (Martijn Verburg) Date: Tue, 3 Mar 2015 11:13:42 +0000 Subject: JCov and coverage numbers on OpenJDK In-Reply-To: <54F590DA.7040109@oracle.com> References: <54F590DA.7040109@oracle.com> Message-ID: https://adopt-openjdk.ci.cloudbees.com/view/OpenJDK/job/openjdk-1.9-linux-x86_64/ws/testoutput/jdk_core/JTreport/jcov/index.html Cheers, Martijn On 3 March 2015 at 10:45, Rory O'Donnell wrote: > Hi Martijn, > > Can you send out a link to the lastest results ? > > Tuesday, will work for me, checking if Balchandra is available. > > Rgds,Rory > > On 03/03/2015 10:32, Martijn Verburg wrote: > >> Hi All, >> >> As some of you know we've been running some experiments in the Cloudbees >> incubator to see if we can get accurate code coverage numbers using JCov >> on >> the jdk9 forest in particular. >> >> John Oliver has gone back and reviewed the process and the numbers and we >> *think* we've gone about it the right way. >> >> Before we even think about taking the next step to start producing these >> numbers regularly in the incubator, we need to make sure that we've used >> JCov correctly and that the numbers are not misleading. >> >> It would be great to have a technical call with John Oliver, Mani, someone >> from Rory's team (the person who does the internal OpenJDK numbers?) and >> probably Jonathan Gibbons. >> >> Does next Tuesday suit folks? It all depends on timezones (John Oliver, >> Mani and myself are GMT) >> >> Cheers, >> Martijn >> > > -- > Rgds,Rory O'Donnell > Quality Engineering Manager > Oracle EMEA , Dublin, Ireland > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balchandra.vaidya at oracle.com Tue Mar 3 12:00:36 2015 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Tue, 03 Mar 2015 17:30:36 +0530 Subject: JDK 9 early access b52 test results now available Message-ID: <54F5A264.7040909@oracle.com> JDK 9 ea b52 test results are now available at : http://www.java.net/download/openjdk/testresults/9/testresults.html The jdk test results contain 6 differences from the b51 test results. No new testcase failures found. 0: /home/jtest/merge9/b51/jdk/JTwork pass: 4,987; fail: 11; not run: 1,723 1: /home/jtest/merge9/b52/jdk/JTwork pass: 4,993; fail: 11; not run: 1,733 0 1 Test --- pass com/sun/nio/sctp/SctpMultiChannel/SendFailed.java --- pass java/lang/invoke/lambda/LambdaStackTrace.java --- pass java/lang/management/MemoryMXBean/LowMemoryTest.java --- pass java/security/ProtectionDomain/PreserveCombinerTest.java --- pass javax/imageio/plugins/jpeg/MagentaEXIFTest.java --- pass javax/imageio/plugins/shared/WriteAfterAbort.java 6 differences The hotspot test results contain 9 differences from the b51 test results. There are 2 testcase failures, those failures are under investigation. 0: /home/jtest/merge9/b51/hotspot/JTwork pass: 741; fail: 37; error: 1; not run: 33 1: /home/jtest/merge9/b52/hotspot/JTwork pass: 738; fail: 38; error: 1; not run: 37 0 1 Test --- pass compiler/arraycopy/TestArrayCopyNoInitDeopt.java pass fail compiler/c2/6857159/Test6857159.java pass --- compiler/codecache/stress/OverloadCompileQueueTest.java pass fail compiler/whitebox/DeoptimizeFramesTest.java --- pass runtime/ErrorHandling/SecondaryErrorTest.java pass --- serviceability/dcmd/framework/HelpTest.java pass --- serviceability/dcmd/framework/InvalidCommandTest.java pass --- serviceability/dcmd/framework/VMVersionTest.java fail pass serviceability/dcmd/gc/RunGCTest.java 9 differences The langtools test results contain 2 differences from the b51 test results. No new testcase failures found. 0: /home/jtest/merge9/b51/langtools/JTwork pass: 3,201; not run: 14 1: /home/jtest/merge9/b52/langtools/JTwork pass: 3,203; not run: 14 0 1 Test --- pass tools/javac/file/BootClassPathPrepend.java --- pass tools/javac/file/ExplodedImage.java 2 differences The nashorn test result is available at below link. The test failures are under investigation. http://download.java.net/openjdk/testresults/9/archives/b52/emailable-report.html Thanks Balchandra -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin.john.evans at gmail.com Wed Mar 4 13:25:02 2015 From: benjamin.john.evans at gmail.com (Ben Evans) Date: Wed, 4 Mar 2015 13:25:02 +0000 Subject: JCov and coverage numbers on OpenJDK In-Reply-To: References: Message-ID: Depending on timings, I can probably be free on Tuesday (I'm on GMT too). Thanks, Ben On Tue, Mar 3, 2015 at 10:32 AM, Martijn Verburg wrote: > Hi All, > > As some of you know we've been running some experiments in the Cloudbees > incubator to see if we can get accurate code coverage numbers using JCov on > the jdk9 forest in particular. > > John Oliver has gone back and reviewed the process and the numbers and we > *think* we've gone about it the right way. > > Before we even think about taking the next step to start producing these > numbers regularly in the incubator, we need to make sure that we've used > JCov correctly and that the numbers are not misleading. > > It would be great to have a technical call with John Oliver, Mani, someone > from Rory's team (the person who does the internal OpenJDK numbers?) and > probably Jonathan Gibbons. > > Does next Tuesday suit folks? It all depends on timezones (John Oliver, > Mani and myself are GMT) > > Cheers, > Martijn From balchandra.vaidya at oracle.com Fri Mar 6 17:05:52 2015 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Fri, 06 Mar 2015 22:35:52 +0530 Subject: JDK 8u60 ea b05 test results now available Message-ID: <54F9DE70.9020009@oracle.com> JDK 8u60 ea b05 test results are now available at http://www.java.net/download/openjdk/testresults/8/testresults.html The nashorn test result is available at http://download.java.net/openjdk/testresults/8/archives8/jdk8u60-b05/emailable-report.html Thanks Balchandra From balchandra.vaidya at oracle.com Tue Mar 24 04:59:54 2015 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Tue, 24 Mar 2015 10:29:54 +0530 Subject: JDK 8u60 ea b07 test results now available Message-ID: <5510EF4A.4020306@oracle.com> JDK 8u60 ea b07 test results are now available at http://www.java.net/download/openjdk/testresults/8/testresults.html The jdk test results contain 1 difference from the b06 test results. The results contain 0 testcase failures. 0: /home/jtest/merge8/jdk8u60-b06/jdk/JTwork pass: 4,783; fail: 14; error: 1; not run: 1,008 1: /home/jtest/merge8/jdk8u60-b07/jdk/JTwork pass: 4,784; fail: 14; not run: 1,008 0 1 Test error pass sun/tools/jstatd/TestJstatdExternalRegistry.java 1 differences The hotspot test results contain 3 differences from the b06 test results. There are 2 testcase failures, those failures are under investigation. 0: /home/jtest/merge8/jdk8u60-b06/hotspot/JTwork pass: 623; fail: 33; error: 2; not run: 16 1: /home/jtest/merge8/jdk8u60-b07/hotspot/JTwork pass: 623; fail: 35; error: 1; not run: 16 0 1 Test --- pass compiler/escapeAnalysis/TestEscapeThroughInvoke.java pass fail gc/g1/TestShrinkAuxiliaryData25.java error fail serviceability/sa/jmap-hashcode/Test8028623.java 3 differences The langtools test results contain 0 differences from the b06 test results. The nashorn test result is available at http://download.java.net/openjdk/testresults/8/archives8/jdk8u60-b07/emailable-report.html Thanks Balchandra -------------- next part -------------- An HTML attachment was scrubbed... URL: From martijnverburg at gmail.com Sat Mar 28 15:03:17 2015 From: martijnverburg at gmail.com (Martijn Verburg) Date: Sat, 28 Mar 2015 15:03:17 +0000 Subject: JCov and coverage numbers on OpenJDK In-Reply-To: References: Message-ID: Hi all, Oracle's internal QA team were able to confirm that the numbers that the Adoption Group were producing are very close (not a statistical significant difference) to their numbers. With validation that the numbers are accurate, it would be good to start publishing these for the purpose of guiding OpenJDK developers to areas that need more test coverage! What steps would people like to take next? I think the right home for these reports is in the quality group. They could host the code coverage reports and pro-actively release test coverage numbers alongside the # tests passing/failing (as they do currently). @Rory, is that feasible in the short term? I understand that there's potentially some technical work to do and other hoops to jump through. If it's not possible in the short term then perhaps the quality group could reference the reports that the Adoption Group are hosting (with a caveat) in the short term until that work can be completed. Special thanks to John Oliver and Alexandre Iline for digging into this! Cheers, Martijn On 4 March 2015 at 13:25, Ben Evans wrote: > Depending on timings, I can probably be free on Tuesday (I'm on GMT too). > > Thanks, > > Ben > > On Tue, Mar 3, 2015 at 10:32 AM, Martijn Verburg > wrote: > > Hi All, > > > > As some of you know we've been running some experiments in the Cloudbees > > incubator to see if we can get accurate code coverage numbers using JCov > on > > the jdk9 forest in particular. > > > > John Oliver has gone back and reviewed the process and the numbers and we > > *think* we've gone about it the right way. > > > > Before we even think about taking the next step to start producing these > > numbers regularly in the incubator, we need to make sure that we've used > > JCov correctly and that the numbers are not misleading. > > > > It would be great to have a technical call with John Oliver, Mani, > someone > > from Rory's team (the person who does the internal OpenJDK numbers?) and > > probably Jonathan Gibbons. > > > > Does next Tuesday suit folks? It all depends on timezones (John Oliver, > > Mani and myself are GMT) > > > > Cheers, > > Martijn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rory.odonnell at oracle.com Mon Mar 30 09:12:06 2015 From: rory.odonnell at oracle.com (Rory O'Donnell) Date: Mon, 30 Mar 2015 10:12:06 +0100 Subject: JCov and coverage numbers on OpenJDK In-Reply-To: References: Message-ID: <55191366.3060309@oracle.com> On 28/03/2015 15:03, Martijn Verburg wrote: > Hi all, > > Oracle's internal QA team were able to confirm that the numbers that > the Adoption Group were producing are very close (not a statistical > significant difference) to their numbers. With validation that the > numbers are accurate, it would be good to start publishing these for > the purpose of guiding OpenJDK developers to areas that need more test > coverage! > > What steps would people like to take next? > > I think the right home for these reports is in the quality group. > They could host the code coverage reports and pro-actively release > test coverage numbers alongside the # tests passing/failing (as they > do currently). > > @Rory, is that feasible in the short term? I understand that there's > potentially some technical work to do and other hoops to jump through. > If it's not possible in the short term then perhaps the quality group > could reference the reports that the Adoption Group are hosting (with > a caveat) in the short term until that work can be completed. Hi Martijn, I think it makes sense for the person generating the reports to host and post a pointer to the mailing list. I mentioned before comparing our internal numbers with yours is like comparing apples with pears. Rgds,Rory > > Special thanks to John Oliver and Alexandre Iline for digging into this! > > > Cheers, > Martijn > > On 4 March 2015 at 13:25, Ben Evans > wrote: > > Depending on timings, I can probably be free on Tuesday (I'm on > GMT too). > > Thanks, > > Ben > > On Tue, Mar 3, 2015 at 10:32 AM, Martijn Verburg > > wrote: > > Hi All, > > > > As some of you know we've been running some experiments in the > Cloudbees > > incubator to see if we can get accurate code coverage numbers > using JCov on > > the jdk9 forest in particular. > > > > John Oliver has gone back and reviewed the process and the > numbers and we > > *think* we've gone about it the right way. > > > > Before we even think about taking the next step to start > producing these > > numbers regularly in the incubator, we need to make sure that > we've used > > JCov correctly and that the numbers are not misleading. > > > > It would be great to have a technical call with John Oliver, > Mani, someone > > from Rory's team (the person who does the internal OpenJDK > numbers?) and > > probably Jonathan Gibbons. > > > > Does next Tuesday suit folks? It all depends on timezones (John > Oliver, > > Mani and myself are GMT) > > > > Cheers, > > Martijn > > -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: From rory.odonnell at oracle.com Tue Mar 31 09:41:05 2015 From: rory.odonnell at oracle.com (Rory O'Donnell) Date: Tue, 31 Mar 2015 10:41:05 +0100 Subject: JDK 9 b55 and JDK 8u60 b08 test results now available Message-ID: <551A6BB1.7030503@oracle.com> *JDK 9 ea b55 test results are now available at * http://www.java.net/download/openjdk/testresults/9/testresults.html The jdk test results contain 7 differences from the b54 test results. No new testcase failures found. 0: /home/jtest/merge9/b54/jdk/JTwork pass: 5,009; fail: 11; error: 1; not run: 1,740 1: /home/jtest/merge9/b55/jdk/JTwork pass: 5,016; fail: 11; not run: 1,740 0 1 Test --- pass com/sun/crypto/provider/Cipher/AES/TestAESCiphers/TestAESWithDefaultProvider.java --- pass com/sun/crypto/provider/Cipher/AES/TestAESCiphers/TestAESWithProviderChange.java --- pass com/sun/crypto/provider/Cipher/AES/TestAESCiphers/TestAESWithRemoveAddProvider.java --- pass java/lang/instrument/ManyMethodsBenchmarkAgent.java --- pass java/util/prefs/CodePointZeroPrefsTest.java error pass java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java --- pass sun/text/resources/Format/Bug8074791.java 7 differences The hotspot test results contain 9 differences from the b54 test results. No new testcase failures found. 0: /home/jtest/merge9/b54/hotspot/JTwork pass: 740; fail: 38; error: 1; not run: 38 1: /home/jtest/merge9/b55/hotspot/JTwork pass: 744; fail: 38; error: 2; not run: 40 0 1 Test --- pass compiler/arraycopy/TestArrayCloneBadAssert.java --- pass compiler/arraycopy/TestArrayCopyAsLoadsStores.java --- pass compiler/c2/FloatingPointFoldingTest.java --- pass compiler/escapeAnalysis/TestEscapeThroughInvoke.java --- pass compiler/loopopts/TestCastIINoLoopLimitCheck.java --- pass compiler/print/TestProfileReturnTypePrinting.java pass --- compiler/tiered/LevelTransitionTest.java pass --- gc/TestSoftReferencesBehaviorOnOOME.java --- error runtime/Thread/Fibonacci.java 9 differences The langtools test results contain 2 differences from the b54 test results. No new testcase failures found. 0: /home/jtest/merge9/b54/langtools/JTwork pass: 3,221; not run: 14 1: /home/jtest/merge9/b55/langtools/JTwork pass: 3,221; not run: 14 0 1 Test pass --- com/sun/javadoc/testNonFrameWarning/TestNonFrameWarning.java --- pass tools/javac/doclint/IncludePackagesTest.java 2 differences The nashorn test result is available at http://download.java.net/openjdk/testresults/9/archives/b55/emailable-report.html *JDK 8u60 ea b08 test results are now available at * http://www.java.net/download/openjdk/testresults/8/testresults.html The jdk test results contain 6 differences from the b07 test results.There is 1 testcase failure, this failure is under investigation. 0: /home/jtest/merge8/jdk8u60-b07/jdk/JTwork pass: 4,784; fail: 14; not run: 1,008 1: /home/jtest/merge8/jdk8u60-b08/jdk/JTwork pass: 4,786; fail: 15; error: 1; not run: 1,009 0 1 Test --- pass java/lang/instrument/ManyMethodsBenchmarkAgent.java --- pass java/lang/invoke/lambda/LambdaStackTrace.java pass fail java/lang/management/MemoryMXBean/LowMemoryTest2.sh --- pass sun/security/ssl/com/sun/net/ssl/internal/ssl/ClientHandshaker/LengthCheckTest.java --- pass sun/text/resources/Format/Bug8074791.java pass error sun/tools/jstatd/TestJstatdExternalRegistry.java 6 differences The hotspot test results contain 2 differences from the b07 test results. No new testcase failures found. 0: /home/jtest/merge8/jdk8u60-b07/hotspot/JTwork pass: 623; fail: 35; error: 1; not run: 16 1: /home/jtest/merge8/jdk8u60-b08/hotspot/JTwork pass: 625; fail: 34; error: 1; not run: 16 0 1 Test --- pass compiler/loopopts/ConstFPVectorization.java fail pass gc/g1/TestShrinkAuxiliaryData25.java 2 differences The langtools test results contain 0 differences from the b07 test results. The nashorn test result is available at http://download.java.net/openjdk/testresults/8/archives8/jdk8u60-b08/emailable-report.html -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: