RFR 8215729: Enhance makefiles to allow collecting code coverage with JCov

Wed Jan 9 19:46:31 UTC 2019

David,

At least these tests appear to be failing because of extra lines in the error output (showing test names with a line or two from jtr file):

java/io/File/DeleteOnExitNPE.java
	java.lang.RuntimeException: Failed: No output should have been received from the process
java/lang/ProcessBuilder/SkipTest.java
	java.lang.AssertionError: header char wrong, expected: 65, actual: 80
java/lang/Runtime/exec/ArgWithSpaceAndFinalBackslash.java
	java.lang.Exception: Test failed: Got "Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
	foo bar\\baz\\", expected "foo bar\\baz\\"
java/lang/Runtime/exec/ExitValue.java
	java.lang.Exception: Unexpected stderr: Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
java/lang/Thread/UncaughtExceptions.sh
	FAIL: Expected "Exception in thread "Thread-0".*Seppuku", got Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
java/net/InetAddress/ptr/Lookup.java
	IPv4 lookup results: [192.0.43.7:icann.orgPicked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
	]
java/net/URLConnection/ContentHandlers/ContentHandlersTest.java
	java.lang.RuntimeException: Expected a different kind of failure: Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
java/net/spi/URLStreamHandlerProvider/Basic.java
	java.lang.RuntimeException: Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java
	 stderr: [Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
	Bad response or premature EOF, bytes read:
	java.lang.RuntimeException: Bad response or premature EOF from service
java/security/AccessController/DoPrivAccompliceTest.java
	java.lang.RuntimeException: stderr was not empty
java/util/logging/LoggingDeadlock2.java
	Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
	 not equal to 
java/util/zip/EntryCount64k.java
	java.lang.RuntimeException: '\A\Z' missing from stderr
javax/naming/spi/providers/InitialContextTest.java
	java.lang.RuntimeException: Expected a different kind of failure: Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
	Picked up _JAVA_OPTIONS: -Xms64m -Xmx4g
sun/rmi/runtime/Log/6409194/NoConsoleOutput.java
	java.lang.Error: TEST FAILED: unexpected subprocess output
sun/tools/jrunscript/jrunscript-eTest.sh
sun/tools/jrunscript/jrunscript-fTest.sh
sun/tools/jrunscript/jrunscriptTest.sh
	Output of jrunscript -e differ from expected output. Failed.
tools/jar/LeadingGarbage.java
	test LeadingGarbage.test_canListWithLeadingGarbage(): failure
	java.lang.RuntimeException: '\\A\\Z' missing from stderr 
tools/jar/modularJar/Basic.java
	java.lang.AssertionError: Unknown value Picked up JAVA_TOOL_OPTIONS: -Xms64m -Xmx4g
tools/jar/multiRelease/Basic.java
	java.lang.RuntimeException: stderr was not empty
tools/jjs/jjs-fileTest.sh
	Output of jjs file.js differ from expected output. Failed.	

Other failures are:

java/io/CharArrayReader/OverflowInRead.java
java/lang/ProcessBuilder/InheritIO/InheritIO.sh
java/lang/StackTraceElement/ModuleFrames.java
java/lang/StackTraceElement/SerialTest.java
java/lang/StackWalker/VerifyStackTrace.java
java/lang/invoke/lambda/LogGeneratedClassesTest.java
java/net/MulticastSocket/PromiscuousIPv6.java
java/net/URL/JarHandlerPkgPrefix/JarHandlerPkgPrefix.java
java/nio/Buffer/LimitDirectMemory.java
java/nio/channels/DatagramChannel/PromiscuousIPv6.java
java/util/concurrent/locks/Lock/TimedAcquireLeak.java
jdk/internal/misc/VM/RuntimeArguments.java
jdk/modules/etc/JdkQualifiedExportTest.java
tools/jar/multiRelease/ApiValidatorTest.java

Again, a thorough analysis will need to be done - here I just quickly scanned the jtr files.

Shura

> On Jan 8, 2019, at 6:15 PM, David Holmes <David.Holmes at oracle.com> wrote:
> 
> Hi Shura,
> 
> Sorry not a review just a comment on one thing ...
> 
> On 9/01/2019 4:35 am, Alexandre (Shura) Iline wrote:
>> Hi,
>> Could you please take a look on a change which allows to run tests while collecting code coverage with JCov. This is a continuation of work done in
>> JDK-8214309: Enhance makefiles to allow generating JCov instrumented build.
>> This adds make targets jcov-test and others like jcov-test-tier1 etc. After running the tests, one is left with a coverage report and also a data file containing coverage, for further analysis.
>> Part of the changes are related to increasing maximum and initial heap size, which is needed to be done because instrumented classes are bigger in size. The way it is done for JTReg tests is by setting both _JAVA_OPTIONS and JAVA_TOOL_OPTIONS environment variables.
>> As it stands now, not all tests pass while running with code coverage, which pass normally. This particular enhancement is not addressing failing tests. The tests will need to be fixed separately if ever. For example, there are 35 tests which fail in open/test/jdk:jdk_core. A quick look tells that the majority of the failures are in tests which verify error output and discover an unexpected line "Picked up _JAVA_OPTIONS: -Xmx4g”.
> 
> We've been bitten in the past with VM warnings causing tests to fail due to the unexpected output and made a conscious effort to fix all such tests. Unfortunately a few still remain. Do your 35 failures match up with the tests in the blocking issues for this bug:
> 
> https://bugs.openjdk.java.net/browse/JDK-8196741
> 
> ? 35 seems a lot. I hope we haven't regressed in this area.
> 
> Thanks,
> David
> 
>> Enhancement: https://bugs.openjdk.java.net/browse/JDK-8215729
>> Webrev: http://cr.openjdk.java.net/~shurailine/8215729/webrev.01/
>> Shura