Code Coverage

[Jonathan Gibbons]

On 05/18/2013 08:49 AM, Alan Bateman wrote:
> On 18/05/2013 11:26, Richard Warburton wrote:
>> Hi,
>>
>> I'm currently helping Steve Poole run an openjdk test-fest in 
>> Krakow.  The goal of the event to be incrementally improving test 
>> coverage in Openjdk.  One of the issues that has cropped up is the 
>> lack of available coverage reports in openjdk.
>>
>> When this has been previously discussed it was commented there is 
>> already internal code coverage tooling, but this information wasn't 
>> publicly available.  Any update or clarification on this situation 
>> would be most appreciated.
>>
>> If there is a barrier to making the tooling publicly available then 
>> even publicly publishing HTML or text reports to provide visibility 
>> into the situation would be helpful.  The results of test runs are 
>> already regularly posted to this list.
>>
> In Oracle, the tool that we use is called "jcov", it's proprietary so 
> I can't point you at anything. Maybe someday it could be proposed to 
> the Code Tool Projects but I have no knowledge of specific plans on that.
>
> I wonder if anyone has tried using EMMA or any of the other code 
> coverage tools out there? At least for the JDK releases in development 
> then one challenge would be keeping the tool up to the date with the 
> class file format, another would be taking care when instrumenting 
> very core classes that are used early in VM startup. It's the same 
> challenge that profiles and other tools that do bytecode 
> instrumentation have used on a JDK version that is still in development.
>
> -Alan.
>
>
>
>

Alan,

I looked at EMMA.  I see one notable problem with using it to get 
coverage data for OpenJDK while running the OpenJDK test suites. These 
test suites create *lots* of JVMs, for a variety of reasons. Even in 
"sameVM" mode, the langtools test suite creates about 80+ JVMs, last 
time I checked. This poses a problem for coverage tools, when they try 
and write out coverage data.

It looks like EMMA tries to write out all data to a single file [1].  
You can specify whether data in that file is to be overwritten by 
subsequent runs, or merged.  In experiments in times past, merging has 
not been successful in our crazy worlds of concurrent JVMs coming and going.

There are two possible ways a good coverage tool could get around this 
problem. It should either be possible to specify that each JVM should 
write its coverage data to a new uniquely named file (for later offline 
merging) or each JVM could write data to a server process which can 
coordinate access to a single shared file.

-- Jon


[1] http://emma.sourceforge.net/reference/ch03s02.html#prop-ref.table.file