Code Coverage

[Jonathan Gibbons]

Richard, all,

Yes, I agree it would be nice if we could get code-coverage reports 
published.

But that being said, I have a couple of comments, which I will say up 
front appear contradictory, even to me, as I write them.

1. The Java SQE org does do what you're suggesting: they run coverage 
reports and look to plug the holes.

2. With respect, I think you're looking for needles in haystacks, and 
missing the giant low hanging fruit all around us. In other words, 
you're looking at the wrong granularity.

When I was writing my recent blog entry last week, I went looking for 
examples of bad shell scripts. I started looking at the first 10 shell 
scripts that find showed me, and found problems. Within an hour, I had 
written scripts to identify more instances of those problems and filed 
bugs in the system.  Separately, as you may know if you follow the 
various aliases, there is an ongoing effort to improve the reliability 
of many areas of the test suite, and/or to "fix" tests that appear on 
the Problem list".  In other words, it's not so much that we need /more/ 
tests; we need /better/ tests.

If you do want to write more, new tests, I think that is as easy as 
deciding what area you're interested in, and then asking on the relevant 
aliases. For example, even though the code coverage numbers for javac 
are pretty good, we do not have many tests that verify the debug 
attributes generated by javac, such LineNumberTable, LocalVariableTable, 
etc. Mostly, class files generated by javac are tested by running them. 
We do not have any/many tests that check javac's behavior with respect 
to reading invalid class files.  We went *15 years* before anyone 
noticed and/or got concerned that javadoc generated invalid HTML. (Ooops.)

So, bottom line, even without code coverage tools, there's still easy 
pickings ;-)

-- Jon

On 05/18/2013 10:54 AM, Richard Warburton wrote:
> Hi,
>
> Thanks for the replies everyone.
>
>     In Oracle, the tool that we use is called "jcov", it's proprietary
>     so I can't point you at anything. Maybe someday it could be
>     proposed to the Code Tool Projects but I have no knowledge of
>     specific plans on that.
>
>
> If jcov produces HTML reports - would it be possible to have the 
> reports from your internal CI builds published?  This would at least 
> improve the situation even in the absence of jcov being made available.
>
>     I wonder if anyone has tried using EMMA or any of the other code
>     coverage tools out there? At least for the JDK releases in
>     development then one challenge would be keeping the tool up to the
>     date with the class file format, another would be taking care when
>     instrumenting very core classes that are used early in VM startup.
>     It's the same challenge that profiles and other tools that do
>     bytecode instrumentation have used on a JDK version that is still
>     in development.
>
>
> I imagine with sufficient effort Jacoco could be made to work - though 
> I was hoping to avoid debugging Jacoco with Java 8.  I'll have a look 
> into this.  At work we've got a patch against Jacoco in order to merge 
> together the results from different sub module builds in maven, so 
> I'll see if its possible to use this this same approach in order to 
> make Jacoco work with merging separate files from different openjdk 
> VMs as Johnathan has suggested.
>
> regards,
>
>   Richard Warburton
>
> http://insightfullogic.com
> @RichardWarburto <http://twitter.com/richardwarburto>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130518/e3028ec5/attachment.html