Gathering metrics for the Adoption programme

[dalibor topic]

On 05.02.2014 18:00, Martijn Verburg wrote:
> Hi all,
>
> **Important - please remember to sign-up to the
> adoption-discuss at openjdk.java.net mailing list and reply to this thread
> there please**
>
> We'd like to measure & report to the OpenJDK governing board some
> quantifiable metrics on the impact and effectiveness of the Adoption
> programme. These metrics will also be useful to the group in terms of
> helping us determine whether we are fulfilling our mission statement and
> purpose.

Thanks for kicking off the discussion on this item, Martijn!

To set some context, here's what the Bylaws require the Group Lead to do:
http://openjdk.java.net/bylaws#group-lead

"The obligation, once per quarter, to publish a written report 
summarizing the recent activities of the Group."

As there is no specific format required, we have some liberty in terms 
of what information we want to report, where to report it, and so on.

Let's start with the 'where' and 'when' parts first.

While Martijn's e-mail says 'report to the OpenJDK governing board', 
that's actually not what the Bylaws ask for, and not what the Governing 
Board asked for (see 
http://mail.openjdk.java.net/pipermail/gb-discuss/2013-December/000255.html 
for the actual request:
"Unlike other groups/projects, keeping to a schedule
of quarterly reports seems appropriate."). So I would suggest that

	* the quarterly report is compiled on the Group's Wiki,
	* starting in April 2014, and
	* published by the Group Lead as a finished copy to this mailing list, 
allowing followup and discussion by all Participants

which includes Governing Board Members, of course, should they chose to 
participate on this mailing list.

Now, on to the 'what' question.

What activities are there to report about for Groups? Again, looking at 
the Bylaws we find:
http://openjdk.java.net/bylaws#_4
"A Group is a collection of Participants who engage in open conversation 
about a common interest. "

"Groups may have web content and one or more mailing lists. Groups do 
not have code repositories of their own but they may sponsor Projects, 
which do."

so based on that, the metrics that we should be looking for are around 
conversations, the Group's web content and mailing lists.

If the Adoption Group were to sponsor Projects which publish quarterly 
report themselves, then such reports could be referenced by the 
quarterly report of this Group, but compiling such reports would be the 
responsibility of their respective Project Leads, as defined in 
http://openjdk.java.net/bylaws#project-lead - and not this Group's task.

Some further exploration of notable activities in the Bylaws can be 
found in http://openjdk.java.net/bylaws#_5

"A Member of a Group has write access to the Group’s web content and 
file repositories."

"A Member of a Group may nominate any Contributor to be a new Member of 
that Group."

"A Group’s Lead has:

     The authority to sponsor Projects;

     The obligation to act as a contact point for the Group and to look 
after the Group’s mailing lists and web content; and

     The obligation, once per quarter, to publish a written report 
summarizing the recent activities of the Group."

"A Group’s Lead may delegate selected obligations, but not authorities, 
to other of that Group’s Members as desired."

So further Group activities to measure and report could be

	* web content updates and changes (like Wiki spring cleaning)
	* Group Member nominations and other Group membership changes
	* sponsoring of Projects
	* publication of written quarterly reports

Another thing to keep in mind is that just because something could be 
measured, it may not actually be useful information. For example, while 
a quarterly report could include a word count of e-mails sent to this 
mailing list in a given period, it would provide no actually useful 
information for the OpenJDK Community or this Group to do so. ;)

So instead of thinking of the things that we could measure, I think it's 
important to apply the concept of 'Datensparsamkeit' 
(http://martinfowler.com/bliki/Datensparsamkeit.html for some mildly 
related thoughts) and ask what we really have to measure, and then go 
from there.

Another important consideration from my point of view is that 
information included in the reports should either be publicly available 
or compilable from publicly available sources. As any such reports would 
be compiled by people, and "people are people", as a wise philosopher 
once said, exclusively relying on publicly available or compilable 
information would allow errors to be detected and corrected prior to the 
publication.

Finally, an important consideration is that a Group Lead could only 
measure those activities directly taking place within OpenJDK, as those 
would be the activities they would be responsible in some way or another 
for coordinating and directing, as defined in the Bylaws.

That doesn't mean that other information would not be interesting to 
reference in quarterly reports, and if someone compiles quarterly 
updates on, say, edits of OpenJDK pages on Wikipedia, then that may be 
interesting information to reference, but it should not be something 
that is expected to be tracked and reported on an ongoing basis as an 
activity of this Group.

That was a lot to read. Now on to the specifics!

> Initial Onboarding:
> =============
>
> * Number of mailing list users

That number is not publicly available. As the mailing list moderator, I 
could find out, and report on that, but I don't think that it's an 
actually useful number, and besides, it's trivial to game.

A more important number is the number of users who posted to the mailing 
lists in a quarter, I think. It's not that trivial to compile, but since 
the archives are publicly available, it can be publicly verified, by 
looking at the archives:
http://mail.openjdk.java.net/pipermail/adoption-discuss/2014-February/author.html#start 
for February (I believe the current count is 13).

The reason why the number is interesting is that activity on the mailing 
list is a usual way for new Contributors to make their mark on the 
Group, and through lasting contributions become Group Members 
themselves. So in a way, that number, when offset against the number of 
Group Members, gives us an very rough idea of whether the Group is 
attracting fresh ideas outside its core membership.

> * Message volume on the mailing list

That is public information available on the page cited above - " 
Messages: 63" it says, and should be at least 64 by the time this e-mail 
arrives. ;)

I think that's a useful number to track, though I don't think that its 
resolution as such is very useful. If mailing list traffic goes from 216 
e-mails in a month to 252 in the next month, is it relevant? Probably 
not very much. If it goes to 2045? It may be.

I don't have a better resolution other than "none, few, dozens, 
hundreds" to propose though - and that seems a bit arbitrary. So let's 
stick with reporting the number as is.

> * Number of new developers successfully building OpenJDK / Downloads of
> OpenJDK development virtual machine

I don't think that the former number could be tracked, as there is no 
'callback' mechanism in the build. Nor should there be one, I think.

The latter number is a no op for this Group, as Groups do not produce 
artifacts you can download - that's something Projects do.

> * Number of unique visitors to Adoption page/wiki

I'm not aware of any public information on that, or if it exists, to 
begin with. I don't think that's actually very useful information as it 
is trivial to game, Datensparsamkeit aside.

On the other hand, since Adoption Group Members can edit the wiki, I 
think information on the number of edits, new pages and active editors 
in a quarter could be useful information to collect and track, as that 
would be a measure of how well the Group is utilizing that part of its 
infrastructure.

Looking at the page information menu entry point in the wiki, I don't 
see any further items on which information is available that would be 
worth tracking.

> ** Perhaps measure impact on overall OpenJDK visits

I'm not aware of any public source of such information, either, as above.

> Betterrev:
> =======
>
> Betterev is a proposed patch workflow & build farm system. It will act as
> community clearing house for new contributors and as a prototype for
> OpenJDK infrastructure.
>

I think that if a Betterrev Project were to be established in OpenJDK, 
then its quarterly report could contain all kinds of interesting data on 
its activities, but I don't think a Project's data belongs in a Group's 
quarterly report. A reference to a Project's quarterly report would be 
perfectly fine, though.

> QA:
> ===
>

I'd relabel that category 'QA Outreach' to differentiate it from ongoing 
OpenJDK development focused QA efforts which some Projects may have. 
This is a great example of contributions users can make to improve the 
JDK without having to be or become experts in the implementation themselves.

> * Number of FOSS projects involved in testing OpenJDK nightly builds

I think that this is an important metric, though, but not at that 
granularity - basically, I think the metric we should generally care 
about is the number of FOSS projects approached by this Group's Members 
providing active feedback on OpenJDK builds, whether their own, or from 
someone else, whether those are nightly, weekly, monthly, or release 
builds, and the volume of such feedback.

So - an example of great information to report would be: P1 issues filed 
and fixed in JDK Bug System based on this Group's QA outreach efforts in 
a given quarter. I think that would be an a lot more interesting list 
than a line like '18 projects' in a column on a report or '19' the next 
quarter.

Assuming the QA outreach effort was tracked on our wiki, along with the 
reported issues, such a list should not be very hard to compile and verify.

> * Number of bugs/issues reviewed by adoption group for quality control
> purposes.

Technically, Reviewer is a Project-specific role, rather than 
Group-specific:
http://openjdk.java.net/bylaws#reviewer

If we redefine this item to mean something like 'major issues this Group 
assisted with' then I think we have something more interesting to 
report, which may not be as neatly quantifiable as a number, but is 
probably more interesting as a report on activity than a line that says 
'reviewed 10 issues this quarter'. ;)

> Longterm Onboarding:
> ================
>
> * Number of 'graduates' from Adoption programme into official OpenJDK
> contributors, authors, committers and reviewers.

This could be partly derived from the OpenJDK census at
http://openjdk.java.net/census . I don't think that tracking Contributors
is that useful, as being a Contributor does not imply having made a
code contribution or ever getting around to making one.

On the other hand, you need at least two code contributions to a Project
to become an Author on a Project, so that's interesting information to
track, per se.

But ... a challenge here is the connection between the Adoption Group 
membership and Project members.

A way to do it is to look at the progress in roles of Group Members of 
this Group in different Projects. But becoming a Group Member requires 
continuous contributions to a Group over a longer period of time: 
http://openjdk.java.net/groups/#group-member

"As a rough guide, a Contributor should be regularly active in a Group 
or its sponsored Projects for at least a year before being nominated to 
be a Group Member."

In other words, the set of this Group Members, whose 'graduations' we 
could try to measure, is not going to change a lot over the next twelve 
months.

And so, while this is an interesting measurement overall, as the name 
says 'long term', it's not something I think we should include in our 
initial quarterly reports. Over the long term, in 12 months, or so, it 
would be interesting to look at the additional roles this Group's 
Members have been voted into in other OpenJDK Groups and Projects, but 
it seems premature to me to report on it in the Group's first year of 
existence.

cheers,
dalibor topic

-- 
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
<tel:+491737185961>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment