OpenJDK GB Minutes: 2007/8/23

[Mark Reinhold]

Attached please find the minutes of the first meeting of the Governance
Board.  The minutes are also available on the web:

    http://openjdk.java.net/groups/gb/2007-08-23.html

Respectfully submitted,
- Mark

-------------- next part --------------

OpenJDK Governance Board Minutes: 2007/8/23
Mark Reinhold <mr at sun.com>

The third meeting of the OpenJDK Governance Board took place via conference
call on Thursday, 23 August 2007 at 15:00 UTC.  The topic was the OpenJDK
Community TCK License, which had been announced two weeks prior.

All GB members were present: Doug Lea, Fabiane Nardon, Dalibor Topic, Simon
Phipps, and Mark Reinhold.  Carla Schroer of Sun, who was deeply involved in
the crafting of the new license, joined the call in order to answer questions
and provide context.

The meeting was essentially a long question-and-answer session, with no agreed
resolutions, so these minutes are structured in a Q&A format in the following
sections:

  1 License-application process
  2 License eligibility
  3 Running the TCK
  4 Certifying a platform implementation
  5 Confidentiality
  6 Miscellaneous

On the whole the GB membership was pleased with the new TCK license.  Most
members would like to see the TCK itself open-sourced in the long term, and
also made available to certify any platform implementation rather than only
those substantially derived from the OpenJDK code base.  The license was,
nonetheless, generally praised as a big step in the right direction and
something with which most Linux distributors, in particular, will be able 
to work.


1 License-application process

    Q1: When will the online TCK license-application process be available?
        There are developers who want to start working with the TCK as soon
        as possible.

  Sun's TCK team and the JCP Program Management Office are working on that.
  It's been delayed a bit by other responsibilities, but it should be done in
  the next few weeks.  [Note: As of early December the online application page
  is not yet available, but interested parties have been able to execute the
  license agreement and acquire the TCK by contacting Sun directly.]

    Q2: How long should it take for someone to receive the TCK once they've
        submitted an online application?

  It depends partly on how many people apply; a large number would slow things
  down.  With a moderate number it should be possible to approve an application
  in under a month.  The TCK will be delivered to licensees using the same
  mechanism aleady in place for Sun's commercial partners.

    Q3: Is the OpenJDK TCK license meant to be executed by individuals, by
        organizations, or either?

  It's written to accommodate both cases.  Which one is best depends upon your
  specific situation.

    Q4: Suppose that a student at a university is working with the OpenJDK
        code and wants to get the TCK.  Should he sign his own license, or
        should the university sign for him?  What's the best way to handle
        that kind of situation?

  Students at universities are different than employees of corporations because
  they aren't employees of the university and often do not have the ability to
  control access to university computers.  That said, the recommendation is
  that students sign the TCK license as individuals if they want to put the TCK
  on computers that they can control.  If a group of students is working on a
  project together then they can sign a single license and fill in the names of
  all the students working on the project.  This could include a professor too,
  if that's appropriate.  In other words, students should act as individuals or
  as groups of individuals, and install the software on computers they control.


2 License eligibility

    Q5: What about hybrid implementations, for example what the Cacao folks
        are working on, where they're using the OpenJDK class libraries on
        top of their own VM to run on PowerPC or S/390 machines?  Would they
        be eligible for the TCK license?

  Absolutely!  As long as they're GPL-licensed and contain substantial amounts
  of OpenJDK code then it's in everybody's interest to help them build a
  compatible implementation.

    Q6: What does it mean for an implementation to be "substantially derived,"
        as the license says, from the OpenJDK code?  For example, would the
        Classpath library combined with the HotSpot VM count as "substantially
        derived?"

  There's no precise definition of "substantial."  It can't be expressed in
  terms of, e.g., a specific number of lines of code.  It's more of a principle
  which says that you can't just take a few small things from OpenJDK into your
  own code base and claim that your code is "substantially derived."

  The Classpath library running with the HotSpot VM would certainly count as
  substantially derived, since the HotSpot VM is a significant chunk of code.

  Another obvious kind of case is where someone has replaced a few packages
  with alternative implementations, e.g., to customize for a particular
  operating system or windowing environment.  That would also be considered to
  be substantially derived.

    Q7: Suppose it's 2008.  The Classpath library has moved to GPLv3, and
        someone ships it with the VM from Apache Harmony.  Does that count
        as "substantially derived?"

  No.  Aside from the likely absence of significant amounts of OpenJDK code,
  such a combination would not be considered derived because the OpenJDK TCK
  license requires that a tested implementation be released under the same
  license as the OpenJDK code itself, which at the moment is GPL version 2
  only.

    Q8: Would the Harmony class libraries combined with the HotSpot VM be
        considered "substantially derived?" 

  In terms of the code, yes, this combination would be considered to be
  substantially derived.

  To be eligible for the OpenJDK TCK License, however, you'd have to be
  planning to release the whole combination under GPLv2, and that's likely a
  problem.  Neither Apache, the FSF, nor Sun view the Apache license as
  compatible with GPLv2, at least not if you're incorporating Apache code
  as-is.  You'd have to decide for yourself whether such a combination is
  legitimate.


3 Running the TCK

    Q9: So I get the license application approved, I download the TCK from 
        Sun, and then I can run it locally against my implementation, right?

  Right.

    Q10: Are the TCK tests fully automated, or are they interactive?

  The selection of tests and logging of results is fully automated.  Most of
  the tests themselves are automated, but some of the GUI tests are
  interactive.  When you run an interactive test a dialog box pops up, tells
  you what to do (e.g., "click the mouse here") and what to look for, and gives
  you a button to click if the right thing happens.  The test harness then
  records the result in its log file.  You can run the automated tests
  separately from the interactive tests.

    Q11: How long does it take to run the whole TCK suite?

  There are tens of thousands of tests in the suite.  It obviously depends on
  the speed of your machine, but running the automated tests on reasonably
  modern hardware is a matter of hours, not days.  Many of Sun's engineering
  teams have machines set up to run the automated tests every night, so they
  can review the results each morning.

  It's also easy to configure the harness to run just a subset of the tests,
  e.g., those for a specific package or feature, and that's how most developers
  tend to use it when they're working in a specific area.  The test harness,
  which has been open-sourced in the "JT Harness" project on java.net, has
  fairly sophisticated test-selection facilities using keywords and other
  criteria.  You can also easily re-run just the tests that failed in your most
  recent run.


4 Certifying a platform implementation

    Q12: How complete is the TCK?

  The primary quality metric for the TCK suite is specification coverage rather
  code coverage, as for most other kinds of tests.  The breadth and depth of
  coverage varies across the platform.  The VM, language, and core libraries
  are pretty well covered by now; some of the newer areas, especially those
  further away from the core, are not as well covered.

    Q13: Does an implementation have to pass every single test?

  Yes, every single valid test in the suite must pass.  You can challenge the
  validity of specific tests, however, and sometimes tests are thrown out as a
  result of such challenges.

    Q14: What's the difference between passing the test suite and actually
         certifying an implementation as compatible?

  That's an important distinction.  A compatible implementation must meet all
  the requirements that are spelled out in the TCK User's Guide, specifically
  those in Chapter 2.  It must not only pass the test suite but also satisfy a
  set of rules that are, inherently, untestable.  In other words, there are
  parts of the certification process that amount to you asserting that, to the
  best of your knowledge, you're not breaking compatibility.  It doesn't mean
  that you don't have any bugs, it just means that you're not knowingly
  violating any of the relevant specifications and other requirements.

  The TCK rules also cover things like how you run the tests; e.g., you must
  use the provided test harness, unmodified, since the harness itself
  determines what passes and fails.  That's especially important for negative
  tests, e.g., compiler tests that are not intended to compile and should be
  considered to fail if they do.  The harness contains all that pass/fail
  logic, so obviously if you could modify the harness then that could change
  the results.

  Perhaps the most important rule is the famous "all modes" rule, which
  requires that an implementation be able to pass in all modes, i.e., in all
  possible combinations of command-line flags, environment variables, and
  whatever other configuration data may govern its behavior.  It's not
  sufficient to say that the test suite passes in some configuration, it must
  in principle be able to pass in all configurations.  An implementation that
  has a flag to disable array-bounds checking, e.g., would fail this rule.

  Now the practical reality, of course, is that there will be modes that don't
  pass, e.g., modes that just print a version string and exit, or that provide
  certain kinds of debugging information.  The rules cover all these sorts of
  special cases and exceptions.

    Q15: How hard is it to certify a new implementation?

  A lot of it depends on how that implementation is constructed.  Historically
  Sun has found that when people have something that's almost completely based
  on Sun's source code, and built in pretty much the way that Sun builds it,
  then it doesn't take that much effort to certify.

  When people have written their own VMs or other significant components then
  the tests will find bugs, and they'll have to be fixed.  The time to certify
  such an implementation tends to be dominated by the time required to find and
  fix the bugs rather than the time required to run the tests.  This can take
  weeks or even months, if the code in question has a lot of issues.

  People who replace a lot of Sun's code often find that the hardest part of
  certifying is in handling error conditions correctly.  They'll implement all
  of the required positive behaviors and then discover that the TCK contains
  pretty thorough tests for negative behaviors too, e.g., throwing the correct
  exception for particular invalid inputs.

    Q16: Have any of Sun's commercial licensees replaced significant portions
         of the Sun codebase?

  Yes.  Several Sun partners have written their very own VMs, e.g., and the
  first couple of times that happened some issues in the tests were identified.
  Some of the testing techniques even had to be changed to eliminate
  assumptions that'd been true only of Sun's VMs.  At this point the suite has
  been pretty well vetted, especially in the VM and compiler areas, by third
  parties testing implementations that are only partly derived from Sun's code.

    Q17: What about source-based distributions such as Gentoo, in which
         binaries don't exist until the source code is compiled on the
         end-user's machine?  Will the TCK be of interest to people working
	 on such distributions?

  Claims of compatibility in the Java world apply only to binaries, not to
  source code.  If people working on a source-based distribution make
  modifications to the OpenJDK code they'll still probably want to get their
  hands on the TCK, however, to help flush out bugs and to verify that they can
  build a compatible implementation -- even if that isn't what they'll actually
  ship in the end.

  The trademark license, by the way, requires compatibility, so a source-based
  distribution containing OpenJDK code wouldn't be able to carry the Java
  brand.

    Q18: Suppose that the maintainer of an OpenJDK-derived implementation 
         needs to issue an emergency release to fix a security hole.  Is it
         okay to do that without actually running the TCK all over again?

  Yes.  In general Sun tries to handle situations like this in a reasonable,
  common-sense sort of way.  (Sun is in the business of shipping software too,
  so they understand.)

  Claims of compatibility rest upon what is essentially a self-certification
  process.  You get the test suite, you get the rules, you look at them.  When
  you think you're ready, you write back to Sun and say, "I certified that this
  product meets your requirements using this version of the TCK."

  It's understood that implementors sometimes have to do things quickly, like
  ship security fixes.  Whether to retest is a judgement call that you, as an
  implementor, would have to make for yourself.  If the fix involves a
  low-level change in the VM that could have broad consequences then you'd
  obviously want to retest before you continue to claim compatibility.  If it's
  an isolated fix further up the stack, however, and you judge the risk to be
  low, then you might reasonably move ahead.  There's certainly not the
  expectation that you'll always test every combination on every build; that's
  just not practical.

  The expectation is that people are acting in good faith, trying to meet the
  compatibility requirements, and that they aren't knowingly doing something
  that would violate them.  They're expected to make their own choices around
  risk, with the understanding that if a mistake is made and an implementation
  isn't compatible then Sun could ask them to stop claiming it's compatible and
  stop using the Java logo with it.

  This is pretty much how Sun has worked with its commercial licensees for many
  years now.  This should scale to the relatively modest expected number of new
  OpenJDK TCK licensees that will want to self-certify and carry the brand.

    Q19: If an OpenJDK-derived implementation is still in beta, or is 
         declared final but isn't compatible, can it still be shipped?

  Yes.  This is a key difference between the OpenJDK TCK license and the
  traditional TCK license available under commercial terms from Sun.

  Under the commercial license an implementor can ship early-access or beta
  releases and not worry about compatibility, so long as they don't claim
  compatibility and don't allow production use.  Once they're ready to ship the
  final release to customers and support it in production, however, then it
  must be compatible -- otherwise they can't ship it.

  The GPL, of course, doesn't permit such constraints.  An incompatible
  OpenJDK-derived implementation can go final and be shipped -- it just can't
  be claimed to be compatible, nor can it carry the Java brand or logo.

    Q20: Suppose that an implementation depends upon some libraries provided
         by the underlying system, e.g., the X11 library.  When that library 
	 is updated then does the implementation have to be re-certified?

  Claims of compatibility are always made against a specific set of software
  configurations.  An implementation might be compatible on some other,
  possibly related configuration, or it might not; there's no guarantee.  When
  Microsoft shipped Windows Vista, e.g., Sun had to re-certify its
  implementations on Vista in order to be able to claim compatibility on Vista.


5 Confidentiality

    Q21: What about the confidentiality clause in the OpenJDK TCK license?
         Does it allow for the publication of test results?  That is, can 
	 one talk about what passes and fails in public?

  Yes.  Sharing the results of test runs is fine, per section 5.1 of the
  license.  You can say things like "I passed all the tests," or "I passed all
  the VM tests, but failed these seven compiler tests," or you could even, if
  you really wanted to, publish the entire log of a TCK run.

  Sun doesn't, however, want people to make comparative claims of
  compatibility, e.g., "I'm just as compatible as that other guy over there,
  but not really actually compatible," since that will tend to confuse people
  about what compatibility means.

  The confidentiality clause is intended to protect the tests themselves, not
  the results of running the tests.  You can't disclose the details of the
  tests publicly, though you can discuss them with other parties who've also
  signed the OpenJDK TCK license.


6 Miscellaneous

    Q22: How does one go about writing TCK tests?

  Since specification coverage is the goal, the process is best started by
  marking up the specification to be tested in order to identify all the
  assertions it makes, and then figuring out which assertions are testable.

    Q23: Will Sun accept contributions of new TCK tests?

  Yes, but since TCK tests are so different from other kinds of tests any
  contributions will have to meet the standards of the rest of the suite.
  Given the overhead involved it's probably not worthwhile for people to
  contribute just a few tests here and there.  Sun does, however, very much
  welcome large sets of new tests, especially for new features.  Doug Lea and
  the JSR-166 Expert Group, e.g., have made significant contributions to the
  TCK, as have some of Sun's commercial partners over the years.

    Q24: What other kinds of tests does Sun have, besides the TCK?

  Sun has several different kinds of tests.

  The TCK tests are fairly special given their role in defining what it means
  to be a compatible implementation, being able to run in a wide variety of
  environments, having to be robust enough to stand up to challenges of
  validity, and being the basis of various commercial contracts.  These
  qualities tend to make compatibility tests harder to write than most other
  kinds of tests.

  Aside from the TCK, Sun also has a large collection of regression and unit
  tests, written and maintained by the development team.  Many of these tests
  have already been published under the GPL.  The long-term goal is to publish
  as many of them as possible, subject to legal constraints.

  Finally there are functional, performance, and reliability test suites.  Some
  of these will likely be open-sourced over time although some cannot, again
  due to legal constraints.

    Q25: When will the regression/unit-test harness, jtreg, be open-sourced?

  The jtreg harness is actually just a small plugin for JT Harness, which has
  already been open-sourced.  JT Harness is really more of a harness framework
  than a simple harness; it has some special features for compatibility testing
  but can actually be used for a wide variety of different kinds of tests.

  It's understood that people are interested in an open-source version of
  jtreg, and Sun does intend to release it, but doing so hasn't been among the
  highest priorities.

[end]