OpenJDK 8u/11u release information

[Gil Tene]

> On May 27, 2019, at 6:34 AM, Matthias Klose <doko at ubuntu.com> wrote:
> 
> Hi guys,

The following is meant to be constructive and informational, so please don't
read it as anything other than that. With that said, any and all "you are plain
wrong, and here is why" followups to what I say below are welcome.

> 
> until recently we didn't have any source code releases for OpenJDK at all.

Not sure what you mean by this. Can you clarify what you mean when you
use the term "source code releases"? And how the thing you mean differs
from historical practice in OpenJDK?

By my understanding OpenJDK, as a source code project, has been producing
source code releases pretty much since OpenJDK 6 was first
released, and has never stopped doing so, with continuing update releases
happening within major version project on a pretty regular basis.

Version formats have changed across major java versions, tagging and
development processes going into a release also evolved and changed over
time. But at all points in the past several years, it's been pretty clear when a
release happened, and what the source code for that release actually was.

Binary builds of those released sources have been around for quite a
while as well. But in the past, these may not have been created consistently
by the project lead for every update.

What is "new" (AFAIK), specifically in 8u and 11u (and not in other projects
thus far), is the posting of project-lead blessed, TCK-tested builds of "vanilla"
released (as well as EA) sources of 8u and 11u (see e.g. https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-April/009105.html
and subsequent discussions) at some external location. That's a very positive
development, which I certainly hope will last and continue. But as noted in
that e-mail thread, those binaries are simply good, fully-TCK-tested builds of
the released sources, posted to a known location. They did not change the
release process in any way.

> I appreciate the effort to have source releases, however the gap to produce
> binaries based on the source release is way to high.  Instead of having a dozen
> of configuration options to build a binary, please can we have a default option
> which builds consumable packages by default?

As things stand, across the various java version projects, the default build modes
for all sources (including releases) produce consumable-by-the-builder things,
intentionally labeled in ways that would scare people away from using the bits as
a release. This is intentional, and is aimed to make sure that things that look
like an actual release will only happen through intent, assertive actions and
real choices about reported versions.

E.g., when I build things using the defaults, I'd get stuff like:

# bash ./configure
# make all
# build/linux-x86_64-normal-server-release/jdk/bin/java -version
openjdk version "1.8.0-internal"
OpenJDK Runtime Environment (build 1.8.0-internal-gil_2019_05_27_10_48-b00)
OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)

And that's a good thing. It prevents the accidental leakage of unfinished bits
to others.

To produce things you would give other people ("consumable" by others),
you need to intentionally make assertive (non-default) choices about
configuration. Normally, stating your update version, build version, and
milestone (or "pre" version) at the very least, and likely things like
vendor version, etc. Non of those have good "defaults", as you don't
want any of those to appear "by accident" in a build.

The same is true for the sources of the actual quarterly releases, which are
no more than a snapshot of the sources at the release point.

To create a binary build that does not carry the "-internal" identification, and
that has specific update and build numbers, you have to assertively declare those
in the build configuration. But that's not very complicated. E.g.

# bash ./configure --with-update-version=972 --with-build-number=b41 --with-milestone="snapshot"
# make clean; make all
# build/linux-x86_64-normal-server-release/jdk/bin/java -version
openjdk version "1.8.0_972-snapshot"
OpenJDK Runtime Environment (build 1.8.0_972-snapshot-b41)
OpenJDK 64-Bit Server VM (build 25.972-b41, mixed mode)

And with the special milestone called "fcs":

# bash ./configure --with-update-version=982 --with-build-number=b42 --with-milestone="fcs"
# make clean; make all
# build/linux-x86_64-normal-server-release/jdk/bin/java -version
openjdk version "1.8.0_982"
OpenJDK Runtime Environment (build 1.8.0_982-b42)
OpenJDK 64-Bit Server VM (build 25.982-b42, mixed mode)

Note:
For OpenJDK 8u "fcs" is a special milestone. It is the specific thing you need to do to
get rid of the milestone part of the versions string. Any other value (including "", which
will revert to "internal") will show up in the version, because of this in common/autoconf/spec.gmk.in :

# These variables need to be generated here so that MILESTONE and
# JDK_BUILD_NUMBER can be overridden on the make command line.
ifeq ($(MILESTONE), fcs)
  RELEASE=$(JDK_VERSION)$(BUILD_VARIANT_RELEASE)
else
  RELEASE=$(JDK_VERSION)-$(MILESTONE)$(BUILD_VARIANT_RELEASE)
endif

11u is a little different. The release versioning is more specifically defined (JEP322),
and the term for that "disqualifier" part for the version that tells you it is not an
actual release (what 8u sets using --with-milestone) is the "pre-release identifier"
($PRE) part of the version string (set --with-version-pre).

The default behaves the same with regards to "-internal" in the sense that $PRE defaults
to "internal", a =nd must be assertively set to "" to get rid of it. But in 11u a default for the
UPDATE portion of the JEP322 version string is typically baked into
make/autoconf/version-numbers (as opposed to having to be set via --with-update-version
in 8u)

so e.g.:

# bash ./configure
# make all
# ./build/linux-x86_64-normal-server-release/jdk/bin/java -version
openjdk version "11.0.3-internal" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3-internal+0-adhoc.gil.11u1103rel)
OpenJDK 64-Bit Server VM (build 11.0.3-internal+0-adhoc.gil.11u1103rel, mixed mode)

But when you build something that is fit for others to consume, you will still want to provide
a build number with --with-build-number, use -with-version-pre="" if you want to say it is an
actual release (use e.g. "ea", or just leave blank to make it "internal" otherwise), and in
addition set --with-version-opt to control that thing that defaults to
"adhoc.$USERNAME.$basedirname". Finally, --with-vendor-version-string="18.9" is
considered "vanilla" (the year.month for Java SE 11), and that's where non-vanilla
distros tend put their name in.

So e.g.:

# bash ./configure --with-version-build=42 --with-version-opt="" --with-version-pre="" --with-vendor-version-string="18.9"
# make all
# ./build/linux-x86_64-normal-server-release/jdk/bin/java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+42)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+42, mixed mode)

> 
> OpenJDK doesn't have source releases until recently.  Now we have, and from my
> point of view, such a source release should only have a minimal set of
> configuration options to build a usable image.  Things I would like to see
> 
> - I see it's important to display the version string as the first line
>  of java -version.  The source release should set that correctly.

See above discussion about why (by long standing OpenJDK historical
practice) version strings are controlled by the build configuration, and not
by the source code of a release.

> 
> - The OpenJDK source release ships with the vendor set to Oracle.
>  Distributors set that to Azul, AdoptJDK, Debian, and probably other
>  values.  The binaries built by the OpenJDK itself set that to some
>  sort of version string.  An "unknown" vendor causes issue, because some
>  software (LibreOffice, Gradle) uses or at least used that to check
>  for a valid java installation.

I'll assume you are referring to the "18.9" part in the version string reported
by the "blessed by project lead" builds of the OpenJDK 11u 11.0.3 release:

# ./openjdk-11.0.3+7/bin/java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode)

That 18.9 part is the "vendor version string" as defined by JEP322 (a new
property that did not exist in 8u). The choice of "18.9" (the year.month that
Java SE 11 was released) is both the actual example vendor version string
shown in the JEP, and follows the precedent already set for "vanilla" OpenJDK
builds of 11, 11.0.1, and 11.0.2 produced by the previous project lead during
the first 6 months of 11u. (note that those were the OpenJDK build posted by
Oracle, not the Oracle JDK builds).

Since "18.9" was there as a vendor version string form the very start of
OpenJDK 11, we can assume that anything that knows how to parse JEP322
versions will not have a problem with it.

> 
> - The version number should be used for both the source release and the
>  binary package.  E.g. the 11.0.3 source release is missing the -ga
>  modifier.

I'll assume that you are referring to version string reported by the binary,
e.g. in response to -version, and not to package names, file names, or tags
in source control.

The version string conventions for each existing Java SE version are
already established, and they (unfortunately) differ by major version. Thus
far, for all Java SE versions I know of which had an OpenJDK project
(6, 7, 8, 9, 10, 11, 12), neither Oracle nor OpenJDK builds have included a
positive (e.g. "-ga") indicator in the version strings of actually-released
versions. Instead, the long standing convention has been to include a
disqualifier ("-ea", "-internal", "-rc1") in non-released builds. The build
system defaults to using the -internal qualifier in all versions to date.

Thus the current output from the project-lead blessed builds of 8u212 is:
# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

(which, since there is no "disqualifier" in the version, means that it
reports as a build of an actual release).

The technical term for this "disqualifier" has varied over the years. As
shown above, 8u calls it "milestone", 11u calls it a "pre-release identifier".
But in all versions so far, *not* having it there means that it is a release.

Changing version string formats and conventions for indicating things
within a major java version is a generally bad idea. Adding "-ga" to actual
update releases of existing Java SE major versions is likely to blow up
a whole bunch of things. For one: a common ways to determine
that a version of OpenJDK or Oracle JDK is an actual released version
(as opposed to some ea/beta/rc1/internal/experimental thing) is to verify
that there is no "-XYZ" part (the milestone or $PRE) in the version string.

Making changes to conventions across versions is quite possible, and
has been done multiple times (e.g. JEP 322, JEP 223). JEP 322 is what
we currently follow for 11u and above. If it insufficient for some reason,
and we want to change it yet again for e.g. OpenJDK 14, that's something
to discuss, I guess., But such changes should not affect existing Java SE
versions (12 and below).

> 
> - To include a package in a Linux distro, you have to use a monotonically
>  increasing version number, and you have to follow the versioning
>  constraints for the distro.  So sometimes you have to juggle with the
>  version numbers.  E.g. for Debian/Ubuntu a second dash is problematic
>  for version comparisons.  I consider uploads to a development distro
>  as essential, so I have to plan for these version numbers as well.
>  Last time when I asked on the mailing lists, people seemed to be fine
>  with the versioning, however if needed, we could document such
>  versioning on the OpenJDK wiki?

This is a good point (make sure your *package* names retain monotonically
growing version numbers), but since these rules apply to the package names
as seen by the distro (and not to the version strings) simple distro-specific file
name conversion conventions seem to suffice. This is often necessary simply
because conventions and file name requirements can vary widely
between OSs (some don't like dots, some don't like dashes, some deal with
capitalization in interesting ways).

E.g. you can convert every dash in the version string to to an underscore in
the .deb or .rpm package names.

An interesting problem shows up if the parsing of the version string
related fields appears to "go backwards" in your monotonic comparisons
(e.g. if the sorting is text based rather than numeric, and 11.0.10 would be
considered earlier than 11.0.9, or 8u92 was considered to come after 8u102).
I can think of a few ways to resolve that with package name conventions,
but I don't really know if this is a problem. Does debian compare
numeric sequence substrings in package name numerically, or
lexicographically?

> 
> - It would be very helpful to see directly in the binary how the build
>  was configured.  GCC is showing this information with
>    gcc -v
>  Python is showing that with
>    python -c 'import sysconfig; print(sysconfig.get_config_var("CONFIG_ARGS"))'
>  Or maybe this already exists?
> 
>  Usually binaries in a distro come with a changelog, however sometimes
>  even that is stripped away by re-distributors of binaries.
> 
> - The configure system has some issues with invalid configure arguments, e.g.
>  configuring --with-version-build='' leads to a failing build.

Setting --with-version-build="" works on 8u (it is a string).

In 11u, you'd use --with-version-build=0 (it is an integer).

> 
> Matthias