JDK-8025705

Tue Apr 29 14:39:21 UTC 2014

I have a different issue with closed and custom source that maybe can be addressed while you're looking at this area.  As a licensee, I build with at least some of the closed source, but I also need to customize the build.  Ideally, I'd like to be able to insert my custom make files and still have the closed files used.  The solution might be as simple as having a configure definition for the location of my custom files.  Chaining to the closed files would be my responsibility (though I'm not sure how I would change the closed files, if I needed to do that.)

For JDK8, we took the path of least resistance and just modified the make files in much the same way we had for earlier releases.

Tom Salter

-------------------------------------------------------------

Date: Tue, 29 Apr 2014 14:52:32 +1000
From: David Holmes <david.holmes at oracle.com>
To: Keith McGuigan <kmcguigan at twitter.com>, mark.reinhold at oracle.com
Cc: build-dev <build-dev at openjdk.java.net>
Subject: Re: JDK-8025705
Message-ID: <535F3010.5070607 at oracle.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi Keith,

As requested via email I'm responding on your response to Mark.

On 25/04/2014 8:10 AM, Keith McGuigan wrote:
> Hi Mark,
>
> Well first off, the existence of src/closed triggers OPENJDK to be
> unset, so even having such directories (or subrepos) present turns on
> all the logic that looks for files in src/closed which probably don't
> exist in a non-Oracle distribution.

Yes that is the default logic from configure. It's not perfect but it 
sets OPENJDK correctly for the majority of people that only have the 
OpenJDK sources.

As I said early on part of the problem you have is that you chose to use 
src/closed for your own customizations and that conflicts with the 
far-from-perfect use of src/closed existence checks. Obviously "we" 
don't have exclusive rights to that name, but it doesn't seem a prudent 
choice given the current imperfect separation.

That aside configure supports building an OpenJDK configuration, 
--enable-openjdk-only, to allow for building the OpenJDK even when you 
also have custom sources present**. That said I think some recent 
changes regarding licensee builds may have broken that somewhat as any 
existence check for src/closed outside of "ifndef OPENJDK" would trigger.

** In that logic everywhere you see "closed" it should read "custom". :(

Building "OPENJDK" does have other side-effects as you have mentioned 
elsewhere, specifically the version information used. But presumably you 
are customizing that information anyway. Plus if the code were changed 
to use !ORACLEJDK you'd still get the default OpenJDK version 
information. So it seems to me that version string customization is 
something you will need to manage explicitly either way.

> So having an ORACLEJDK test instead
> around that code means we can control it with a single switch somewhere
> in a top-level Makefile (maybe even a makefile in make/closed). This
> would let us use the src/closed mechanism and at the same time unset
> ORACLEJDK (OPENJDK would also be unset).

I'm not convinced this really makes a significant difference to things, 
and I am convinced it doesn't help us along the path to a proper 
separation. But if you and others think differently I'm not going to 
waste more energy fighting it.

> If we need to have a private modification to futz with ORACLEJDK (to
> turn it off), that's not so bad -- it'd be just in one place so
> maintenance would be easier.  But it's possible that we can put this
> logic in a make/closed file that gets conditionally included, in which
> case we wouldn't even have to worry about that.
>
> As to how to get custom code compiled, I think we have a couple options
> to try.  In hotspot the code is just magically picked up and treated as
> if it were sitting in the non-closed directory, augmenting or overriding
> the open sources (depending on the filename).

To the best of my understanding this is all handled with the lists of 
src dirs to build. If you add a custom src dir then you can build those 
custom sources. If you add a custom src dir and remove the equivalent 
open src dir, then you can effectively override. Of course in a custom 
makefile you can do whatever you like to pick and choose files to 
compile etc.

> If we could do something
> like that in the jdk makefiles, that would be ideal (IMO).  But it may
> not be feasible to do this -- I'd have to investigate.  Another,
> probably more realistic alternative is to add a couple generic hooks (as
> few as possible) in the open source to make/closed makefiles which may
> or may not exist depending on which distribution you are building.
>   Something like:
> -include make/closed/custom.gmk
> somewhere in a top-level Makefile which would include custom makefile if
> it exists, and silently skip over it if it is not there.

Such logic exists already in many places in the build system. But we did 
not pro-actively add a custom include for every single file; rather we 
have added the includes as needed - knowing that this may need expansion 
in the future, but being content to wait until there was something that 
would actually test the use of the new "hook". That said (and I have 
said this a few times over the past couple of years) the hook placement 
may not be flexible enough nor suit all possible customizations. 
Sometimes we need to include at the start to control the logic in the 
open file; sometimes at the end, to override definitions in the open 
file; and sometimes in the "middle" because that's the only way we could 
get it to work. :(

Also note that a number of the customizations are applied at configure 
time, through custom .m4 file hooks, not necessarily via the .gmk files. 
We use CUSTOM_CONFIG_DIR to allow customization of the configure process.

> It would be up
> to the customizer to provide that file and have it include whatever is
> needed in order to build the custom code.  For example, I could put
> "ORACLEJDK=" in there perhaps to turn off the Oracle extensions in the
> Makefile, as well as adding whatever other logic I need.  Oracle's
> version of make/closed/custom.gmk maybe just sets "ORACLEJDK=true" and
> that's it (for now, once the artifacts are removed from the Makefiles
> you wouldn't need ORACLEJDK at all).

OPENJDK/ORACLEJDK would be set/unset in the custom .m4 files for configure.

> My vision of this is that I'd like to get to the point where OpenJDK
> makefiles work fine as is, but will automatically pick up new logic and
> new code from make/closed and src/closed if those directories exist (and
> contain the "hook" files).  This gives custom code a nice place on the
> side where to live, with fewer worries about having to merge whenever
> anything in OpenJDK changes.  Of course it wouldn't solve all the
> problems, but it would be a very nice start.

I have been trying to facilitate such a scheme with numerous changes I 
have made to the build system over the past couple of years - working in 
conjunction with the build(-infra) team. The use of CUSTOM_MAKE_DIR etc 
was part of that

http://mail.openjdk.java.net/pipermail/build-infra-dev/2012-July/001142.html

and I've flagged the need for CUSTOM_SRC_DIR as well. Alas I could not 
address all the issues, nor move all the OracleJDK specific instructions 
to the "closed" side - but that is the longer term goal. I hoped what we 
had in place would at least enable some external customization.

David
-----

>
> On Thu, Apr 24, 2014 at 12:43 PM, <mark.reinhold at oracle.com
> <mailto:mark.reinhold at oracle.com>> wrote:
>
>     2014/4/22 21:23 -0700, kmcguigan at twitter.com
>     <mailto:kmcguigan at twitter.com>:
>      > Yes, I did consider using some ifeq tricks like that -- but they
>     are rather
>      > ugly and unreadable and have the same problem that you want to avoid:
>      > adding distribution-specific code into the open-source makefiles.
>      >
>      > My goal here is to have the public OpenJDK makefiles be in a
>     state such
>      > that custom distribution code can be added (in make/closed,
>     src/closed, or
>      > some such alternative location) without having to perform surgery
>     on the
>      > Makefiles and maintain the private changes.  The mechanism is
>     already in
>      > place,it's just some leftover OracleJDK that hasn't made it out
>     of the open
>      > makefiles yet.  If we could just cordon that off somehow, then
>     anyone could
>      > make a custom distribution by augmenting OpenJDK with 'closed' style
>      > repositories -- without having to maintain private, unrelated
>     edits to jdk
>      > Makefiles.
>
>     I'm confused.
>
>     Even if we replace `ifndef OPENJDK` with `ifdef ORACLEJDK`, how does
>     this help you?  In those cases where you want an open Makefile to refer
>     to code in Twitter's internal src/closed directory aren't you still
>     going to have to create and maintain your own patches to the open
>     Makefile?
>
>     Just trying to understand the problem here ...
>
>     - Mark
>
>
>
>
> --
>
> twitter-icon-large.png
>
> 	
>
> Keith McGuigan
>
> @kamggg
>
> kmcguigan at twitter.com <mailto:kmcguigan at twitter.com>
>

Tom Salter  |  Software Engineer  |  Java & Middleware Development 
Unisys  |  2476 Swedesford Road  |  Malvern, PA  19355   |  610-648-2568 |  N385-2568 