JDK-8025705

[David Holmes]

Hi Tom,

On 30/04/2014 12:39 AM, Salter, Thomas A wrote:
> I have a different issue with closed and custom source that maybe can be addressed while you're looking at this area.  As a licensee, I build with at least some of the closed source, but I also need to customize the build.  Ideally, I'd like to be able to insert my custom make files and still have the closed files used.  The solution might be as simple as having a configure definition for the location of my custom files.  Chaining to the closed files would be my responsibility (though I'm not sure how I would change the closed files, if I needed to do that.)

Probably not a discussion for an OpenJDK list but there is only a single 
notion of a "custom" file location, and that is "closed" in your case. 
So if you want to further customize you would need to do it within that 
closed directory structure.

David

> For JDK8, we took the path of least resistance and just modified the make files in much the same way we had for earlier releases.
>
> Tom Salter
>
> -------------------------------------------------------------
>
> Date: Tue, 29 Apr 2014 14:52:32 +1000
> From: David Holmes <david.holmes at oracle.com>
> To: Keith McGuigan <kmcguigan at twitter.com>, mark.reinhold at oracle.com
> Cc: build-dev <build-dev at openjdk.java.net>
> Subject: Re: JDK-8025705
> Message-ID: <535F3010.5070607 at oracle.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Keith,
>
> As requested via email I'm responding on your response to Mark.
>
> On 25/04/2014 8:10 AM, Keith McGuigan wrote:
>> Hi Mark,
>>
>> Well first off, the existence of src/closed triggers OPENJDK to be
>> unset, so even having such directories (or subrepos) present turns on
>> all the logic that looks for files in src/closed which probably don't
>> exist in a non-Oracle distribution.
>
> Yes that is the default logic from configure. It's not perfect but it
> sets OPENJDK correctly for the majority of people that only have the
> OpenJDK sources.
>
> As I said early on part of the problem you have is that you chose to use
> src/closed for your own customizations and that conflicts with the
> far-from-perfect use of src/closed existence checks. Obviously "we"
> don't have exclusive rights to that name, but it doesn't seem a prudent
> choice given the current imperfect separation.
>
> That aside configure supports building an OpenJDK configuration,
> --enable-openjdk-only, to allow for building the OpenJDK even when you
> also have custom sources present**. That said I think some recent
> changes regarding licensee builds may have broken that somewhat as any
> existence check for src/closed outside of "ifndef OPENJDK" would trigger.
>
> ** In that logic everywhere you see "closed" it should read "custom". :(
>
> Building "OPENJDK" does have other side-effects as you have mentioned
> elsewhere, specifically the version information used. But presumably you
> are customizing that information anyway. Plus if the code were changed
> to use !ORACLEJDK you'd still get the default OpenJDK version
> information. So it seems to me that version string customization is
> something you will need to manage explicitly either way.
>
>> So having an ORACLEJDK test instead
>> around that code means we can control it with a single switch somewhere
>> in a top-level Makefile (maybe even a makefile in make/closed). This
>> would let us use the src/closed mechanism and at the same time unset
>> ORACLEJDK (OPENJDK would also be unset).
>
> I'm not convinced this really makes a significant difference to things,
> and I am convinced it doesn't help us along the path to a proper
> separation. But if you and others think differently I'm not going to
> waste more energy fighting it.
>
>> If we need to have a private modification to futz with ORACLEJDK (to
>> turn it off), that's not so bad -- it'd be just in one place so
>> maintenance would be easier.  But it's possible that we can put this
>> logic in a make/closed file that gets conditionally included, in which
>> case we wouldn't even have to worry about that.
>>
>> As to how to get custom code compiled, I think we have a couple options
>> to try.  In hotspot the code is just magically picked up and treated as
>> if it were sitting in the non-closed directory, augmenting or overriding
>> the open sources (depending on the filename).
>
> To the best of my understanding this is all handled with the lists of
> src dirs to build. If you add a custom src dir then you can build those
> custom sources. If you add a custom src dir and remove the equivalent
> open src dir, then you can effectively override. Of course in a custom
> makefile you can do whatever you like to pick and choose files to
> compile etc.
>
>> If we could do something
>> like that in the jdk makefiles, that would be ideal (IMO).  But it may
>> not be feasible to do this -- I'd have to investigate.  Another,
>> probably more realistic alternative is to add a couple generic hooks (as
>> few as possible) in the open source to make/closed makefiles which may
>> or may not exist depending on which distribution you are building.
>>    Something like:
>> -include make/closed/custom.gmk
>> somewhere in a top-level Makefile which would include custom makefile if
>> it exists, and silently skip over it if it is not there.
>
> Such logic exists already in many places in the build system. But we did
> not pro-actively add a custom include for every single file; rather we
> have added the includes as needed - knowing that this may need expansion
> in the future, but being content to wait until there was something that
> would actually test the use of the new "hook". That said (and I have
> said this a few times over the past couple of years) the hook placement
> may not be flexible enough nor suit all possible customizations.
> Sometimes we need to include at the start to control the logic in the
> open file; sometimes at the end, to override definitions in the open
> file; and sometimes in the "middle" because that's the only way we could
> get it to work. :(
>
> Also note that a number of the customizations are applied at configure
> time, through custom .m4 file hooks, not necessarily via the .gmk files.
> We use CUSTOM_CONFIG_DIR to allow customization of the configure process.
>
>> It would be up
>> to the customizer to provide that file and have it include whatever is
>> needed in order to build the custom code.  For example, I could put
>> "ORACLEJDK=" in there perhaps to turn off the Oracle extensions in the
>> Makefile, as well as adding whatever other logic I need.  Oracle's
>> version of make/closed/custom.gmk maybe just sets "ORACLEJDK=true" and
>> that's it (for now, once the artifacts are removed from the Makefiles
>> you wouldn't need ORACLEJDK at all).
>
> OPENJDK/ORACLEJDK would be set/unset in the custom .m4 files for configure.
>
>> My vision of this is that I'd like to get to the point where OpenJDK
>> makefiles work fine as is, but will automatically pick up new logic and
>> new code from make/closed and src/closed if those directories exist (and
>> contain the "hook" files).  This gives custom code a nice place on the
>> side where to live, with fewer worries about having to merge whenever
>> anything in OpenJDK changes.  Of course it wouldn't solve all the
>> problems, but it would be a very nice start.
>
> I have been trying to facilitate such a scheme with numerous changes I
> have made to the build system over the past couple of years - working in
> conjunction with the build(-infra) team. The use of CUSTOM_MAKE_DIR etc
> was part of that
>
> http://mail.openjdk.java.net/pipermail/build-infra-dev/2012-July/001142.html
>
> and I've flagged the need for CUSTOM_SRC_DIR as well. Alas I could not
> address all the issues, nor move all the OracleJDK specific instructions
> to the "closed" side - but that is the longer term goal. I hoped what we
> had in place would at least enable some external customization.
>
> David
> -----
>
>>
>> On Thu, Apr 24, 2014 at 12:43 PM, <mark.reinhold at oracle.com
>> <mailto:mark.reinhold at oracle.com>> wrote:
>>
>>      2014/4/22 21:23 -0700, kmcguigan at twitter.com
>>      <mailto:kmcguigan at twitter.com>:
>>       > Yes, I did consider using some ifeq tricks like that -- but they
>>      are rather
>>       > ugly and unreadable and have the same problem that you want to avoid:
>>       > adding distribution-specific code into the open-source makefiles.
>>       >
>>       > My goal here is to have the public OpenJDK makefiles be in a
>>      state such
>>       > that custom distribution code can be added (in make/closed,
>>      src/closed, or
>>       > some such alternative location) without having to perform surgery
>>      on the
>>       > Makefiles and maintain the private changes.  The mechanism is
>>      already in
>>       > place,it's just some leftover OracleJDK that hasn't made it out
>>      of the open
>>       > makefiles yet.  If we could just cordon that off somehow, then
>>      anyone could
>>       > make a custom distribution by augmenting OpenJDK with 'closed' style
>>       > repositories -- without having to maintain private, unrelated
>>      edits to jdk
>>       > Makefiles.
>>
>>      I'm confused.
>>
>>      Even if we replace `ifndef OPENJDK` with `ifdef ORACLEJDK`, how does
>>      this help you?  In those cases where you want an open Makefile to refer
>>      to code in Twitter's internal src/closed directory aren't you still
>>      going to have to create and maintain your own patches to the open
>>      Makefile?
>>
>>      Just trying to understand the problem here ...
>>
>>      - Mark
>>
>>
>>
>>
>> --
>>
>> twitter-icon-large.png
>>
>> 	
>>
>> Keith McGuigan
>>
>> @kamggg
>>
>> kmcguigan at twitter.com <mailto:kmcguigan at twitter.com>
>>
>
>
> Tom Salter  |  Software Engineer  |  Java & Middleware Development
> Unisys  |  2476 Swedesford Road  |  Malvern, PA  19355   |  610-648-2568 |  N385-2568
>
>
>