JDK8 Preliminary Repository Layout

[Mike Swingler]

On Mar 11, 2011, at 10:07 AM, Kelly O'Hair wrote:

> On Mar 11, 2011, at 2:11 AM, Steve Poole wrote:
> 
>> Kelly - can you explain for us newbies why you have separate repositories?  I'm sure I can list any number of reasons but it would be good to get your view.   It may sound like a dumb question but it does help in these sort of discussions to know some of the history :-)
> 
> This is probably tainted, but I will try and provide hopefully a honest view, with some humor thrown in. ;^)
> 
> Prior to Mercurial, we used the Sun product Teamware and we had separate workspaces (what Teamware called
> a repository) for: control, hotspot, j2se, deploy, install, etc. (deploy and install were Sun plugins & installers).
> So that set a pattern. Teamware basically managed SCCS files, so as a workspace grew, it did not scale well,
> and Teamware relied on NFS access to share these files (80,000+ files, when you count SCCS s.* and * files).
> So this separation initially, in my view, was done for developer productivity.
> I don't have any history on why the other workspaces existed as separate workspaces, but I just assume
> it was for the same reasons as hotspot, nobody wanted to part of the big j2se gorilla in the room,
> and having your own workspace created more of a separate silo for that team to work in I suspect.
> The control workspace was a small batch of makefiles that built all the workspaces, used by Release
> Engineering mostly.
> 
> Note that Teamware allowed for partial workspaces, since it was only managing SCCS and individual
> file edits, you could trim a Teamware workspace down to just the directories you were working in, and
> still sync and push with subset workspaces. This flexibility was taken advantage of by the j2se team
> to minimize the NFS traffic and improve productivity too. Mercurial doesn't allow for subset repositories.
> 
> The hotspot team found that their smaller 5,000 file workspace was easier to deal with, and in fact
> the VM was a natural interface boundary, easy to isolate, controlled APIs, pre-built VMs could be
> dropped into a JDK, testing/experiments were easy. Hotspot was also mostly C++ and native code.
> Later, a "Hotspot Express" delivery model was possible so that the same sources could be delivered
> to completely separate JDK releases.
> The hotspot developers were happy, well, as happy as a hotspot developer can be I suppose ;^)
> (The Serviceability Agent or SA was developed by the hotspot team and was/is very tightly integrated
> with hotspot, so it became part of hotspot, not the j2se).
> 
> The j2se workspace was much larger, maybe 35,000 source files, it initially included all the sources from the
> corba, jaxp. jaxws, and langtools repositories that exist now.
> This j2se workspace was very hard to deal with and many of the sources were copy&paste from other projects
> that weren't even managed by the JDK team, new deliveries created lost fix situations and an unreliable state.
> The build process was complicated because part of the workspace had the javac sources, which had to be built
> first, then that used to build the sources all over again.
> 
> So just prior to OpenJDK, or about then, we decided to try and split up the j2se workspace to better manage our build
> and source importing issues. The corba, jaxp, and jaxws workspaces were created and those files were pulled
> from the j2se workspace, as was the javac and "language tools" sources into a langtools workspace.
> The j2se workspace was then renamed "jdk".
> 
> That gave us the workspaces: corba, jaxp, jaxws, langtools, jdk, hotspot, ...
> 
> These Teamware workspaces eventually became what you see today as the openjdk7 Mercurial repositories,
> but we had to push some files down into smaller closed repositories: src/closed, test/closed, and make/closed
> for jdk, and src/closed, test/closed, and build/closed for hotspot. The fact that hotspot had managed sources
> in a build directory was a thorn in our sides for a while and it was eventually removed along with build/closed.
> Makefile logic is pretty much 100% open right now.
> 
> I'm not sure that the open sourcing influenced this, but note that corba, jaxp, jaxws, and langtools are pure
> open source, and 100% Java (except for one .c file in corba initially). Managing pure open Java projects is a
> joy if you ask me. ;^)
> 
> For langtools, the team wanted this separate repository and lobbied hard for it as a productivity aid and also to allow
> them to use the NetBeans IDE on just their sources (NetBeans and some IDEs had a hard time swallowing the entire
> j2se sources), but they also needed to try and ship a separate javac product somewhere, I forget the details.
> Maybe some work with some outside developers, Jonathan Gibbons would remember.
> I'm sure if you asked him, there is no way they would want back into a larger repository.
> 
> The corba sources haven't changed much since then, makefile changes and all native code has been removed.
> Originally, we wanted an ant script for a faster build and to allow for NetBeans/IDE use as it became pure
> Java. That hasn't happened. We keep thinking that these sources should be updated with newer Corba
> sources and use whatever build process the J2EE Corba team has. Not sure what the plans are here.
> 
> The jaxp and jaxws repositories got the source drop model and the sources originally managed were deleted
> in favor of source drops from these teams, where they manage the master sources for these products that also
> ship in other forms in other products. This is still a work in progress in terms of finding the best way to
> manage this. We need the sources (can't just get class/jar drops) so that we can build classes with -target 7,
> but changes really need to go through these teams so they can be managed properly.
> 
> Mercurial's changeset model and the need for "merge changesets" when two changesets were created from
> the same parent changeset is another aspect to this. Many teams that changed from a file based management
> system to Mercurial have encountered "merge mania", the NetBeans team ran into this.
> It's an issue with too many developers trying to push changes into a single large repository.
> You can't push a changeset into Mercurial unless you have done a pull and sync'd up with the latest changesets
> in the repository. If there are frequent pushes going on, either from too much activity or too many developers,
> someone may experience a:
>   hg push    # fails because you need to do a pull "too many heads message"
>   hg pull -u && hg merge && hg commit -m Merge    #  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new one
>   hg pull -u && hg merge && hg commit -m Merge    #  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new one
>   ...
> This is minimized by reducing the "fan in", smaller repositories, fewer developers pushing into the same
> repository, etc. Our team forests minimize this, and our separate repositories minimize this.
> Now some people might say this is a flaw in Mercurial, and I disagree.
> By having one "tip", and explicit merge changesets, the sources have a singular state, with one simple
> changeset ID, you know the state of all 20,000 files in the jdk repository.
> 
> Mercurial handles very large repositories very well in my opinion, tremendously fast when using local
> disk and not NFS file systems. So having Mercurial manage one repository of 50,000 files is not an issue,
> except needing the disk space.
> 
> Hope this helps and I wasn't too long winded.

This should go up on a wiki somewhere. I'd love to point folks interested in the macosx-port project to it, and the exact state will change over time as well.

Cheers,
Mike Swingler
Java Engineering
Apple Inc.