container format for jigsaw modules

[Dalibor Topic]

Hi,

Back at JavaOne, one of the questions raised in the Jigsaw BOF was
what deployment format makes sense on platforms that aren't lucky
enough to have their own, native package management systems.

The basic way jpkg works is take a compiled module as input,
along with associated resource data, and other files that go into
a module's deployment unit, typically a native package. For systems
that don't have a native deployment unit format, we need something
that lets us 

* store a module's metadata
* store a module's binary representation
* store a module's resources 
* store other files in a module's deployment unit (docs, etc.)

So, a container format, basically, that needs to provide support for
metadata and different kinds of (file-based) payload in a hierarchical
structure.

Desirable properties of such a container format would be

* extractable by open source tools

  - Well, obviously.

* cross-platform (solaris, windows, bsd, linux, ...)

  - Another obvious one.

* ability to create archives on one platform and extract them on another

  - And another obvious one (and not the same thing as above).

* immediate access to metadata

  - I.e. being able to fetch and decode metadata for a jigsaw module,
  while downloading its payload. I.e not having to pull many different 
  small pieces of metadata to figure out where the payload is, and then 
  having to open another connection to download it separately.

* random access to elements of payload

  - I.e. not having to extract all of a module's contents in order to
  get at a single file (the tar effect). Useful for installation of 
  just the module's binary representation and its resources, without
  other files in a module's deployment unit.

* compression of elements of payload

  - I.e. not having to uncompress all of the payload to get at elements
  of the payload (the tar.bz2 effect)

* compression of metadata

  - I.e. metadata isn't necessarily always going to be tiny, so being
  able to squeeze it down is a good thing.

* support for different compression schemes

  - I.e. being able to pick different compression schemes for different
  elements of the payload to trade size vs. speed, from no compression
  to gzip, bzip2 and lzma.

* hashing of metadata

  - I.e. being able to detect damaged metadata.

* hashing of elements of payload 

  - I.e. being able to detect damaged payload.

* support for different hashing schemes

  - I.e. being able to pick a different hashing scheme for different
  deployment scenarios to trade size vs. speed.

* support for files with interesting names (long, unusual characters)

  - I.e. being able to archive and transfer files with long names 
  (again, tar stands out as ugly there), and with local, non-US characters
  in their names (a source of much joy with the zip format)

* support for files of arbitrary sizes (4GB and above)

  - I.e. while I hope that 4GB should be enough for anyone's jigsaw modules,
  there is probably going to be something, somewhere in a resource file that's
  going to someday push that limit. A bad example here is zip, again.

* structured, extensible format for metadata

  - I.e. something that can be parsed easily, allows metadata to be in any
  language, and allows metadata to be extended to (for example) index all 
  exported classes and the modules that provide them.

* ability to carry multiple modules in a single file

  - I.e. not being forced to have a 1:1 correspondence between modules 
  (i.e. the logical unit of abstraction during development) and corresponding
  file representation (i.e. the unit of delivery). For example, some modules
  may only be 'virtual' to fit into corresponding nodes in the metadata graph
  for a set of modules - they shouldn't have to be tranferred separately from
  the rest of modules that belong to their metadata subgraph. In other words -
  make it possible and easy to take a set of modules and wrap them in a single
  file for transfer, along with their metadata.

More ideas and thoughts on this are welcome, of course.

I've looked around a bit, and a format that looks like it may fit reasonably well 
to the points above is the XAR format used in Mac OS X 10.5 and later, underpinning
their pkg format for installation packages. It's cross-platform, open-source, etc.
and available for your favorite distribution (including Cygwin ;). [1] 

So, for the container format for jigsaw modules, I'm currently thinking about 
using xar files with a .jxr extension rather then .xar to differentiate them
from regular xar archives. As there are multiple claims to the .xar
extension, while I've seen no one lay claim to .jxr ...



[1] http://en.wikipedia.org/wiki/Xar_%28archiver%29

-- 
*******************************************************************
Dalibor Topic                   Tel: (+49 40) 23 646 738
Java F/OSS Ambassador           AIM: robiladonaim
Sun Microsystems GmbH           Mobile: (+49 177) 2664 192
Nagelsweg 55                    http://openjdk.java.net
D-20097 Hamburg                 mailto:Dalibor.Topic at sun.com
Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten
Amtsgericht München: HRB 161028
Geschäftsführer: Thomas Schröder, Wolfgang Engels, Wolf Frenkel
Vorsitzender des Aufsichtsrates: Martin Häring