Module-file format (DRAFT)

[Alex Buckley]

- Does "length of filename" mean a) number of bytes in the 'name' item, 
or b) the number of Unicode code points in the 'name' item ?

- Filenames can include path separators, implying relative filenames. 
What are they relative to, or does it not matter?

- A top-level ModuleFile structure would contain ModuleFileHeader + 
ModuleInfoSection + an array of section objects + HashSection. A count 
of the number of section objects is useful as a minimal well-formedness 
check.

- In ModuleFileHeader, it would be super-smart to add a 'u2 
meta_version' item that indicates the version scheme used for 'major' 
and 'minor'. This allows corrections to the semantics of major/minor 
over time.

- Is a u4 really enough for every size item?

Alex

Mark Reinhold wrote:
> Here's a first cut at a simple module-file format.
> 
> Comments welcome!
> 
> - Mark
> 
> ----
> 
> Jigsaw module-file format DRAFT
> ===============================
> 
> Mark Reinhold <mr at sun.com>  
> 15 January 2010
> 
> 
> Goals
> -----
> 
>   - Optimized for streamed reading, from beginning to end.  Random access
>     should not be required when reading a module file, though it can be
>     required when writing.
> 
>   - Content-specific compression: Pack200+gzip/lzma for classes, bzip2 for
>     native code, etc.
> 
>   - Independent of any specific installed-module format or target-filesystem
>     capability.
> 
> 
> Layout
> ------
> 
> A module file has the following sections, in the order listed.
> 
> The first two sections are required:
> 
>   - Header -- Magic number, format version, sizes
>   - module-info.class (never compressed, so as to be easily readable by tools)
> 
> These sections are optional:
> 
>   - Classes (excluding module-info.class)
>   - Resources
>   - Native libraries
>   - Native launchers
>   - Configuration (i.e., properties) files
> 
> The final section is required:
> 
>   - Secure hash of the entire file (except for this section)
> 
> 
> Module file header
> ------------------
> 
>     ModuleFileHeader {
>       u4 magic;                 // FileConstants.MAGIC
>       u2 type;                  // FileConstants.Type.MODULE_FILE
>       u2 major;                 // FileConstants.ModuleFile.MAJOR_VERSION
>       u2 minor;                 // FileConstants.ModuleFile.MINOR_VERSION
>       u4 csize;                 // Size of entire file, compressed
>       u4 usize;                 // Space required for uncompressed contents
>                                 //   (upper bound; need not be exact)
>     }
> 
> All integers are in network byte order.
> 
> 
> Section headers
> ---------------
> 
> Each section has a header:
> 
>     SectionHeader {
>       u2 type;                  // One of FileConstants.ModuleFile.SectionType
>       u2 compressor;            // One of FileConstants.ModuleFile.Compressor
>     }
> 
> Compression is only relevant to section content; it never applies to the
> headers defined here.
> 
> For sections that contain only one entity (i.e., module-info, classes, hash)
> the section header is followed by a size pair, which is then followed by the
> content:
> 
>     SectionSize {
>       u4 csize;                 // Size of section content, compressed
>       u4 usize;                 // Size of section content, uncompressed
>     }
> 
> If a section contains named files (i.e., resources, native code, config) then
> each file within that section has a header, which is then followed by the
> content:
> 
>     SectionFileHeader {
>       u4 csize;                 // Size of file, compressed
>       u4 usize;                 // Size of file, uncompressed
>       u2 nameLength;            // Length of file name
>       b* name;                  // File name, in UTF-8
>     }
> 
> File names may include directory separators, though they should not be used for
> native-code files.
> 
> There is no mode field on files within a section.  The module-installation code
> must set the local mode appropriately; e.g., it must make native-code files
> executable as appropriate.
> 
> A hash section has the content:
> 
>     HashSection {
>       u2 type;                  // One of FileConstants.ModuleFile.HashType
>       u2 size;                  // Never compressed
>       b* hash;
>     }
> 
> 
> Open issues
> -----------
> 
>   - Signing: Should module files carry their own digital-signature information,
>     as jar files do today, or should such information be provided elsewhere?