Auto-update of native application bundles

[Tobias Bley]

From Apples point of view it's only allowed to save any app specific preferences in ~/Library/Application Support/<ApplicationName>

Best regards,
Tobi



Am 18.06.2012 um 00:31 schrieb Igor Nekrestyanov <igor.nekrestyanov at oracle.com>:

> Hi,
>> 
>>  * In your proposal the "app-cast" system is used, where each update
>>    is released as a patch (i.e. the 'diff') to get from one version
>>    to the next. If an app was out in the wild running at version 2,
>>    and the latest release was actually version 5, then to upgrade we
>>    would download the v2->v3 patch, install it, then the v3->v4 patch
>>    install it, and then the v4->v5 patch and install that (correct me
>>    if I got this wrong).
>> 
> Sparkle appcast is simply update descriptor (kind of RSS) wth news about update availability.
> It is unrelated to the way update is shipped.
> 
> Moreover but default update is full image of application and can be installed on top of any version.
> Developer may choose to have delta updates if he wants to optimize but he is not required too.
> We are using Sparkle to AU JRE on Mac and it works fine.
>> 
>>  * In my POC I've gone for the "app-profile" approach, which
>>    basically defines the latest release of the system in it's
>>    entirety. It's up to each app out there to determine what has
>>    changed between it's version and the latest version and download
>>    what it needs (i.e. the 'diff'). To go from version 2 to version
>>    5, the app would compare the version of the JARs in its own
>>    app-profile against the new one, and download the ones that have
>>    changed. There is no download of the 'in-between' versions of 3
>>    and 4.
>> 
> Sparkle approach does not assume this either.
> Update description must have full image. If there is (optional) delta image it is used but otherwise full image is used.
> 
> IMHO, the biggest differences between appcast xml and app-profile.xml are:
>   * appcast is strictly update descriptor and app-profile.xml is application descriptor + versioning info
>   * app-profile.xml corresponds to subset of "item" element in the appcast
>   * appcast xml may have multiple items - e.g. define application updates for different platforms or several apps
>   * item in the appcast describes UPDATE, not application. Application is kind of blackbox and this is good.
>     I.e. no knowledge of classpath or specific directory structure. Only what driver knows how to apply update.
>   * item has update specific information, e.g. may include release notes that can be used to justify why to upgrade.
> 
> It does not mean we can not use efficient compression (e.g. pack200 for jars) or update parts of app package (e.g. leave alone Java runtime).
> This is up for driver implementation.
> 
> And another important difference is simplier model in Sparkle:
>    * ONE version to check (whats available vs update version in appcast)
>        [no need to check every jar, resource file, etc]
>    * ONE file with update to sign and stage (less hassle for developer)
>        [instead of every jar, fewer chance to have staging error (not all files are uploaded, etc)]
>    * ONE file to download
>        [less chance to have mixed state, e.g. developer may decide to restage/resign app with same version but some jars are already loaded]
> 
> JNLP model is ok but it tries to solve "remote launch" problem and here we are solving other task.
> 
> IMHO, if we separate launch descriptor and update descriptor we will have better model for this problem.
>> 
>> It was with this type of usage in mind that I ended up leaning towards the app-profile approach. I imagined my initial upload to my server would be all the JARs for my app, and then when I release subsequent versions I only upload the couple of JARs that have changed and upload the new app-profile.xml (and the build tools could potentially do the upload as well and they would know which jars had changed).
> Essentially you are implementing specific kind of "delta update".
> Except you can dynamically define what delta is needed for arbitrary initial state.
> 
> In sparkle model specifics of delta update are up to the update driver that knows what to do with update package.
> We can support update packages that only update "important" subset of jars, etc.
> 
> For background type of updates size is often not an issue at all (although there are scenarios where it needs to be optimized).
> 
> Anyway, with "update driver" plugins we can extend set of delta updates supported as needed.
> 
>> The flip side of this is that you couldn't choose to go to an in-between version if you wanted to, but I'm not sure that's an overly valid use case?
> Agree, this is not important use case.
> But you may have "different" latest versions - one for XP and another for Vista and another for Mac.
>> 
>> With this approach we also have the option of providing a "repair" function. So a client could run a utility to just completely download the entire app and get fresh copies of all the JARs. This is not critical but it could be quite useful. With the app-cast approach, you have to uninstall and then reinstall with your original MSI and then patch (or install with the new MSI) - the uninstall may delete your 'appdata' though, whereas a "repair" could leave this intact.
> IMHO, repair by reinstalling still needs to be supported and existing mechanisms (like MSI or rpm) have means to do this.
> It does not seem as critical requirements for AU module.
>> 
>> One other niggling thing for me is how to do things like system properties or VM settings (like -Xmx) without an app-profile. Currently you would be hard coding those into your 'exe' I imagine? So if an update wanted to change these things it would have to recompile the launcher exe and that would have to be part of the 'patch'?
> Bundled app has application descriptor/config file. It is called package.cfg and very simple now.
> It may and need to be improved to pass in things like JVM options.
> 
> In fact launcher is fixed, it is not compiled per app.
> It is embedded into ant task and simply renamed when application bundle is generated.
> All variable parameters are coming from package.cfg
>> There'd be no easy (possible?) way for the user to change these settings on their machine, not sure if that's a good thing or a bad thing though?
> Well, if we need it (it is questionable) then user may tweak package.cfg directly (imho, bad idea. especially if it is system wide) or we could have
> some abstraction of "config" API and application may update its settings per user. But this is unrelated to AU.
> User settings should be left intact after AU. Default app settings are part of application descriptor. So i think we are good with this.
>> 
>> And finally one other edge use case I thought of today: it's not uncommon to have several 'apps' included in the one install.  For example you might have a user UI and an admin UI. In Java-land, these would ideally share the same libraries but have different 'main' classes. I was thinking the app-profile approach could potentially be extended to cover this but it needs more thinking about, and may just be an unsupported use case. Not sure how/if the app-cast approach could handle this if we said it was a requirement.
> It does not seem hard to me.
> 
> Key question here -- are they shipped as separate installs or one bundle with multiple launchers.
> If they are separate installs they are updated separately and there should be application specific means to ensure they are consistent
> (because user may simply install incompatible versions to start with). Application specific solution may use AU APIs to force autoupdate
> if user tries to launch app that is compatible with others.
> 
> If they are shipped as one bundle then nothing special to be done by AU. AU will update folder with multiple launchers easily as they are just files for AU module.
> It is different story how to bundle msi/exe/dmg for this scenario but this is separate from AU discussion,
> 
> -igor
>> 
>> Cheers,
>> Dan
>> 
>> 
>> On Sun, Jun 17, 2012 at 5:26 PM, Igor Nekrestyanov <igor.nekrestyanov at oracle.com <mailto:igor.nekrestyanov at oracle.com>> wrote:
>> 
>>    Hi Dan,
>> 
>>    thanks for the great writeup (and pushing this ahead).
>>    I finally started to look into this more actively. Some
>>    questions/suggestions below.
>> 
>>    On 6/15/12 5:02 PM, Daniel Zwolenski wrote:
>> 
>>        *The General Approach*
>> 
>>        Richard made reference to
>>        Sparkle<http://sparkle.andymatuschak.org/>  and
>> 
>>        like him, I feel this is a nice reference point for the style
>>        of solution
>>        we want. The neat thing that Sparkle does is it provides an
>>        API that
>>        developers can use inside their code to completely control and
>>        customise
>>        the upgrade of their application. This is the approach I've
>>        gone for as
>>        well. My POC includes a Java API that you can call to check
>>        for updates and
>>        trigger downloads.
>> 
>>    IMHO, main great thing about Sparkle is that it is easy to add
>>    basic AU functionality to the application
>>    (see https://github.com/andymatuschak/Sparkle/wiki)
>>    You do not need to use API unless you want custom solution.
>> 
>>    Therefore i am not focusing here on the public API part (keeping
>>    this for later).
>> 
>> 
>>        I think this approach is neat and powerful, but it's worth
>>        mentioning that
>>        there is still some debate amongst the JFX team about whether
>>        to go down
>>        this road or avoid the Java API and use off-the-shelf native
>>        updaters. Your
>>        input will very likely influence this decision so if you have
>>        a preference
>>        one way or the other (or have other suggestions), be vocal.
>> 
>> 
>>    I think there is no debate on that having some "all in java with
>>    minimal native code solution" is good idea.
>>    Debate is about whether it should be the only option or it also
>>    make sense to support other native update mechanisms
>>    (e.g. use Sparkle directly on Mac).
>> 
>>    As Dan said, thoughts on this are very welcome.
>> 
>> 
>> 
>>        An XML deployment descriptor, called 'app-profile.xml' is used
>>        to describe
>>        the JARs, JRE versions and other launch details - it is
>>        semantically
>>        comparable to a traditional JNLP file. When an update is
>>        triggered via the
>>        API, the code checks the remote URL for an updated
>>        app-profile.xml. If it
>>        finds one it downloads this and checks this with its local
>>        app-profile.xml.
>>        Any jars that have different version numbers are downloaded,
>>        and if a new
>>        JRE version is defined in the app-profile then the zip for this is
>>        downloaded and extracted as well.
>> 
>>    Where these files are placed after download?
>> 
>>        Once all the downloading is complete, the
>>        local app-profile is then overwritten with the newer one and
>>        this newer one
>>        will be used by the native launcher to set the classpath next
>>        time the app
>>        is started.
>> 
>> 
>>    IMHO, we can simplify this model following Sparkle ideas.
>>    Sparkle does not know anything about application structure, etc.
>>    and focus only on update process.
>>    This makes it easy to use, simple  and robust as it focuses on
>>    solves one problem.
>> 
>>    Important points are:
>>       A. Sparkle uses "appcast" to describe what updates are available
>>                (https://github.com/andymatuschak/Sparkle/wiki/publishing-an-update)
>>           Appcast includes
>>             - version of updates available
>>             - meta info about each update (type of update (e.g.
>>    regular or mandatory security), release notes, system requirements
>>    (e.g. Windows version 7+), etc.)
>>             - set of available update packages (aka enclosures) (e.g.
>>    full app image, delta patches for several versions, etc.)
>> 
>>      B. Sparkle knows where to look for update meta info in the
>>    application package (on Mac it is part of Info.plist that is some
>>    kind of application manifest).
>>          However, it only needs to know:
>>             - URL of appcast
>>             - location of key to verify update signatures
>>             - application update settings/preferences
>>          The expectation is that once app is updated this info will
>>    be updated (as it is part of "update package")
>>          [Note: this is not so simple on Windows if we want to make
>>    sure we are good citizens and need to update registry]
>> 
>>      C. Update package is black box for Sparkle core.
>>          It is modeled as
>>             - file hosted at given URL
>>             - signed with given key (signature is part of appcast)
>>             - package type defines what "update driver" will be
>>    handling this update
>>                 (this allows to add support for various update
>>    packages as plugins/modules)
>>             - for most type for packages (except running installers)
>>    Sparkle goal is to update set of application files only
>>                [Windows or linux require more work as we may need to
>>    update registry or package info]
>> 
>>      D. If admin permissions are needed they only asked before update
>>    is finally installed.
>>          Sparkle does not keep any internal state in the app folder
>>    as it may be only writable for admins.
>> 
>>      E. Application relaunch after update
>> 
>>    This is close to your approach but there are some important
>>    differences.
>>    Following Sparkle more closely may simplify things and at the same
>>    time add some desired features
>>    (e.g. ability to validate origin of autoupdate)
>> 
>>    My current thoughts on "ideal" basic experience are as follows.
>> 
>>    ========
>>    Developer experience (no java coding needed):
>> 
>>     1. At the package time developer requests autoupdate support in
>>    the native cobundle by providing:
>>           * http URL to appcast + public key/cert to validate update
>>    signatures
>>           * or https url
>>        and define application update mode (mandatory update, ask user
>>    and allow him to skip version, etc.)
>> 
>>     2. Once update is needed developer creates "update package"
>>    (single file).
>>       For this he will need to run same packager utility/ant task but
>>    specify base image, certificate to sign package and
>>       and desired type of update (full or incremental)
>> 
>>     3. Developer need to create an appcast xml file (default can be
>>    generated in step 2 but he may want to customize it)
>> 
>>     4. Developer stages both appcast and package bundle
>>    =========
>> 
>>    End user (background updates):
>>      * installs application with background update policy
>>      * every time application is launched it will check appcast (if
>>    online)
>>      * if update is detected it will be loaded silently while app is
>>    running
>>      * Once update is loaded and verified user will be prompted to
>>    install it (it may be notification if update is mandatory)
>>      * If he agrees then he may see UAC dialog (unless installation
>>    is on per user basis)
>>      * application relaunch after update
>>      * attempt to launch app being updated should fail with
>>    reasonable dialog and should not fail update process (it may need
>>    to wait will .exe is unlocked)
>>    =========
>> 
>>    Implementation wise i was thinking along the lines of what you
>>    have (only glimpsed though your code it though) with following
>>    notable differences:
>> 
>>      * update package is single file and we only worry about it version.
>>        No need to worry about version of individual components as we
>>    do not want arbitrary mix&match.
>> 
>>      * updates are loaded to ~/.javafx/app-key (where app-key could
>>    be derivation from app install location)
>>        We also keep user answers for this app in that folder (e.g.
>>    had we offered him this version already? when was last time we
>>    checked for update, etc)
>> 
>>      * Update process is:
>>           - build temp full app image in the ~/.javafx/app-key/version
>>           - if update accepted then exit application and run helper
>>    app co copy image over
>>           - possibly run "post upgrade script" as part of helper app
>>               (this script is specific to "bundler" technology used.
>>    E.g. for EXE/MSI bundlers we may want to update few keys in registry
>>           - relaunch app
>>           - AU module running in the relaunched app could detect temp
>>    "image" that is the same as current version and erase it
>> 
>>         Note that there could be several alternative helper apps.
>>    E.g. one that pops UAC and one that does not for per user installs.
>>         Or helper up to install update packaged as MSI (by running
>>    msiexec silently and showing progress)
>> 
>>         I was even thinking that "helper" can be implemented in java
>>    with minimal native code (to elevate permissions and update
>>    registry if needed).
>> 
>>      * Integration with application bundle
>>          - details on whether AU is needed and what type is it may be
>>    embedded application manifest for main app jar
>>          - embedded jauncher will read them and use internal AU APIs
>>    accordingly
>>             (this is why no user code changes are needed)
>>          - current app version, update url and signature are part of
>>    package.cfg or manifest in the main app jar
>>          - AU support itself is part of jfxrt.jar, bundle also
>>    includes helper app to "move image into final location" and
>>              script to do "post update config"
>> 
>>    What do you think?
>> 
>>    -igor
>> 
>> 
>