Auto-update of native application bundles

[Igor Nekrestyanov]

Hi,
>
>   * In your proposal the "app-cast" system is used, where each update
>     is released as a patch (i.e. the 'diff') to get from one version
>     to the next. If an app was out in the wild running at version 2,
>     and the latest release was actually version 5, then to upgrade we
>     would download the v2->v3 patch, install it, then the v3->v4 patch
>     install it, and then the v4->v5 patch and install that (correct me
>     if I got this wrong).
>
Sparkle appcast is simply update descriptor (kind of RSS) wth news about 
update availability.
It is unrelated to the way update is shipped.

Moreover but default update is full image of application and can be 
installed on top of any version.
Developer may choose to have delta updates if he wants to optimize but 
he is not required too.
We are using Sparkle to AU JRE on Mac and it works fine.
>
>   * In my POC I've gone for the "app-profile" approach, which
>     basically defines the latest release of the system in it's
>     entirety. It's up to each app out there to determine what has
>     changed between it's version and the latest version and download
>     what it needs (i.e. the 'diff'). To go from version 2 to version
>     5, the app would compare the version of the JARs in its own
>     app-profile against the new one, and download the ones that have
>     changed. There is no download of the 'in-between' versions of 3
>     and 4.
>
Sparkle approach does not assume this either.
Update description must have full image. If there is (optional) delta 
image it is used but otherwise full image is used.

IMHO, the biggest differences between appcast xml and app-profile.xml are:
    * appcast is strictly update descriptor and app-profile.xml is 
application descriptor + versioning info
    * app-profile.xml corresponds to subset of "item" element in the appcast
    * appcast xml may have multiple items - e.g. define application 
updates for different platforms or several apps
    * item in the appcast describes UPDATE, not application. Application 
is kind of blackbox and this is good.
      I.e. no knowledge of classpath or specific directory structure. 
Only what driver knows how to apply update.
    * item has update specific information, e.g. may include release 
notes that can be used to justify why to upgrade.

It does not mean we can not use efficient compression (e.g. pack200 for 
jars) or update parts of app package (e.g. leave alone Java runtime).
This is up for driver implementation.

And another important difference is simplier model in Sparkle:
     * ONE version to check (whats available vs update version in appcast)
         [no need to check every jar, resource file, etc]
     * ONE file with update to sign and stage (less hassle for developer)
         [instead of every jar, fewer chance to have staging error (not 
all files are uploaded, etc)]
     * ONE file to download
         [less chance to have mixed state, e.g. developer may decide to 
restage/resign app with same version but some jars are already loaded]

JNLP model is ok but it tries to solve "remote launch" problem and here 
we are solving other task.

IMHO, if we separate launch descriptor and update descriptor we will 
have better model for this problem.
>
> It was with this type of usage in mind that I ended up leaning towards 
> the app-profile approach. I imagined my initial upload to my server 
> would be all the JARs for my app, and then when I release subsequent 
> versions I only upload the couple of JARs that have changed and upload 
> the new app-profile.xml (and the build tools could potentially do the 
> upload as well and they would know which jars had changed).
Essentially you are implementing specific kind of "delta update".
Except you can dynamically define what delta is needed for arbitrary 
initial state.

In sparkle model specifics of delta update are up to the update driver 
that knows what to do with update package.
We can support update packages that only update "important" subset of 
jars, etc.

For background type of updates size is often not an issue at all 
(although there are scenarios where it needs to be optimized).

Anyway, with "update driver" plugins we can extend set of delta updates 
supported as needed.

> The flip side of this is that you couldn't choose to go to an 
> in-between version if you wanted to, but I'm not sure that's an overly 
> valid use case?
Agree, this is not important use case.
But you may have "different" latest versions - one for XP and another 
for Vista and another for Mac.
>
> With this approach we also have the option of providing a "repair" 
> function. So a client could run a utility to just completely download 
> the entire app and get fresh copies of all the JARs. This is not 
> critical but it could be quite useful. With the app-cast approach, you 
> have to uninstall and then reinstall with your original MSI and then 
> patch (or install with the new MSI) - the uninstall may delete your 
> 'appdata' though, whereas a "repair" could leave this intact.
IMHO, repair by reinstalling still needs to be supported and existing 
mechanisms (like MSI or rpm) have means to do this.
It does not seem as critical requirements for AU module.
>
> One other niggling thing for me is how to do things like system 
> properties or VM settings (like -Xmx) without an app-profile. 
> Currently you would be hard coding those into your 'exe' I imagine? So 
> if an update wanted to change these things it would have to recompile 
> the launcher exe and that would have to be part of the 'patch'?
Bundled app has application descriptor/config file. It is called 
package.cfg and very simple now.
It may and need to be improved to pass in things like JVM options.

In fact launcher is fixed, it is not compiled per app.
It is embedded into ant task and simply renamed when application bundle 
is generated.
All variable parameters are coming from package.cfg
> There'd be no easy (possible?) way for the user to change these 
> settings on their machine, not sure if that's a good thing or a bad 
> thing though?
Well, if we need it (it is questionable) then user may tweak package.cfg 
directly (imho, bad idea. especially if it is system wide) or we could have
some abstraction of "config" API and application may update its settings 
per user. But this is unrelated to AU.
User settings should be left intact after AU. Default app settings are 
part of application descriptor. So i think we are good with this.
>
> And finally one other edge use case I thought of today: it's not 
> uncommon to have several 'apps' included in the one install.  For 
> example you might have a user UI and an admin UI. In Java-land, these 
> would ideally share the same libraries but have different 'main' 
> classes. I was thinking the app-profile approach could potentially be 
> extended to cover this but it needs more thinking about, and may just 
> be an unsupported use case. Not sure how/if the app-cast approach 
> could handle this if we said it was a requirement.
It does not seem hard to me.

Key question here -- are they shipped as separate installs or one bundle 
with multiple launchers.
If they are separate installs they are updated separately and there 
should be application specific means to ensure they are consistent
(because user may simply install incompatible versions to start with). 
Application specific solution may use AU APIs to force autoupdate
if user tries to launch app that is compatible with others.

If they are shipped as one bundle then nothing special to be done by AU. 
AU will update folder with multiple launchers easily as they are just 
files for AU module.
It is different story how to bundle msi/exe/dmg for this scenario but 
this is separate from AU discussion,

-igor
>
> Cheers,
> Dan
>
>
> On Sun, Jun 17, 2012 at 5:26 PM, Igor Nekrestyanov 
> <igor.nekrestyanov at oracle.com <mailto:igor.nekrestyanov at oracle.com>> 
> wrote:
>
>     Hi Dan,
>
>     thanks for the great writeup (and pushing this ahead).
>     I finally started to look into this more actively. Some
>     questions/suggestions below.
>
>     On 6/15/12 5:02 PM, Daniel Zwolenski wrote:
>
>         *The General Approach*
>
>         Richard made reference to
>         Sparkle<http://sparkle.andymatuschak.org/>  and
>
>         like him, I feel this is a nice reference point for the style
>         of solution
>         we want. The neat thing that Sparkle does is it provides an
>         API that
>         developers can use inside their code to completely control and
>         customise
>         the upgrade of their application. This is the approach I've
>         gone for as
>         well. My POC includes a Java API that you can call to check
>         for updates and
>         trigger downloads.
>
>     IMHO, main great thing about Sparkle is that it is easy to add
>     basic AU functionality to the application
>     (see https://github.com/andymatuschak/Sparkle/wiki)
>     You do not need to use API unless you want custom solution.
>
>     Therefore i am not focusing here on the public API part (keeping
>     this for later).
>
>
>         I think this approach is neat and powerful, but it's worth
>         mentioning that
>         there is still some debate amongst the JFX team about whether
>         to go down
>         this road or avoid the Java API and use off-the-shelf native
>         updaters. Your
>         input will very likely influence this decision so if you have
>         a preference
>         one way or the other (or have other suggestions), be vocal.
>
>
>     I think there is no debate on that having some "all in java with
>     minimal native code solution" is good idea.
>     Debate is about whether it should be the only option or it also
>     make sense to support other native update mechanisms
>     (e.g. use Sparkle directly on Mac).
>
>     As Dan said, thoughts on this are very welcome.
>
>
>
>         An XML deployment descriptor, called 'app-profile.xml' is used
>         to describe
>         the JARs, JRE versions and other launch details - it is
>         semantically
>         comparable to a traditional JNLP file. When an update is
>         triggered via the
>         API, the code checks the remote URL for an updated
>         app-profile.xml. If it
>         finds one it downloads this and checks this with its local
>         app-profile.xml.
>         Any jars that have different version numbers are downloaded,
>         and if a new
>         JRE version is defined in the app-profile then the zip for this is
>         downloaded and extracted as well.
>
>     Where these files are placed after download?
>
>         Once all the downloading is complete, the
>         local app-profile is then overwritten with the newer one and
>         this newer one
>         will be used by the native launcher to set the classpath next
>         time the app
>         is started.
>
>
>     IMHO, we can simplify this model following Sparkle ideas.
>     Sparkle does not know anything about application structure, etc.
>     and focus only on update process.
>     This makes it easy to use, simple  and robust as it focuses on
>     solves one problem.
>
>     Important points are:
>        A. Sparkle uses "appcast" to describe what updates are available
>            
>      (https://github.com/andymatuschak/Sparkle/wiki/publishing-an-update)
>            Appcast includes
>              - version of updates available
>              - meta info about each update (type of update (e.g.
>     regular or mandatory security), release notes, system requirements
>     (e.g. Windows version 7+), etc.)
>              - set of available update packages (aka enclosures) (e.g.
>     full app image, delta patches for several versions, etc.)
>
>       B. Sparkle knows where to look for update meta info in the
>     application package (on Mac it is part of Info.plist that is some
>     kind of application manifest).
>           However, it only needs to know:
>              - URL of appcast
>              - location of key to verify update signatures
>              - application update settings/preferences
>           The expectation is that once app is updated this info will
>     be updated (as it is part of "update package")
>           [Note: this is not so simple on Windows if we want to make
>     sure we are good citizens and need to update registry]
>
>       C. Update package is black box for Sparkle core.
>           It is modeled as
>              - file hosted at given URL
>              - signed with given key (signature is part of appcast)
>              - package type defines what "update driver" will be
>     handling this update
>                  (this allows to add support for various update
>     packages as plugins/modules)
>              - for most type for packages (except running installers)
>     Sparkle goal is to update set of application files only
>                 [Windows or linux require more work as we may need to
>     update registry or package info]
>
>       D. If admin permissions are needed they only asked before update
>     is finally installed.
>           Sparkle does not keep any internal state in the app folder
>     as it may be only writable for admins.
>
>       E. Application relaunch after update
>
>     This is close to your approach but there are some important
>     differences.
>     Following Sparkle more closely may simplify things and at the same
>     time add some desired features
>     (e.g. ability to validate origin of autoupdate)
>
>     My current thoughts on "ideal" basic experience are as follows.
>
>     ========
>     Developer experience (no java coding needed):
>
>      1. At the package time developer requests autoupdate support in
>     the native cobundle by providing:
>            * http URL to appcast + public key/cert to validate update
>     signatures
>            * or https url
>         and define application update mode (mandatory update, ask user
>     and allow him to skip version, etc.)
>
>      2. Once update is needed developer creates "update package"
>     (single file).
>        For this he will need to run same packager utility/ant task but
>     specify base image, certificate to sign package and
>        and desired type of update (full or incremental)
>
>      3. Developer need to create an appcast xml file (default can be
>     generated in step 2 but he may want to customize it)
>
>      4. Developer stages both appcast and package bundle
>     =========
>
>     End user (background updates):
>       * installs application with background update policy
>       * every time application is launched it will check appcast (if
>     online)
>       * if update is detected it will be loaded silently while app is
>     running
>       * Once update is loaded and verified user will be prompted to
>     install it (it may be notification if update is mandatory)
>       * If he agrees then he may see UAC dialog (unless installation
>     is on per user basis)
>       * application relaunch after update
>       * attempt to launch app being updated should fail with
>     reasonable dialog and should not fail update process (it may need
>     to wait will .exe is unlocked)
>     =========
>
>     Implementation wise i was thinking along the lines of what you
>     have (only glimpsed though your code it though) with following
>     notable differences:
>
>       * update package is single file and we only worry about it version.
>         No need to worry about version of individual components as we
>     do not want arbitrary mix&match.
>
>       * updates are loaded to ~/.javafx/app-key (where app-key could
>     be derivation from app install location)
>         We also keep user answers for this app in that folder (e.g.
>     had we offered him this version already? when was last time we
>     checked for update, etc)
>
>       * Update process is:
>            - build temp full app image in the ~/.javafx/app-key/version
>            - if update accepted then exit application and run helper
>     app co copy image over
>            - possibly run "post upgrade script" as part of helper app
>                (this script is specific to "bundler" technology used.
>     E.g. for EXE/MSI bundlers we may want to update few keys in registry
>            - relaunch app
>            - AU module running in the relaunched app could detect temp
>     "image" that is the same as current version and erase it
>
>          Note that there could be several alternative helper apps.
>     E.g. one that pops UAC and one that does not for per user installs.
>          Or helper up to install update packaged as MSI (by running
>     msiexec silently and showing progress)
>
>          I was even thinking that "helper" can be implemented in java
>     with minimal native code (to elevate permissions and update
>     registry if needed).
>
>       * Integration with application bundle
>           - details on whether AU is needed and what type is it may be
>     embedded application manifest for main app jar
>           - embedded jauncher will read them and use internal AU APIs
>     accordingly
>              (this is why no user code changes are needed)
>           - current app version, update url and signature are part of
>     package.cfg or manifest in the main app jar
>           - AU support itself is part of jfxrt.jar, bundle also
>     includes helper app to "move image into final location" and
>               script to do "post update config"
>
>     What do you think?
>
>     -igor
>
>