Auto-update of native application bundles

[Daniel Zwolenski]

Hey Igor,

Generally liking the discussion. It's a big topic with lots of twists and
turns so for now I'm just going to pull out one of your key points to delve
into further, as I think a lot of other things will be dependent on this.

Currently we have two proposed styles of app updates:

   - In your proposal the "app-cast" system is used, where each update is
   released as a patch (i.e. the 'diff') to get from one version to the next.
   If an app was out in the wild running at version 2, and the latest release
   was actually version 5, then to upgrade we would download the v2->v3 patch,
   install it, then the v3->v4 patch install it, and then the v4->v5 patch and
   install that (correct me if I got this wrong).

   - In my POC I've gone for the "app-profile" approach, which basically
   defines the latest release of the system in it's entirety. It's up to each
   app out there to determine what has changed between it's version and the
   latest version and download what it needs (i.e. the 'diff'). To go from
   version 2 to version 5, the app would compare the version of the JARs in
   its own app-profile against the new one, and download the ones that have
   changed. There is no download of the 'in-between' versions of 3 and 4.

I think both options have merit, each with its own advantages and
disadvantages. I looked at the app-cast approach first and decided the
app-profile worked better for the Java scenario than the app-cast one
(primarily because we have JARs and these work well with app-profile), but
I'm definitely good to look at this again.

One thing it would be good to work out up front however is: what does the
typical (80% case) upgrade look like? It would be great to get community
input on this guys?

For me, I will typically have one or two JARs that contain my code, then I
will have another 20 or so jars making up all my third-party libraries. The
one or two JARs containing my code will be the things that change most
between releases of my product. I might occasionally add, remove, or
upgrade a third-party dependency but this would be much less frequent. For
example, my app might have monthly releases with my latest code, but it
would be only once every year or two that I change the Hibernate or Spring
versions I'm using.

It was with this type of usage in mind that I ended up leaning towards the
app-profile approach. I imagined my initial upload to my server would be
all the JARs for my app, and then when I release subsequent versions I only
upload the couple of JARs that have changed and upload the new
app-profile.xml (and the build tools could potentially do the upload as
well and they would know which jars had changed).

So looking at the benefits of each approach in turn:

The big benefit of the app-cast approach that I can see is the single file
encapsulating the update. If lots of dependencies have changed then
downloading one file will be quicker than downloading lots of individual
JARs. Additionally, for the developer, uploading a single file with just
the changes in it is quicker and easier than uploading the full updated
application.

The complexity around signing the update file looks a little easier too,
but I think tools would mask the complexity in this area in either case so
the developer won't be overly affected (?). Similarly, there is less
complexity in checking new version numbers, but I think this complexity is
all in our code so its not really a big win for end users/developers.
Developers would still be maintaining versions in their local environment
in order to produce the 'patch' file (I assume - or how do you see the
contents of the patch file being determined?).

Did I miss any major advantages of the app-cast approach, or am I off on
any of my assessments here?


With the app-profile approach I think the main benefit is that the latest,
working code is available in full from the server. There is no need to
store patch files for older versions. These would be fairly hard to
regenerate if you ever lost them I imagine as you may not even have the
source code to reproduce them (this would be my biggest concern with the
app-cast approach)? In the app-profile approach the only thing you store on
your server is the latest MSI and the latest extracted copy of your app -
all of which can be easily pulled by doing a build out of your latest
source code tree. You don't have to keep around old patches just in case
someone out there is still using an old version.

Another benefit is that the updating app would just jump straight to the
latest version, and there would be no need of the "in-between" downloads
and installs. In my typical deployment scenario it is the same few JARs
that are being updated in each release, so the in-between downloads are
redundant as they immediately get overwritten (e.g. myapp.jar is
overwritten in v3, v4 and again in v5 so why not just jump straight to the
v5 file). The flip side of this is that you couldn't choose to go to an
in-between version if you wanted to, but I'm not sure that's an overly
valid use case?

With this approach we also have the option of providing a "repair"
function. So a client could run a utility to just completely download the
entire app and get fresh copies of all the JARs. This is not critical but
it could be quite useful. With the app-cast approach, you have to uninstall
and then reinstall with your original MSI and then patch (or install with
the new MSI) - the uninstall may delete your 'appdata' though, whereas a
"repair" could leave this intact.

One other niggling thing for me is how to do things like system properties
or VM settings (like -Xmx) without an app-profile. Currently you would be
hard coding those into your 'exe' I imagine? So if an update wanted to
change these things it would have to recompile the launcher exe and that
would have to be part of the 'patch'? There'd be no easy (possible?) way
for the user to change these settings on their machine, not sure if that's
a good thing or a bad thing though?

And finally one other edge use case I thought of today: it's not uncommon
to have several 'apps' included in the one install.  For example you might
have a user UI and an admin UI. In Java-land, these would ideally share the
same libraries but have different 'main' classes. I was thinking the
app-profile approach could potentially be extended to cover this but it
needs more thinking about, and may just be an unsupported use case. Not
sure how/if the app-cast approach could handle this if we said it was a
requirement.

Cheers,
Dan


On Sun, Jun 17, 2012 at 5:26 PM, Igor Nekrestyanov <
igor.nekrestyanov at oracle.com> wrote:

> Hi Dan,
>
> thanks for the great writeup (and pushing this ahead).
> I finally started to look into this more actively. Some
> questions/suggestions below.
>
> On 6/15/12 5:02 PM, Daniel Zwolenski wrote:
>
>> *The General Approach*
>>
>> Richard made reference to Sparkle<http://sparkle.**andymatuschak.org/<http://sparkle.andymatuschak.org/>>
>>  and
>>
>> like him, I feel this is a nice reference point for the style of solution
>> we want. The neat thing that Sparkle does is it provides an API that
>> developers can use inside their code to completely control and customise
>> the upgrade of their application. This is the approach I've gone for as
>> well. My POC includes a Java API that you can call to check for updates
>> and
>> trigger downloads.
>>
> IMHO, main great thing about Sparkle is that it is easy to add basic AU
> functionality to the application
> (see https://github.com/**andymatuschak/Sparkle/wiki<https://github.com/andymatuschak/Sparkle/wiki>
> )
> You do not need to use API unless you want custom solution.
>
> Therefore i am not focusing here on the public API part (keeping this for
> later).
>
>
>  I think this approach is neat and powerful, but it's worth mentioning that
>> there is still some debate amongst the JFX team about whether to go down
>> this road or avoid the Java API and use off-the-shelf native updaters.
>> Your
>> input will very likely influence this decision so if you have a preference
>> one way or the other (or have other suggestions), be vocal.
>>
>
> I think there is no debate on that having some "all in java with minimal
> native code solution" is good idea.
> Debate is about whether it should be the only option or it also make sense
> to support other native update mechanisms
> (e.g. use Sparkle directly on Mac).
>
> As Dan said, thoughts on this are very welcome.
>
>
>>
>> An XML deployment descriptor, called 'app-profile.xml' is used to describe
>> the JARs, JRE versions and other launch details - it is semantically
>> comparable to a traditional JNLP file. When an update is triggered via the
>> API, the code checks the remote URL for an updated app-profile.xml. If it
>> finds one it downloads this and checks this with its local
>> app-profile.xml.
>> Any jars that have different version numbers are downloaded, and if a new
>> JRE version is defined in the app-profile then the zip for this is
>> downloaded and extracted as well.
>>
> Where these files are placed after download?
>
>  Once all the downloading is complete, the
>> local app-profile is then overwritten with the newer one and this newer
>> one
>> will be used by the native launcher to set the classpath next time the app
>> is started.
>>
>>
>>  IMHO, we can simplify this model following Sparkle ideas.
> Sparkle does not know anything about application structure, etc. and focus
> only on update process.
> This makes it easy to use, simple  and robust as it focuses on solves one
> problem.
>
> Important points are:
>    A. Sparkle uses "appcast" to describe what updates are available
>          (https://github.com/**andymatuschak/Sparkle/wiki/**
> publishing-an-update<https://github.com/andymatuschak/Sparkle/wiki/publishing-an-update>
> )
>        Appcast includes
>          - version of updates available
>          - meta info about each update (type of update (e.g. regular or
> mandatory security), release notes, system requirements (e.g. Windows
> version 7+), etc.)
>          - set of available update packages (aka enclosures) (e.g. full
> app image, delta patches for several versions, etc.)
>
>   B. Sparkle knows where to look for update meta info in the application
> package (on Mac it is part of Info.plist that is some kind of application
> manifest).
>       However, it only needs to know:
>          - URL of appcast
>          - location of key to verify update signatures
>          - application update settings/preferences
>       The expectation is that once app is updated this info will be
> updated (as it is part of "update package")
>       [Note: this is not so simple on Windows if we want to make sure we
> are good citizens and need to update registry]
>
>   C. Update package is black box for Sparkle core.
>       It is modeled as
>          - file hosted at given URL
>          - signed with given key (signature is part of appcast)
>          - package type defines what "update driver" will be handling this
> update
>              (this allows to add support for various update packages as
> plugins/modules)
>          - for most type for packages (except running installers) Sparkle
> goal is to update set of application files only
>             [Windows or linux require more work as we may need to update
> registry or package info]
>
>   D. If admin permissions are needed they only asked before update is
> finally installed.
>       Sparkle does not keep any internal state in the app folder as it may
> be only writable for admins.
>
>   E. Application relaunch after update
>
> This is close to your approach but there are some important differences.
> Following Sparkle more closely may simplify things and at the same time
> add some desired features
> (e.g. ability to validate origin of autoupdate)
>
> My current thoughts on "ideal" basic experience are as follows.
>
> ========
> Developer experience (no java coding needed):
>
>  1. At the package time developer requests autoupdate support in the
> native cobundle by providing:
>        * http URL to appcast + public key/cert to validate update
> signatures
>        * or https url
>     and define application update mode (mandatory update, ask user and
> allow him to skip version, etc.)
>
>  2. Once update is needed developer creates "update package" (single file).
>    For this he will need to run same packager utility/ant task but specify
> base image, certificate to sign package and
>    and desired type of update (full or incremental)
>
>  3. Developer need to create an appcast xml file (default can be generated
> in step 2 but he may want to customize it)
>
>  4. Developer stages both appcast and package bundle
> =========
>
> End user (background updates):
>   * installs application with background update policy
>   * every time application is launched it will check appcast (if online)
>   * if update is detected it will be loaded silently while app is running
>   * Once update is loaded and verified user will be prompted to install it
> (it may be notification if update is mandatory)
>   * If he agrees then he may see UAC dialog (unless installation is on per
> user basis)
>   * application relaunch after update
>   * attempt to launch app being updated should fail with reasonable dialog
> and should not fail update process (it may need to wait will .exe is
> unlocked)
> =========
>
> Implementation wise i was thinking along the lines of what you have (only
> glimpsed though your code it though) with following
> notable differences:
>
>   * update package is single file and we only worry about it version.
>     No need to worry about version of individual components as we do not
> want arbitrary mix&match.
>
>   * updates are loaded to ~/.javafx/app-key (where app-key could be
> derivation from app install location)
>     We also keep user answers for this app in that folder (e.g. had we
> offered him this version already? when was last time we checked for update,
> etc)
>
>   * Update process is:
>        - build temp full app image in the ~/.javafx/app-key/version
>        - if update accepted then exit application and run helper app co
> copy image over
>        - possibly run "post upgrade script" as part of helper app
>            (this script is specific to "bundler" technology used. E.g. for
> EXE/MSI bundlers we may want to update few keys in registry
>        - relaunch app
>        - AU module running in the relaunched app could detect temp "image"
> that is the same as current version and erase it
>
>      Note that there could be several alternative helper apps. E.g. one
> that pops UAC and one that does not for per user installs.
>      Or helper up to install update packaged as MSI (by running msiexec
> silently and showing progress)
>
>      I was even thinking that "helper" can be implemented in java with
> minimal native code (to elevate permissions and update registry if needed).
>
>   * Integration with application bundle
>       - details on whether AU is needed and what type is it may be
> embedded application manifest for main app jar
>       - embedded jauncher will read them and use internal AU APIs
> accordingly
>          (this is why no user code changes are needed)
>       - current app version, update url and signature are part of
> package.cfg or manifest in the main app jar
>       - AU support itself is part of jfxrt.jar, bundle also includes
> helper app to "move image into final location" and
>           script to do "post update config"
>
> What do you think?
>
> -igor
>
>