Auto-update of native application bundles

[Daniel Zwolenski]

Hey guys,

Those of you who have been on this list for a while will know this topic of
deployment is one I'm pretty passionate about: in my opinion JFX cannot
succeed without a simple, effective deployment model. I've been pretty
vocal on this in the past (probably to the point of irritating) and I'm
extremely excited to see the developments in this area.

As Richard mentioned, I've been talking to him about the direction this is
heading for a little while now and decided to stop talking and throw my hat
in the ring. Over the last couple of weeks I've been prototyping a solution
based on these early discussions. This is really just a proof of concept -
the code is rough and the features limited, but the idea is to demonstrate
one way it could work. I stress "one way", as there are also many other
possible solutions out there and the one I am putting forward should be
seen as a starting point, not a final locked in solution.

Currently my prototype works only on Windows (the platform I develop on,
and am most comfortable with) but the solution is designed so that it can
be extended to all desktop platforms - just the native components need to
be replaced for each platform. At a code level, I've tried to section off
these adaptation points to make it easy to extend the prototype to other
platforms. I've also developed the UI in Swing at this stage as I wanted to
put off the whole native/JFX versioning issue for now. This is high on my
list to address.

While I was developing this, Igor has been working on the native packaging
tools that he introduced in his recent blog
post<https://blogs.oracle.com/talkingjavadeployment/entry/native_packaging_for_javafx>.
Igor's focus has been on native installing and launching and he's ignored
auto-updating at this stage. My prototype focuses on auto-updating and as
yet does not worry about the actual installer too much (I cheated and used
a free, third-party MSI builder <http://www.advancedinstaller.com/>), since
I knew Igor was working in this area.

The design is similar: we both co-bundle a JRE and use a native compiled
launcher (e.g. 'exe') to setup the appropriate classpath and launch the
Java process. However due to the fact that Igor is developing behind the
closed doors of Oracle the solutions have been developed independently of
each other and share no code. Although Igor focuses on installation, and I
focus on auto-updating, the common 'launcher' functionality is duplicated
between the two solutions. i.e. I've had to write my own C++ native
launcher for Windows since I couldn't get access to Igor's (yet).

The intent (or at least my intent) is to flesh out my POC, extend it to
more OS's, add more features, and get feedback and input from you guys.
Then eventually Igor can feed back whatever works well into his packaging
tools as part of future official JFX releases. Both Igor and Richard are on
board with this general idea, so treat my POC as a way to communicate and
demonstrate what we want in JFX.

Your feedback and comments will be critical in getting a great solution in
place. Your code contributions will be even more valuable and anyone who
wants to pitch in is very encourages to do so (you will need to sign the
Oracle Contributors agreement, but this is pretty easy).

So what follows is a an outline of the general approach used, a working
example, and a more detailed look at the code and current implementation. I
look forward to hearing from anyone and everyone who has an opinion on this
topic.


*The General Approach*

The solution is designed entirely around co-bundled JREs - i.e. the JRE
instance is included in the distribution and is used only for that
application. It is not installed (it does not need to be), it is simply
extracted into the application folder and runs directly from there. There
is no use of a system-wide JRE in this approach. Richard made the case for
this in his blog<http://fxexperience.com/2012/06/application-deployment-with-javafx/>,
but to reiterate the core drivers for going down this path are:

   - Ability to pin an app to a JRE version - each app has its own JRE so
   each can upgrade at their own pace and co-exist with each other
   - No need for a separate Java installation - a horrible experience for
   non-technical users, and one that requires admin access to the system
   - No need for browser plugin installations, etc - the applet deployment
   model and its peculiar requirements are now separated from desktop
   deployment
   - Better integration with the emerging Desktop 'app stores', where the
   app must be fully contained in the deployed app store bundle (installing a
   JRE will likely be nigh on impossible on Mac before long)

A native package is developed for each platform, so on Windows an 'MSI',
Mac a 'dmg' etc. This installer is actually fairly light-weight. It's main
job is to extract the JRE and jars for the app to a directory and then
include the 'launcher.exe' (or platform equivalent) to launch the app. On
windows, the MSI I use simply extracts these files, adds a shortcut to the
Start Bar and registers an uninstaller with the windows registry. This is
about as lightweight as an installer gets - there are no additional
registry entries or registered DLLs etc.

So far, the approach described is much in line with what Igor has done. The
main area where my POC diverges is in the area of auto-updates. That is,
the ability to check for a newer version of the application (including the
move to a higher JRE version), and then download and use that new version
if it exists.

Richard made reference to Sparkle <http://sparkle.andymatuschak.org/> and
like him, I feel this is a nice reference point for the style of solution
we want. The neat thing that Sparkle does is it provides an API that
developers can use inside their code to completely control and customise
the upgrade of their application. This is the approach I've gone for as
well. My POC includes a Java API that you can call to check for updates and
trigger downloads.

It comes with it's own UI components for the most common workflows but if
you want to override these and hook in your own UI or workflow components
you can. Maybe you want to force the customer to accept new T&C's before
upgrading, or maybe you just want to theme the upgrade dialogs to look more
like your UI.

It's all nice, clean Java so you are back in control. Unlike the current
JNLP/Java way of doing updates there will be no more ugly, random 'You need
to update Java' dialogs popping up to scare your users. The auto-updating
is inside your app and only happens when your app is in use (and when you
trigger it from your code).

An XML deployment descriptor, called 'app-profile.xml' is used to describe
the JARs, JRE versions and other launch details - it is semantically
comparable to a traditional JNLP file. When an update is triggered via the
API, the code checks the remote URL for an updated app-profile.xml. If it
finds one it downloads this and checks this with its local app-profile.xml.
Any jars that have different version numbers are downloaded, and if a new
JRE version is defined in the app-profile then the zip for this is
downloaded and extracted as well. Once all the downloading is complete, the
local app-profile is then overwritten with the newer one and this newer one
will be used by the native launcher to set the classpath next time the app
is started.

I think this approach is neat and powerful, but it's worth mentioning that
there is still some debate amongst the JFX team about whether to go down
this road or avoid the Java API and use off-the-shelf native updaters. Your
input will very likely influence this decision so if you have a preference
one way or the other (or have other suggestions), be vocal.


*A Working Example*

To test and demonstrate my POC, I've created a little test-app, assembled
it as an MSI, deployed it to my server and then uploaded a patch update to
my server as well so that when you check for updates, an upgrade will take
place. Just for fun, I made the first MSI installed version use Java6 and
the upgrade one use Java7, so when you do an upgrade it will also download
a new JRE (from my site as well - distribution, licencing issues are still
a little hazy and we're looking into this in more detail. Technically we
are legally allowed to co-bundle for the initial install but this area of
auto-updating and JRE redistribution is a bit loose in the T&C's).

You can download the initial installer from here:
http://zenjava.com/demo/update/testapp.msi

Unlike Igor's installer, this will install in Program Files (and so will
prompt a UAC security notice). It also installs an uninstaller so you can
always go to "Control Panel -> Add/Remove Programs" to completely remove
it. It adds a shortcut to the Start Menu (called "Zen Deploy Test
Application"), and when it installs it unpacks a directory structure like
so:

installdir/
+ launcher.exe
+ app-profile.xml
+ admin-exec.exe

installdir/app
+ zen-deploy-api-1.0-SNAPSHOT.jar
+ zen-deploy-testapp-1.0-SNAPSHOT.jar
+ zen-deploy-win-impl-1.0-SNAPSHOT.jar

installdir/jre
+ 1.6/ *(direct copy of a full JRE installation is under here)*

When you run (by selecting the shortcut from the start menu, or you could
double click launcher.exe) it will popup a little Swing GUI which is the
test application. This could do anything, but I've made my app check for an
update on startup. If an update is found it shows the details of the update
and an option to upgrade. Click it and it will go through the motions of
upgrading and finally restart itself.

To make it clear what version you are on, the first installed version is a
lovely yellow, the updated version is an even more attractive green.

Note that the feedback of the installation process is poor. You only see a
single, indeterminate progress bar for the whole time. I intend to address
this pretty soon with more informative progress information and the ability
to cancel the upgrade. Error handing is also very poor, so if something
goes wrong you may not get great feedback at this stage of the POC.

After installation you can see that a new JAR (version 2.0 of
zen-deploy-testapp) has been included in the 'app' directory, and also a
new JRE (1.7) has been downloaded to the 'jre' directory. Currently the old
JAR and JRE do not get deleted so they are still hanging around in the
directories but they are not used since they are not referenced in the
app-profile.xml. Deleting these while the app is running proved difficult
(as they are locked), so I need a more creative solution (probably delete
them on next startup) - this is also in the todo list.


*The Implementation*

As I mentioned earlier the code is pretty rough. It's a rapid prototype,
which means I haven't done any unit tests or pretty JavaDoc and I don't
handle errors well. I have tried to be fairly clean in my code structure
however and I hope it's not too hard to follow. If you need more direction,
just ask.

One disclaimer - my C++ is extremely rusty (it's been over 10 years since I
last touched it), so don't judge me by this :) I would welcome any and all
input from veteran C++ developers out there who want to substitute my
clunky code for something a little more hard core and impressive.

The full source code can be found at:
http://code.google.com/p/zen-deploy/source/checkout

I've called it 'Zen Deploy' and used my zenjava domain just to have
something to reference it by. This is not intended to be a stand-alone,
long term product (unless the JFX team want to go that way but I would
prefer not), anything that works should eventually get rolled into JFX
proper and Zen Deploy eventually decommissioned. I use 'zen' cause it's
small and distinctive.

It's a Maven project but don't let that scare you off if you are not a
Maven user. There are pretty much no external dependencies on other
libraries so just open the source code and build however you normally do.
It's not overly automated anyway, due to the project consisting of native
modules and MSI packagers, etc, and if you want to actually assemble it
there's a few steps to go through. If anyone is actually keen to build and
is having trouble let me know, and I'll talk you thought it. If you're just
wanting to get your head around it and feedback, you can probably get away
with just flicking through the code.

Following Maven's convention I've split it into a bunch of sub-modules so
it looks a little large at first, but there's not actually that much code
in there. You can probably get away with just checking out a few choice
classes if you're interested in how it works. Here's a little site map to
help you out:

Modules:

   1. *zen-deploy*: top level parent module for everything else (open the
   pom.xml in intellij and you will have access to all the Java code in one
   project)

   2. *zen-deploy-api **(creates zen-deploy-api.jar)*: the generic (i.e. OS
   independent) API and utility classes for auto-updating. I've also included
   the higher level gui/workflow components here but that was slightly lazy as
   these should probably be in a module of their own (one for Swing, one for
   JFX, one for HTML, etc).

   3. *zen-deploy-win-impl ** (creates zen-deploy-win-impl.jar) *: this is
   the windows implementation of the zen-deploy-api. This implements the
   'updateNow' methods defined in the API to do windows specific stuff (such
   as playing nice with the UAC).

   4. *zen-deploy-win-launch **(creates launcher.exe) *: this is a Visual
   C++ project which generates the 'launch.exe' file that goes into the final
   bundle. This file is the main launching point for the app and it starts up
   a Java process based on the app-profile.xml file.

   5. *zen-deploy-win-admin-exec **(creates admin-exec.exe) *: another
   little Visual C++ project. This one generates 'admin-exec.exe' which is
   included in the final bundle and is used by the windows AU Java
   implementation to move files into 'Program Files' by playing friendly with
   windows UAC/security.

   6. *zen-deploy-testapp* *(creates zen-deploy-testapp.jar)* : this is a
   very small sample application that is used to demonstrate the auto-update
   API. This is what's deployed as an MSI at:
   http://zenjava.com/demo/update/testapp.msi There are two versions to
   this, a yellow one and a green one. I just build these two manually (i.e. I
   comment out some code, build then comment it back in, and build again).

   7. *zen-deploy-win-msi **(creates testapp.msi)*  : this is an 'Advanced
   Installer <http://www.advancedinstaller.com/>' project used to create
   the MSI. You need to have Advanced Installer installed and then you need to
   manually copy the output files from all the other projects into the
   'bundle' directory so it can be packaged up. This wouldn't exist in a more
   complete solution, the MSI would be generated off the project files of
   'testapp' using built-in tools (i.e. similar to the installers Igor is
   putting together for jfx2.2).


The main classes/files you might find interesting are:

*in zen-deploy-testapp: *

   - *ZenDeployTestApp.java*<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-testapp/src/main/java/com/zenjava/zendeploy/testapp/ZenDeployTestApp.java>
:
   the example 'app' that uses the updater API. It's a normal, everyday
   (Swing) app that just embeds the 'wizard' component from the API into it's
   own panel but it could just as easily pop this up in a dialog, etc.
   - *app-profile.xml<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-testapp/src/main/deploy/app-profile.xml>
   *: the XML deployment descriptor for launching the TestApp. This is just
   a PoC syntax, plenty of things missing and plenty of room to clean-up.

*in zen-deploy-win-launch: *

   - *ZenLauncherWin32.cpp<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-win-launch/ZenLauncherWin32.cpp>
   ** :* the C++ launcher file that reads the 'app-profile.xml' file and
   launches a Java process configured appropriately (e.g. sets the classpath).

*in zen-deploy-api:*

   - *ZenDeployManager.java*
<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-api/src/main/java/zom/zenjava/zendeploy/ZenDeployManager.java>:
   the main entry point into the lower level AU API. This provides the methods
   for updating and checking versions, some of which is implemented directly
   in this class and some is deferred to the OS specific sub-classes.
   - *ApplicationUpdateWizard.java*<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-api/src/main/java/zom/zenjava/zendeploy/gui/swing/ApplicationUpdateWizard.java>
:
   a basic Wizard component (in Swing) that steps a user through the update
   process. This would obviously be a lot prettier and feature rich in a real
   implementation (and written in JFX!).

in zen-deploy-win-impl:

   - *ZenDeployManagerWinImpl.java<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-win-impl/src/main/java/com/zenjava/zendeploy/win/ZenDeployManagerWinImpl.java>
   * : the windows specific implementation of the ZenDeployManager. This
   adds all the logic for downloading the files, unzipping them and copying in
   ways that Windows is happy with.
   - *WinUacFriendlyFileMover.java*<http://code.google.com/p/zen-deploy/source/browse/trunk/zen-deploy/zen-deploy-win-impl/src/main/java/com/zenjava/zendeploy/win/WinUacFriendlyFileMover.java>
:
   a class I'd prefer not to need, but this is here to support UAC on windows.
   This class re-launches itself in an 'elevated' process (i.e. run as admin)
   in order to copy files into the 'Program Files' directory.


*Next Steps*

This first POC works but there are still quite a lot of things to add
before the POC can really be considered proof that the concept will work.

In particular I want to look into the following (in rough order of
priority):

   - Including native files in the app distribution, especially JFX
   - Updating the GUI components to use JFX instead of Swing (or rather in
   addition to - the framerwork is intended to be UI agnostic)
   - Extending the POC to other platforms, especially Mac and Linux
   - Providing better feedback on the upgrade progress and allowing the
   user to cancel
   - Cleaning up files after an upgrade (i.e. removing obsolete files)
   - Possibly looking at ANT and/or Maven tasks to assemble deployment
   bundles

Additionally I'd like to start working more with Igor to make my launcher
more inline with what he is doing. Ideally the auto-updating code should
work with the packaging tool he has created, regardless of whether the code
gets rolled back into JFX proper or not.

As I've not-so-subtly hinted at a few times above, there's plenty of work
for anyone who wants to stick their hand-up and all of this is in need of
heavy review and feedback. I'd be particularly interested in hearing from
anyone who wants to help port this to Mac and Linux as I don't have easy
access to these environments to develop and test on.

If you want an awesome deployment solution for Java, now's your chance to
have your say.

Cheers,
Dan



On Sat, Jun 16, 2012 at 6:58 AM, Jonathan Giles
<jonathan.giles at oracle.com>wrote:

> After much talking I'm very pleased to see code is being written. This is
> of the utmost importance in my opinion.
>
> I often tell the story of how impressed I am with other UI toolkits. I
> occassionally see that they need an update, and show a nice looking and
> succinct update interface that does everything painlessly and
> automatically. What blows me away further is that applications that are
> written in this UI toolkit are able to access exactly the same updating
> mechanism for their own application updates. The fact that an application
> developer can painlessly update their application is incredibly wonderful,
> coming from my background of having to write custom update preloaders for
> clients in the past. The complexity in these things is rather high - you
> have to have some form of script to contain the update instructions, and be
> a preloader for the application (so you need some way of starting up the
> application that works on most operating systems).
>
> Key words from the above paragraph: 'nice looking', 'succinct update
> interface', 'painlessly', and 'automatically'. I really hope we can pull
> something along these lines of - it would be great!
>
> In short, I am pleased that this is something we are now taking seriously,
> and something that I think end-developers will be pleased about to. This
> should be a simple, painless process that we provide, and I look forward to
> testing it out! Feel free to drag me into any form of feedback cycle and or
> end-user testing.
>
> -- Jonathan
>
>
> On 16/06/2012 8:31 a.m., Richard Bair wrote:
>
>> Hi,
>>
>> On my blog[1] I posted an article linking to Igor's blog about
>> application deployment. I went on to describe scenarios where native app
>> bundles do not require auto-update (primarily, when dealing with IT
>> department's distributing MSI files or when using an app store), and
>> conversely those that do require auto-update (just about everything else).
>> I mentioned that I've been talking with Dan Zwolenski (aka Zonski) about
>> Sparkle and auto-update requirements and he's been building a prototype.
>>
>> There are basically two ways we can take this. The first is that we can
>> try to find some native auto-update system on each of the desktop operating
>> systems, and then build a think Java API that sits on top of these. This
>> approach would allow us to leverage existing code and get some robust
>> auto-update in place much quicker. On mac for instance we would just use
>> Sparkle and it would be rock solid from day one.
>>
>> I have concerns about this approach. First, the mechanism by which you
>> communicate with the application that it is time to update is likely going
>> to be different for different native auto-update frameworks. Sparkle for
>> example uses "app casting" -- an RSS feed with a URL to the new version.
>> Google's auto-update frameworks probably use some other mechanism. I think
>> it is unacceptable to ask developers to use multiple different deployment
>> mechanisms for telling the app that it is time to update. So that would
>> then suggest we have to have some custom code + Java API + native
>> auto-update frameworks -- now things are getting complicated. Plus,
>> different frameworks have different feature sets, so we either have to use
>> the intersection of APIs supported by all, or we have to modify frameworks
>> with features that are missing.
>>
>> I don't think it should really be that complicated. I think that
>> essentially doing an auto-update mechanism in nearly all Java (with a minor
>> bit of native code) would reduce the numbers of bugs we have to deal with
>> over all, and would also ensure the same API across all of the desktop
>> operating systems (or embedded systems that don't force the use of an app
>> store). Sparkle is very versatile and yet essentially maintained by one
>> guy, so I think it should be feasible that we could support a similar
>> feature set in Java.
>>
>> I've asked Dan if he wanted to spearhead this work and he's been
>> enthusiastic about doing so. With that introduction, Dan, the rest is
>> your's :-).
>>
>> I'm hoping we can put the auto-update mechanism into the lombard
>> repositories (what we're calling 3.0) as soon as they open. In the meantime
>> I know he has a prototype for Windows.
>>
>> Cheers
>> Richard
>>
>> [1] http://fxexperience.com/2012/**06/application-deployment-**
>> with-javafx/<http://fxexperience.com/2012/06/application-deployment-with-javafx/>
>>
>
>
>