Alternative to fatJar - modular solution?

[Ioi Lam]

As a proof of concept, I wrote a quick-and-dirty demo that can load 
Jigsaw modules from a Uber-JAR file:

     https://github.com/iklam/tools/tree/main/jigsaw/uberjar

Glavo, would something like this fit your needs?

I think something like this could be done outside the JDK (in frameworks 
like SpringBoot) or inside the JDK (as some sort of extension to "java 
-jar", for example).

As the demo shows, we already have the core API support to enable this. 
I would encourage other member of the Java community to experiment with 
it and see if we should proceed further in this direction.

Thanks
- Ioi

On 10/7/21 9:31 PM, Ioi Lam wrote:
> I am wondering if there are 3rd party solutions that support loading 
> Jigsaw modules from uber jars. The JDK should have all the APIs to 
> support such a solution.
>
> E.g., I looked at SpringBoot, which has uber jar support, but it 
> doesn't seem to support modules ("java.lang.module" doesn't appear in 
> any of the source files).
>
> Thanks
> -  Ioi
>
> On 10/7/21 6:43 AM, Gregg Wonderly wrote:
>> The URL class loader is the easiest way to solve conditional 
>> loading.  In Jini, now Apache River, we’ve long used this mechanism 
>> to “get” the implementation of all interfaces that a remote client 
>> application needed to talk to a particular server (versioning makes 
>> this necessary and powerfully easy as a solution).  But the security 
>> manager didn’t seem useful as of late and the removal of the security 
>> manager support for managing per jar security is a bit problematic 
>> for this kind of mobile code use in Java.   Yet, this is the primary 
>> way that javascript works in the web browser as the mobile code 
>> interface to remote services.  It really feels like Oracle and the 
>> Java team have no interest in what the desktop environment represents…
>>
>> Gregg Wonderly
>>
>>> On Oct 7, 2021, at 7:41 AM, Glavo <zjx001202 at gmail.com> wrote:
>>>
>>>> *Bandwidth optimization and rare machines.* This is interesting 
>>>> because
>>>> it's a requirement that feels like it may be more common in China than
>>>> elsewhere. I'd be keen to learn more about your bandwidth constraints,
>>>> unless this is more of a theoretical concern?
>>>
>>> Ah, in fact, in Chinese mainland, server bandwidth is a very real
>>> problem. In China, many websites such as GitHub and cloudflare cannot
>>> provide services normally. The cost of civil broadband is low, but
>>> commercial bandwidth is more expensive, which costs several times or
>>> even more than ten times higher than other parts of the world.
>>> Our average income level also lags behind that of developed countries,
>>> so we will pay more attention to the cost of bandwidth.
>>>
>>> Mike Hearn <mike at plan99.net> 于2021年10月7日周四 下午7:31写道：
>>>
>>>> Thanks for your insightful reply, Glavo. Here are some thoughts. I 
>>>> should
>>>> note that I don't work for Oracle or on OpenJDK, in case that wasn't
>>>> already clear.
>>>>
>>>> *Forum.* Although it's logical that you ended up on this list,
>>>> realistically the JPMS is "done" and not being worked on since Java 
>>>> 9. Any
>>>> solutions or improvements have to come from the user community so 
>>>> it may
>>>> make more sense to have this discussion on Reddit, or some other 
>>>> Java forum.
>>>>
>>>> *Alternative approach. *Given this constraint, it can make sense to 
>>>> think
>>>> wider or bigger than just updating previous approaches. Would your 
>>>> needs be
>>>> met or even met better by a re-imagining of Web Start, but one 
>>>> suitable for
>>>> servers and the CLI? For example:
>>>>
>>>> $ alias glavos="jrun glavos-cool-app.com"
>>>> $ glavos --flag --another-flag
>>>>
>>>> Here an imaginary "jrun" command (re)downloads an app and stores it 
>>>> to a
>>>> local cache, perhaps downloading an appropriate JVM/jlinked image 
>>>> alongside
>>>> it if none is available already locally. It's given a URL but in a
>>>> convenient form for typing, e.g. with assumed protocols and paths 
>>>> if only a
>>>> domain name is specified. The tool would occasionally check for 
>>>> updates and
>>>> run from the cache the rest of the time. This doesn't make apps into a
>>>> single file but it tackles other problems you mention having to 
>>>> roll your
>>>> own solutions for, like writing your own update checker and asking 
>>>> users to
>>>> download the right file. Unlike tools like apt-get or brew there 
>>>> would be
>>>> no notion of adding a repository beforehand, so for CLI / server 
>>>> apps, it
>>>> retains its usability.
>>>>
>>>> For desktop apps a simple .jrun file association could be used to 
>>>> do the
>>>> same thing.
>>>>
>>>> For building Docker images you could have:
>>>>
>>>> $ jrun --cache-only glavos-cool-app.com
>>>>
>>>> which would populate a cache during the docker build, but not run the
>>>> program itself.
>>>>
>>>> I've often wished for such a tool. At one point I built one that 
>>>> did Maven
>>>> resolution, but it for GUI apps. Although my new venture is about
>>>> self-updating desktop/server app packages, I've been planning an 
>>>> extension
>>>> in this direction later because once you can distribute a generic 
>>>> runtime
>>>> as a self-updating "app" you can easily bring back the JRE model 
>>>> for those
>>>> who want it.
>>>>
>>>> *Jimage.* In your first mail you proposed a new kind of fat-jar 
>>>> based on
>>>> the jimage format the modules file uses. JImage isn't a documented 
>>>> format,
>>>> or rather, it's documented only in the source code, but it has quite a
>>>> clever design. The upside is that it's highly optimized. The 
>>>> downsides are:
>>>>
>>>> 1. Write only. ZIPs have some basic support for editing but jimage
>>>> doesn't. This is a pain for things like config files, where you may 
>>>> want to
>>>> make specialized versions of an app by adjusting the internal 
>>>> files. It can
>>>> be easily fixed using a classloader that checks local disk for 
>>>> resources
>>>> first.
>>>>
>>>> 2. No built-in support for native code libraries. There was a related
>>>> discussion of this problem a week or so ago on this list. Of 
>>>> course, JARs
>>>> have the same problem.
>>>>
>>>> 3. No support for multiple versions of the same JAR in the same 
>>>> file, even
>>>> though the core JPMS *can* support this via the
>>>> defineModulesWithManyLoaders API, and even though this would be a very
>>>> useful thing to support. Fat JARs have the same problem so this is 
>>>> not a
>>>> downside compared to the status quo.
>>>>
>>>> 4. The format is deliberately undocumented so it can be changed in 
>>>> future
>>>> JVM versions. Thus using it would actually mean cloning it, and/or
>>>> rewriting parts of the code because otherwise the GPL2 might kick in.
>>>>
>>>> Overall, the downsides are not that big! The worst is the need to 
>>>> clone
>>>> the format to avoid depending on JVM internals. On the other hand, 
>>>> ZIPs
>>>> work well enough and don't require writing any new code except a 
>>>> little
>>>> stub entry point that uses custom classloaders.
>>>>
>>>> *Bandwidth optimization and rare machines.* This is interesting 
>>>> because
>>>> it's a requirement that feels like it may be more common in China than
>>>> elsewhere. I'd be keen to learn more about your bandwidth constraints,
>>>> unless this is more of a theoretical concern?
>>>>
>>>> You mention you actually have users on LoongArch64 for example. 
>>>> Indeed,
>>>> the chances that non-Chinese developers will produce jlinked images 
>>>> for
>>>> this CPU any time soon is very low.
>>>>
>>>> *Product potential.* As mentioned, I'm setting up a new venture 
>>>> that is
>>>> starting with app distribution, and particularly distribution for 
>>>> the JVM
>>>> world. JPackage is good as far as it goes, but it doesn't solve all 
>>>> the
>>>> problems developers face. Given your list of target machines it 
>>>> feels like
>>>> you're probably a commercial organization with a wide customer 
>>>> base. If
>>>> you're in the market for better approaches please send an email to
>>>> mike at hydraulic.software and maybe your needs can influence our product
>>>> direction.
>>>>
>>>>
>>>>
>