June 2022 - leyden-dev

Re: Experimentation with build time and runtime class initialization in qbicc
by David P Grove 06 Jun '22

06 Jun '22

Hi, I’ve appended the contents of the referenced wiki page in this email. Apologies in advance if the formatting doesn’t come through as intended. There is a full implementation of this (GPLv2 + Classpath exception) as part of the qbicc project on GitHub. There is also a GitHub discussion in the qbicc project that links to various GitHub issues that capture the history that led to the current design. I will not hyperlink to those here so that if people have any IP concerns, they can avoid seeing them. They are easily findable. Regards, --dave ## Overview One of the goals of the qbicc project is to explore technical approaches for adapting Java's specification of class initialization to fully support native image compilation. Enabling build-time evaluation of complex class initialization logic is essential for obtaining much of the benefits of native image compilation: reduced memory footprint and fast startup. However, both the core JDK and many frameworks will not be primarily be used in native image scenarios. Therefore, it is essential that the approach taken for build-time initialization enables both the existing runtime class initialization and the new build-time class initialization logic to co-exist. Furthermore, for as many cases as possible, the class initialization code should be shared between the two usage scenarios and have non-surprising semantics in both. ## Build-time Initialization In qbicc, all classes are initialized at build-time. Class initialization at build time is performed according to the existing semantics of Java class initialization driven by build-time execution of the `<clinit>` methods of reachable classes. The set of reachable classes is determined iteratively, starting with the program entrypoints and adding the methods and classes they utilize until no further reachable classes are discovered (a fixed point is reached). After build-time initialization has completed, a build-time heap has been constructed that contains the objects that were created during the build-time execution of the `<clinit>` methods. Using the reachable static fields of the reachable program as roots, this build-time heap is serialized into the native image. This set of objects will form the initial runtime heap of the program when it is executed. ## Runtime Initializers There are cases where one or more initialization actions of a class **must** be executed at program runtime. Most typically these involve the creation of native resources (open files, threads, etc) that cannot be successfully serialized into the build time heap. Qbicc supports runtime initialization by allowing static fields of a classes to be declared as runtime initialized. These fields will be initialized lazily, at first access, by executing a runtime initializer (`<rtinit>`) associated with the accessed field. Runtime initialization is localized: accessing a particular static field will cause its runtime initializer to be executed but has no implications for other runtime initializers defined either in the field's defining class or any superclass or implemented interface of the field's defining class. When serialized from the build-time heap to the runtime heap, all runtime-initialized fields will be serialized with the zero (uninitialized) value appropriate for their type. Qbicc allows related static fields in the same class to share a common `<rtinit>` method. The first access to any of the fields will cause the execution of the associated `<rtinit>` method and the initialization of all the fields. ## Adjusting Heap Serialization For some objects it is necessary to initialize them during build-time initialization, but "reset" them before they are used at runtime. Qbicc supports this by allowing fields to be annotated to be serialized as the type-appropriate zero value or as a primitive constant value. This value replacement happens as the build time heap is serialized. One common scenario is to invalidate objects that are wrapping native resources. For example, when a `FileDescriptor` is serialized its `fd` and `handle` instance fields are serialized as `-1` and its `closed` field is serialize as `true`. Thus, any attempt to use the build-time FileDescriptor at runtime will raise the appropriate exception. ## Patching: Migration for Existing Classes The runtime initialization mechanisms described above are currently enabled via a set of annotations. This allows qbicc to implement the desired semantics without requiring any changes to the Java compiler, class file format, or language specification. In the long term, we believe small modifications to the Java specification, for example defining a `rtinit { ... }` similar to the existing `static { ... }` construct could enable a simpler specification. The primary annotation for runtime initialization is `RuntimeAspect`. This annotation is defined on a class and is interpreted as meaning that the `<clinit>` method of the class should be interpreted as an `<rtinit>` method. This method will not be executed during build-time initialization and instead will be deferred until the first access of one of the static fields defined in the class. To allow us to "externally" modify JDK core classes for qbicc, we have developed an annotation-driven patcher infrastructure. The patcher allows the declaration of patch classes that add, remove, and modify the methods and fields of an existing class. This modification includes the replacement of the `<clinit>` method and the declaration of multiple `RuntimeAspect` patch classes. The best way to explore what is possible with the patcher is to examine the java.base/src directory in the qbicc-class-library project. It makes extensive use of the patcher annotations to adapt the core JDK classes to qbicc while still allowing us to consume the upstream OpenJDK code base via an unmodified git submodule. ## Design Alternatives A number of alternatives were considered before arriving at the final design documented here. The technical discussions and options considered can be explored starting in qbicc discussion #764 on GitHub. From: Brian Goetz <brian.goetz(a)oracle.com> Date: Thursday, May 26, 2022 at 2:21 PM To: David P Grove <groved(a)us.ibm.com>, "leyden-dev(a)openjdk.java.net" <leyden-dev(a)openjdk.java.net> Subject: [EXTERNAL] Re: Experimentation with build time and runtime class initialization in qbicc Hi David; Would like to understand more about this, but first, from an IP-hygiene perspective, documents linked from this list should be under the OpenJDK terms and conditions. Can you post the contents of that document here, so there are no ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi David; Would like to understand more about this, but first, from an IP-hygiene perspective, documents linked from this list should be under the OpenJDK terms and conditions. Can you post the contents of that document here, so there are no issues there? Thanks, -Brian On 5/26/2022 12:35 PM, David P Grove wrote: Hi, In the qbicc project, we’ve been exploring options for adapting Java’s class initialization semantics for native images. In particular, we are trying to arrive at a non-surprising semantics that in a native-image scenarios allows most initialization to happen at build-time while still enabling runtime initialization of selected static fields. Our current design and experience is captured here: https://github.com/qbicc/qbicc/wiki/Class-Initialization-in-qbicc<https://github.com/qbicc/qbicc/wiki/Class-Initialization-in-qbicc>. In a nutshell, the idea is to initialize classes via build-time execution of existing <clinit> methods as per normal Java semantics while adding per-static-field <rtinit> methods to provide a capability for runtime-reinitialization of a field before its first access. --dave

6 13

AppCDS / AOT thoughts based on CLI app experience
by Mike Hearn 03 Jun '22

03 Jun '22

Hi, It feels like most of the interest in static Java comes from the microservices / functions-as-a-service community. My new company spent the last year creating a developer tool that runs on the JVM (which will be useful for Java developers actually, but what it does is irrelevant here). Internally it's a kind of build system and is thus a large(ish) CLI app in which startup time and throughput are what matter most. We also have a separate internal tool that uses Kotlin scripting to implement a bash-like scripting language, and which is sensitive in the same ways. Today the JVM is often overlooked for writing CLI apps due to startup time, 'lightness' and packaging issues. I figured I'd write down some notes based on our experiences. They cover workflow, performance, implementation costs and security issues. Hopefully it's helpful. 1. I really like AppCDS because: a. It can't break the app so switching it on/off a no-brainer. Unlike native-image/static java, no additional testing overhead is created by it. b. It's effective even without heap snapshotting. We see a ~40% speedup for executing --help c. It's pay-as-you-go. We can use a small archive that's fast to create to accelerate just the most latency sensitive startup paths, or we can use it for the whole app, but ultimately costs are controllable. d. Archives are deterministic. Modern client-side packaging systems support delta updates, and CDS plays nicely with them. GraalVM native images are non-deterministic so every update is going to replace the entire app, which isn't much fun from an update speed or bandwidth consumption perspective. Startup time is dominated by PicoCLI which is a common problem for Java CLI apps. Supposedly the slowest part is building the model of the CLI interface using reflection, so it's a perfect candidate for AppCDS heap snapshotting. I say supposedly, because I haven't seen concrete evidence that this is actually where the time goes, but it seems like a plausible belief. There's a long standing bug filed to replace reflection with code generation but it's a big job and so nobody did it. Unfortunately the app will ship without using AppCDS. Some workflow issues remain. These can be solved in the app itself, but it'd be nice if the JVM does it. The obvious way to use CDS is to ship an archive with the app. We might do this as a first iteration, but longer term don't want to for two reasons: a. The archive can get huge. b. Signature verification penalties on macOS (see below). For just making --help and similar short commands faster size isn't so bad (~6-10mb for us), but if it's used for a whole execution the archive size for a standard run is nearly the same as total bytecode size of the app. As more stuff gets cached this will get worse. Download size might not matter much for this particular app, but as a general principle it does. So a nice improvement would be to generate it client side. CDS files are caches and different platforms have different conventions for where those go. The JVM doesn't know about those conventions but our app does, so we'd need our custom native code launcher (which exists anyway for other reasons) to set the right paths for CDS. Then you have to pick the right flags depending on whether the CDS file exists or not. I follow CDS related changes and believe this is fixed in latest Java versions but maybe (?) not released yet. Even once that's fixed it's not quite obvious that we'd use it. The JVM runs much slower when dumping a dynamic CDS archive and the first run is when first impressions are made. Whilst for cloud stuff this is a matter of (artificially?) expensive resources, for CLI apps it's about more subjective things like feeling snappy. One idea is to delay dumping a CDS archive until after the first run is exiting, so it doesn't get in the way. The first run wouldn't benefit from the archive which is a pity (except on Linux where the package managers make it easy to run code post-install), but it at least wouldn't be slowed down by creating it either. The native launcher can schedule this. Alternatively there could be a brief pause on first run when the user is told explicitly that the app is optimizing itself, but how feasible that is depends very much on dump speed. Finally we could ship a small archive that only covers startup, and then in parallel make a dump of a full run in the background. Speaking of which, there's a need for some protocol to drive an app through a representative 'trial run'. Whether it's generating the class list or the archive itself, it could be as simple as an alternative static method that sits next to main. If it were to be standardized the rest of the infrastructure becomes more re-usable, for instance build systems can take care of generating classlists, or the end-user packaging can take care of dynamic dumping. CDS has two modes and it's not clear which is better. I'm unusually obsessive about this stuff to the extent of reading the CDS source code, but despite that I have absolutely no idea if I should be trying to use static or dynamic archives. There used to be a performance difference between them but maybe it's fixed now? There's a lack of end-to-end guidance on how to exploit this feature best. The ideal would obviously be losing the dump/exec split and make dynamic dumping continuous, incremental and imposing no performance penalty. Then we could just supply a path to where the CDS file should go and things magically warm up across executions. I have no idea how feasible that is. Once AppCDS archives are in place and being created at the right times, a @Snapshotted annotation for fields (or similar) should be an easy win to eliminate the bulk of the rest of the PicoCLI time. Dynamically loaded heaps would also be useful to eliminate the overhead of loading configs and instantiating the (build) task graph without a Gradle-style daemon. 2. AppCDS archives can open a subtle security issue when distributing code to desktop platforms. Because they're full of vtables anyone who can write to them can (we assume) take over any JVM that loads the archive and gain whatever privileges have been granted to that app. The archive file is fully trusted. On Windows and Linux this doesn't matter. On Linux sensitive files can be packaged or created in postinst scripts. On Windows either an app comes with a legacy installer/MSI file and thus doesn't have any recognized package identity that can be granted extra permissions, or it uses the current gen MSIX system. In the latter case Windows has a notion of app identity and so you can request permissions to access e.g. keychain entries, the user's calendar etc, but in that case Windows also gives you a private directory that's protected from other apps where sensitive files can be stashed. AppCDS archives can go there and we're done. MacOS is a problem child. There are two situations that matter. In the first case archives are shipped as data files with the app. Security is not an issue here, but there's a subtle performance footgun. On most platforms signatures of files shipped with an app are checked at install time but on macOS they aren't. Thanks to its NeXT roots it doesn't really have an installation concept, and thus the kernel checks signatures of files on first use then caches the signature check in the kernel vnode. By default the entire file is hashed in order to link it back to the root signature, which for large files can impose a small but noticeable delay before the app can open them. This first run penalty is unfortunate given that AppCDS exists partly to improve startup time. You can argue it doesn't matter much due to the caching, but it's worth being aware of - very large AppCDS archives would get fully paged in and hashed before the app even gets to do anything. In turn that means people might enable AppCDS with a big classlist expecting it to speed things up, not noticing that for Mac users only it slowed things down instead. There are ways to fix this using supported Apple APIs. One is to supply a CodeDirectory structure stored in extended attributes: you should get incremental hashing and normal page fault behaviour (untested!). Another is to wrap the data in a Mach-O file. In the second case the CDS archive is being generated client side. Mac apps don't have anywhere they can create tamperproof data, except for very small amounts in the keychain. Thus if a Mac app opens a malicious cache file that can take control of it that's a security bug, because it'd allow one program to grab any special privileges the user granted to another. The fact that the grabbing program has passed GateKeeper and notarization doesn't necessarily matter (Apple's guidance on this is unclear, but it seems plausible that this is their stance). In this case the key chain can be used as a root of trust by storing a hash of the CDS archive in it and checking that after mmap/before use. Alternatively, again, Apple provides an API that lets you associate an on-disk (xattr) CodeDirectory structure with a file which will then be checked incrementally at page fault time. Extreme care must be taken to avoid race conditions, but in theory, a CodeDirectory structure can be computed at dump time, written to disk as an xattr, and then stored again in the key chain (e.g. by pretending it's a "key" or "password"). After the security API is instructed to associate a CD with the file, it can be checked against the tamperproofed version stored in the key chain and if they match, the archive can then be mmapped and used as normal. Native images don't have these issues because the state snapshot is stored inside the Mach-O file and thus gets covered by the normal mechanisms. However once it adds support for persisted heaps, the same issue may arise. Whether it's worth doing the extra work to solve this is unclear. Macs are guaranteed to come with very fast NVMe disks and CPUs. Still, it's worth being aware of the issue. 3. Why not just use a native image then? Maybe we'll do that because the performance wins are really compelling, but again, v1 will ship without this for the following reasons: a. Static minification can break things. Our integration tests currently invoke the entry point of the app directly, but that could be fixed to run the tool in an external process. For unit tests the situation is far murkier. It's a bit unclear how to run JUnit tests against the statically compiled version and it may not even make sense (because the tests would pin a bunch of code that might get stripped in the real app so what are you really testing?). b. It'd break delta updates. Not the end of the world, but a factor. c. I have no idea if we're using any libraries that spin bytecode dynamically. Even if we're not today, what if tomorrow we want to use such a library? Do we have to avoid using it and increase the cost of feature development, or roll back the native image and give our users a nasty performance downgrade? Neither option is attractive. Ideally SubstrateVM would contain a bytecode interpreter and use it when necessary. Lots of issues there but e.g. it'd probably be OK if it's not a general classloader and the code dependencies have to be known AOT. d. Similar to (c), fully AOT compilation can explode code and thus download size even though many codepaths are cold and only execute once. It'd be nice if a native image could include a mix of bytecode and AOT compiled hotspots. e. Once you're past the initial interactive stage the program is throughput sensitive. How much of a perf downgrade over HotSpot would we get, if any? With GraalVM EE we could use PGO and not lose any, but the ISV pricing is opaque. At any rate to answer this we have to fix the compatibility issues first. The prospect of improving startup time and then discovering we slowed down the actual builds isn't really appealing (though I suspect in our case AOT wouldn't really hurt much). f. What if we want to support in-process plugins? Maybe we can use Espresso, but this is a road less travelled (lack of tutorials, well documented examples etc). An interesting possibility is using a mix of approaches. For the bash competitor I mentioned earlier dynamic code loading is needed because the script bytecode is loaded into the host JVM, but the Kotlin compiler itself could theoretically be statically compiled to a JNI or Panama-accessible library. We tried this before and hit compatibility errors, but didn't make any effort to resolve them. 4. What about CRaC? It's Linux only so isn't interesting to us, given that most devs are on Windows/macOS. The benefits for Linux servers are clear though. Obvious question - can you make a snapshot on one machine/Linux distro, and resume them on a totally different one, or does it require a homogenous infrastructure? 5. A big reason AppCDS is nice is we get to keep the open world. This isn't only about compatibility, open worlds are just better. The most popular way to get software to desktop machines is Chrome and the web is totally open world. Apps are downloaded incrementally as the user navigates around, and companies exploit this fact aggressively. Large web sites can be far larger than would be considered practical to distribute to end user machines, and can easily update 50 times a day. Web developers have to think about latency on specific interactions, but they don't have to think about the size of the entire app and that allows them to scale up feature sets as fast as funding allows. In contrast the closed world mobile versions of their sites are a parade of horror stories in which firms have to e.g. hotpatch Dalvik to work around method count limits (Facebook), or in which code size issues nearly wrecked the entire company (Uber): https://twitter.com/StanTwinB/status/1336914412708405248 Right now code size isn't a particularly serious problem for us, but the ease of including open source libraries means footprint grows all the time. Especially for our shell scripting tool, there are tons of cool features that could be added but if we did all of them we'd probably end up with 500mb of bytecode. With an open world features can be downloaded on the fly as they get used and you can build a plugin ecosystem. The new more incremental direction of Leyden is thus welcomed and appreciated, because it feels like a lot of ground can be covered by "small" changes like upgrading AppCDS and caching compiled hotspots. Even if the results aren't as impressive as with native-image, the benefits of keeping an open world can probably make up for it, at least for our use cases.

4 6

Re: Project Leyden: Beginnings
by Anton Kozlov 01 Jun '22

01 Jun '22

Great to see Leyden started! I also think that Leyden and CRaC are working toward the same goal from different sides. Assuming a program to be a series of actions, A B C D E F In CRaC, we don't need to specify dependencies and correctness of all permutations, the only difference from the usual Java programming model is that between each subsequent pair of actions checkpoint and restore may happen. Those are rather transparent, except they may invalidate some assumptions about the program environment. So a non-trivial task for the CRaC Project is how to find these assumptions implemented in the program state. If an explicit language-level mechanism will be developed in Leyden to mark executions for build time, the same mechanism will probably be useful to mark now-wrong assumptions after restore for CRaC. So I'm excited and curious about what's going to be developed in Leyden. I also hope CRaC Project to be useful for Leyden. Not sure about pieces of the program or VM state, like a standalone JIT compilations, that would be orthogonal tasks for CRaC. But why not a closed world start image that is restored into an open world Java instance, to achieve a great start-up, warm-up, and footprint of Leyden, without sacrificing the dynamic Java features in the runtime. Thanks, Anton

2 2

Can Ahead of Time code benefit regular Java applications too?
by Julian Waters 01 Jun '22

01 Jun '22

Hi all, Since Leyden's goal has shifted from originally exploring only binaries compiled directly to native code, to "address the long-term pain points of Java’s slow startup time, slow time to peak performance, and large footprint", would there be any merit in looking at allowing native code to be embedded within jars to bypass the Interpreter at runtime? (Maybe have Ahead of Time code that replaces the Interpreter be compiled by C1, and treat it as part of the C1 pipeline so it can be profiled while being run) Ideally it'd be similar to the now defunct jaotc, but more compact (within the jar itself or perhaps the classfiles somehow) instead of compiling the Ahead of Time code into an entirely separate file which then needs to be explicitly passed to the JVM at runtime. This may or may not be a good starting point before advancing to entirely standalone Java binaries, but I digress. Perhaps the experience of the CraC team would be of some help in this area? best regards, Julian

3 4

Improve determinism in the Java language
by Ioi Lam 01 Jun '22

01 Jun '22

A lot of the recent Leyden discussion has been around "what optimizations can be done ahead of time" (e.g., static field initialization). However, I think we also need to look at a lower level. One reason that Java has been difficult to optimize ahead-of-time is the tremendous dynamism in the language. Here are a few things that I think we can do to make Java programs more deterministic so that ahead of time optimizations can be applied: 1 Deterministic Program Code A Java program can essentially rewrite itself and even the libraries it uses. Here's an example: class App { static { if (...) { MethodHandles.lookup() .defineClass(.. hacked App$Bar ...); } } static final Bar bar = new Bar(); static class Bar { .... } } - We can't effectively AOT-compile the program code because the native code may not match the runtime generated bytecodes. - We can't pre-initialize the App.bar field because its shape may be different. One idea is to disallow such code patching when Leyden is enabled. For example, we can require that to use Leyden, an application must be "prelinked", which means that as soon as the application is loaded, the classes App and App$Bar are already loaded. The defineClass() call will fail with a LinkageError (duplicated class definition). 2 Decouple class namespaces from dynamic bytecode generation This is a corollary of the above item. Java uses ClassLoader.defineClass() for BOTH namespace and dynamic bytecode generation. I would stipulate that most users of Leyden want to do the former and not the latter. We should have a new API to load a fixed set of classes into a namespace. 3 <clinit> order Java allows <clinit>s that recursively depend on each other. The result depends on the reference order of these classes. class A { static final int a = B.b++; } class B { static final int b = A.a++; } We could have a problem if the application assumes that A is always initialized before B, but the Leyden optimizer initializes them in the opposite order. We could: - Refuse to optimize classes that have mutually recursive <clinit>, or - Change the language spec to give the JVM more freedom to decide the initialization order.

2 1