Creating snapshots with dynamically loaded JARs

[Nikola Grcevski]

It's actually our bug :), which I ported to the simple example, the
Files.list() stream is never closed. Sorry for the noise.

Nikola

On Fri, Dec 2, 2022 at 6:53 PM Nikola Grcevski <grcevski at gmail.com> wrote:
>
> That didn't seem to work, I guess something isn't quite closed.
>
> I changed the code as this:
>
> final URLClassLoader classLoader = buildClassLoader();
> final Supplier<?> console = buildConsoleLoader(classLoader);
> classLoader.close();
> Core.checkpointRestore();
>
> One thing that's confusing to me is that it seems to complain that
> the directory is open, rather than any files? I think it might be a bug?
>
> Exception in thread "main" jdk.crac.CheckpointException
>     at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
>     at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
>     at java.base/jdk.crac.Core.checkpointRestore(Core.java:231)
>     at co.elastic.simple.SimpleModule.main(SimpleModule.java:21)
>     Suppressed: jdk.crac.impl.CheckpointOpenFileException:
> /<snip>/simplemodule/lib/ansi-console
>         at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
>         at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
>         ... 3 more
>     Suppressed: jdk.crac.impl.CheckpointOpenFileException:
> /<snip>/simplemodule/lib/ansi-console
>         at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
>         at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
>
> The URL class loader is created with a
> directory(simplemodule/lib/ansi-console), which contains multiple
> jars.
>
> Thanks again,
> Nikola
>
> On Fri, Dec 2, 2022 at 3:09 PM Nikola Grcevski <grcevski at gmail.com> wrote:
> >
> > Thanks Dan, I’ll give that a shot. We’ll need the classloader after restore, but I can recreate it.
> > ________________________________
> > From: Dan Heidinga <heidinga at redhat.com>
> > Sent: Friday, December 2, 2022 2:53:20 PM
> > To: Nikola Grcevski <grcevski at gmail.com>
> > Cc: crac-dev at openjdk.org <crac-dev at openjdk.org>
> > Subject: Re: Creating snapshots with dynamically loaded JARs
> >
> > Do you need the classloader to be able to load new classes after the checkpoint is restored?  If not, there's a "close()" method on URLClassLoader that should close the jar and file URLs and "makes a best effort attempt to close all opened files".
> >
> > Obviously not a complete solution but it may work around your issue for now.
> >
> > --Dan
> >
> > On Fri, Dec 2, 2022 at 2:30 PM Nikola Grcevski <grcevski at gmail.com> wrote:
> >
> > Hello crac-dev,
> >
> > While experimenting more with CRaC and Elasticsearch I encountered one
> > more issue and I thought I should bring it up for discussion.
> >
> > We load some JARs dynamically and these don't get marked as 'persistent'.
> >
> > For example:
> >
> > final ClassLoader classLoader = buildClassLoader();
> > try {
> >    final Class<? extends Supplier<?>> cls = (Class<? extends
> > Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
> > ...
> > } catch (ReflectiveOperationException e) {
> >    throw new RuntimeException("Failed to load ANSI console", e);
> > }
> >
> > private static ClassLoader buildClassLoader() {
> >    final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");
> >
> >    try {
> >       final URL[] urls = Files.list(libDir)
> >                                   .filter(each ->
> > each.getFileName().toString().endsWith(".jar"))
> >                                    .map(SimpleModule::pathToURL)
> >                                    .toArray(URL[]::new);
> >
> >        return URLClassLoader.newInstance(urls,
> > SimpleModule.class.getClassLoader());
> >     } catch (IOException e) {
> >        throw new RuntimeException("Failed to list jars in [" + libDir
> > + "]: " + e.getMessage(), e);
> >     }
> > }
> >
> > I think typically we'd say that these should be closed and recreated
> > on snapshot and restore,
> > but it's not easy to do with ClassLoaders. I think we'd need to lose
> > the reference and have
> > GC collect the class loader for all the files to close.
> >
> > I was able to work around this issue by putting all the jars we might
> > load dynamically
> > on the class path with -cp.
> >
> > I'm wondering if the work around is the correct 'fix' or is there something
> > else that should be done? If the URLs are files, I suppose we can mark those as
> > persistent?
> >
> > I made a GitHub repo with all the code that shows the issue here:
> > https://github.com/grcevski/openjdk_CRaC_simplemodule
> >
> > Thanks,
> > Nikola
> >