From grcevski at gmail.com  Thu Dec  1 00:23:16 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Wed, 30 Nov 2022 19:23:16 -0500
Subject: Unable to create snapshots with Java modules
In-Reply-To: <d83171cc-6871-cf6c-81b8-9a448d8f9ef8@azul.com>
References: <CA+OqXEFZOROu8YiND_gyD1rgy8h_tDwJHeymka=2F6kuu39c_w@mail.gmail.com>
 <e84078f6-1537-a15e-acdf-a3dc8fb52bec@azul.com>
 <CA+OqXEGSEfFPe0AhJhvqYHQH=t7-PR04LamLJ=VUDNhoGtg5yQ@mail.gmail.com>
 <CA+OqXEEw6pJbUYbcWzZbO6ntY6GT=7D+PqYJO_gC-eJGNnY1-Q@mail.gmail.com>
 <ff614d81-e14d-ba48-d3e7-7b322bb0b775@azul.com>
 <CA+OqXEHd343o=UKbQ3VJ8Xdga-gTU7xMxXVMij2-FsmScmpp0A@mail.gmail.com>
 <d83171cc-6871-cf6c-81b8-9a448d8f9ef8@azul.com>
Message-ID: <CA+OqXEFUFRCR_04MtVuvLJVbENg_OTeU+md6p5CC19ksLeZ-zw@mail.gmail.com>

Great, thanks, I'll follow up with a PR to clean up some of the VM code
that's
not needed.

On Wed, Nov 30, 2022 at 9:18 AM Anton Kozlov <akozlov at azul.com> wrote:

> On 11/29/22 20:51, Nikola Grcevski wrote:
> > Thanks again Anton! I confirm that your patch works and it
> > is much simpler than the hacks I did in os_linux:
> > https://github.com/grcevski/crac/pull/1
>
> Thanks for confirming!
>
> > I'm guessing since the jars are recorded in URLClassPath
> > and in ModuleReferences now, those mark_classpaths are
> > probably not needed. I did a quick experiment by commenting
> > out
>
> It seems the VM code around that needs to be cleaned up. I will happily
> review a PR for that.
>
> Thanks,
> Anton
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/crac-dev/attachments/20221130/27556d24/attachment.htm>

From duke at openjdk.org  Thu Dec  1 07:24:47 2022
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 1 Dec 2022 07:24:47 GMT
Subject: [crac] RFR: Fix [ASSERT] compilation error after
 fdf4eb8dd5c2a11176eb1ec8f40b3f4133a872aa
Message-ID: <IIJsVvhEQVvxqmYEIA6iey3DtI1pLsb1c6sbWnVVwbU=.df8400e6-4b78-4d4b-aca2-330491f575c5@github.com>

../../src/hotspot/os/linux/attachListener_linux.cpp: In function 'void assert_listener_thread()':
../../src/hotspot/os/linux/attachListener_linux.cpp:389:16: error: aggregate 'ResourceMark rm' has incomplete type and cannot be defined
  389 |   ResourceMark rm; // For retrieving the thread names

-------------

Commit messages:
 - Fix [ASSERT] compilation error after fdf4eb8dd5c2a11176eb1ec8f40b3f4133a872aa

Changes: https://git.openjdk.org/crac/pull/36/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=36&range=00
  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/36.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/36/head:pull/36

PR: https://git.openjdk.org/crac/pull/36

From duke at openjdk.org  Thu Dec  1 07:54:09 2022
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 1 Dec 2022 07:54:09 GMT
Subject: [crac] RFR: 8272472: StackGuardPages test doesn't build with glibc
 2.34
Message-ID: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>

[8272472](https://bugs.openjdk.org/browse/JDK-8272472): StackGuardPages test doesn't build with glibc 2.34

../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c: In function 'set_signal_handler':
../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c:72:15: error: storage size of 'altstack' isn't constant
   72 |   static char altstack[SIGSTKSZ];
      |               ^~~~~~~~

-------------

Commit messages:
 - Update full name
 - 8272472: StackGuardPages test doesn't build with glibc 2.34

Changes: https://git.openjdk.org/crac/pull/37/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=37&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8272472
  Stats: 11 lines in 1 file changed: 9 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/crac/pull/37.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/37/head:pull/37

PR: https://git.openjdk.org/crac/pull/37

From akozlov at openjdk.org  Thu Dec  1 10:04:21 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 1 Dec 2022 10:04:21 GMT
Subject: [crac] RFR: Assume module jars persistent [v3]
In-Reply-To: <L0kfEoQLhOrJZzEFZlsRE9xzjZiKfk0YzKhISJD5Das=.05fd6afc-c74e-40ec-8395-eb256f4fea4d@github.com>
References: <k-CyV7NeVJdEC85-Gjb_l8SirJkvHMGrHA709RKJnSI=.d9983901-5216-4125-81c2-1090e666e9da@github.com>
 <L0kfEoQLhOrJZzEFZlsRE9xzjZiKfk0YzKhISJD5Das=.05fd6afc-c74e-40ec-8395-eb256f4fea4d@github.com>
Message-ID: <LAm4Dx12s80T8wa6qONM4KixWa7y0g0QflBXTnhBLIE=.45ebba16-172b-4737-9bb0-e57c9ef1af43@github.com>

On Wed, 30 Nov 2022 17:36:48 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Jar files opened by URLClassLoaders are assumed persistent. We need to handle module Jars the same way to prevent CheckpointException as reported by [1] 
>> 
>> [1] https://mail.openjdk.org/pipermail/crac-dev/2022-November/000380.html
>
> Anton Kozlov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Use SharedSecrets instead of reflection

Thanks for review!

-------------

PR: https://git.openjdk.org/crac/pull/35

From akozlov at openjdk.org  Thu Dec  1 10:06:44 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 1 Dec 2022 10:06:44 GMT
Subject: [crac] Integrated: Assume module jars persistent
In-Reply-To: <k-CyV7NeVJdEC85-Gjb_l8SirJkvHMGrHA709RKJnSI=.d9983901-5216-4125-81c2-1090e666e9da@github.com>
References: <k-CyV7NeVJdEC85-Gjb_l8SirJkvHMGrHA709RKJnSI=.d9983901-5216-4125-81c2-1090e666e9da@github.com>
Message-ID: <ZAnjz5FGX4VI9-QkgS2EMO4cp2jWQ-pwax-_IR6_31Y=.0467d966-4035-479c-a5eb-3aa7dcb8cf6f@github.com>

On Tue, 29 Nov 2022 18:06:30 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> Jar files opened by URLClassLoaders are assumed persistent. We need to handle module Jars the same way to prevent CheckpointException as reported by [1] 
> 
> [1] https://mail.openjdk.org/pipermail/crac-dev/2022-November/000380.html

This pull request has now been integrated.

Changeset: 68419ca2
Author:    Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/68419ca269d3168c9d99c662c5fc30b3ef1f3cdc
Stats:     111 lines in 5 files changed: 65 ins; 41 del; 5 mod

Assume module jars persistent

Reviewed-by: heidinga

-------------

PR: https://git.openjdk.org/crac/pull/35

From akozlov at openjdk.org  Thu Dec  1 11:34:56 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 1 Dec 2022 11:34:56 GMT
Subject: [crac] RFR: Fix [ASSERT] compilation error after
 fdf4eb8dd5c2a11176eb1ec8f40b3f4133a872aa
In-Reply-To: <IIJsVvhEQVvxqmYEIA6iey3DtI1pLsb1c6sbWnVVwbU=.df8400e6-4b78-4d4b-aca2-330491f575c5@github.com>
References: <IIJsVvhEQVvxqmYEIA6iey3DtI1pLsb1c6sbWnVVwbU=.df8400e6-4b78-4d4b-aca2-330491f575c5@github.com>
Message-ID: <f791eeceXlSAX7rwqnDXGNQgHM1yj3WMzTVyNOs7DwE=.53829336-7cf6-41d2-b5ea-40fafa89f266@github.com>

On Thu, 1 Dec 2022 07:16:54 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> ../../src/hotspot/os/linux/attachListener_linux.cpp: In function 'void assert_listener_thread()':
> ../../src/hotspot/os/linux/attachListener_linux.cpp:389:16: error: aggregate 'ResourceMark rm' has incomplete type and cannot be defined
>   389 |   ResourceMark rm; // For retrieving the thread names

Indeed, the build fails if configured with --disable-precompiled-headers, thanks!

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/36

From akozlov at openjdk.org  Thu Dec  1 11:56:14 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 1 Dec 2022 11:56:14 GMT
Subject: [crac] RFR: 8272472: StackGuardPages test doesn't build with glibc
 2.34
In-Reply-To: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>
References: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>
Message-ID: <7zYjWfcPPRr8wPqHnZjLbraTCaEKsRusL5WRGtA6S_s=.701b4f6c-16c8-44c6-bbd5-74554d8b3e64@github.com>

On Thu, 1 Dec 2022 07:44:21 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> [8272472](https://bugs.openjdk.org/browse/JDK-8272472): StackGuardPages test doesn't build with glibc 2.34
> 
> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c: In function 'set_signal_handler':
> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c:72:15: error: storage size of 'altstack' isn't constant
>    72 |   static char altstack[SIGSTKSZ];
>       |               ^~~~~~~~

I'm not sure about this. The change was made for 18 and backported to JDK17.0.2. Instead of this, I'd prefer to continue CRaC development on JDK17u [1] or on openjdk/jdk master, to avoid backporting changes useful in general. Maintaining both will create some overhead though.

[1] https://mail.openjdk.org/pipermail/crac-dev/2022-May/000230.html

-------------

PR: https://git.openjdk.org/crac/pull/37

From duke at openjdk.org  Thu Dec  1 12:27:15 2022
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 1 Dec 2022 12:27:15 GMT
Subject: [crac] Withdrawn: 8272472: StackGuardPages test doesn't build with
 glibc 2.34
In-Reply-To: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>
References: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>
Message-ID: <f8ggpPjQNsvF997FHZlmNo5qBrBbeAUYB1ILYOtkPOY=.361ca477-d1c6-4aa9-bab4-2d8870c442e2@github.com>

On Thu, 1 Dec 2022 07:44:21 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> [8272472](https://bugs.openjdk.org/browse/JDK-8272472): StackGuardPages test doesn't build with glibc 2.34
> 
> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c: In function 'set_signal_handler':
> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c:72:15: error: storage size of 'altstack' isn't constant
>    72 |   static char altstack[SIGSTKSZ];
>       |               ^~~~~~~~

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/crac/pull/37

From duke at openjdk.org  Thu Dec  1 12:27:20 2022
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 1 Dec 2022 12:27:20 GMT
Subject: [crac] Integrated: Fix [ASSERT] compilation error after
 fdf4eb8dd5c2a11176eb1ec8f40b3f4133a872aa
In-Reply-To: <IIJsVvhEQVvxqmYEIA6iey3DtI1pLsb1c6sbWnVVwbU=.df8400e6-4b78-4d4b-aca2-330491f575c5@github.com>
References: <IIJsVvhEQVvxqmYEIA6iey3DtI1pLsb1c6sbWnVVwbU=.df8400e6-4b78-4d4b-aca2-330491f575c5@github.com>
Message-ID: <k4vtRsIKWO7sjkEBrjxSGC4f4PWLaEylHM4J6ET2fbY=.646c8952-da36-40c3-b318-c35c02c28dc3@github.com>

On Thu, 1 Dec 2022 07:16:54 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> ../../src/hotspot/os/linux/attachListener_linux.cpp: In function 'void assert_listener_thread()':
> ../../src/hotspot/os/linux/attachListener_linux.cpp:389:16: error: aggregate 'ResourceMark rm' has incomplete type and cannot be defined
>   389 |   ResourceMark rm; // For retrieving the thread names

This pull request has now been integrated.

Changeset: 9272770a
Author:    Jan Kratochvil <jkratochvil at azul.com>
Committer: Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/9272770a1fcd3a4c58373ca90ef04737e2409e28
Stats:     1 line in 1 file changed: 1 ins; 0 del; 0 mod

Fix [ASSERT] compilation error after fdf4eb8dd5c2a11176eb1ec8f40b3f4133a872aa

Reviewed-by: akozlov

-------------

PR: https://git.openjdk.org/crac/pull/36

From grcevski at gmail.com  Fri Dec  2 19:30:02 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Fri, 2 Dec 2022 14:30:02 -0500
Subject: Creating snapshots with dynamically loaded JARs
Message-ID: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>

Hello crac-dev,

While experimenting more with CRaC and Elasticsearch I encountered one
more issue and I thought I should bring it up for discussion.

We load some JARs dynamically and these don't get marked as 'persistent'.

For example:

final ClassLoader classLoader = buildClassLoader();
try {
   final Class<? extends Supplier<?>> cls = (Class<? extends
Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
...
} catch (ReflectiveOperationException e) {
   throw new RuntimeException("Failed to load ANSI console", e);
}

private static ClassLoader buildClassLoader() {
   final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");

   try {
      final URL[] urls = Files.list(libDir)
                                  .filter(each ->
each.getFileName().toString().endsWith(".jar"))
                                   .map(SimpleModule::pathToURL)
                                   .toArray(URL[]::new);

       return URLClassLoader.newInstance(urls,
SimpleModule.class.getClassLoader());
    } catch (IOException e) {
       throw new RuntimeException("Failed to list jars in [" + libDir
+ "]: " + e.getMessage(), e);
    }
}

I think typically we'd say that these should be closed and recreated
on snapshot and restore,
but it's not easy to do with ClassLoaders. I think we'd need to lose
the reference and have
GC collect the class loader for all the files to close.

I was able to work around this issue by putting all the jars we might
load dynamically
on the class path with -cp.

I'm wondering if the work around is the correct 'fix' or is there something
else that should be done? If the URLs are files, I suppose we can mark those as
persistent?

I made a GitHub repo with all the code that shows the issue here:
https://github.com/grcevski/openjdk_CRaC_simplemodule

Thanks,
Nikola

From heidinga at redhat.com  Fri Dec  2 19:53:20 2022
From: heidinga at redhat.com (Dan Heidinga)
Date: Fri, 2 Dec 2022 14:53:20 -0500
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
Message-ID: <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>

Do you need the classloader to be able to load new classes after the
checkpoint is restored?  If not, there's a "close()" method on
URLClassLoader that should close the jar and file URLs and "makes a best
effort attempt to close all opened files".

Obviously not a complete solution but it may work around your issue for now.

--Dan

On Fri, Dec 2, 2022 at 2:30 PM Nikola Grcevski <grcevski at gmail.com> wrote:

> Hello crac-dev,
>
> While experimenting more with CRaC and Elasticsearch I encountered one
> more issue and I thought I should bring it up for discussion.
>
> We load some JARs dynamically and these don't get marked as 'persistent'.
>
> For example:
>
> final ClassLoader classLoader = buildClassLoader();
> try {
>    final Class<? extends Supplier<?>> cls = (Class<? extends
>
> Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
> ...
> } catch (ReflectiveOperationException e) {
>    throw new RuntimeException("Failed to load ANSI console", e);
> }
>
> private static ClassLoader buildClassLoader() {
>    final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");
>
>    try {
>       final URL[] urls = Files.list(libDir)
>                                   .filter(each ->
> each.getFileName().toString().endsWith(".jar"))
>                                    .map(SimpleModule::pathToURL)
>                                    .toArray(URL[]::new);
>
>        return URLClassLoader.newInstance(urls,
> SimpleModule.class.getClassLoader());
>     } catch (IOException e) {
>        throw new RuntimeException("Failed to list jars in [" + libDir
> + "]: " + e.getMessage(), e);
>     }
> }
>
> I think typically we'd say that these should be closed and recreated
> on snapshot and restore,
> but it's not easy to do with ClassLoaders. I think we'd need to lose
> the reference and have
> GC collect the class loader for all the files to close.
>
> I was able to work around this issue by putting all the jars we might
> load dynamically
> on the class path with -cp.
>
> I'm wondering if the work around is the correct 'fix' or is there something
> else that should be done? If the URLs are files, I suppose we can mark
> those as
> persistent?
>
> I made a GitHub repo with all the code that shows the issue here:
> https://github.com/grcevski/openjdk_CRaC_simplemodule
>
> Thanks,
> Nikola
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/crac-dev/attachments/20221202/207784f7/attachment.htm>

From grcevski at gmail.com  Fri Dec  2 20:09:42 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Fri, 2 Dec 2022 20:09:42 +0000
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
 <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
Message-ID: <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>

Thanks Dan, I?ll give that a shot. We?ll need the classloader after restore, but I can recreate it.
________________________________
From: Dan Heidinga <heidinga at redhat.com>
Sent: Friday, December 2, 2022 2:53:20 PM
To: Nikola Grcevski <grcevski at gmail.com>
Cc: crac-dev at openjdk.org <crac-dev at openjdk.org>
Subject: Re: Creating snapshots with dynamically loaded JARs

Do you need the classloader to be able to load new classes after the checkpoint is restored?  If not, there's a "close()" method on URLClassLoader that should close the jar and file URLs and "makes a best effort attempt to close all opened files".

Obviously not a complete solution but it may work around your issue for now.

--Dan

On Fri, Dec 2, 2022 at 2:30 PM Nikola Grcevski <grcevski at gmail.com<mailto:grcevski at gmail.com>> wrote:
Hello crac-dev,

While experimenting more with CRaC and Elasticsearch I encountered one
more issue and I thought I should bring it up for discussion.

We load some JARs dynamically and these don't get marked as 'persistent'.

For example:

final ClassLoader classLoader = buildClassLoader();
try {
   final Class<? extends Supplier<?>> cls = (Class<? extends
Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
...
} catch (ReflectiveOperationException e) {
   throw new RuntimeException("Failed to load ANSI console", e);
}

private static ClassLoader buildClassLoader() {
   final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");

   try {
      final URL[] urls = Files.list(libDir)
                                  .filter(each ->
each.getFileName().toString().endsWith(".jar"))
                                   .map(SimpleModule::pathToURL)
                                   .toArray(URL[]::new);

       return URLClassLoader.newInstance(urls,
SimpleModule.class.getClassLoader());
    } catch (IOException e) {
       throw new RuntimeException("Failed to list jars in [" + libDir
+ "]: " + e.getMessage(), e);
    }
}

I think typically we'd say that these should be closed and recreated
on snapshot and restore,
but it's not easy to do with ClassLoaders. I think we'd need to lose
the reference and have
GC collect the class loader for all the files to close.

I was able to work around this issue by putting all the jars we might
load dynamically
on the class path with -cp.

I'm wondering if the work around is the correct 'fix' or is there something
else that should be done? If the URLs are files, I suppose we can mark those as
persistent?

I made a GitHub repo with all the code that shows the issue here:
https://github.com/grcevski/openjdk_CRaC_simplemodule

Thanks,
Nikola

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/crac-dev/attachments/20221202/28f0c779/attachment-0001.htm>

From grcevski at gmail.com  Fri Dec  2 23:53:58 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Fri, 2 Dec 2022 18:53:58 -0500
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
 <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
 <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>
Message-ID: <CA+OqXEGm_mG=87LEvzOn1PMcGboALoGbKsKxVTDmAFuJoKQBjg@mail.gmail.com>

That didn't seem to work, I guess something isn't quite closed.

I changed the code as this:

final URLClassLoader classLoader = buildClassLoader();
final Supplier<?> console = buildConsoleLoader(classLoader);
classLoader.close();
Core.checkpointRestore();

One thing that's confusing to me is that it seems to complain that
the directory is open, rather than any files? I think it might be a bug?

Exception in thread "main" jdk.crac.CheckpointException
    at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
    at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
    at java.base/jdk.crac.Core.checkpointRestore(Core.java:231)
    at co.elastic.simple.SimpleModule.main(SimpleModule.java:21)
    Suppressed: jdk.crac.impl.CheckpointOpenFileException:
/<snip>/simplemodule/lib/ansi-console
        at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
        at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
        ... 3 more
    Suppressed: jdk.crac.impl.CheckpointOpenFileException:
/<snip>/simplemodule/lib/ansi-console
        at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
        at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)

The URL class loader is created with a
directory(simplemodule/lib/ansi-console), which contains multiple
jars.

Thanks again,
Nikola

On Fri, Dec 2, 2022 at 3:09 PM Nikola Grcevski <grcevski at gmail.com> wrote:
>
> Thanks Dan, I?ll give that a shot. We?ll need the classloader after restore, but I can recreate it.
> ________________________________
> From: Dan Heidinga <heidinga at redhat.com>
> Sent: Friday, December 2, 2022 2:53:20 PM
> To: Nikola Grcevski <grcevski at gmail.com>
> Cc: crac-dev at openjdk.org <crac-dev at openjdk.org>
> Subject: Re: Creating snapshots with dynamically loaded JARs
>
> Do you need the classloader to be able to load new classes after the checkpoint is restored?  If not, there's a "close()" method on URLClassLoader that should close the jar and file URLs and "makes a best effort attempt to close all opened files".
>
> Obviously not a complete solution but it may work around your issue for now.
>
> --Dan
>
> On Fri, Dec 2, 2022 at 2:30 PM Nikola Grcevski <grcevski at gmail.com> wrote:
>
> Hello crac-dev,
>
> While experimenting more with CRaC and Elasticsearch I encountered one
> more issue and I thought I should bring it up for discussion.
>
> We load some JARs dynamically and these don't get marked as 'persistent'.
>
> For example:
>
> final ClassLoader classLoader = buildClassLoader();
> try {
>    final Class<? extends Supplier<?>> cls = (Class<? extends
> Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
> ...
> } catch (ReflectiveOperationException e) {
>    throw new RuntimeException("Failed to load ANSI console", e);
> }
>
> private static ClassLoader buildClassLoader() {
>    final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");
>
>    try {
>       final URL[] urls = Files.list(libDir)
>                                   .filter(each ->
> each.getFileName().toString().endsWith(".jar"))
>                                    .map(SimpleModule::pathToURL)
>                                    .toArray(URL[]::new);
>
>        return URLClassLoader.newInstance(urls,
> SimpleModule.class.getClassLoader());
>     } catch (IOException e) {
>        throw new RuntimeException("Failed to list jars in [" + libDir
> + "]: " + e.getMessage(), e);
>     }
> }
>
> I think typically we'd say that these should be closed and recreated
> on snapshot and restore,
> but it's not easy to do with ClassLoaders. I think we'd need to lose
> the reference and have
> GC collect the class loader for all the files to close.
>
> I was able to work around this issue by putting all the jars we might
> load dynamically
> on the class path with -cp.
>
> I'm wondering if the work around is the correct 'fix' or is there something
> else that should be done? If the URLs are files, I suppose we can mark those as
> persistent?
>
> I made a GitHub repo with all the code that shows the issue here:
> https://github.com/grcevski/openjdk_CRaC_simplemodule
>
> Thanks,
> Nikola
>

From grcevski at gmail.com  Mon Dec  5 17:48:04 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Mon, 5 Dec 2022 12:48:04 -0500
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <CA+OqXEGm_mG=87LEvzOn1PMcGboALoGbKsKxVTDmAFuJoKQBjg@mail.gmail.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
 <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
 <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>
 <CA+OqXEGm_mG=87LEvzOn1PMcGboALoGbKsKxVTDmAFuJoKQBjg@mail.gmail.com>
Message-ID: <CA+OqXEFymt5Tesi+BeG=NqEx=qoaoSYENHOCywQ3MSx9LGRQ0Q@mail.gmail.com>

It's actually our bug :), which I ported to the simple example, the
Files.list() stream is never closed. Sorry for the noise.

Nikola

On Fri, Dec 2, 2022 at 6:53 PM Nikola Grcevski <grcevski at gmail.com> wrote:
>
> That didn't seem to work, I guess something isn't quite closed.
>
> I changed the code as this:
>
> final URLClassLoader classLoader = buildClassLoader();
> final Supplier<?> console = buildConsoleLoader(classLoader);
> classLoader.close();
> Core.checkpointRestore();
>
> One thing that's confusing to me is that it seems to complain that
> the directory is open, rather than any files? I think it might be a bug?
>
> Exception in thread "main" jdk.crac.CheckpointException
>     at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
>     at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
>     at java.base/jdk.crac.Core.checkpointRestore(Core.java:231)
>     at co.elastic.simple.SimpleModule.main(SimpleModule.java:21)
>     Suppressed: jdk.crac.impl.CheckpointOpenFileException:
> /<snip>/simplemodule/lib/ansi-console
>         at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
>         at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
>         ... 3 more
>     Suppressed: jdk.crac.impl.CheckpointOpenFileException:
> /<snip>/simplemodule/lib/ansi-console
>         at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:87)
>         at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
>
> The URL class loader is created with a
> directory(simplemodule/lib/ansi-console), which contains multiple
> jars.
>
> Thanks again,
> Nikola
>
> On Fri, Dec 2, 2022 at 3:09 PM Nikola Grcevski <grcevski at gmail.com> wrote:
> >
> > Thanks Dan, I?ll give that a shot. We?ll need the classloader after restore, but I can recreate it.
> > ________________________________
> > From: Dan Heidinga <heidinga at redhat.com>
> > Sent: Friday, December 2, 2022 2:53:20 PM
> > To: Nikola Grcevski <grcevski at gmail.com>
> > Cc: crac-dev at openjdk.org <crac-dev at openjdk.org>
> > Subject: Re: Creating snapshots with dynamically loaded JARs
> >
> > Do you need the classloader to be able to load new classes after the checkpoint is restored?  If not, there's a "close()" method on URLClassLoader that should close the jar and file URLs and "makes a best effort attempt to close all opened files".
> >
> > Obviously not a complete solution but it may work around your issue for now.
> >
> > --Dan
> >
> > On Fri, Dec 2, 2022 at 2:30 PM Nikola Grcevski <grcevski at gmail.com> wrote:
> >
> > Hello crac-dev,
> >
> > While experimenting more with CRaC and Elasticsearch I encountered one
> > more issue and I thought I should bring it up for discussion.
> >
> > We load some JARs dynamically and these don't get marked as 'persistent'.
> >
> > For example:
> >
> > final ClassLoader classLoader = buildClassLoader();
> > try {
> >    final Class<? extends Supplier<?>> cls = (Class<? extends
> > Supplier<?>>)classLoader.loadClass("org.elasticsearch.io.ansi.AnsiConsoleLoader");
> > ...
> > } catch (ReflectiveOperationException e) {
> >    throw new RuntimeException("Failed to load ANSI console", e);
> > }
> >
> > private static ClassLoader buildClassLoader() {
> >    final Path libDir = Path.of(".").resolve("lib").resolve("ansi-console");
> >
> >    try {
> >       final URL[] urls = Files.list(libDir)
> >                                   .filter(each ->
> > each.getFileName().toString().endsWith(".jar"))
> >                                    .map(SimpleModule::pathToURL)
> >                                    .toArray(URL[]::new);
> >
> >        return URLClassLoader.newInstance(urls,
> > SimpleModule.class.getClassLoader());
> >     } catch (IOException e) {
> >        throw new RuntimeException("Failed to list jars in [" + libDir
> > + "]: " + e.getMessage(), e);
> >     }
> > }
> >
> > I think typically we'd say that these should be closed and recreated
> > on snapshot and restore,
> > but it's not easy to do with ClassLoaders. I think we'd need to lose
> > the reference and have
> > GC collect the class loader for all the files to close.
> >
> > I was able to work around this issue by putting all the jars we might
> > load dynamically
> > on the class path with -cp.
> >
> > I'm wondering if the work around is the correct 'fix' or is there something
> > else that should be done? If the URLs are files, I suppose we can mark those as
> > persistent?
> >
> > I made a GitHub repo with all the code that shows the issue here:
> > https://github.com/grcevski/openjdk_CRaC_simplemodule
> >
> > Thanks,
> > Nikola
> >

From akozlov at azul.com  Wed Dec  7 16:46:44 2022
From: akozlov at azul.com (Anton Kozlov)
Date: Wed, 7 Dec 2022 18:46:44 +0200
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <CA+OqXEFymt5Tesi+BeG=NqEx=qoaoSYENHOCywQ3MSx9LGRQ0Q@mail.gmail.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
 <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
 <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>
 <CA+OqXEGm_mG=87LEvzOn1PMcGboALoGbKsKxVTDmAFuJoKQBjg@mail.gmail.com>
 <CA+OqXEFymt5Tesi+BeG=NqEx=qoaoSYENHOCywQ3MSx9LGRQ0Q@mail.gmail.com>
Message-ID: <f12ee5bb-eda1-a673-c246-14c8d3e88984@azul.com>

On 12/5/22 19:48, Nikola Grcevski wrote:
> It's actually our bug :), which I ported to the simple example, the
> Files.list() stream is never closed. Sorry for the noise.


Thanks for the example to play with. So after closing the stream for Files.list(), the dynamically loaded jar is marked "persistent" and the checkpoint is successful, right? It may be a bit of a problem that the checkpoint implicitly catches everything an application may open. Would it be useful to print all persistent jars, like [1]

Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/./lib/ansi-console/jansi-2.4.0.jar
Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/./lib/ansi-console/elasticsearch-ansi-console-8.7.0-SNAPSHOT.jar
Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/target/simplemodule.jar
CR: Checkpoint ...
Killed

I'm not a big fan of persistent files, but while they are there, some additional information should at least make assumptions that got into the image more visible.

[1] https://github.com/AntonKozlov/crac/commit/492fb87e8371529750701969604a863cacadf724

Thanks,
Anton

From grcevski at gmail.com  Thu Dec  8 15:18:53 2022
From: grcevski at gmail.com (Nikola Grcevski)
Date: Thu, 8 Dec 2022 10:18:53 -0500
Subject: Creating snapshots with dynamically loaded JARs
In-Reply-To: <f12ee5bb-eda1-a673-c246-14c8d3e88984@azul.com>
References: <CA+OqXEEYc9GcKnvSo3ryqSTQBygC3xCyhLQUx8droTp_LoRoBQ@mail.gmail.com>
 <CAJq4Gi4xG4nvL2oz_SiiPvPfwjYiGRJHzZa7TyaDdQNBY=PDPw@mail.gmail.com>
 <MW4PR10MB639623E3C723A7D07DC90E73F7179@MW4PR10MB6396.namprd10.prod.outlook.com>
 <CA+OqXEGm_mG=87LEvzOn1PMcGboALoGbKsKxVTDmAFuJoKQBjg@mail.gmail.com>
 <CA+OqXEFymt5Tesi+BeG=NqEx=qoaoSYENHOCywQ3MSx9LGRQ0Q@mail.gmail.com>
 <f12ee5bb-eda1-a673-c246-14c8d3e88984@azul.com>
Message-ID: <CA+OqXEFHSrCSgcKRR1n1cUu48hc_-sOs6SUox-OsBWsbziOxoQ@mail.gmail.com>

Yes, I think it would be super useful when debugging problems of why a
path wasn't marked as "persistent". An INFO log message would be
awesome.

Thanks,
Nikola

On Wed, Dec 7, 2022 at 11:46 AM Anton Kozlov <akozlov at azul.com> wrote:
>
> On 12/5/22 19:48, Nikola Grcevski wrote:
> > It's actually our bug :), which I ported to the simple example, the
> > Files.list() stream is never closed. Sorry for the noise.
>
>
> Thanks for the example to play with. So after closing the stream for Files.list(), the dynamically loaded jar is marked "persistent" and the checkpoint is successful, right? It may be a bit of a problem that the checkpoint implicitly catches everything an application may open. Would it be useful to print all persistent jars, like [1]
>
> Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
> INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/./lib/ansi-console/jansi-2.4.0.jar
> Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
> INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/./lib/ansi-console/elasticsearch-ansi-console-8.7.0-SNAPSHOT.jar
> Dec 07, 2022 6:32:21 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
> INFO: Assuming persistent /home/anton/Downloads/openjdk_CRaC_simplemodule/target/simplemodule.jar
> CR: Checkpoint ...
> Killed
>
> I'm not a big fan of persistent files, but while they are there, some additional information should at least make assumptions that got into the image more visible.
>
> [1] https://github.com/AntonKozlov/crac/commit/492fb87e8371529750701969604a863cacadf724
>
> Thanks,
> Anton

From akozlov at openjdk.org  Fri Dec  9 12:03:34 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 9 Dec 2022 12:03:34 GMT
Subject: [crac] RFR: Print peristent JAR assumptions
Message-ID: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>

When an application creates URLClassLoader with the JAR, the JAR file will be assumed persistent, that is, it should be available on the restore. To make this more evident, the PR adds reporting for such JARs.

I'm not a big fan of persistent files and the implications, I think it may be done better, but while they are there, better to make them more usable.

-------------

Commit messages:
 - Update message
 - Print persistent jars on checkpoint

Changes: https://git.openjdk.org/crac/pull/38/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=38&range=00
  Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/38.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/38/head:pull/38

PR: https://git.openjdk.org/crac/pull/38

From heidinga at openjdk.org  Fri Dec 16 19:26:19 2022
From: heidinga at openjdk.org (Dan Heidinga)
Date: Fri, 16 Dec 2022 19:26:19 GMT
Subject: [crac] RFR: Print peristent JAR assumptions
In-Reply-To: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
References: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
Message-ID: <vRZBN6UF8Atd7TN-0xfCbTWVRhfa5Pvf6OM6acT7WTY=.f0d18340-1f18-4931-9ddb-688b0ac13f40@github.com>

On Fri, 9 Dec 2022 11:56:46 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> When an application creates URLClassLoader with the JAR, the JAR file will be assumed persistent, that is, it should be available on the restore. To make this more evident, the PR adds reporting for such JARs.
> 
> I'm not a big fan of persistent files and the implications, I think it may be done better, but while they are there, better to make them more usable.

Looks straight forward

-------------

Marked as reviewed by heidinga (Committer).

PR: https://git.openjdk.org/crac/pull/38

From akozlov at openjdk.org  Mon Dec 19 17:53:28 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Mon, 19 Dec 2022 17:53:28 GMT
Subject: [crac] RFR: Print peristent JAR assumptions
In-Reply-To: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
References: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
Message-ID: <RqX1YBUqN8KNbfrW-neO4ry4R572zVOf6TLvD2KP47o=.e352a75b-d426-499b-8fe3-cb4212888ba8@github.com>

On Fri, 9 Dec 2022 11:56:46 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> When an application creates URLClassLoader with the JAR, the JAR file will be assumed persistent, that is, it should be available on the restore. To make this more evident, the PR adds reporting for such JARs.
> 
> I'm not a big fan of persistent files and the implications, I think it may be done better, but while they are there, better to make them more usable.

Thanks for review!

-------------

PR: https://git.openjdk.org/crac/pull/38

From akozlov at openjdk.org  Mon Dec 19 17:56:21 2022
From: akozlov at openjdk.org (Anton Kozlov)
Date: Mon, 19 Dec 2022 17:56:21 GMT
Subject: [crac] Integrated: Print peristent JAR assumptions
In-Reply-To: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
References: <svxiRHVlGEY38NoKR5nYoHRIsmgt_qkaxK-oYFMezMA=.2db1906a-2cc1-48a6-8adf-26f112abb6d2@github.com>
Message-ID: <aVP5sMojubBJuOhB7Focez9l_UN0aXI3JjsBLodjEaY=.35356906-ab82-4fc1-ade9-3c5d36e15bda@github.com>

On Fri, 9 Dec 2022 11:56:46 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> When an application creates URLClassLoader with the JAR, the JAR file will be assumed persistent, that is, it should be available on the restore. To make this more evident, the PR adds reporting for such JARs.
> 
> I'm not a big fan of persistent files and the implications, I think it may be done better, but while they are there, better to make them more usable.

This pull request has now been integrated.

Changeset: 65e0785e
Author:    Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/65e0785ed583abbd995965de7de8490623321054
Stats:     2 lines in 1 file changed: 2 ins; 0 del; 0 mod

Print peristent JAR assumptions

Reviewed-by: heidinga

-------------

PR: https://git.openjdk.org/crac/pull/38