From duke at openjdk.org  Wed Feb  1 08:17:22 2023
From: duke at openjdk.org (Radim Vansa)
Date: Wed, 1 Feb 2023 08:17:22 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v2]
In-Reply-To: <iLSQMMaMupvH8eEfwFnlZXHntSb0adPDnG5L9P7kKS0=.bdcd8d07-41c5-4732-9572-0efec59f802a@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <iLSQMMaMupvH8eEfwFnlZXHntSb0adPDnG5L9P7kKS0=.bdcd8d07-41c5-4732-9572-0efec59f802a@github.com>
Message-ID: <-BxmxgjSXmGIISwgHPyrTfg_6c8ciz25ZH-ASqbM12w=.bfff8c26-1fe3-4d1e-b5df-f44d8491eede@github.com>

On Tue, 31 Jan 2023 17:11:24 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Tracks `java.io.FileDescriptor` instances as CRaC resource; before checkpoint these are reported and if not allow-listed (e.g. as opened as standard descriptors) an exception is thrown. Further investigation can use system property `jdk.crac.collect-fd-stacktraces=true` to record origin of those file descriptors.
>> File descriptors claimed in Java code are passed to native; native code checks all open file descriptors and reports error if there's an unexpected FD that is not included in the list passed previously.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Drop native FDs tracking

@AntonKozlov I've dropped the native FDs tracking.

-------------

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Wed Feb  1 14:30:36 2023
From: duke at openjdk.org (Radim Vansa)
Date: Wed, 1 Feb 2023 14:30:36 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
Message-ID: <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>

> Tracks `java.io.FileDescriptor` instances as CRaC resource; before checkpoint these are reported and if not allow-listed (e.g. as opened as standard descriptors) an exception is thrown. Further investigation can use system property `jdk.crac.collect-fd-stacktraces=true` to record origin of those file descriptors.
> File descriptors claimed in Java code are passed to native; native code checks all open file descriptors and reports error if there's an unexpected FD that is not included in the list passed previously.

Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:

  Empty commit to trigger GHA

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/43/files
  - new: https://git.openjdk.org/crac/pull/43/files/91b223ef..723d6f32

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=43&range=02
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=43&range=01-02

  Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/43.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/43/head:pull/43

PR: https://git.openjdk.org/crac/pull/43

From heidinga at openjdk.org  Wed Feb  1 14:34:31 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Wed, 1 Feb 2023 14:34:31 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>

On Mon, 30 Jan 2023 02:44:23 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

This is a nice piece of engineering to make it work.  I'm concerned about the maintainability and correctness of the approach though, when running on different systems / glibcs and over time as glibc is recompiled.

Ashutosh suggested an alternative approach back in Feb 2022 [0] [1] of providing a single "-XX:[+-]PortableCode" with a common enough baseline set of features and the use of GLIBC_TUNABLES to control the glibc bindings.  

This is very similar to the approach the OpenJ9 team has taken in their CRIU efforts and have found that it works reasonable well for the use cases they have targeted (mostly restore inside a container).

[0] http://cr.openjdk.java.net/~heidinga/crac/Portability_of_checkpoints.pdf
[1] https://mail.openjdk.org/pipermail/crac-dev/2022-February/000114.html

src/hotspot/os/linux/os_linux.cpp line 6050:

> 6048:         unsigned long l = strtoul(name, &endptr, 10);
> 6049:         pid_t ent_tid = l;
> 6050:         if (l >= LONG_MAX || ent_tid != (long) l) {

`l` is an `unsigned long` and I think `LONG_MAX` is a signed value as there's also a `ULONG_MAX`.  Are we using this to validate that `l` is a positive value?  Otherwise, I'm not sure I understand this condition.

src/hotspot/os/linux/os_linux.cpp line 6074:

> 6072:   static pthread_mutex_t signalled_mutex;
> 6073:   static pthread_cond_t signalled_cond;
> 6074:   static size_t signalled;

The above `template<class Callback> bool all_threads` also uses a "signalled" local variable.  It might be clearer to rename one of these so future readers don't have to think about scoping and shadowing rules

src/hotspot/os/linux/os_linux.cpp line 6086:

> 6084:     err = pthread_cond_signal(&signalled_cond);
> 6085:     assert(!err, "pthread error");
> 6086:     pthread_mutex_t unused_mutex = PTHREAD_MUTEX_INITIALIZER;

Waiting on a local mutex deserves a comment to explain the approach.  I think we're waiting until the thaw operation signals the resume condition but better to be explicit in the code on the intended interactions between the waiters

src/hotspot/os/linux/os_linux.cpp line 6177:

> 6175:   assert(sig == RESTORE_SIGNAL, "got what requested");
> 6176: 
> 6177:   linux_ifunc_reset();

Assuming we're freezing all the threads to allow resetting the linux_ifuncs, it would be good to comment explicitly about why that's the right approach and/or necessary.

Also, what if we stopped a thread that was about to be call one of these ifuncs - that had already loaded the old address into a register - before we did the reset?  Wouldn't it still have the wrong value and with the right (wrong?) thread scheduling, still abort after the restore?

Basically, I think there is probably a race here still.

src/hotspot/os/linux/os_linux_ifunc.cpp line 43:

> 41: // These offsets are for Debian 12 x86_64:
> 42: // gdb -batch /lib64/ld-linux-x86-64.so.2 -ex
> 43: static unsigned   l_scope_offset = 0x3b0;

Committing this into the repo seems very suspect unless we have some guarantee that these offsets are stable.  Is there such a guarantee?

src/hotspot/os/linux/os_linux_ifunc.cpp line 145:

> 143:   unsigned sht;
> 144: };
> 145: static int symtab_lookup_iterate_phdr(struct dl_phdr_info *info, size_t size, void *data_voidp) {

This seems like some really cool "I made it work"-style hacking and something that would be utterly unsupportable long term.

I'd be pretty concerned if we merged this approach into the repo in this form.  If glibc provided an API to reset the ifunc bindings, then I'd be way more comfortable with the approach.

-------------

PR: https://git.openjdk.org/crac/pull/41

From heidinga at openjdk.org  Wed Feb  1 18:46:37 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Wed, 1 Feb 2023 18:46:37 GMT
Subject: [crac] RFR: Environment vars propagation into restored process
 [v3]
In-Reply-To: <j-OMCmG8CByxJL--yts3uyNvJBWnK--kIaBRFtx9aa4=.7e15beae-319c-499e-8338-0dedf233ae06@github.com>
References: <wPtfozJKMqH5af3L5xmJWkgjpvzEXkMgGUaB4JFgrUU=.47b706b2-a02c-4414-9ddd-dd376a32f23f@github.com>
 <j-OMCmG8CByxJL--yts3uyNvJBWnK--kIaBRFtx9aa4=.7e15beae-319c-499e-8338-0dedf233ae06@github.com>
Message-ID: <UsdhjLR8wjOitukUu_Q5bHkFhWXTEy2L9tgfcPRj5yE=.a2355f89-0647-4957-8678-ca895fde8bef@github.com>

On Mon, 3 Oct 2022 08:21:56 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

>> This PR provides functionality to propagate actual environment variables to a restored process, as well as the test for this functionality.
>> 
>> Env propagation is done in few steps:
>> - Store the actual environment before restoring
>> - After restoring, replace the restored `environ` with a new one.
>> - On `afterRestore` event, propagate the new environment into a restored process via `ProcessEnvironment`.
>
> Roman Marchenko has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fixing review comments

The updated test doesn't really address the concern though.  The key concern was:
> limit the env var changes to only add new env vars (no inconsistencies).

OpenJ9 did this by constraining the checkpoint env to only capture a limited subset of env vars and allowing restore to expand that set with new env vars.

The current CRaC approach still allows overriding env vars, which is something Java (and developers, and library developers) doesn't expect to happen so they haven't written their code to be aware these values can change.  Changing existing env vars on restore will just lead to inconsistent systems.  See [0] for more details on why this is the case.

[0] https://github.com/openjdk/crac/pull/30#issuecomment-1268433596

-------------

PR: https://git.openjdk.org/crac/pull/30

From duke at openjdk.org  Thu Feb  2 06:06:28 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 2 Feb 2023 06:06:28 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:

 - +comment; found by Dan Heidinga.
 - Rename signalled->retry; found by Dan Heidinga.

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/328e6b5d..b4030f8e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=01
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=00-01

  Stats: 5 lines in 1 file changed: 1 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb  2 06:06:31 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 2 Feb 2023 06:06:31 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
Message-ID: <8TxWv5mTtjySdPTIIyM_FZt3ter2Wtja5keA3J-wbws=.f06b0556-5439-4b9b-a9dc-1d355a3eb837@github.com>

On Wed, 1 Feb 2023 14:06:01 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

>> Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - +comment; found by Dan Heidinga.
>>  - Rename signalled->retry; found by Dan Heidinga.
>
> src/hotspot/os/linux/os_linux.cpp line 6050:
> 
>> 6048:         unsigned long l = strtoul(name, &endptr, 10);
>> 6049:         pid_t ent_tid = l;
>> 6050:         if (l >= LONG_MAX || ent_tid != (long) l) {
> 
> `l` is an `unsigned long` and I think `LONG_MAX` is a signed value as there's also a `ULONG_MAX`.  Are we using this to validate that `l` is a positive value?  Otherwise, I'm not sure I understand this condition.

The number must fit in `pid_t` (otherwise it is a weird Linux kernel). Therefore `int`. But there is no `strtoi`. Also PID must not be negative (otherwise it is a weird Linux kernel again). So I found this as the most simple and safe code to validate `name` contains a non-negative number which can fit in `pid_t`.

> src/hotspot/os/linux/os_linux.cpp line 6074:
> 
>> 6072:   static pthread_mutex_t signalled_mutex;
>> 6073:   static pthread_cond_t signalled_cond;
>> 6074:   static size_t signalled;
> 
> The above `template<class Callback> bool all_threads` also uses a "signalled" local variable.  It might be clearer to rename one of these so future readers don't have to think about scoping and shadowing rules

OK, yes, renamed the local variable to `retry`.

> src/hotspot/os/linux/os_linux.cpp line 6086:
> 
>> 6084:     err = pthread_cond_signal(&signalled_cond);
>> 6085:     assert(!err, "pthread error");
>> 6086:     pthread_mutex_t unused_mutex = PTHREAD_MUTEX_INITIALIZER;
> 
> Waiting on a local mutex deserves a comment to explain the approach.  I think we're waiting until the thaw operation signals the resume condition but better to be explicit in the code on the intended interactions between the waiters

Added: `// Just wait until thaw() is called. No mutex is needed for that.`

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb  2 06:11:56 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 2 Feb 2023 06:11:56 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
Message-ID: <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>

On Wed, 1 Feb 2023 14:32:01 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

> I'm concerned about the maintainability and correctness of the approach though, when running on different systems / glibcs and over time as glibc is recompiled.

Me too. This is why I also called it just "RFC". But it is not up to me whether this temporary and/or a bit fragile solution gets accepted for CRaC or not.

> Ashutosh suggested an alternative approach back in Feb 2022 [0] [1] of providing a single "-XX:[+-]PortableCode" with a common enough baseline set of features

IIUC you can just use `-XX:CPUFeatures=0` with this patch for the same effect. As my employer is all about Java Performance I find useful to provide the option of more optimal runtime JIT features.

> and the use of GLIBC_TUNABLES to control the glibc bindings.

This is what I say in Comment 0:

> If upstream glibc maintainers do not like the IFUNC reset idea [...]. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, [...]

> This is very similar to the approach the OpenJ9 team has taken in their CRIU efforts and have found that it works reasonable well for the use cases they have targeted (mostly restore inside a container).

Could you send some more specific link for that functionality?

> src/hotspot/os/linux/os_linux.cpp line 6177:
> 
>> 6175:   assert(sig == RESTORE_SIGNAL, "got what requested");
>> 6176: 
>> 6177:   linux_ifunc_reset();
> 
> Assuming we're freezing all the threads to allow resetting the linux_ifuncs, it would be good to comment explicitly about why that's the right approach and/or necessary.
> 
> Also, what if we stopped a thread that was about to be call one of these ifuncs - that had already loaded the old address into a register - before we did the reset?  Wouldn't it still have the wrong value and with the right (wrong?) thread scheduling, still abort after the restore?
> 
> Basically, I think there is probably a race here still.

You are right, there is a race, I will fix that, thanks.

> src/hotspot/os/linux/os_linux_ifunc.cpp line 43:
> 
>> 41: // These offsets are for Debian 12 x86_64:
>> 42: // gdb -batch /lib64/ld-linux-x86-64.so.2 -ex
>> 43: static unsigned   l_scope_offset = 0x3b0;
> 
> Committing this into the repo seems very suspect unless we have some guarantee that these offsets are stable.  Is there such a guarantee?

These offsets are not stable, I hope they won't change in one stable OS release but still they can. If there are any problems one can always install `*-debuginfo.rpm` or `*-dbg.deb` which will override all these assumed values. Maybe if there is no debuginfo available there should be a database of `libc.so.`6 binary hashes and refuse to run if none is found.
Still I find this patch only as a proof of concept and a some codebase to ask upstream glibc for inclusion of an IFUNC-reset functionality into glibc mainline. As I stated in Comment 0 if glibc refuses such upstreaming I believe CRaC should rather use `GLIBC_TUNABLES` anyway.
This IFUNC-reset functionality looked simple originally but in the end it got much more complicated than I expected. I would not probably implement it if I could see it will not be simple enough. But when it was already mostly written I have finished it.

> src/hotspot/os/linux/os_linux_ifunc.cpp line 145:
> 
>> 143:   unsigned sht;
>> 144: };
>> 145: static int symtab_lookup_iterate_phdr(struct dl_phdr_info *info, size_t size, void *data_voidp) {
> 
> This seems like some really cool "I made it work"-style hacking and something that would be utterly unsupportable long term.
> 
> I'd be pretty concerned if we merged this approach into the repo in this form.  If glibc provided an API to reset the ifunc bindings, then I'd be way more comfortable with the approach.

Already discussed above.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb  2 09:19:26 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 2 Feb 2023 09:19:26 GMT
Subject: [crac] RFR: 8272472: StackGuardPages test doesn't build with glibc
 2.34
In-Reply-To: <7zYjWfcPPRr8wPqHnZjLbraTCaEKsRusL5WRGtA6S_s=.701b4f6c-16c8-44c6-bbd5-74554d8b3e64@github.com>
References: <QGMxJic_3VAJjZ84VzAvpNZYC9C3NTAxWomK6oeWIEY=.d507f651-bb00-4535-bc51-c4fbbd8b2447@github.com>
 <7zYjWfcPPRr8wPqHnZjLbraTCaEKsRusL5WRGtA6S_s=.701b4f6c-16c8-44c6-bbd5-74554d8b3e64@github.com>
Message-ID: <Nn7rDCcMGuj7BFX9C_vKrfY4jqJCLSOmgwBTCuexqZ8=.d2e7e844-3435-42d0-a0ea-fbe3f1f4b553@github.com>

On Thu, 1 Dec 2022 11:53:29 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> [8272472](https://bugs.openjdk.org/browse/JDK-8272472): StackGuardPages test doesn't build with glibc 2.34
>> 
>> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c: In function 'set_signal_handler':
>> ../test/hotspot/jtreg/runtime/StackGuardPages/exeinvoke.c:72:15: error: storage size of 'altstack' isn't constant
>>    72 |   static char altstack[SIGSTKSZ];
>>       |               ^~~~~~~~
>
> I'm not sure about this. The change was made for 18 and backported to JDK17.0.2. Instead of this, I'd prefer to continue CRaC development on JDK17u [1] or on openjdk/jdk master, to avoid backporting changes useful in general. Maintaining both will create some overhead though.
> 
> [1] https://mail.openjdk.org/pipermail/crac-dev/2022-May/000230.html

@AntonKozlov So what's the plan to get CRaC builiding with new glibc? Should I create a PR to merge in the most recent jdk17u-dev?

-------------

PR: https://git.openjdk.org/crac/pull/37

From heidinga at openjdk.org  Thu Feb  2 14:38:07 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Thu, 2 Feb 2023 14:38:07 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <8TxWv5mTtjySdPTIIyM_FZt3ter2Wtja5keA3J-wbws=.f06b0556-5439-4b9b-a9dc-1d355a3eb837@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
 <8TxWv5mTtjySdPTIIyM_FZt3ter2Wtja5keA3J-wbws=.f06b0556-5439-4b9b-a9dc-1d355a3eb837@github.com>
Message-ID: <TrMfHDSnmwuRATkoZGGOwdxtj_8zAwR7FnwQRcb49aE=.d18e0bec-2adc-45b2-adc9-c4e8bed971ed@github.com>

On Thu, 2 Feb 2023 05:57:28 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> The number must fit in `pid_t` (otherwise it is a weird Linux kernel). Therefore `int`. But there is no `strtoi`.

If we're boxing to an int, then I think we should check if `l >= INT_MAX` rather than `LONG_MAX`.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb  2 14:47:08 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 2 Feb 2023 14:47:08 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <TrMfHDSnmwuRATkoZGGOwdxtj_8zAwR7FnwQRcb49aE=.d18e0bec-2adc-45b2-adc9-c4e8bed971ed@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
 <8TxWv5mTtjySdPTIIyM_FZt3ter2Wtja5keA3J-wbws=.f06b0556-5439-4b9b-a9dc-1d355a3eb837@github.com>
 <TrMfHDSnmwuRATkoZGGOwdxtj_8zAwR7FnwQRcb49aE=.d18e0bec-2adc-45b2-adc9-c4e8bed971ed@github.com>
Message-ID: <dU4SNQx1hLmcKW5WZUp7y4UZppvyd32bgECTZXP-5hQ=.fa9aa23d-15de-4ad6-8313-472e47158d7b@github.com>

On Thu, 2 Feb 2023 14:34:48 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

>> The number must fit in `pid_t` (otherwise it is a weird Linux kernel). Therefore `int`. But there is no `strtoi`. Also PID must not be negative (otherwise it is a weird Linux kernel again). So I found this as the most simple and safe code to validate `name` contains a non-negative number which can fit in `pid_t`.
>
>> The number must fit in `pid_t` (otherwise it is a weird Linux kernel). Therefore `int`. But there is no `strtoi`.
> 
> If we're boxing to an int, then I think we should check if `l >= INT_MAX` rather than `LONG_MAX`.

The `int` check is there `ent_tid != (long) l` (as nobody says `pid_t` is really an `int`). The `l >= LONG_MAX` is there to make valid the cast `(long) l`. But this is IMO a [bikeshed](https://en.wikipedia.org/wiki/Law_of_triviality).

-------------

PR: https://git.openjdk.org/crac/pull/41

From heidinga at openjdk.org  Thu Feb  2 14:53:10 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Thu, 2 Feb 2023 14:53:10 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
 <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>
Message-ID: <lHtCFrUsm3TB0V3hnynJO8_RTw3GIKhEHiAJppCqEN0=.841dfab5-ff6c-4125-b58d-75df7e897ca2@github.com>

On Thu, 2 Feb 2023 06:07:59 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> > This is very similar to the approach the OpenJ9 team has taken in their CRIU efforts and have found that it works reasonable well for the use cases they have targeted (mostly restore inside a container).
> 
> Could you send some more specific link for that functionality?

Details on the glibc tunables used by OpenJ9 are in: https://github.com/eclipse-openj9/openj9/issues/14253

A script setting them for some of the tests: https://github.com/eclipse-openj9/openj9/commit/172bcf7244116186d4edde019473db69fb5d9559

Open Liberty also sets the glibc tuneables as part of their server scripts for use with CRIU which I think shows consumers need to be aware of the issue: https://github.com/OpenLiberty/open-liberty/pull/22132

> > Ashutosh suggested an alternative approach back in Feb 2022 [0] [1] of providing a single "-XX:[+-]PortableCode" with a common enough baseline set of features
> 
> IIUC you can just use `-XX:CPUFeatures=0` with this patch for the same effect. As my employer is all about Java Performance I find useful to provide the option of more optimal runtime JIT features.

Also a big fan of Java performance and a fan of avoiding foot-guns =)  There are benefits to both approaches, I think most users will want a simple "make it portable given a good baseline system" (aka how most native binaries are built) and those who need the most performance can use the existing Hotspot enable/disable flags to allow particular cpu features.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb  2 15:31:04 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 2 Feb 2023 15:31:04 GMT
Subject: [crac] RFR: Fix failures in testsuite
Message-ID: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>

This PR fixes some failures I've noticed in the pre-submit checks and could also reproduce locally.

-------------

Commit messages:
 - Fix failures in testsuite

Changes: https://git.openjdk.org/crac/pull/44/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=44&range=00
  Stats: 8 lines in 1 file changed: 6 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/crac/pull/44.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/44/head:pull/44

PR: https://git.openjdk.org/crac/pull/44

From duke at openjdk.org  Fri Feb  3 07:59:26 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Fri, 3 Feb 2023 07:59:26 GMT
Subject: [crac] RFR: Fix failures in testsuite
In-Reply-To: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
References: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
Message-ID: <x6_SCd9ghn8jbWaNcuWO3QKjcaS1i9oZxQ4g-_88HwU=.48fad230-2a55-453d-a33b-17a62f3e7859@github.com>

On Thu, 2 Feb 2023 15:23:39 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This PR fixes some failures I've noticed in the pre-submit checks and could also reproduce locally.

Could you list some testcases which get fixed by this patch? Running whole testsuite takes some time to try that.

-------------

PR: https://git.openjdk.org/crac/pull/44

From heidinga at openjdk.org  Mon Feb  6 06:01:18 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Mon, 6 Feb 2023 06:01:18 GMT
Subject: [crac] RFR: RestoreEnvironmentTest refactoring
In-Reply-To: <KW5lX0EghZi8BswVsYQ6kv_GZRI_vyjVohqk2U5b9EA=.6bf7e8ea-4a77-499a-a70c-1cf87ac1bcd8@github.com>
References: <KW5lX0EghZi8BswVsYQ6kv_GZRI_vyjVohqk2U5b9EA=.6bf7e8ea-4a77-499a-a70c-1cf87ac1bcd8@github.com>
Message-ID: <-_0ODn7EiyYECUVcvSrQZRv2iypXz89BSTZibHLsBRU=.9e9d96af-4c45-419b-ae15-60767009edfe@github.com>

On Tue, 31 Jan 2023 07:23:11 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

> The test was extended with the example to illustrate the scenario when an user don't like to propagate some of environment variables into a restored process (see `RESTORE_ENVIRONMENT_TEST_VAR2` in `RestoreEnvironmentTest.sh`). See the initial discussion here #30

I'll quote my response from the (closed) PR #30:
> The current CRaC approach still allows overriding env vars, which is something Java (and developers, and library developers) doesn't expect to happen so they haven't written their code to be aware these values can change. Changing existing env vars on restore will just lead to inconsistent systems.

Full response in https://github.com/openjdk/crac/pull/30#issuecomment-1412545649

-------------

PR: https://git.openjdk.org/crac/pull/42

From duke at openjdk.org  Mon Feb  6 07:11:21 2023
From: duke at openjdk.org (Radim Vansa)
Date: Mon, 6 Feb 2023 07:11:21 GMT
Subject: [crac] RFR: Fix failures in testsuite
In-Reply-To: <x6_SCd9ghn8jbWaNcuWO3QKjcaS1i9oZxQ4g-_88HwU=.48fad230-2a55-453d-a33b-17a62f3e7859@github.com>
References: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
 <x6_SCd9ghn8jbWaNcuWO3QKjcaS1i9oZxQ4g-_88HwU=.48fad230-2a55-453d-a33b-17a62f3e7859@github.com>
Message-ID: <vrW25grLr_XfXsoLQnS_GeYA6OkNlnQerXTF1a5T11Q=.f4b61a33-5ca2-4a3f-af4b-1c45ed28b023@github.com>

On Fri, 3 Feb 2023 07:56:21 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> This PR fixes some failures I've noticed in the pre-submit checks and could also reproduce locally.
>
> Could you list some testcases which get fixed by this patch? Running whole testsuite takes some time to try that.

@jankratochvil I had these [failing in GHA](https://github.com/rvansa/crac/actions/runs/4065379704/jobs/7014856738):

java/lang/ClassLoader/securityManager/ClassLoaderTest.java 
jdk/modules/incubator/ImageModules.java 
jdk/modules/scenarios/automaticmodules/RunWithAutomaticModules.java 
jdk/modules/scenarios/container/ContainerTest.java 
java/util/ServiceLoader/basic/ServiceLoaderBasicTest.java 
java/util/logging/Logger/bundleLeak/BundleTest.java
tools/javac/Paths/MineField.sh 
tools/javac/modules/AddLimitMods.java
serviceability/dcmd/framework/VMVersionTest.java 

TBH I've investigated only the first one and left for GHA to validate that others get fixed, too.

-------------

PR: https://git.openjdk.org/crac/pull/44

From duke at openjdk.org  Thu Feb  9 08:11:06 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 9 Feb 2023 08:11:06 GMT
Subject: [crac] RFR: DNS cache maintenance
Message-ID: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>

This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

-------------

Commit messages:
 - Address review comments
 - Prevent Resource being garbage-collected
 - DNS cache maintainance

Changes: https://git.openjdk.org/crac/pull/40/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=40&range=00
  Stats: 251 lines in 3 files changed: 251 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/40.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/40/head:pull/40

PR: https://git.openjdk.org/crac/pull/40

From akozlov at openjdk.org  Thu Feb  9 08:11:10 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 9 Feb 2023 08:11:10 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <BehdmR3BCBZvJdcpRoCpXoyktzTqLPSFswXvo8W0c7o=.ed178ca2-1419-4b30-bd7e-fe48348dfe0d@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

Looks very nice! I don't see this to the Draft anymore :)

src/java.base/share/classes/java/net/InetAddress.java line 363:

> 361:             }
> 362:         };
> 363:         Core.getGlobalContext().register(checkpointListener);

For JDK internal Resources, please prefer jdk.internal.crac.Core.getJDKContext() https://github.com/openjdk/crac/blob/crac/src/java.base/share/classes/jdk/internal/crac/Core.java#L41

test/jdk/jdk/crac/ResolveInetAddress.java line 28:

> 26: import java.nio.file.Path;
> 27: 
> 28: public class ResolveInetAddress {

As this is not a test itself, should this be moved into a subdir along with TestInetAddress.java?

test/jdk/jdk/crac/TestInetAddress.java line 130:

> 128:                 .filter(line -> !line.isBlank() && !line.contains("Jps"))
> 129:                 .mapToInt(line -> Integer.parseInt(line.split("\\s")[0]))
> 130:                 .min().orElseThrow();

Not something really need to fix, but it's possible to specify just e.g. class-name and not parse jps output: https://github.com/openjdk/crac/blob/master/src/jdk.jcmd/share/man/jcmd.1#L58

-------------

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Thu Feb  9 08:11:11 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 9 Feb 2023 08:11:11 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <15SM3gCdH_09WPBeOpNq0ad2DKhridZBu0oy2rRJBrs=.35e8da3b-f7c7-44ec-a0a0-3c127271fc02@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

That glibc fix I did submit but it was rejected: https://github.com/openjdk/crac/pull/37#issuecomment-1333650424

-------------

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Thu Feb  9 08:11:12 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 9 Feb 2023 08:11:12 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <BehdmR3BCBZvJdcpRoCpXoyktzTqLPSFswXvo8W0c7o=.ed178ca2-1419-4b30-bd7e-fe48348dfe0d@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
 <BehdmR3BCBZvJdcpRoCpXoyktzTqLPSFswXvo8W0c7o=.ed178ca2-1419-4b30-bd7e-fe48348dfe0d@github.com>
Message-ID: <oQRwo8W6lbU99E3GEBnGt0jjMTSWAc4NC2tSTiDRyr4=.4a21e4ab-6a65-4b95-9a52-aa85786de058@github.com>

On Wed, 8 Feb 2023 18:39:39 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.
>
> Looks very nice! I don't see this to the Draft anymore :)

@AntonKozlov Thanks for the review, updated.

-------------

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Thu Feb  9 10:03:15 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 9 Feb 2023 10:03:15 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <fl6B9XKJc4TM3slrP8HxFbPS7H3Humxj-zuXLu-vy0I=.9c70b1ce-3510-4cd5-88ee-1e6b4d354b75@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

GHA failures should be fixed by https://github.com/openjdk/crac/pull/44

-------------

PR: https://git.openjdk.org/crac/pull/40

From akozlov at openjdk.org  Thu Feb  9 14:02:39 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 9 Feb 2023 14:02:39 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
Message-ID: <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>

On Thu, 2 Feb 2023 06:06:28 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - +comment; found by Dan Heidinga.
>  - Rename signalled->retry; found by Dan Heidinga.

> I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the -XX:CPUFeatures OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: #31 (comment)

Could you explore that futher? The concerns about #31 were merely about complexity of implementation vs reward. Looking backward, I probably overthougth about documenting the workaround, although unlikely any user would notice the workaround exists.

But here, re-exec seems more simple solution. That would also make `Freeze` mechanism unnecessary, as I understand that is another implementation of safepoint.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Thu Feb  9 14:25:23 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 9 Feb 2023 14:25:23 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <lHtCFrUsm3TB0V3hnynJO8_RTw3GIKhEHiAJppCqEN0=.841dfab5-ff6c-4125-b58d-75df7e897ca2@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
 <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>
 <lHtCFrUsm3TB0V3hnynJO8_RTw3GIKhEHiAJppCqEN0=.841dfab5-ff6c-4125-b58d-75df7e897ca2@github.com>
Message-ID: <Jyb6z4ZE0Hf2xvoBUntSAtzKbanG7j4hgNMl2ksWqag=.347b96a2-a9c5-434c-87db-b58da021492c@github.com>

On Thu, 2 Feb 2023 14:50:18 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

> > Ashutosh suggested an alternative approach back in Feb 2022 [0] [1] of providing a single "-XX:[+-]PortableCode" with a common enough baseline set of features
> 
> IIUC you can just use `-XX:CPUFeatures=0` with this patch for the same effect.

That's nice! Since portability affects the final perfomance on restore, the user should have a control between portability and performance. CPUFeatures=0 looks indeed as an impelmentation of some -XX:+PortableCode that was proposed before.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Thu Feb  9 15:08:47 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Thu, 9 Feb 2023 15:08:47 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <gYfYONW0l9RPrwij3GIE4Aq97NVipUUqMIhuS5xR6FQ=.2780f821-b60e-417b-bcde-b6ac71585015@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

Thank you! LGTM

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Thu Feb  9 16:41:34 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Thu, 9 Feb 2023 16:41:34 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
Message-ID: <cE0HuQeQMLCIB2fLQsMOhvz8cbTMyTWYwSgVloyO16w=.efe612e5-0a92-4c42-8ce0-4c9a7cbc3866@github.com>

On Thu, 2 Feb 2023 06:06:28 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - +comment; found by Dan Heidinga.
>  - Rename signalled->retry; found by Dan Heidinga.

IIUC `CPUFeatures=0` disables all the CPU features, which was never the intention behind PortableCode option. With `PortableCode` we wanted to enable a baseline set of feature which would cater to most commonly used microarchitectures.

Selection of default cpu features is a trade-off between portability and performance. Currently the cpu features are selected based on the native architecture which is one extreme as it makes the generated code less portable (unless you run on a sufficiently old arch). Similarly CPUFeatures=0 is another extreme of that which would make the code portable but is highly likely to degrade the performance. PortableCode lies somewhere in between.

In this context gcc provides a good analogy. It does not target i386 instruction set by default just to make the code more portable. The default target arch used by gcc is x86-64 which corresponds to "A generic CPU with 64-bit extensions". And if you look at the features that are enabled by default, you will find sse and sse2 extended instructions sets in there [0].

Now, I also understand CPUFeatures=<n> can be used to achieve the same affect that PortableCode option would. However, with PortableCode the default set of features would be automatically selected by the JVM; the user doesn't need to figure out the value to pass to CPUFeatures to make the code portable.
And as Dan mentioned earlier, even with PortableCode if the user wants to enable a particular extended instruction set, existing Hotspot options can be used.

[0] Use `gcc -Q --help=target`to see the list of options enabled or disabled

-------------

PR: https://git.openjdk.org/crac/pull/41

From rmarchenko at openjdk.org  Fri Feb 10 10:18:18 2023
From: rmarchenko at openjdk.org (Roman Marchenko)
Date: Fri, 10 Feb 2023 10:18:18 GMT
Subject: [crac] RFR: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <0lfEg2TVSyuBlky5nMYI3jko8DNfsCt6xcPKivTgPQQ=.1cc21e4b-a54d-4fd7-8525-120241044bac@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

test/jdk/jdk/crac/java/net/InetAddress/ResolveTest.java line 48:

> 46:  * @modules java.base/jdk.crac
> 47:  * @build ResolveInetAddress
> 48:  * @run main/timeout=360 TestInetAddress

I guess TestInetAddress needs to be changed here also. Currently the test fails due to absense of TestInetAddress

-------------

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Fri Feb 10 10:26:59 2023
From: duke at openjdk.org (Radim Vansa)
Date: Fri, 10 Feb 2023 10:26:59 GMT
Subject: [crac] RFR: DNS cache maintenance [v2]
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <CAi67QxbYIrHanH3LSu_4UdXApPglxgqTq1D1kBrGYA=.dc44bd18-ee99-446a-924d-0c235fa29254@github.com>

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:

  Fix test name in jtreg annotations

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/40/files
  - new: https://git.openjdk.org/crac/pull/40/files/819c60b8..31a1943b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=40&range=01
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=40&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/crac/pull/40.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/40/head:pull/40

PR: https://git.openjdk.org/crac/pull/40

From duke at openjdk.org  Fri Feb 10 10:27:02 2023
From: duke at openjdk.org (Radim Vansa)
Date: Fri, 10 Feb 2023 10:27:02 GMT
Subject: [crac] RFR: DNS cache maintenance [v2]
In-Reply-To: <0lfEg2TVSyuBlky5nMYI3jko8DNfsCt6xcPKivTgPQQ=.1cc21e4b-a54d-4fd7-8525-120241044bac@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
 <0lfEg2TVSyuBlky5nMYI3jko8DNfsCt6xcPKivTgPQQ=.1cc21e4b-a54d-4fd7-8525-120241044bac@github.com>
Message-ID: <mP_wFXY0V12FHJ9UtF_CRACpjAAv-aYuH6-PVoGd3GI=.89202e6f-ee44-4f32-a567-d6d28f425359@github.com>

On Fri, 10 Feb 2023 10:15:39 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

>> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix test name in jtreg annotations
>
> test/jdk/jdk/crac/java/net/InetAddress/ResolveTest.java line 48:
> 
>> 46:  * @modules java.base/jdk.crac
>> 47:  * @build ResolveInetAddress
>> 48:  * @run main/timeout=360 TestInetAddress
> 
> I guess TestInetAddress needs to be changed here also. Currently the test fails due to absense of TestInetAddress

Oops, thanks for spotting this. Looks like I still had the old class cached when running the test...

-------------

PR: https://git.openjdk.org/crac/pull/40

From akozlov at openjdk.org  Fri Feb 10 12:43:16 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 10 Feb 2023 12:43:16 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
Message-ID: <LGpAFDsoosQuM8-Lm19qQy3lQaoX7M_Sd9j6yHsulMs=.0bc19ca0-a2e9-4f7e-ab5e-d19a3e5e76ad@github.com>

On Thu, 2 Feb 2023 06:06:28 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - +comment; found by Dan Heidinga.
>  - Rename signalled->retry; found by Dan Heidinga.

Agree, a VM option is likely to be very similar to a C compiler target microarch option, except maybe default can be different (e.g. `native` in gcc's terms, that's closer to Hotspot's default maximum optimization).

On usability topic, I probably missed that in your report, so are you proposing -XX:+PortableCode to change default CPU features to some conservative set, that still can be extended by existing Hotpost options? That's also reasonable. I wonder, how does that compared with setting CPUFeatures= and then also e.g. UseSSE=4, won't that provide the same effect? But yes, even for CPUFeatures we may want values like `generic`, `native` (again, in gcc terms).

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Fri Feb 10 14:15:18 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Fri, 10 Feb 2023 14:15:18 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <LGpAFDsoosQuM8-Lm19qQy3lQaoX7M_Sd9j6yHsulMs=.0bc19ca0-a2e9-4f7e-ab5e-d19a3e5e76ad@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <LGpAFDsoosQuM8-Lm19qQy3lQaoX7M_Sd9j6yHsulMs=.0bc19ca0-a2e9-4f7e-ab5e-d19a3e5e76ad@github.com>
Message-ID: <bW1NlFqCmgoD6ZXgULulQiecHF8I6a4L_e4XlmVfs6A=.e5cc5416-a3b1-4f91-8b89-019a5bbb56e8@github.com>

On Fri, 10 Feb 2023 12:40:06 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> so are you proposing -XX:+PortableCode to change default CPU features to some conservative set, that still can be extended by existing Hotpost options?

That's right.

> even for CPUFeatures we may want values like generic, native (again, in gcc terms)

Yes, something like CPUFeatures=generic would serve the same purpose as PortableCode.  IMO adding something like `-XX:CPUFeatures=[generic|native]` or `-XX:[+-]PortableCode` is more appropriate. To enable/disable extended instruction sets, hotspot already has options (like UseSSE). Combining those options with `CPUFeatures=[generic|native]` or `PortableCode` would provide an easier way for the user to control the cpu features that hotspot should use.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Fri Feb 10 18:24:18 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 10 Feb 2023 18:24:18 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
Message-ID: <Pf8T4ZYFwIJrcAbwRTcQqUfVnVCaLOIv7msosRYJams=.6b190310-31f3-4b2c-be2e-092add01c8e8@github.com>

On Thu, 2 Feb 2023 06:06:28 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - +comment; found by Dan Heidinga.
>  - Rename signalled->retry; found by Dan Heidinga.

There is some room anyway for XX:CPUFeatures= to take a specification of a feature set in uniform format, that is clear for user without a need to look in hotspot sources. It may be more simple in implementation, that will be easier to verify. The subsequent -XX: options will likely be still available, if user to prefer them; I assume that it's a rare situation when they need to be altered, but CPUFeatures will likely be more frequently used. CPUFeatures and -XX: flags looks to be on different level of abstractions for users. Even though they are not orthogonal.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Fri Feb 10 18:51:13 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 10 Feb 2023 18:51:13 GMT
Subject: [crac] RFR: Fix failures in testsuite
In-Reply-To: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
References: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
Message-ID: <7gimdMOdO4-98cAQZuyf1GFFo4F5Jx2r9-cM3_VFU7I=.a6dfe157-280e-460f-9cad-39ae63d71105@github.com>

On Thu, 2 Feb 2023 15:23:39 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This PR fixes some failures I've noticed in the pre-submit checks and could also reproduce locally.

LGTM, thanks!

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/44

From duke at openjdk.org  Mon Feb 13 05:13:55 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Mon, 13 Feb 2023 05:13:55 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <bW1NlFqCmgoD6ZXgULulQiecHF8I6a4L_e4XlmVfs6A=.e5cc5416-a3b1-4f91-8b89-019a5bbb56e8@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <LGpAFDsoosQuM8-Lm19qQy3lQaoX7M_Sd9j6yHsulMs=.0bc19ca0-a2e9-4f7e-ab5e-d19a3e5e76ad@github.com>
 <bW1NlFqCmgoD6ZXgULulQiecHF8I6a4L_e4XlmVfs6A=.e5cc5416-a3b1-4f91-8b89-019a5bbb56e8@github.com>
Message-ID: <6i9NUwk04YysEu7dtfCNwezt19hBLwDPxUUfsW93BBU=.3249483a-89c2-4617-b89e-cd55c991c02e@github.com>

On Fri, 10 Feb 2023 14:12:22 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> To enable/disable extended instruction sets, hotspot already has options (like UseSSE). Combining those options with `CPUFeatures=[generic|native]` or `PortableCode` would provide an easier way for the user to control the cpu features that hotspot should use.

I did not expect anyone would study which CPU model should use which `UseSSE`, `UseAVX`, `UseCLMUL` and other options.

The current code just prints `You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo` as described in the Comment 0. So one should just run it once on the oldest farm machine and copy/paste the number to the snapshotting script. Or just to run it on all of the farm machines and do a logical AND of all the numbers if one is unsure which machine is the oldest one in your farm.

I sure do not mind which options should be used. I agree `-XX:CPUFeatures=[generic|native]` would be useful (`native` not so useful as it is the current OpenJDK default).

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Mon Feb 13 05:18:54 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Mon, 13 Feb 2023 05:18:54 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>
Message-ID: <12ceIHKZD7AN5mR3hVavIUlHyuGG9rBgYOZgkiWtpOI=.99505719-3aa2-4978-948e-dc1eaa874906@github.com>

On Thu, 9 Feb 2023 13:59:34 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> But here, re-exec seems more simple solution.

OK, if the glibc upstreaming files then there is at least agreement on this way forward. Also maybe OpenJDK should use that anyway to be compatible with older (current) glibc versions.

> That would also make `Freeze` mechanism unnecessary, as I understand that is another implementation of safepoint.

I was originally expecting all threads are at a safepoint during CRaC snapshot. But in practice I have found out they are not. This is why I had to implement the `Freeze` mechanism. Is there some way to stop all the threads at a safepoint?

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Mon Feb 13 08:04:30 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Mon, 13 Feb 2023 08:04:30 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v3]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <S1tCufbhl6GfpqGmUbV7x8c1Wo7osBUSLzP5YwrgvKI=.24b54be9-92ba-4633-9e1e-fee55b9db032@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with three additional commits since the last revision:

 - 0f1d91bc: <no commit message found>
 - 22e5f966: <no commit message found>
 - 59308128: <no commit message found>

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/b4030f8e..0f1d91bc

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=02
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=01-02

  Stats: 114 lines in 2 files changed: 82 ins; 4 del; 28 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Mon Feb 13 08:04:32 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Mon, 13 Feb 2023 08:04:32 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v3]
In-Reply-To: <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <uSO9fLo-Uh_09YrBm4ov6pxs5G1Rf29qb3jgRR10uWY=.9ae899d0-bc90-4d8d-8414-39babde46e17@github.com>
 <NFYxFlkyHNNzIBue6w6Ob6k5W9dWqgdnaQPynr9tczQ=.c7a81447-62a2-4688-9366-ac45de206aef@github.com>
Message-ID: <enXi1jknfi95o_boqrs4OXeZ-VWzh3forFJdMc_gK_I=.63d7eba1-0d81-4c6a-9ab8-183b6e993c4b@github.com>

On Thu, 2 Feb 2023 06:09:18 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> src/hotspot/os/linux/os_linux.cpp line 6177:
>> 
>>> 6175:   assert(sig == RESTORE_SIGNAL, "got what requested");
>>> 6176: 
>>> 6177:   linux_ifunc_reset();
>> 
>> Assuming we're freezing all the threads to allow resetting the linux_ifuncs, it would be good to comment explicitly about why that's the right approach and/or necessary.
>> 
>> Also, what if we stopped a thread that was about to be call one of these ifuncs - that had already loaded the old address into a register - before we did the reset?  Wouldn't it still have the wrong value and with the right (wrong?) thread scheduling, still abort after the restore?
>> 
>> Basically, I think there is probably a race here still.
>
> You are right, there is a race, I will fix that, thanks.

The race should be fixed now. Although still all threads of OpenJDK should be at a safepoint but in practice they are not, why so?

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Mon Feb 13 18:21:06 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Mon, 13 Feb 2023 18:21:06 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <12ceIHKZD7AN5mR3hVavIUlHyuGG9rBgYOZgkiWtpOI=.99505719-3aa2-4978-948e-dc1eaa874906@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>
 <12ceIHKZD7AN5mR3hVavIUlHyuGG9rBgYOZgkiWtpOI=.99505719-3aa2-4978-948e-dc1eaa874906@github.com>
Message-ID: <ABF6R3PbI--Ldl1m0eEpozvFH7OQA9LF-jqzLTNcLFc=.827473d6-3d45-4576-bc97-b2f61bad6287@github.com>

On Mon, 13 Feb 2023 05:16:34 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> This is why I had to implement the Freeze mechanism. Is there some way to stop all the threads at a safepoint?

That should not happen with java threads. Could that be a native thread?

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Mon Feb 13 19:49:19 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Mon, 13 Feb 2023 19:49:19 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
Message-ID: <hr4w3Rl4Mm9hxy1dR90SEqBA1Nvaz7t69E4UDITbag4=.e465d830-4a10-480e-8cf6-71f79baa9ede@github.com>

On Wed, 1 Feb 2023 14:30:36 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Tracks `java.io.FileDescriptor` instances as CRaC resource; before checkpoint these are reported and if not allow-listed (e.g. as opened as standard descriptors) an exception is thrown. Further investigation can use system property `jdk.crac.collect-fd-stacktraces=true` to record origin of those file descriptors.
>> File descriptors claimed in Java code are passed to native; native code checks all open file descriptors and reports error if there's an unexpected FD that is not included in the list passed previously.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Empty commit to trigger GHA

src/hotspot/os/linux/os_linux.cpp line 57:

> 55: #include "runtime/javaCalls.hpp"
> 56: #include "runtime/jniHandles.hpp"
> 57: #include "runtime/jniHandles.inline.hpp"

These ones probably not needed

src/java.base/linux/classes/sun/nio/ch/EPollSelectorImpl.java line 117:

> 115:             // This FileDescriptor is a one-time use, the actual FD will be closed from EventFD
> 116:             fd.markClosedByNIO();
> 117:             IOUtil.configureBlocking(fd, false);

The eventfd was actually closed on line 149, so it's incorrect to mark it is closed here. I think everything was fine before this chunk

src/java.base/share/classes/java/io/FileDescriptor.java line 67:

> 65:      * Called by FileDispatcherImpl when the file descriptor is about to be closed natively.
> 66:      */
> 67:     public void markClosedByNIO() {

Can we avoid changing the public interface? This is not a method we'd like users to call

src/java.base/share/classes/java/io/FileDescriptor.java line 73:

> 71:     class Resource implements jdk.internal.crac.JDKResource {
> 72:         private static final boolean COLLECT_FD_STACKTRACES =
> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);

Should not be moved to JDKContext or jdk.internal.Core ? To avoid any possible user to get the property value the similar way

src/java.base/share/classes/java/io/FileDescriptor.java line 75:

> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
> 74: 
> 75:         private boolean closedByNIO;

I think it may perfectly reside in FileDesriptor, as it looks like a bug for me that FileDispatcherImpl does not change valid() status of the FileDesriptor. Or even make FileDispatcherImpl to reset FileDescriptor.fd? Are there reasons not to do that?

src/java.base/share/classes/java/io/RandomAccessFile.java line 96:

> 94:         public void beforeCheckpoint(Context<? extends jdk.crac.Resource> context) throws Exception {
> 95:             if (Core.getJDKContext().claimFdWeak(fd, this)) {
> 96:                 if (Core.getJDKContext().matchClasspath(path)) {

Actually it would be more natural to match with PersistentJarFile who claimed their FDs already. So we'll be able to handle dynamically loaded JARs that are not a part of the `java.class.path`

src/java.base/share/classes/java/io/RandomAccessFile.java line 100:

> 98:                     return;
> 99:                 }
> 100:                 throw new CheckpointOpenFileException("RandomAccessFile " + path + " left open." + JDKContext.COLLECT_FD_STACKTRACES_HINT,

Stack-traces are good idea! 

Here I would suggest add file desriptor number as well.

-------------

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Tue Feb 14 06:15:13 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Tue, 14 Feb 2023 06:15:13 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <ABF6R3PbI--Ldl1m0eEpozvFH7OQA9LF-jqzLTNcLFc=.827473d6-3d45-4576-bc97-b2f61bad6287@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>
 <12ceIHKZD7AN5mR3hVavIUlHyuGG9rBgYOZgkiWtpOI=.99505719-3aa2-4978-948e-dc1eaa874906@github.com>
 <ABF6R3PbI--Ldl1m0eEpozvFH7OQA9LF-jqzLTNcLFc=.827473d6-3d45-4576-bc97-b2f61bad6287@github.com>
Message-ID: <hlBYNDKrTlt2DXnOUoGmShd5Oh0DNQBV-RfjYDmet0k=.815eb9f2-4ee1-4299-a896-cb0280d7a343@github.com>

On Mon, 13 Feb 2023 18:18:30 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> That should not happen with java threads. Could that be a native thread?

It's a pity but I can no longer reproduce the problem (and I no longer have the original core file). So I think I can remove the Freezer for now and if anyone reports it we can discuss whether to reintroduce the Freezer or choose some other solution.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Tue Feb 14 06:25:47 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Tue, 14 Feb 2023 06:25:47 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v4]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <N7UsLYdeN-nR0yr2x3XZyyK-S0rN-Bb2qEEmwubPUlE=.0a2934a5-f5d6-4bf9-83f1-e5e8c1e0f54a@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:

  Remove Freeze; the problem is no longer reproducible for me.

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/0f1d91bc..5d3de5a1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=03
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=02-03

  Stats: 210 lines in 1 file changed: 0 ins; 210 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Tue Feb 14 08:00:19 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 14 Feb 2023 08:00:19 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <hr4w3Rl4Mm9hxy1dR90SEqBA1Nvaz7t69E4UDITbag4=.e465d830-4a10-480e-8cf6-71f79baa9ede@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
 <hr4w3Rl4Mm9hxy1dR90SEqBA1Nvaz7t69E4UDITbag4=.e465d830-4a10-480e-8cf6-71f79baa9ede@github.com>
Message-ID: <E4GTT4YBzav5LY9mgK5zYrsz_vshqUGMzt6QbhCiwNU=.1e1e8be2-d9de-4d74-bfe1-39ab6d4872d0@github.com>

On Tue, 31 Jan 2023 12:20:01 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Empty commit to trigger GHA
>
> src/hotspot/os/linux/os_linux.cpp line 57:
> 
>> 55: #include "runtime/javaCalls.hpp"
>> 56: #include "runtime/jniHandles.hpp"
>> 57: #include "runtime/jniHandles.inline.hpp"
> 
> These ones probably not needed

OK, that comes from your commits, I'll clean it up.

> src/java.base/linux/classes/sun/nio/ch/EPollSelectorImpl.java line 117:
> 
>> 115:             // This FileDescriptor is a one-time use, the actual FD will be closed from EventFD
>> 116:             fd.markClosedByNIO();
>> 117:             IOUtil.configureBlocking(fd, false);
> 
> The eventfd was actually closed on line 149, so it's incorrect to mark it is closed here. I think everything was fine before this chunk

The native descriptor is closed, but the FileDescriptor instance is created and forgotten. If the GC doesn't remove it before checkpoint it may be reported as a FileDescriptor leak. I think I saw that happening.

> src/java.base/share/classes/java/io/FileDescriptor.java line 67:
> 
>> 65:      * Called by FileDispatcherImpl when the file descriptor is about to be closed natively.
>> 66:      */
>> 67:     public void markClosedByNIO() {
> 
> Can we avoid changing the public interface? This is not a method we'd like users to call

I've checked if it would be possible to extend FileDescriptor and put this method there, but there's several non-trivial places where this would have to be changed (e.g. in `sun.nio.ch.InheritedChannel` the FD is constructed using a non-public ctor via reflection). So we can make this package-private and use reflection, or probablz a better way would be through `JavaIOFileDescriptorAccess`.

> src/java.base/share/classes/java/io/FileDescriptor.java line 73:
> 
>> 71:     class Resource implements jdk.internal.crac.JDKResource {
>> 72:         private static final boolean COLLECT_FD_STACKTRACES =
>> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
> 
> Should not be moved to JDKContext or jdk.internal.Core ? To avoid any possible user to get the property value the similar way

Not sure what you're trying to prevent, user code reading a property? The field here is private.

> src/java.base/share/classes/java/io/FileDescriptor.java line 75:
> 
>> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
>> 74: 
>> 75:         private boolean closedByNIO;
> 
> I think it may perfectly reside in FileDesriptor, as it looks like a bug for me that FileDispatcherImpl does not change valid() status of the FileDesriptor. Or even make FileDispatcherImpl to reset FileDescriptor.fd? Are there reasons not to do that?

I haven't changed that to minimize the impact of changes; I still assume at one point it made sense to close the native FD without going through FileDescriptor. The reply I got on nio-dev was this:

> In the java.io APIs, FIS/FOS/RAF define getFD methods so the FileDescriptor can be obtained by applications. This is problematic for a bunch of reasons but for the discussion here, setting it to -1 is needed for the FileDescriptor::valid. Another point is that java.io pre-dates the more robust async close mechanism that is NIO and setting it to -1 helps avoid trying to use a file descriptor that has been closed and/or recycled.
> In the NIO area, FileDescriptor objects do not leak to applications. The only case where it is necessary to set to -1 is on Windows with sockets and this is because there is no equivalent of dup2 there. Another point here is that FileDescriptors aren't really needed as the channels implementation just need the raw file descriptor or handle. 

Having that in FileDescriptor seemed more consolidated but if you prefer it that way I won't object. So, should I do that or risk breaking stuff by setting fd to -1?

> src/java.base/share/classes/java/io/RandomAccessFile.java line 96:
> 
>> 94:         public void beforeCheckpoint(Context<? extends jdk.crac.Resource> context) throws Exception {
>> 95:             if (Core.getJDKContext().claimFdWeak(fd, this)) {
>> 96:                 if (Core.getJDKContext().matchClasspath(path)) {
> 
> Actually it would be more natural to match with PersistentJarFile who claimed their FDs already. So we'll be able to handle dynamically loaded JARs that are not a part of the `java.class.path`

This case is trying to prevent a situation when the FD is **not** opened using `PersistentJarFile`, as Spring Boot uses own implementation.

-------------

PR: https://git.openjdk.org/crac/pull/43

From akozlov at openjdk.org  Tue Feb 14 15:54:04 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Tue, 14 Feb 2023 15:54:04 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <E4GTT4YBzav5LY9mgK5zYrsz_vshqUGMzt6QbhCiwNU=.1e1e8be2-d9de-4d74-bfe1-39ab6d4872d0@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
 <hr4w3Rl4Mm9hxy1dR90SEqBA1Nvaz7t69E4UDITbag4=.e465d830-4a10-480e-8cf6-71f79baa9ede@github.com>
 <E4GTT4YBzav5LY9mgK5zYrsz_vshqUGMzt6QbhCiwNU=.1e1e8be2-d9de-4d74-bfe1-39ab6d4872d0@github.com>
Message-ID: <9aXXQ9zQ5-GyoGxNCFauwDJouM8AJWPhwZBE1HGFzP8=.02cbef65-78ba-4f64-8e4d-710a6fff923f@github.com>

On Tue, 14 Feb 2023 07:20:01 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> src/java.base/linux/classes/sun/nio/ch/EPollSelectorImpl.java line 117:
>> 
>>> 115:             // This FileDescriptor is a one-time use, the actual FD will be closed from EventFD
>>> 116:             fd.markClosedByNIO();
>>> 117:             IOUtil.configureBlocking(fd, false);
>> 
>> The eventfd was actually closed on line 149, so it's incorrect to mark it is closed here. I think everything was fine before this chunk
>
> The native descriptor is closed, but the FileDescriptor instance is created and forgotten. If the GC doesn't remove it before checkpoint it may be reported as a FileDescriptor leak. I think I saw that happening.

Agree, I've missed the configureBlocking requires a temporary FileDescriptor just to match its interface.

>> src/java.base/share/classes/java/io/FileDescriptor.java line 73:
>> 
>>> 71:     class Resource implements jdk.internal.crac.JDKResource {
>>> 72:         private static final boolean COLLECT_FD_STACKTRACES =
>>> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
>> 
>> Should not be moved to JDKContext or jdk.internal.Core ? To avoid any possible user to get the property value the similar way
>
> Not sure what you're trying to prevent, user code reading a property? The field here is private.

My note is minor, I'm trying to save future users of COLLECT_FD_STACKTRACES_PROPERTY from readingthe property, instead the property can be used, parsed in JDKContext 

class JDKContext {
        static final boolean COLLECT_FD_STACKTRACES =
               GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
}

// In this class: just use JDKContext.COLLECT_FD_STACKTRACES without GetProperty

>> src/java.base/share/classes/java/io/FileDescriptor.java line 75:
>> 
>>> 73:                 GetBooleanAction.privilegedGetProperty(JDKContext.COLLECT_FD_STACKTRACES_PROPERTY);
>>> 74: 
>>> 75:         private boolean closedByNIO;
>> 
>> I think it may perfectly reside in FileDesriptor, as it looks like a bug for me that FileDispatcherImpl does not change valid() status of the FileDesriptor. Or even make FileDispatcherImpl to reset FileDescriptor.fd? Are there reasons not to do that?
>
> I haven't changed that to minimize the impact of changes; I still assume at one point it made sense to close the native FD without going through FileDescriptor. The reply I got on nio-dev was this:
> 
>> In the java.io APIs, FIS/FOS/RAF define getFD methods so the FileDescriptor can be obtained by applications. This is problematic for a bunch of reasons but for the discussion here, setting it to -1 is needed for the FileDescriptor::valid. Another point is that java.io pre-dates the more robust async close mechanism that is NIO and setting it to -1 helps avoid trying to use a file descriptor that has been closed and/or recycled.
>> In the NIO area, FileDescriptor objects do not leak to applications. The only case where it is necessary to set to -1 is on Windows with sockets and this is because there is no equivalent of dup2 there. Another point here is that FileDescriptors aren't really needed as the channels implementation just need the raw file descriptor or handle. 
> 
> Having that in FileDescriptor seemed more consolidated but if you prefer it that way I won't object. So, should I do that or risk breaking stuff by setting fd to -1?

>From the reply [1] I understand that there was no reason to keep track of the native fd/valid() status in FileDescriptors used by NIO, so that was not done. And for me the status of the FD is closer to FD itself, but I don't have a strong preference.

[1] https://mail.openjdk.org/pipermail/nio-dev/2023-January/013075.html, just for bookkeeping

>> src/java.base/share/classes/java/io/RandomAccessFile.java line 96:
>> 
>>> 94:         public void beforeCheckpoint(Context<? extends jdk.crac.Resource> context) throws Exception {
>>> 95:             if (Core.getJDKContext().claimFdWeak(fd, this)) {
>>> 96:                 if (Core.getJDKContext().matchClasspath(path)) {
>> 
>> Actually it would be more natural to match with PersistentJarFile who claimed their FDs already. So we'll be able to handle dynamically loaded JARs that are not a part of the `java.class.path`
>
> This case is trying to prevent a situation when the FD is **not** opened using `PersistentJarFile`, as Spring Boot uses own implementation.

Indeed! This is a required transition from the old logic in hotspot, which ignored all files (by path) specified in classpath. 

I was thinking about another use-case, when we have an instance of `PersistentJarFile` (which allows underlying fd), and RandomAccessFile with another underlying fd, but reffering to the same path on file system.

-------------

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Tue Feb 14 16:55:30 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 14 Feb 2023 16:55:30 GMT
Subject: [crac] Integrated: Fix failures in testsuite
In-Reply-To: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
References: <5fku54LYdHZw2RfXe1iFlnd1o8NHxwTqp547DCi7EcU=.cce8c25b-6da4-4343-a068-f70b2858c46e@github.com>
Message-ID: <v0Jx_skXuQDOR15pNQ_mOAOXf4clcYkXCoU5kqZS5to=.116ee271-e4ae-445d-bb97-21b2023085d7@github.com>

On Thu, 2 Feb 2023 15:23:39 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This PR fixes some failures I've noticed in the pre-submit checks and could also reproduce locally.

This pull request has now been integrated.

Changeset: 5925959f
Author:    Radim Vansa <rvansa at azul.com>
Committer: Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/5925959f742e33cc3255575dca727192dfb8cb86
Stats:     8 lines in 1 file changed: 6 ins; 0 del; 2 mod

Fix failures in testsuite

Reviewed-by: akozlov

-------------

PR: https://git.openjdk.org/crac/pull/44

From duke at openjdk.org  Wed Feb 15 18:28:21 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Wed, 15 Feb 2023 18:28:21 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <6i9NUwk04YysEu7dtfCNwezt19hBLwDPxUUfsW93BBU=.3249483a-89c2-4617-b89e-cd55c991c02e@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <LGpAFDsoosQuM8-Lm19qQy3lQaoX7M_Sd9j6yHsulMs=.0bc19ca0-a2e9-4f7e-ab5e-d19a3e5e76ad@github.com>
 <bW1NlFqCmgoD6ZXgULulQiecHF8I6a4L_e4XlmVfs6A=.e5cc5416-a3b1-4f91-8b89-019a5bbb56e8@github.com>
 <6i9NUwk04YysEu7dtfCNwezt19hBLwDPxUUfsW93BBU=.3249483a-89c2-4617-b89e-cd55c991c02e@github.com>
Message-ID: <jf7IWh1zYQ06iuxW4fbxx-kNdMoD9zdkvt_U-Ie5y6w=.a0a429a3-2ead-4ef5-898f-78faa328adeb@github.com>

On Mon, 13 Feb 2023 05:11:07 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> So one should just run it once on the oldest farm machine and copy/paste the number to the snapshotting script. Or just to run it on all of the farm machines and do a logical AND of all the numbers if one is unsure which machine is the oldest one in your farm.

Challenge with this approach is the user needs to have access to all the farm machines. This may not always be the case. If I am running my application on cloud, it wouldn't be possible for me to do this in anyway.
For cloud workloads where the user does not know the cpu features of the systems, I think it makes sense to enable only a baseline set of cpu features.

The mechanism currently proposed would work in the scenario where systems are known, but it sounds cumbersome to use from the way it is described. The user has to create a checkpoint, migrate it to another system and restore it just to find out the cpu features to use. We can achieve the same without the need to create any checkpoint. All we need is an option to display the hex value of feature flags for the cpu features currently enabled.
eg java -XX:CPUFeatures=native -XX:+ShowFeatureFlagsInHex would show the feature flags enabled by the JVM based on native system features.
Now, as mentioned earlier in the comment , user can run  -XX:CPUFeatures=native -XX:+ShowFeatureFlagsInHex on all the systems and do a logical AND to find the common set of features to enable.

Nonetheless, we should keep the proposed mechanism as well to detect incompatible CPU features, but IMO it should not be used as a way to discover the common CPU features for a set of systems, and it also does not solve the problem when the set of target systems are not known.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb 16 07:44:03 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 07:44:03 GMT
Subject: [crac] RFR: Close extraneous file descriptors
Message-ID: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>

Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.

-------------

Commit messages:
 - Remove unneeded check
 - Applied review suggestions
 - Close non-standard file descriptors on boot

Changes: https://git.openjdk.org/crac/pull/45/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=45&range=00
  Stats: 253 lines in 7 files changed: 237 ins; 12 del; 4 mod
  Patch: https://git.openjdk.org/crac/pull/45.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/45/head:pull/45

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Thu Feb 16 09:27:07 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 09:27:07 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
Message-ID: <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>

> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.

Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:

 - Remove unneeded check
 - Applied review suggestions
 - Close non-standard file descriptors on boot
   
   If the process is configured for Checkpoint and is started with open file descriptors (e.g. inherited from parent process), under some circumstances CRIU cannot properly execute the checkpoint. Prevent this be proactively closing file descriptors > 2 on boot.
   This behaviour can be overriden by settings -XX:CRIgnoreFileDescriptors to a comma-separated list of FD numbers or paths these file descriptors map to.

-------------

Changes: https://git.openjdk.org/crac/pull/45/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=45&range=01
  Stats: 253 lines in 7 files changed: 237 ins; 12 del; 4 mod
  Patch: https://git.openjdk.org/crac/pull/45.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/45/head:pull/45

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Thu Feb 16 09:27:08 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 09:27:08 GMT
Subject: [crac] RFR: Close extraneous file descriptors
In-Reply-To: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
Message-ID: <9f47CSRJzbUIALtv2gbLfZENXSPOJmQckciShlQy41I=.69648508-4762-4686-bf4f-7b7e97752da5@github.com>

On Thu, 16 Feb 2023 07:36:43 GMT, Radim Vansa <duke at openjdk.org> wrote:

> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.

I've rebased on top of most recent crac branch but there are no comments.

-------------

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Thu Feb 16 11:01:00 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 11:01:00 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <E4GTT4YBzav5LY9mgK5zYrsz_vshqUGMzt6QbhCiwNU=.1e1e8be2-d9de-4d74-bfe1-39ab6d4872d0@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
 <hr4w3Rl4Mm9hxy1dR90SEqBA1Nvaz7t69E4UDITbag4=.e465d830-4a10-480e-8cf6-71f79baa9ede@github.com>
 <E4GTT4YBzav5LY9mgK5zYrsz_vshqUGMzt6QbhCiwNU=.1e1e8be2-d9de-4d74-bfe1-39ab6d4872d0@github.com>
Message-ID: <lTZojHl-EnA4qPvjupF4vcNjdTgM2kp2z7pkOIjM44Y=.518807f4-405e-4a33-8e3c-4ba8877e1e20@github.com>

On Tue, 14 Feb 2023 07:14:31 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> src/hotspot/os/linux/os_linux.cpp line 57:
>> 
>>> 55: #include "runtime/javaCalls.hpp"
>>> 56: #include "runtime/jniHandles.hpp"
>>> 57: #include "runtime/jniHandles.inline.hpp"
>> 
>> These ones probably not needed
>
> OK, that comes from your commits, I'll clean it up.

Turns out this is needed, doesn't compile without.

-------------

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Thu Feb 16 11:18:34 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 11:18:34 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v4]
In-Reply-To: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
Message-ID: <Q-75iizW7wdYFdJEkh6gS6l_9o3YGMf1yHW-BMRcvZE=.9734af8f-b5e3-41c3-899f-b6a9d4bcb135@github.com>

> Tracks `java.io.FileDescriptor` instances as CRaC resource; before checkpoint these are reported and if not allow-listed (e.g. as opened as standard descriptors) an exception is thrown. Further investigation can use system property `jdk.crac.collect-fd-stacktraces=true` to record origin of those file descriptors.
> File descriptors claimed in Java code are passed to native; native code checks all open file descriptors and reports error if there's an unexpected FD that is not included in the list passed previously.

Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 16 additional commits since the last revision:

 - Merge remote-tracking branch 'origin/crac' into newfd
 - Use descriptor access rather than extending API
 - 8272472: StackGuardPages test doesn't build with glibc 2.34
   
   Backport-of: f77a1a156f3da9068d012d9227c7ee0fee58f571
 - Empty commit to trigger GHA
 - Drop native FDs tracking
 - Avoid claiming invalid FileDescriptor
 - Whitelist RandomAccessFile opening classpath files
   
   This is a workaround for some frameworks opening classpath files in
   a non-standard way.
 - Add tracking of FD origin
 - Track FileDescriptors closed by NIO
 - Track native FDs from EPoll
 - ... and 6 more: https://git.openjdk.org/crac/compare/474db523...9b5e7edd

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/43/files
  - new: https://git.openjdk.org/crac/pull/43/files/723d6f32..9b5e7edd

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=43&range=03
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=43&range=02-03

  Stats: 53 lines in 8 files changed: 34 ins; 11 del; 8 mod
  Patch: https://git.openjdk.org/crac/pull/43.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/43/head:pull/43

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Thu Feb 16 11:18:37 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 16 Feb 2023 11:18:37 GMT
Subject: [crac] RFR: Improved open file descriptors tracking [v3]
In-Reply-To: <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
References: <uXeWJdux27M_3jKa48IhHbZrhUt_2dDHeyfOqLQEcuc=.b0f95f82-1ff3-4116-98d8-a966311c17dd@github.com>
 <z7uY3QM9kUzx6ysjFqLYa87-6M0IcD_KgYsWjYMuUKc=.37c0c249-2eae-4f52-976b-90b8d8310cfc@github.com>
Message-ID: <CuX2E5hT1Kx-icAbGpZI-A2pil8o8LZmd6c9CnnBWBY=.1ebd2771-1f25-45cf-bfad-b247c52fe85a@github.com>

On Wed, 1 Feb 2023 14:30:36 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Tracks `java.io.FileDescriptor` instances as CRaC resource; before checkpoint these are reported and if not allow-listed (e.g. as opened as standard descriptors) an exception is thrown. Further investigation can use system property `jdk.crac.collect-fd-stacktraces=true` to record origin of those file descriptors.
>> File descriptors claimed in Java code are passed to native; native code checks all open file descriptors and reports error if there's an unexpected FD that is not included in the list passed previously.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Empty commit to trigger GHA

@AntonKozlov Updated.

-------------

PR: https://git.openjdk.org/crac/pull/43

From duke at openjdk.org  Fri Feb 17 12:40:24 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Fri, 17 Feb 2023 12:40:24 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v5]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <PDrCuJWhVLTspBS1cpO82oFQfWNTEg-siPdtOSbwpFc=.c46a3834-4772-4fc7-9bda-cce59b92c8d1@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:

  Support -XX:CPUFeatures=native and -XX:CPUFeatures=generic.

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/5d3de5a1..a7583e98

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=04
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=03-04

  Stats: 56 lines in 4 files changed: 34 ins; 5 del; 17 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Fri Feb 17 13:42:55 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Fri, 17 Feb 2023 13:42:55 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v6]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <0UhQzQugBHBrYrmridXbw4vGdm2uUe4meqlUmrgbfag=.5faf976f-ef02-44a5-a566-bd8dead4a1c3@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:

  Add: -XX:+ShowCPUFeatures

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/a7583e98..1b631aa2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=05
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=04-05

  Stats: 8 lines in 2 files changed: 7 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Fri Feb 17 13:44:28 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Fri, 17 Feb 2023 13:44:28 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v5]
In-Reply-To: <PDrCuJWhVLTspBS1cpO82oFQfWNTEg-siPdtOSbwpFc=.c46a3834-4772-4fc7-9bda-cce59b92c8d1@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <PDrCuJWhVLTspBS1cpO82oFQfWNTEg-siPdtOSbwpFc=.c46a3834-4772-4fc7-9bda-cce59b92c8d1@github.com>
Message-ID: <SOeBDTa3bWNA16UqaStxT56Q27TJPLcqhfND-LLQzf8=.e75a3303-2708-4662-a191-5d104ea285d9@github.com>

On Fri, 17 Feb 2023 12:40:24 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Support -XX:CPUFeatures=native and -XX:CPUFeatures=generic.

I hope I have addressed all the comments.
- `-XX:+ShowCPUFeatures` produces: `This machine's CPU features are: -XX:CPUFeatures=0x7fff9dfcfbf7`
- `-XX:CPUFeatures=native`
- `-XX:CPUFeatures=generic` - the only problem is what features should be the default, the current one is definitely wrong

-------------

PR: https://git.openjdk.org/crac/pull/41

From heidinga at openjdk.org  Fri Feb 17 14:47:59 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Fri, 17 Feb 2023 14:47:59 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v5]
In-Reply-To: <SOeBDTa3bWNA16UqaStxT56Q27TJPLcqhfND-LLQzf8=.e75a3303-2708-4662-a191-5d104ea285d9@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <PDrCuJWhVLTspBS1cpO82oFQfWNTEg-siPdtOSbwpFc=.c46a3834-4772-4fc7-9bda-cce59b92c8d1@github.com>
 <SOeBDTa3bWNA16UqaStxT56Q27TJPLcqhfND-LLQzf8=.e75a3303-2708-4662-a191-5d104ea285d9@github.com>
Message-ID: <W-KqrcGsdUIcPFCny_Oe-d6HgGsCWgUaLxEjwBucyIc=.ff316387-c0b7-4d09-94ff-98790ac335ea@github.com>

On Fri, 17 Feb 2023 13:33:55 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> I hope I have addressed all the comments.
> 
> * `-XX:+ShowCPUFeatures` produces: `This machine's CPU features are: -XX:CPUFeatures=0x7fff9dfcfbf7`
> * `-XX:CPUFeatures=native`
> * `-XX:CPUFeatures=generic` - the only problem is what features should be the default, the current one is definitely wrong

One reasonable design point for `generic` is to match the set of features allowed when compiling the JVM so that a checkpoint is as portable as running the app with the java executable.  The extra trick is to then remove any features from that set that C2 doesn't use so we have the best possible portability based on what is actually required.

Given the current code for `=generic`:

    // FIXME: This is for: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
    return 0 | CPU_CX8 | CPU_CMOV | CPU_FXSR | CPU_MMX | CPU_SSE | CPU_SSE2
      | CPU_SSE3 | CPU_SSSE3 | CPU_SSE4_1 | CPU_SSE4_2 | CPU_POPCNT | CPU_LZCNT
      | CPU_TSC | CPU_AVX | CPU_AVX2 | CPU_AES | CPU_ERMS | CPU_CLMUL
      | CPU_BMI1 | CPU_BMI2 | CPU_FMA | CPU_VZEROUPPER | CPU_FLUSH | CPU_HV;
  }

Are there any features that can be pruned from that set?  Does Hotspot actually use all of the CPU_XX capabilities?

Is Xeon a reasonable proxy for the default `make` configuration on a linux x86-64 system?  Grepping the code base I failed to turn up a good `-mtune=` or `-march=` for x86-64.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Fri Feb 17 16:18:20 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Fri, 17 Feb 2023 16:18:20 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v5]
In-Reply-To: <W-KqrcGsdUIcPFCny_Oe-d6HgGsCWgUaLxEjwBucyIc=.ff316387-c0b7-4d09-94ff-98790ac335ea@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <PDrCuJWhVLTspBS1cpO82oFQfWNTEg-siPdtOSbwpFc=.c46a3834-4772-4fc7-9bda-cce59b92c8d1@github.com>
 <SOeBDTa3bWNA16UqaStxT56Q27TJPLcqhfND-LLQzf8=.e75a3303-2708-4662-a191-5d104ea285d9@github.com>
 <W-KqrcGsdUIcPFCny_Oe-d6HgGsCWgUaLxEjwBucyIc=.ff316387-c0b7-4d09-94ff-98790ac335ea@github.com>
Message-ID: <JsacI8DD0hG_tLuYtjP4nufNmHrfHOR7HXCqEYkXwy4=.444cdef3-6ade-4763-ad11-fe3245b0e61f@github.com>

On Fri, 17 Feb 2023 14:42:20 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

> One reasonable design point for generic is to match the set of features allowed when compiling the JVM so that a checkpoint is as portable as running the app with the java executable. 

I used `LOG=cmdlines make images` to see the command lines used during compilation (with `make images`). I don't see any `march` or `mtune` option in the command line. As I mentioned earlier we can use `gcc -Q --help=target` to see the extended instructions sets enabled by gcc (would need some filtering I believe). I guess we can use that set as the baseline to begin with.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Mon Feb 20 15:58:03 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Mon, 20 Feb 2023 15:58:03 GMT
Subject: [crac] RFR: DNS cache maintenance [v2]
In-Reply-To: <CAi67QxbYIrHanH3LSu_4UdXApPglxgqTq1D1kBrGYA=.dc44bd18-ee99-446a-924d-0c235fa29254@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
 <CAi67QxbYIrHanH3LSu_4UdXApPglxgqTq1D1kBrGYA=.dc44bd18-ee99-446a-924d-0c235fa29254@github.com>
Message-ID: <7D3ERs3Jc4D_hXc7EqW-18qmkj2u598znYdCScFC2ek=.ca8fef84-21af-4188-b7b4-75ff19d5df72@github.com>

On Fri, 10 Feb 2023 10:26:59 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix test name in jtreg annotations

I've run the new test on the latest revision and it passes

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/40

From rmarchenko at openjdk.org  Tue Feb 21 13:56:46 2023
From: rmarchenko at openjdk.org (Roman Marchenko)
Date: Tue, 21 Feb 2023 13:56:46 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed
Message-ID: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>

When running CRaC with docker, java may exit before CRIU is finished dumpring because CRIU kills the original java process, and then docker immediately exits.

It could be reproduced with a simple Java test:

import jdk.crac.Core;
public class Test {
    public static void main(String args[]) throws Exception {
        Core.checkpointRestore();
        System.out.println("finish");
    }
}

and run with docker:
`docker <docker_options> $JAVA_HOME/java -XX:CRaCCheckpointTo=./cr Test.java`

After the command above finishes, `cr/cppath` is absent in the case of failure. Or/and it will fail on restore:
`docker <docker_options> $JAVA_HOME/java -XX:CRaCRestoreFrom=./cr`

This change fixes the issue by forkin'g the main process in case of PID=1 (pid=1 means it was run with docker), and waiting for children processes are finished. This makes us sure that CRIU finalized the dump, if any. At the same time, there is no conflict with PIDs on restore, since the process being restored has PID not equal to 1, if restoring with the command above..

-------------

Commit messages:
 - Minor change
 - Implemented fork'ing for pid=1

Changes: https://git.openjdk.org/crac/pull/46/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=46&range=00
  Stats: 36 lines in 1 file changed: 36 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/46.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/46/head:pull/46

PR: https://git.openjdk.org/crac/pull/46

From heidinga at openjdk.org  Tue Feb 21 14:31:00 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Tue, 21 Feb 2023 14:31:00 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed
In-Reply-To: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
Message-ID: <ITgBgnNHPkgY6FTKSu_Y-lQ4ZrUc7b0LG1HLcsoB0tw=.e25d1c73-65ad-4301-abac-03597f6c7d42@github.com>

On Tue, 21 Feb 2023 13:50:23 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

> When running CRaC with docker, java may exit before CRIU is finished dumpring because CRIU kills the original java process, and then docker immediately exits.
> 
> It could be reproduced with a simple Java test:
> 
> import jdk.crac.Core;
> public class Test {
>     public static void main(String args[]) throws Exception {
>         Core.checkpointRestore();
>         System.out.println("finish");
>     }
> }
> 
> and run with docker:
> `docker <docker_options> $JAVA_HOME/java -XX:CRaCCheckpointTo=./cr Test.java`
> 
> After the command above finishes, `cr/cppath` is absent in the case of failure. Or/and it will fail on restore:
> `docker <docker_options> $JAVA_HOME/java -XX:CRaCRestoreFrom=./cr`
> 
> This change fixes the issue by forkin'g the main process in case of PID=1 (pid=1 means it was run with docker), and waiting for children processes are finished. This makes us sure that CRIU finalized the dump, if any. At the same time, there is no conflict with PIDs on restore, since the process being restored has PID not equal to 1, if restoring with the command above..

src/java.base/share/native/launcher/main.c line 236:

> 234: 
> 235:     if (is_checkpoint && 1 == getpid()) {
> 236:         if (0 < fork()) {

Not an expert in this area, but don't we also need to handle `SIGTERM` appropriately by forwarding it to the child process if the forking receives it?

-------------

PR: https://git.openjdk.org/crac/pull/46

From duke at openjdk.org  Tue Feb 21 15:24:08 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 21 Feb 2023 15:24:08 GMT
Subject: [crac] Integrated: DNS cache maintenance
In-Reply-To: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
References: <cWPqxf58XPjB2UH8poXf4DJOSq_alUtV2Q62tGsYpeI=.16db7942-e3f4-4c99-9c5e-0f581f062cee@github.com>
Message-ID: <-sZLFv-8WelPObPgoy9urEP_Vojr5TLw6Nn7B7XF6io=.b06adfe5-8280-4fda-b2c3-f96a60eb73be@github.com>

On Thu, 5 Jan 2023 15:31:17 GMT, Radim Vansa <duke at openjdk.org> wrote:

> This draft PR contains a backport of issue that prevents compilation on systems with newer glibc versions; ideally the backport should be integrated first and then this PR can be rebased.

This pull request has now been integrated.

Changeset: 7dfaf5fc
Author:    Radim Vansa <rvansa at azul.com>
Committer: Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/7dfaf5fc7df13cb02de72aa9c8a02c38d32c43a7
Stats:     251 lines in 3 files changed: 251 ins; 0 del; 0 mod

DNS cache maintenance

Reviewed-by: akozlov

-------------

PR: https://git.openjdk.org/crac/pull/40

From rmarchenko at openjdk.org  Tue Feb 21 16:19:54 2023
From: rmarchenko at openjdk.org (Roman Marchenko)
Date: Tue, 21 Feb 2023 16:19:54 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed
In-Reply-To: <ITgBgnNHPkgY6FTKSu_Y-lQ4ZrUc7b0LG1HLcsoB0tw=.e25d1c73-65ad-4301-abac-03597f6c7d42@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
 <ITgBgnNHPkgY6FTKSu_Y-lQ4ZrUc7b0LG1HLcsoB0tw=.e25d1c73-65ad-4301-abac-03597f6c7d42@github.com>
Message-ID: <5XknoTnEGCe4UI4rYNNnQyLbIZFIfiWBVa_KUCfaBV8=.e2878d44-e8ca-4d32-9a2d-aeaeb7462464@github.com>

On Tue, 21 Feb 2023 14:28:35 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

>> When running CRaC with docker, java may exit before CRIU is finished dumpring because CRIU kills the original java process, and then docker immediately exits.
>> 
>> It could be reproduced with a simple Java test:
>> 
>> import jdk.crac.Core;
>> public class Test {
>>     public static void main(String args[]) throws Exception {
>>         Core.checkpointRestore();
>>         System.out.println("finish");
>>     }
>> }
>> 
>> and run with docker:
>> `docker <docker_options> $JAVA_HOME/java -XX:CRaCCheckpointTo=./cr Test.java`
>> 
>> After the command above finishes, `cr/cppath` is absent in the case of failure. Or/and it will fail on restore:
>> `docker <docker_options> $JAVA_HOME/java -XX:CRaCRestoreFrom=./cr`
>> 
>> This change fixes the issue by forkin'g the main process in case of PID=1 (pid=1 means it was run with docker), and waiting for children processes are finished. This makes us sure that CRIU finalized the dump, if any. At the same time, there is no conflict with PIDs on restore, since the process being restored has PID not equal to 1, if restoring with the command above..
>
> src/java.base/share/native/launcher/main.c line 236:
> 
>> 234: 
>> 235:     if (is_checkpoint && 1 == getpid()) {
>> 236:         if (0 < fork()) {
> 
> Not an expert in this area, but don't we also need to handle `SIGTERM` appropriately by forwarding it to the child process if the forking receives it?

@DanHeidinga 
Do you mean "we need to terminate children if the parent process is terminated"?
It seems like termination of a process with PID=1 running inside a container will stop a container run, so I guess children will be terminated also. I'd appreciate any reproducible scenario, in case I'm mistaken.

-------------

PR: https://git.openjdk.org/crac/pull/46

From heidinga at openjdk.org  Tue Feb 21 16:56:58 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Tue, 21 Feb 2023 16:56:58 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed
In-Reply-To: <5XknoTnEGCe4UI4rYNNnQyLbIZFIfiWBVa_KUCfaBV8=.e2878d44-e8ca-4d32-9a2d-aeaeb7462464@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
 <ITgBgnNHPkgY6FTKSu_Y-lQ4ZrUc7b0LG1HLcsoB0tw=.e25d1c73-65ad-4301-abac-03597f6c7d42@github.com>
 <5XknoTnEGCe4UI4rYNNnQyLbIZFIfiWBVa_KUCfaBV8=.e2878d44-e8ca-4d32-9a2d-aeaeb7462464@github.com>
Message-ID: <WwoMI9ErYvOmcN9fUTkKj1SE7SkFq5GgCSKk0R-6llE=.f7f59bda-853e-438e-a940-613a51fa9d5a@github.com>

On Tue, 21 Feb 2023 16:16:45 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

>> src/java.base/share/native/launcher/main.c line 236:
>> 
>>> 234: 
>>> 235:     if (is_checkpoint && 1 == getpid()) {
>>> 236:         if (0 < fork()) {
>> 
>> Not an expert in this area, but don't we also need to handle `SIGTERM` appropriately by forwarding it to the child process if the forking receives it?
>
> @DanHeidinga 
> Do you mean "we need to terminate children if the parent process is terminated"?
> It seems like termination of a process with PID=1 running inside a container will stop a container run, so I guess children will be terminated also. I'd appreciate any reproducible scenario, in case I'm mistaken.

Sorry, that was unclear as I'm fuzzy on the details.

PID 1 is responsible for `wait()`ing on any spawned child processes to ensure they exit before exiting itself.  It also needs to respond to `SIGTERM` by propagating it to all its child processes so they have a chance to gracefully shutdown before `SIGKILL` is sent.  See [0] and [1]

[0] https://petermalmgren.com/pid-1-child-processes-docker/
[1] https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

-------------

PR: https://git.openjdk.org/crac/pull/46

From akozlov at openjdk.org  Wed Feb 22 10:17:55 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Wed, 22 Feb 2023 10:17:55 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed
In-Reply-To: <WwoMI9ErYvOmcN9fUTkKj1SE7SkFq5GgCSKk0R-6llE=.f7f59bda-853e-438e-a940-613a51fa9d5a@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
 <ITgBgnNHPkgY6FTKSu_Y-lQ4ZrUc7b0LG1HLcsoB0tw=.e25d1c73-65ad-4301-abac-03597f6c7d42@github.com>
 <5XknoTnEGCe4UI4rYNNnQyLbIZFIfiWBVa_KUCfaBV8=.e2878d44-e8ca-4d32-9a2d-aeaeb7462464@github.com>
 <WwoMI9ErYvOmcN9fUTkKj1SE7SkFq5GgCSKk0R-6llE=.f7f59bda-853e-438e-a940-613a51fa9d5a@github.com>
Message-ID: <iQs84Kr40Q_l0mHLJphcc4He3k5L689WjSLSZ4DMlsI=.e89ee913-46fb-4c56-8ea4-d0337cdcaecf@github.com>

On Tue, 21 Feb 2023 16:54:21 GMT, Dan Heidinga <heidinga at openjdk.org> wrote:

>> @DanHeidinga 
>> Do you mean "we need to terminate children if the parent process is terminated"?
>> It seems like termination of a process with PID=1 running inside a container will stop a container run, so I guess children will be terminated also. I'd appreciate any reproducible scenario, in case I'm mistaken.
>
> Sorry, that was unclear as I'm fuzzy on the details.
> 
> PID 1 is responsible for `wait()`ing on any spawned child processes to ensure they exit before exiting itself.  It also needs to respond to `SIGTERM` by propagating it to all its child processes so they have a chance to gracefully shutdown before `SIGKILL` is sent.  See [0] and [1]
> 
> [0] https://petermalmgren.com/pid-1-child-processes-docker/
> [1] https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

Indeed. Since this process was supposed to be the java process, it should be a proxy a good proxy for the real java process. E.g. it should forward SIGTERM and other signals (SIGQUIT to print stack trace, etc). Although this proxy should not necessarily be a good init process.

We had the same problem in restore, this code probably will be useful https://github.com/openjdk/crac/blob/crac/src/java.base/unix/native/criuengine/criuengine.c#L263

-------------

PR: https://git.openjdk.org/crac/pull/46

From duke at openjdk.org  Wed Feb 22 13:11:51 2023
From: duke at openjdk.org (Radim Vansa)
Date: Wed, 22 Feb 2023 13:11:51 GMT
Subject: [crac] RFR: Fix failing CRaC tests
Message-ID: <y83GItfrbGKa3TaSblmot3Gz1dgkSvugtn5ZUiytSCU=.79c13c83-3f7c-40e4-bab8-ef59b74f7c6d@github.com>

Before https://github.com/openjdk/crac/pull/16 the arguments for restored process were ignored; tests were written with this in mind and current behaviour breaks them.

In addition I've added missing `-XX:+UnlockDiagnosticVMOptions` flag and fixed the `recursiveCheckpoint` test runner which got stuck as with `pauseengine` the checkpointed process does not terminate.

-------------

Commit messages:
 - Fix failing CRaC tests

Changes: https://git.openjdk.org/crac/pull/47/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=47&range=00
  Stats: 37 lines in 18 files changed: 3 ins; 1 del; 33 mod
  Patch: https://git.openjdk.org/crac/pull/47.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/47/head:pull/47

PR: https://git.openjdk.org/crac/pull/47

From rmarchenko at openjdk.org  Wed Feb 22 13:21:35 2023
From: rmarchenko at openjdk.org (Roman Marchenko)
Date: Wed, 22 Feb 2023 13:21:35 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed [v2]
In-Reply-To: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
Message-ID: <loJNkXaMqkQsBtTqMjcafGZCg7jMszcC64GR-MOR5ls=.b6de1f8b-365b-466a-a57d-b48fa0342ed7@github.com>

> When running CRaC with docker, java may exit before CRIU is finished dumpring because CRIU kills the original java process, and then docker immediately exits.
> 
> It could be reproduced with a simple Java test:
> 
> public class Test {
>     public static void main(String args[]) throws Exception {
>         jdk.crac.Core.checkpointRestore();
>         System.out.println("finish");
>     }
> }
> 
> and run with docker:
> `docker <docker_options> $JAVA_HOME/java -XX:CRaCCheckpointTo=./cr Test.java`
> 
> After the command above finishes, `cr/cppath` is absent in the case of failure. Or/and it will fail on restore:
> `docker <docker_options> $JAVA_HOME/java -XX:CRaCRestoreFrom=./cr`
> 
> This change fixes the issue by forkin'g the main process in case of PID=1 (pid=1 means it was run with docker), and waiting for children processes are finished. This makes us sure that CRIU finalized the dump, if any. At the same time, there is no conflict with PIDs on restore, since the process being restored has PID not equal to 1, if restoring with the command above..

Roman Marchenko has updated the pull request incrementally with one additional commit since the last revision:

  Added sighandler to the main process

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/46/files
  - new: https://git.openjdk.org/crac/pull/46/files/65a68598..fd074cfc

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=46&range=01
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=46&range=00-01

  Stats: 39 lines in 1 file changed: 35 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/crac/pull/46.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/46/head:pull/46

PR: https://git.openjdk.org/crac/pull/46

From duke at openjdk.org  Wed Feb 22 13:58:50 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Wed, 22 Feb 2023 13:58:50 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v7]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <hFYUV75F7xYpchFyVcYI-O6iciqBUFk885Pa27vU0Tw=.265a4d09-1324-40ef-9e8a-c7286abc78e0@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:

  Set the proper feature set for -XX:CPUFeatures=generic.

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/1b631aa2..81d2008b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=06
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=05-06

  Stats: 13 lines in 1 file changed: 8 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Wed Feb 22 15:07:04 2023
From: duke at openjdk.org (Radim Vansa)
Date: Wed, 22 Feb 2023 15:07:04 GMT
Subject: [crac] RFR: Fix failing CRaC tests [v2]
In-Reply-To: <y83GItfrbGKa3TaSblmot3Gz1dgkSvugtn5ZUiytSCU=.79c13c83-3f7c-40e4-bab8-ef59b74f7c6d@github.com>
References: <y83GItfrbGKa3TaSblmot3Gz1dgkSvugtn5ZUiytSCU=.79c13c83-3f7c-40e4-bab8-ef59b74f7c6d@github.com>
Message-ID: <akg4x2e9OIA2fIDVyjrr6VAa-RVDAkCu40-9fPma2jU=.94fc86a4-0e18-4991-8b79-0ea290df0e14@github.com>

> Before https://github.com/openjdk/crac/pull/16 the arguments for restored process were ignored; tests were written with this in mind and current behaviour breaks them.
> 
> In addition I've added missing `-XX:+UnlockDiagnosticVMOptions` flag and fixed the `recursiveCheckpoint` test runner which got stuck as with `pauseengine` the checkpointed process does not terminate.

Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:

  Remove somebody's forgotten overrides

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/47/files
  - new: https://git.openjdk.org/crac/pull/47/files/a69eb665..3b9598bf

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=47&range=01
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=47&range=00-01

  Stats: 5 lines in 1 file changed: 0 ins; 3 del; 2 mod
  Patch: https://git.openjdk.org/crac/pull/47.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/47/head:pull/47

PR: https://git.openjdk.org/crac/pull/47

From duke at openjdk.org  Wed Feb 22 15:17:35 2023
From: duke at openjdk.org (Radim Vansa)
Date: Wed, 22 Feb 2023 15:17:35 GMT
Subject: [crac] RFR: Add CRaC-specific tests to GHA
Message-ID: <OZX-HO3SKKuZLOBuyT0CaNluMk5cDaV151PWC4qXhqw=.6e9f255e-3c1d-4f45-aa31-e28e09426b59@github.com>

Existing GitHub Actions run test tier1 but since most changes in this project focus on the CRaC capabilities we should run them in an automated fashion, too.
Right now the tests are mostly failing: this should be addressed in https://github.com/openjdk/crac/pull/47

-------------

Commit messages:
 - Add CRaC-specific tests to GHA

Changes: https://git.openjdk.org/crac/pull/48/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=48&range=00
  Stats: 6 lines in 1 file changed: 6 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/48.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/48/head:pull/48

PR: https://git.openjdk.org/crac/pull/48

From akozlov at openjdk.org  Wed Feb 22 16:57:06 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Wed, 22 Feb 2023 16:57:06 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
Message-ID: <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>

On Thu, 16 Feb 2023 09:27:07 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.
>
> Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
> 
>  - Remove unneeded check
>  - Applied review suggestions
>  - Close non-standard file descriptors on boot
>    
>    If the process is configured for Checkpoint and is started with open file descriptors (e.g. inherited from parent process), under some circumstances CRIU cannot properly execute the checkpoint. Prevent this be proactively closing file descriptors > 2 on boot.
>    This behaviour can be overriden by settings -XX:CRIgnoreFileDescriptors to a comma-separated list of FD numbers or paths these file descriptors map to.

Few minor comments

src/hotspot/os/linux/os_linux.cpp line 6501:

> 6499:     const char* fileSep = os::file_separator();
> 6500:     jio_snprintf(modules_path, JVM_MAXPATHLEN, "%s%slib%smodules", Arguments::get_java_home(), fileSep, fileSep);
> 6501:   }

A comment would be useful here. I cannot suggest a way to avoid this particular check, as modules are opened very early during argument parsing, to get possible additional arguments from the module, so let's document this for future readers of the code.

src/hotspot/os/linux/os_linux.cpp line 6512:

> 6510:       int r = readfdlink(fd, path, sizeof(path));
> 6511:       if (!is_fd_ignored(fd, r != -1 ? path : nullptr)) {
> 6512:         tty->print("CRaC closing file descriptor %d: %s\n", fd, path);

`log_trace(os)` to be consistent with line 6484?

src/hotspot/share/runtime/globals.hpp line 2103:

> 2101:       "unavailable")                                                        \
> 2102:                                                                             \
> 2103:   product(ccstr, CRaCIgnoredFileDescriptors, NULL, "Comma-separated list "  \

Should not `0,1,2` be the default option then? So it will be also possible to overwrite this default.

src/hotspot/share/runtime/globals.hpp line 2126:

> 2124:   product(bool, CRTrace, true, "Minimal C/R tracing")                       \
> 2125: 
> 2126: 

Unnecessary whitespace change

-------------

Changes requested by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Thu Feb 23 07:40:37 2023
From: duke at openjdk.org (Radim Vansa)
Date: Thu, 23 Feb 2023 07:40:37 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
 <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
Message-ID: <cCo2sAz4lVrzTXWdWf8fUjR2pi8VkQT2IJ_dgn-vHVc=.ac34b983-bbc7-44cd-93e6-5e500dfdd529@github.com>

On Wed, 22 Feb 2023 16:47:17 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
>> 
>>  - Remove unneeded check
>>  - Applied review suggestions
>>  - Close non-standard file descriptors on boot
>>    
>>    If the process is configured for Checkpoint and is started with open file descriptors (e.g. inherited from parent process), under some circumstances CRIU cannot properly execute the checkpoint. Prevent this be proactively closing file descriptors > 2 on boot.
>>    This behaviour can be overriden by settings -XX:CRIgnoreFileDescriptors to a comma-separated list of FD numbers or paths these file descriptors map to.
>
> src/hotspot/share/runtime/globals.hpp line 2103:
> 
>> 2101:       "unavailable")                                                        \
>> 2102:                                                                             \
>> 2103:   product(ccstr, CRaCIgnoredFileDescriptors, NULL, "Comma-separated list "  \
> 
> Should not `0,1,2` be the default option then? So it will be also possible to overwrite this default.

I didn't use `0,1,2` as if you want to ignore FD 42 you would have to write `0,1,2,42` rather than just 42. Starting with standard FDs closed is extremely non-standard, and in that case we should do something about `System.out` as well to be consistent.

-------------

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Thu Feb 23 13:47:42 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 23 Feb 2023 13:47:42 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v2]
In-Reply-To: <hlBYNDKrTlt2DXnOUoGmShd5Oh0DNQBV-RfjYDmet0k=.815eb9f2-4ee1-4299-a896-cb0280d7a343@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <FJthpOsRH8_Z9mTPwx8p31dShKNAONxwcAJ3wDujNuE=.7265b13c-d3d8-4889-a670-41a9013166ac@github.com>
 <MOu8H6_5WzXMWy35Hlsa4sXmBvaXlNDZqleZkHvi1f0=.85bd1600-267b-4440-80db-104a3a0223fe@github.com>
 <12ceIHKZD7AN5mR3hVavIUlHyuGG9rBgYOZgkiWtpOI=.99505719-3aa2-4978-948e-dc1eaa874906@github.com>
 <ABF6R3PbI--Ldl1m0eEpozvFH7OQA9LF-jqzLTNcLFc=.827473d6-3d45-4576-bc97-b2f61bad6287@github.com>
 <hlBYNDKrTlt2DXnOUoGmShd5Oh0DNQBV-RfjYDmet0k=.815eb9f2-4ee1-4299-a896-cb0280d7a343@github.com>
Message-ID: <AD7-CiqkzZtBEMJ-yvEQ5I-C-PxljyJZutSzTFhgMCY=.5343fa4b-9148-4e57-94cd-4d1f09ffa3c4@github.com>

On Tue, 14 Feb 2023 06:12:38 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

> That should not happen with java threads. Could that be a native thread?

So I have accidentally reproduced it. Maybe it is a native thread. But what else to do with it than the `Freezer`? `checkpoint_restore()` is still executing `linux_ifunc_reset()` that time.


Core was generated by `/home/azul/azul/example-jetty/../crac-git/build/linux-x86_64-server-slowdebug/j'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:69
69	../sysdeps/x86_64/multiarch/strlen-evex.S: No such file or directory.
[Current thread is 1 (Thread 0x7f092cfff6c0 (LWP 26563))]
(gdb) bt -15
#2920 <signal handler called>
#2921 __memset_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:167
#2922 0x00007f0955eb2cfa in GuardedMemory::set_user_bytes(unsigned char) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2923 0x00007f0955eb2c41 in GuardedMemory::release_for_freeing() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2924 0x00007f09564382cf in os::free(void*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2925 0x00007f095592ac4a in ChunkPool::free_all_but(unsigned long) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2926 0x00007f095592af28 in ChunkPool::clean() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2927 0x00007f095592af93 in ChunkPoolCleaner::task() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2928 0x00007f09566ef880 in PeriodicTask::execute_if_pending(int) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2929 0x00007f09566ef3bf in PeriodicTask::real_time_tick(int) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2930 0x00007f09563f0bf5 in WatcherThread::run() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2931 0x00007f09567299cd in Thread::call_run() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2932 0x00007f095643e109 in thread_native_entry(Thread*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2933 0x00007f095763bfd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#2934 0x00007f09576bc66c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) info threads 
  Id   Target Id                         Frame 
* 1    Thread 0x7f092cfff6c0 (LWP 26563) __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:69
  2    Thread 0x7f09553206c0 (LWP 26546) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950024108) at ./nptl/futex-internal.c:57
  3    Thread 0x7f0957592b80 (LWP 26545) __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=26546, futex_word=0x7f0955320990) at ./nptl/futex-internal.c:57
  4    Thread 0x7f093cad56c0 (LWP 26550) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500f8390) at ./nptl/futex-internal.c:57
  5    Thread 0x7f092d2ff6c0 (LWP 26560) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f092d2fec80, op=137, expected=0, futex_word=0x7f09500220c8) at ./nptl/futex-internal.c:57
  6    Thread 0x7f093c6e46c0 (LWP 26552) 0x00007f095644fc4b in strcmp_local(char const*, char const*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
  7    Thread 0x7f093d1df6c0 (LWP 26547) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500729b0) at ./nptl/futex-internal.c:57
  8    Thread 0x7f093d0dd6c0 (LWP 26548) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950020db8) at ./nptl/futex-internal.c:57
  9    Thread 0x7f092d0ff6c0 (LWP 26562) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950022b88) at ./nptl/futex-internal.c:57
  10   Thread 0x7f093cfdb6c0 (LWP 26549) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950085250) at ./nptl/futex-internal.c:57
  11   Thread 0x7f093c5e26c0 (LWP 26553) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950024e38) at ./nptl/futex-internal.c:57
  12   Thread 0x7f093c2e26c0 (LWP 26556) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950022a58) at ./nptl/futex-internal.c:57
  13   Thread 0x7f08dfbff6c0 (LWP 26599) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500729b0) at ./nptl/futex-internal.c:57
  14   Thread 0x7f08dfdff6c0 (LWP 26573) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f08dfdfe580, op=137, expected=0, futex_word=0x7f0950882d80) at ./nptl/futex-internal.c:57
  15   Thread 0x7f093c9d36c0 (LWP 26551) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950020ee8) at ./nptl/futex-internal.c:57
  16   Thread 0x7f092d1ff6c0 (LWP 26561) syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  17   Thread 0x7f093c4e26c0 (LWP 26554) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09502a4584) at ./nptl/futex-internal.c:57
  18   Thread 0x7f092d7ff6c0 (LWP 26558) syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  19   Thread 0x7f093c3e26c0 (LWP 26555) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0950015db0) at ./nptl/futex-internal.c:57
  20   Thread 0x7f092d3ff6c0 (LWP 26559) syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  21   Thread 0x7f08dfcff6c0 (LWP 26598) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500244ac) at ./nptl/futex-internal.c:57
  22   Thread 0x7f08dfafd6c0 (LWP 26600) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500729b0) at ./nptl/futex-internal.c:57
  23   Thread 0x7f093c1e26c0 (LWP 26557) __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f093c1e1ca0, op=137, expected=0, futex_word=0x7f0950022928) at ./nptl/futex-internal.c:57
  24   Thread 0x7f08df0f56c0 (LWP 26601) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f09500729b0) at ./nptl/futex-internal.c:57
(gdb) t 6
[Switching to thread 6 (Thread 0x7f093c6e46c0 (LWP 26552))]
#0  0x00007f095644fc4b in strcmp_local(char const*, char const*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
(gdb) bt
#0  0x00007f095644fc4b in strcmp_local(char const*, char const*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#1  0x00007f0956450404 in symtab_lookup_iterate_phdr(dl_phdr_info*, unsigned long, void*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#2  0x00007f095770211d in __GI___dl_iterate_phdr (callback=0x7f09564500a9 <symtab_lookup_iterate_phdr(dl_phdr_info*, unsigned long, void*)>, data=0x7f093c6e3590) at ./elf/dl-iteratephdr.c:74
#3  0x00007f0956450644 in symtab_lookup(char const*, void const**, unsigned int) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#4  0x00007f0956451f2c in reset_glibc() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#5  0x00007f0956452afd in linux_ifunc_reset() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#6  0x00007f095644a599 in os::Linux::checkpoint_restore(int*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#7  0x00007f095644b737 in VM_Crac::doit() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#8  0x00007f09567d269d in VM_Operation::evaluate() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#9  0x00007f0956801122 in VMThread::evaluate_operation(VM_Operation*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#10 0x00007f0956801917 in VMThread::inner_execute(VM_Operation*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#11 0x00007f0956801cc3 in VMThread::loop() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#12 0x00007f0956800cc0 in VMThread::run() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#13 0x00007f09567299cd in Thread::call_run() () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#14 0x00007f095643e109 in thread_native_entry(Thread*) () from /home/azul/azul/crac-git/build/linux-x86_64-server-slowdebug/jdk/lib/server/libjvm.so
#15 0x00007f095763bfd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x00007f09576bc66c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb 23 14:20:55 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 23 Feb 2023 14:20:55 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v8]
In-Reply-To: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
Message-ID: <siwf3aKNft0lP5KozWrVMvGb0ofHCFBQ4t70y7hV5wA=.52627dfc-b927-473f-a5ad-368cdfa04ba5@github.com>

> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
> 
> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
> 
> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
> 
>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
> 
> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
> 
>> Error occurred during initialization of VM
>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
> 
> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
> 
> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
> 
> That IMO does not preclude trying the same for this case.
> 
> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.

Jan Kratochvil has updated the pull request incrementally with five additional commits since the last revision:

 - Merge branch 'crac-altstack-cpu-cpuexplicit-strip-x' into crac-altstack-cpu-cpuexplicit-strip
 - db3e8e43: <no commit message found>
 - Fix Freezer::in_handler.
 - Revert "Remove Freeze; the problem is no longer reproducible for me."
   
   This reverts commit 5d3de5a10d7ab6f43344a0bb56168aa252b32845.
 - man page: +options

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/41/files
  - new: https://git.openjdk.org/crac/pull/41/files/81d2008b..dd60871e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=41&range=07
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=41&range=06-07

  Stats: 257 lines in 3 files changed: 253 ins; 1 del; 3 mod
  Patch: https://git.openjdk.org/crac/pull/41.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/41/head:pull/41

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Thu Feb 23 14:22:02 2023
From: duke at openjdk.org (Jan Kratochvil)
Date: Thu, 23 Feb 2023 14:22:02 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v7]
In-Reply-To: <hFYUV75F7xYpchFyVcYI-O6iciqBUFk885Pa27vU0Tw=.265a4d09-1324-40ef-9e8a-c7286abc78e0@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <hFYUV75F7xYpchFyVcYI-O6iciqBUFk885Pa27vU0Tw=.265a4d09-1324-40ef-9e8a-c7286abc78e0@github.com>
Message-ID: <-1yluPBwB-6UXs4dzssozhz2FQ4h8RA6CLqpB-T1dq8=.63c4a6d6-ead3-42bb-a812-2a8e5d0cc857@github.com>

On Wed, 22 Feb 2023 13:58:50 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Set the proper feature set for -XX:CPUFeatures=generic.

The patch is now calling `os::Linux::initialize_processor_count()` (extracted from `os::Linux::initialize_system_info()`) as otherwise I was getting (snapshot on 4 vCPUs and restore on 20 vCPUs):

#  Internal Error (../../src/hotspot/os/linux/os_linux.cpp:4979), pid=2870, tid=2877
#  assert(cpu_count > 0 && cpu_count <= os::processor_count()) failed: sanity check

I am not sure what all consequences it can have. @AntonKozlov was talking about these issues such as changing CPU count or memory size. Also I am curious why I did not see this error before.

-------------

PR: https://git.openjdk.org/crac/pull/41

From duke at openjdk.org  Fri Feb 24 11:47:38 2023
From: duke at openjdk.org (Radim Vansa)
Date: Fri, 24 Feb 2023 11:47:38 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
 <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
Message-ID: <vjNs4g6gSIpmFZtUFFEeuEC_W9AyOS9NKiJK3YPJpkw=.a9b7a954-0502-428d-9b56-4fe4bf9beae3@github.com>

On Mon, 20 Feb 2023 18:28:47 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
>> 
>>  - Remove unneeded check
>>  - Applied review suggestions
>>  - Close non-standard file descriptors on boot
>>    
>>    If the process is configured for Checkpoint and is started with open file descriptors (e.g. inherited from parent process), under some circumstances CRIU cannot properly execute the checkpoint. Prevent this be proactively closing file descriptors > 2 on boot.
>>    This behaviour can be overriden by settings -XX:CRIgnoreFileDescriptors to a comma-separated list of FD numbers or paths these file descriptors map to.
>
> src/hotspot/os/linux/os_linux.cpp line 6512:
> 
>> 6510:       int r = readfdlink(fd, path, sizeof(path));
>> 6511:       if (!is_fd_ignored(fd, r != -1 ? path : nullptr)) {
>> 6512:         tty->print("CRaC closing file descriptor %d: %s\n", fd, path);
> 
> `log_trace(os)` to be consistent with line 6484?

It's meant to be more visible than trace-level: for consistency I'll make this `log_warning(os)`

-------------

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Fri Feb 24 11:56:21 2023
From: duke at openjdk.org (Radim Vansa)
Date: Fri, 24 Feb 2023 11:56:21 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v3]
In-Reply-To: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
Message-ID: <jcqs1g9sXPJIwLjOSVJbxS3a6AsLIJh3QqcbPY4Abpw=.4d69c330-79ef-43ce-beae-ad25b2314dd8@github.com>

> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.

Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:

  Comments and change in logging

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/45/files
  - new: https://git.openjdk.org/crac/pull/45/files/4c5afa58..2da79441

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=45&range=02
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=45&range=01-02

  Stats: 6 lines in 2 files changed: 3 ins; 1 del; 2 mod
  Patch: https://git.openjdk.org/crac/pull/45.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/45/head:pull/45

PR: https://git.openjdk.org/crac/pull/45

From duke at openjdk.org  Fri Feb 24 11:56:21 2023
From: duke at openjdk.org (Radim Vansa)
Date: Fri, 24 Feb 2023 11:56:21 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
 <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
Message-ID: <wAPsFsokkLL2gUVmB5MKuZ3pRIoyNjthkL5hIBvI98U=.24136f93-102e-4288-8cde-79413c2bc2aa@github.com>

On Wed, 22 Feb 2023 16:53:59 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Radim Vansa has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
>> 
>>  - Remove unneeded check
>>  - Applied review suggestions
>>  - Close non-standard file descriptors on boot
>>    
>>    If the process is configured for Checkpoint and is started with open file descriptors (e.g. inherited from parent process), under some circumstances CRIU cannot properly execute the checkpoint. Prevent this be proactively closing file descriptors > 2 on boot.
>>    This behaviour can be overriden by settings -XX:CRIgnoreFileDescriptors to a comma-separated list of FD numbers or paths these file descriptors map to.
>
> Few minor comments

@AntonKozlov Updated!

-------------

PR: https://git.openjdk.org/crac/pull/45

From akozlov at openjdk.org  Fri Feb 24 18:45:44 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 24 Feb 2023 18:45:44 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v3]
In-Reply-To: <jcqs1g9sXPJIwLjOSVJbxS3a6AsLIJh3QqcbPY4Abpw=.4d69c330-79ef-43ce-beae-ad25b2314dd8@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <jcqs1g9sXPJIwLjOSVJbxS3a6AsLIJh3QqcbPY4Abpw=.4d69c330-79ef-43ce-beae-ad25b2314dd8@github.com>
Message-ID: <53Boa51O3acXEbxj973xBOlkD8HrGE34_BE_hxjmsa4=.dfadc919-3e9d-4dab-bdac-87da1fdfaf50@github.com>

On Fri, 24 Feb 2023 11:56:21 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Comments and change in logging

LGTM!

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/45

From akozlov at openjdk.org  Fri Feb 24 18:45:44 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Fri, 24 Feb 2023 18:45:44 GMT
Subject: [crac] RFR: Close extraneous file descriptors [v2]
In-Reply-To: <cCo2sAz4lVrzTXWdWf8fUjR2pi8VkQT2IJ_dgn-vHVc=.ac34b983-bbc7-44cd-93e6-5e500dfdd529@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
 <fp9LUvTIqhYt_i-2Qa0TORW2YY7D4Qan7OvutxCRm8c=.9cfc9eb7-fe70-4dd8-8186-65160cbfb199@github.com>
 <uRo2U45e-AJBM9az_G6XXYOoCS7bdVwwDRCOwZ8VdQs=.d242ca39-b2a7-4a4c-8fb3-2cd4641dfaa4@github.com>
 <cCo2sAz4lVrzTXWdWf8fUjR2pi8VkQT2IJ_dgn-vHVc=.ac34b983-bbc7-44cd-93e6-5e500dfdd529@github.com>
Message-ID: <p557kiHnn9_tTOmlH2EpfnsFU6azTuMIjcYj8DiyE0g=.95c73afb-ed30-410c-9842-aea1c7f65411@github.com>

On Thu, 23 Feb 2023 07:37:24 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> src/hotspot/share/runtime/globals.hpp line 2103:
>> 
>>> 2101:       "unavailable")                                                        \
>>> 2102:                                                                             \
>>> 2103:   product(ccstr, CRaCIgnoredFileDescriptors, NULL, "Comma-separated list "  \
>> 
>> Should not `0,1,2` be the default option then? So it will be also possible to overwrite this default.
>
> I didn't use `0,1,2` as if you want to ignore FD 42 you would have to write `0,1,2,42` rather than just 42. Starting with standard FDs closed is extremely non-standard, and in that case we should do something about `System.out` as well to be consistent.

Agree, the current implementation makes sense.

-------------

PR: https://git.openjdk.org/crac/pull/45

From rvansa at azul.com  Mon Feb 27 09:01:28 2023
From: rvansa at azul.com (Radim Vansa)
Date: Mon, 27 Feb 2023 10:01:28 +0100
Subject: Proposal for documentation and snapsafety
Message-ID: <1c0a8870-371f-fe2e-d0a8-f5df241a842d@azul.com>

Hi all,

Anton asked me to have a look on the ways we could communicate 
CRaC-specific behavior to users. I went through this mailing list to see 
this has been discussed in the past quite a bit, but I'd like to get 
some consensus and start working on improving the user experience.

The documentation part should be pretty straightforward; I think that we 
could use custom tag [1] @crac that will usher the CRaC specific 
information. This will make it easier to full-text search it, and 
exclude in builds that won't support CRaC (if there'll be any). While 
the main target is the public API this annotation should be used in 
places that should not be used directly, e.g. sun.security.* packages 
that were 'fixed up' for CRaC. It is perfectly fine to add a 
documentation with this tag to places that are *not* changed, e.g. the 
j.u.Random explaining the reasoning behind that.

While all JDK code can be eventually fixed in a similar way to 
SecureRandom I think that it's clear that not everything can be 
encapsulated. A good example are the environment variables, but also 
number of processors and many others [2]. I propose to tag any 
method/constructor that returns data that could be expected to stay 
constant in non-C/R app but often changes after restore, or an object 
that will need handling through a registered Resource, with 
@CracSensitive (3) annotation. We will provide a tool that will report 
places that could call these methods, unless marked with @CracSafe 
annotation. This tool could work in a static way (scanning set of JARs, 
probably with a thin Maven plugin as well?) and as a javaagent, scanning 
classes as these get loaded.

Naturally not all code invoking non-snap-safe methods is from user code, 
many cases come from the libraries. Alternative way to allow-list places 
calling @CracSensitive methods in places that cannot be changed directly 
would be provided, though eventually we aim at encouraging that the 
libraries adopt the @CracSafe internally.

Non-goal: Any means for *making* the libraries snap-safe (e.g. the 
Substitutions) are out of scope of this proposal.

I will start prototyping this, though I'll welcome any suggestions and 
concerns.

Radim


[1] 
https://docs.oracle.com/javase/8/docs/technotes/tools/windows/javadoc.html#tag

[2] https://github.com/eclipse-openj9/openj9/issues/12484

(3) We could use @SnapSensitive and @SnapSafe instead


From duke at openjdk.org  Tue Feb 28 12:01:25 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 28 Feb 2023 12:01:25 GMT
Subject: [crac] RFR: CRaC related documentation in JDK classes using custom tag
Message-ID: <FV_7yFcATdfiKmuT1n5ez0XW2JVfNI-rJYPYyr2fNFQ=.e959ec7a-d2db-413f-a7c8-3f8248a3f084@github.com>

As proposed in https://mail.openjdk.org/pipermail/crac-dev/2023-February/000496.html I am adding some CRaC related documentation.

-------------

Commit messages:
 - CRaC related documentation in JDK classes using custom tag

Changes: https://git.openjdk.org/crac/pull/51/files
 Webrev: https://webrevs.openjdk.org/?repo=crac&pr=51&range=00
  Stats: 100 lines in 10 files changed: 99 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/51.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/51/head:pull/51

PR: https://git.openjdk.org/crac/pull/51

From duke at openjdk.org  Tue Feb 28 12:33:11 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 28 Feb 2023 12:33:11 GMT
Subject: [crac] RFR: CRaC related documentation in JDK classes using custom
 tag [v2]
In-Reply-To: <FV_7yFcATdfiKmuT1n5ez0XW2JVfNI-rJYPYyr2fNFQ=.e959ec7a-d2db-413f-a7c8-3f8248a3f084@github.com>
References: <FV_7yFcATdfiKmuT1n5ez0XW2JVfNI-rJYPYyr2fNFQ=.e959ec7a-d2db-413f-a7c8-3f8248a3f084@github.com>
Message-ID: <GmYgw0rktUzHbfdHf3LBjqKZHuRAtVGBC1Wb7Ya7WI0=.d7c9e41e-eb63-4593-81a7-1f317cc3f85d@github.com>

> As proposed in https://mail.openjdk.org/pipermail/crac-dev/2023-February/000496.html I am adding some CRaC related documentation.

Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:

  Add timers related docs

-------------

Changes:
  - all: https://git.openjdk.org/crac/pull/51/files
  - new: https://git.openjdk.org/crac/pull/51/files/bd2886bd..0f34254b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=crac&pr=51&range=01
 - incr: https://webrevs.openjdk.org/?repo=crac&pr=51&range=00-01

  Stats: 20 lines in 2 files changed: 20 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/crac/pull/51.diff
  Fetch: git fetch https://git.openjdk.org/crac pull/51/head:pull/51

PR: https://git.openjdk.org/crac/pull/51

From duke at openjdk.org  Tue Feb 28 16:58:19 2023
From: duke at openjdk.org (Radim Vansa)
Date: Tue, 28 Feb 2023 16:58:19 GMT
Subject: [crac] Integrated: Close extraneous file descriptors
In-Reply-To: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
References: <04HYBynBIevciK5RSZlNpUYJWeNyUTmjsx6VToZdWpQ=.0ed790b8-bd84-4d9f-96c2-8e2e1c28fdc8@github.com>
Message-ID: <-1pkfH8-G6ZDpujbpkVhruG0mwzJCsbUL9r2RDxf_-8=.d223e2c7-340d-4e31-a6e6-769a68f2a50a@github.com>

On Thu, 16 Feb 2023 07:36:43 GMT, Radim Vansa <duke at openjdk.org> wrote:

> Java code cannot use inherited non-standard file descriptors; these can cause trouble when trying to checkpoint the process via CRIU, and hence we will proactively close them on boot.

This pull request has now been integrated.

Changeset: 0738da8e
Author:    Radim Vansa <rvansa at azul.com>
Committer: Anton Kozlov <akozlov at openjdk.org>
URL:       https://git.openjdk.org/crac/commit/0738da8efe64f5db0b5a83f7d6de459a482a6e0e
Stats:     255 lines in 7 files changed: 239 ins; 12 del; 4 mod

Close extraneous file descriptors

Reviewed-by: akozlov

-------------

PR: https://git.openjdk.org/crac/pull/45

From akozlov at openjdk.org  Tue Feb 28 17:31:41 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Tue, 28 Feb 2023 17:31:41 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v8]
In-Reply-To: <siwf3aKNft0lP5KozWrVMvGb0ofHCFBQ4t70y7hV5wA=.52627dfc-b927-473f-a5ad-368cdfa04ba5@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <siwf3aKNft0lP5KozWrVMvGb0ofHCFBQ4t70y7hV5wA=.52627dfc-b927-473f-a5ad-368cdfa04ba5@github.com>
Message-ID: <XWuzc95KTxDnGpCNmFCPZiBvnYTYjKr57yRrG1NybJk=.ed9aff3a-46e8-4c2e-8007-381acc4f07a3@github.com>

On Thu, 23 Feb 2023 14:20:55 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with five additional commits since the last revision:
> 
>  - Merge branch 'crac-altstack-cpu-cpuexplicit-strip-x' into crac-altstack-cpu-cpuexplicit-strip
>  - db3e8e43: <no commit message found>
>  - Fix Freezer::in_handler.
>  - Revert "Remove Freeze; the problem is no longer reproducible for me."
>    
>    This reverts commit 5d3de5a10d7ab6f43344a0bb56168aa252b32845.
>  - man page: +options

I think in this PR we can concentrate on CPU features, as CPU core number is a different problem, that can arise even with the same feature set.

As for ifunc change and Watcher Thread problems, this won't happen if we re-execute with GLIBC_TUNABLES. And re-execute will also resolve the maintainability concern.

-------------

PR: https://git.openjdk.org/crac/pull/41

From akozlov at openjdk.org  Tue Feb 28 17:43:35 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Tue, 28 Feb 2023 17:43:35 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed [v2]
In-Reply-To: <loJNkXaMqkQsBtTqMjcafGZCg7jMszcC64GR-MOR5ls=.b6de1f8b-365b-466a-a57d-b48fa0342ed7@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
 <loJNkXaMqkQsBtTqMjcafGZCg7jMszcC64GR-MOR5ls=.b6de1f8b-365b-466a-a57d-b48fa0342ed7@github.com>
Message-ID: <1l4QkI7CfQNqf0Vsyviw0hJGKZL8_hHmfn1BNtRqaDs=.edfb244e-8af4-4262-b072-f875b2959bfa@github.com>

On Wed, 22 Feb 2023 13:21:35 GMT, Roman Marchenko <rmarchenko at openjdk.org> wrote:

>> When running CRaC with docker, java may exit before CRIU is finished dumpring because CRIU kills the original java process, and then docker immediately exits.
>> 
>> It could be reproduced with a simple Java test:
>> 
>> public class Test {
>>     public static void main(String args[]) throws Exception {
>>         jdk.crac.Core.checkpointRestore();
>>         System.out.println("finish");
>>     }
>> }
>> 
>> and run with docker:
>> `docker <docker_options> $JAVA_HOME/java -XX:CRaCCheckpointTo=./cr Test.java`
>> 
>> After the command above finishes, `cr/cppath` is absent in the case of failure. Or/and it will fail on restore:
>> `docker <docker_options> $JAVA_HOME/java -XX:CRaCRestoreFrom=./cr`
>> 
>> This change fixes the issue by forkin'g the main process in case of PID=1 (pid=1 means it was run with docker), and waiting for children processes are finished. This makes us sure that CRIU finalized the dump, if any. At the same time, there is no conflict with PIDs on restore, since the process being restored has PID not equal to 1, if restoring with the command above..
>
> Roman Marchenko has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Added sighandler to the main process

Changes requested by akozlov (Lead).

src/java.base/share/native/launcher/main.c line 118:

> 116:         pid = wait(&st);
> 117:         if (!status && 0 < pid && WIFEXITED(st)) {
> 118:             status = WEXITSTATUS(st);

Some `WEXITSTATUS(st)` can end up 0, so the a next exited child can override this status.

I think here we should watch for the status of the g_child_pid, and ignoring statuses of other childs that are not our immediate java process child.

-------------

PR: https://git.openjdk.org/crac/pull/46

From akozlov at openjdk.org  Tue Feb 28 17:46:41 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Tue, 28 Feb 2023 17:46:41 GMT
Subject: [crac] RFR: Fix failing CRaC tests [v2]
In-Reply-To: <akg4x2e9OIA2fIDVyjrr6VAa-RVDAkCu40-9fPma2jU=.94fc86a4-0e18-4991-8b79-0ea290df0e14@github.com>
References: <y83GItfrbGKa3TaSblmot3Gz1dgkSvugtn5ZUiytSCU=.79c13c83-3f7c-40e4-bab8-ef59b74f7c6d@github.com>
 <akg4x2e9OIA2fIDVyjrr6VAa-RVDAkCu40-9fPma2jU=.94fc86a4-0e18-4991-8b79-0ea290df0e14@github.com>
Message-ID: <WCJ6aC1qWOVAK1ibVMzy2BkKYrv0ly_PXGDucSGzi7c=.18f0fb1c-52e5-4460-a6e4-52203e83a587@github.com>

On Wed, 22 Feb 2023 15:07:04 GMT, Radim Vansa <duke at openjdk.org> wrote:

>> Before https://github.com/openjdk/crac/pull/16 the arguments for restored process were ignored; tests were written with this in mind and current behaviour breaks them.
>> 
>> In addition I've added missing `-XX:+UnlockDiagnosticVMOptions` flag and fixed the `recursiveCheckpoint` test runner which got stuck as with `pauseengine` the checkpointed process does not terminate.
>
> Radim Vansa has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove somebody's forgotten overrides

LGTM, Thanks!

-------------

Marked as reviewed by akozlov (Lead).

PR: https://git.openjdk.org/crac/pull/47

From akozlov at openjdk.org  Tue Feb 28 17:50:36 2023
From: akozlov at openjdk.org (Anton Kozlov)
Date: Tue, 28 Feb 2023 17:50:36 GMT
Subject: [crac] RFR: Add CRaC-specific tests to GHA
In-Reply-To: <OZX-HO3SKKuZLOBuyT0CaNluMk5cDaV151PWC4qXhqw=.6e9f255e-3c1d-4f45-aa31-e28e09426b59@github.com>
References: <OZX-HO3SKKuZLOBuyT0CaNluMk5cDaV151PWC4qXhqw=.6e9f255e-3c1d-4f45-aa31-e28e09426b59@github.com>
Message-ID: <PNj7KulAv3nBlbSQilH9Sp_7-QygqnNybRb3xGaxQ-k=.2e923692-cc4c-4a19-bdda-2d2f161c1c14@github.com>

On Wed, 22 Feb 2023 15:08:29 GMT, Radim Vansa <duke at openjdk.org> wrote:

> Existing GitHub Actions run test tier1 but since most changes in this project focus on the CRaC capabilities we should run them in an automated fashion, too.
> Right now the tests are mostly failing: this should be addressed in https://github.com/openjdk/crac/pull/47

Looks good, but could you rebase the PR after #47 is integrated to demonstrate the GHA is green?

-------------

PR: https://git.openjdk.org/crac/pull/48

From heidinga at openjdk.org  Tue Feb 28 18:43:40 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Tue, 28 Feb 2023 18:43:40 GMT
Subject: [crac] RFR: RFC: -XX:CPUFeatures=0xnumber for CPU migration [v8]
In-Reply-To: <siwf3aKNft0lP5KozWrVMvGb0ofHCFBQ4t70y7hV5wA=.52627dfc-b927-473f-a5ad-368cdfa04ba5@github.com>
References: <eJSSe8BwlSUChwJMbABqpQ9Zv5aEN99k6RBWefipJvw=.044460c8-9e4d-4944-97a8-79cdf7c7cc6f@github.com>
 <siwf3aKNft0lP5KozWrVMvGb0ofHCFBQ4t70y7hV5wA=.52627dfc-b927-473f-a5ad-368cdfa04ba5@github.com>
Message-ID: <69vfao9kbzm6veZFfaN_VVXhWlCMRnBNpuMorbVwGVU=.b058454e-bbe3-4e20-a65e-efc1670db3a8@github.com>

On Thu, 23 Feb 2023 14:20:55 GMT, Jan Kratochvil <duke at openjdk.org> wrote:

>> Currently if you `-XX:CRaCCheckpointTo` on a better CPU and `-XX:CRaCRestoreFrom` on a worse CPU the restored OpenJDK will crash.
>> 
>> 1. An obvious reason is that JIT-compiled code is using CPU features not implemented on the CPU where the image is restored.
>> 2. A second reason is that glibc has a similar problem, its PLT entries point to CPU optimized functions also crashing on the worse CPU. https://sourceware.org/glibc/wiki/GNU_IFUNC
>> 
>> (1) could be solved somehow automatically by deoptimizing and re-JITing all the JIT code. But that would defeat the performance goal of restoring a ready image in the first place. Therefore there had to be implemented a new OpenJDK option:
>> 
>>> use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine
>> 
>> It is intended to specify the lowest common denominator of all CPUs in a farm. Instead of a possible crash of OpenJDK it will now refuse to run:
>> 
>>> Error occurred during initialization of VM
>>> You have to specify -XX:CPUFeatures=0x421801fcfbd7 during -XX:CRaCCheckpointTo making of the checkpoint; specified -XX:CRaCRestoreFrom file contains CPU features 0x7fff9dfcfbf7; this machine's CPU features are 0x421801fcfbd7; missing features of this CPU are 0x3de79c000020 = 3dnowpref, adx, avx512f, avx512dq, avx512cd, avx512bw, avx512vl, sha, avx512_vpopcntdq, avx512_vpclmulqdq, avx512_vaes, avx512_vnni, clflushopt, clwb, avx512_vbmi2, avx512_vbmi
>> 
>> (2) has been implemented according to Anton Kozlov's idea that glibc can just reset its IFUNC PLT entries any time later (after restore), not just during the first initialization of glibc. That has currently a problem that it has turned out to be very invasive into private glibc structures. It could work somehow with glibc debuginfo (*-debuginfo.rpm or *-dbg.deb) installed but that has been considered as unacceptable requirement just to run CRaC. Therefore I have provided this proof of concept while I will propose such feature for glibc upstream where it is sure easily implementable.
>> 
>> If upstream glibc maintainers do not like the IFUNC reset idea then I do not think this hacky IFUNC reset patching many glibc internal data structures is a good way forward for a 3rd party implementation like CRaC/OpenJDK. In such case I believe one should switch to using GLIBC_TUNABLES environment variable, re-execing OpenJDK after converting the `-XX:CPUFeatures` OpenJDK format into glibc GLIBC_TUNABLES format. Unfortunately there is a precedent OpenJDK upstream has already rejected such re-exec idea in the past: https://github.com/openjdk/crac/pull/31#issuecomment-1275707621
>> 
>> That IMO does not preclude trying the same for this case.
>> 
>> - Debian 11 x86_64: It does not work, glibc is too different and inlined there.
>> - Debian 12 x86_64: It works even without libc6-dbg as its offsets are the default.
>> - Fedora 36 x86_64: It works as on Fedoras glibc debuginfo is embedded.
>
> Jan Kratochvil has updated the pull request incrementally with five additional commits since the last revision:
> 
>  - Merge branch 'crac-altstack-cpu-cpuexplicit-strip-x' into crac-altstack-cpu-cpuexplicit-strip
>  - db3e8e43: <no commit message found>
>  - Fix Freezer::in_handler.
>  - Revert "Remove Freeze; the problem is no longer reproducible for me."
>    
>    This reverts commit 5d3de5a10d7ab6f43344a0bb56168aa252b32845.
>  - man page: +options

src/hotspot/cpu/x86/vm_version_x86.cpp line 637:

> 635: #endif
> 636:     ;
> 637:   }

+1 to this part!  Getting a meaningful default for "generic" and the rationale for why it was chosen into the code is right overall approach. 

We can always adapt the set of features if needed in the future and this comment / approach provides the rationale for how it should be approached

-------------

PR: https://git.openjdk.org/crac/pull/41

From heidinga at openjdk.org  Tue Feb 28 18:48:36 2023
From: heidinga at openjdk.org (Dan Heidinga)
Date: Tue, 28 Feb 2023 18:48:36 GMT
Subject: [crac] RFR: CRaC may exit before image dump is completed [v2]
In-Reply-To: <1l4QkI7CfQNqf0Vsyviw0hJGKZL8_hHmfn1BNtRqaDs=.edfb244e-8af4-4262-b072-f875b2959bfa@github.com>
References: <r3bgy7fwAbd-zksRIfIXV_AzTNLmokrZIpqB4m7psN8=.22275a15-b70b-4c66-a98e-c5ae5a34e868@github.com>
 <loJNkXaMqkQsBtTqMjcafGZCg7jMszcC64GR-MOR5ls=.b6de1f8b-365b-466a-a57d-b48fa0342ed7@github.com>
 <1l4QkI7CfQNqf0Vsyviw0hJGKZL8_hHmfn1BNtRqaDs=.edfb244e-8af4-4262-b072-f875b2959bfa@github.com>
Message-ID: <4nHVLWk1AuaRXOGNtOXjwvz2RYK0LGL5ss8MrPuBBmc=.7624f707-b7bc-4c78-b12d-ebc10a1c1030@github.com>

On Tue, 28 Feb 2023 17:39:05 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Roman Marchenko has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Added sighandler to the main process
>
> src/java.base/share/native/launcher/main.c line 118:
> 
>> 116:         pid = wait(&st);
>> 117:         if (!status && 0 < pid && WIFEXITED(st)) {
>> 118:             status = WEXITSTATUS(st);
> 
> Some `WEXITSTATUS(st)` can end up 0, so the a next exited child can override this status.
> 
> I think here we should watch for the status of the g_child_pid, and ignoring statuses of other childs that are not our immediate java process child.

I think Anton's right - we want to exit with the `g_child_pid`'s status so it's the only one we need to save away.  The rest of the wait() loop looks right in that it ensures we correctly wait on all children (even children of the g_child_pid) like an init process should.

-------------

PR: https://git.openjdk.org/crac/pull/46