RFR: 8280468: Crashes in getConfigColormap, getConfigVisualId, XVisualIDFromVisual on Linux

[Maxim Kartashev]

On Wed, 9 Feb 2022 02:56:55 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

> > 2. The idea of testing if the fix actually cures the crash suggested by @mrserb will work, but only to prove that one particular hypothesis about the crash is correct.
> 
> Even this confirmation will be useful.

A reproducer has been created, please see [this commit](https://github.com/mkartashev/jdk/commit/202afe06763ed3ebda6d042239e578af299bb7a0) in a separate branch.
The idea of the reproducer was to introduce artificial synchronization and then delay between the time `X11GraphicsEnvironment.initDevices()` invalidates the device and the time `X11GraphicsDevice.makeConfigurations()` is made aware of that. In the scenario the reproducer helps to create, `makeConfigurations()` starts working on the second monitor (`screen == 1`) and in the middle of that the screen disappears, but the screen number is changed 500+ms later, allowing `getConfigVisualId()` to be called with screen number 1 that is no longer valid. To facilitate a crash rather than random memory corruption, I pad the array of screens (`x11Screens` from `awt_GraphicsEnv.c`) with a couple of zeroed-out elements at the end. This way, when accessing past the effective end, you're de-referencing a zero (or near-zero) pointer.

Steps to reproduce are described in the `Test.java` file in the repository root of the mentioned commit; there's also a sample of the crash there. To repeat:

On a Linux box with exactly two monitors running X11 (no Wayland or XWayland),
- build branch [JDK-8280468-reproducer](https://github.com/mkartashev/jdk/tree/JDK-8280468-reproducer),
- compile `Test.java` from the repository root,
- execute it using the freshly built `java` (both release and fastdebug builds will work),
- when this gets printed to stdout
`makeConfigurations(): waiting for the signal to continue...`
 unplug or disable the second monitor on the system,
- observe the crash.

The reproducer doesn't help to validate the fix, however, as it converts SIGSEGV into a deadlock. It is also quite fragile in the sense that it is built on a number of implicit assumptions about the order and quantity of several things, which may or may not differ between systems. If it doesn't crash on your system, some tweaks may be in order.

Hope this helps.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7182