RFR: 8201567: QuantumRenderer modifies buffer in use by JavaFX Application Thread

[John Neffenger]

On Fri, 26 Jun 2020 03:47:55 GMT, John Neffenger <github.com+1413266+jgneff at openjdk.org> wrote:

> Fixes [JDK-8201567](https://bugs.openjdk.java.net/browse/JDK-8201567).

The method `QueuedPixelSource.usesSameBuffer` calls `Pixels.getPixels` on the QuantumRenderer thread while trying to
find a buffer that's not in use, yet in doing so it rewinds buffers in use on the JavaFX Application Thread.

This pull request modifies `QueuedPixelSource.usesSameBuffer` to call a new method, `Pixels.getBuffer`, that returns
the buffer without rewinding it.

Because the issue only affects the final rendered pixels, I added assertions to catch the error in the Monocle classes
`HeadlessScreen` and `EPDScreen` instead of creating a unit test case. I tried to stay within the guidelines of
[Programming With Assertions][1], which states:

> As a rule, the expressions contained in assertions should be free of *side effects*: evaluating the expression should
> not affect any state that is visible after the evaluation is complete. One exception to this rule is that assertions
> can modify state that is used only from within other assertions.

Below are animated images showing a few frames from a JavaFX animation on the Monocle VNC platform before and after the
fix.

### Before the fix

![monocle-vnc-sw-before](https://user-images.githubusercontent.com/1413266/85825425-0f89e100-b737-11ea-98c6-32fb624dd306.gif)

### After the fix

![monocle-vnc-sw-after](https://user-images.githubusercontent.com/1413266/85825441-19134900-b737-11ea-930f-b7865556d17e.gif)

### Background

The error occurs with software rendering when a subclass of `com.sun.glass.ui.View` uses the buffer position to upload
a pixel buffer to the screen. That can occur in at least two cases:

1. when the pixels are uploaded to a composition byte buffer on the heap, and

2. when the width of the JavaFX scene is less than the width of the screen.

In the first case, the non-direct byte buffer method `IntBuffer.put(IntBuffer)` relies on the source buffer position
when copying the pixels, while the corresponding direct byte buffer method `DirectIntBufferU.put(IntBuffer)` does not.

In the second case, `Framebuffer.composePixels`, for example, uses the buffer position to loop through the source pixel
buffer when its width does not match the width of the destination pixel buffer.

Most subclasses of `View` upload pixels with native methods that do not use the buffer position, so their final
rendered pixels are not corrupted.

Because of implementation choices, only the Monocle platforms end up having visible errors in their rendered pixels.
While there are ways to work around the issue just for Monocle, this pull request is an attempt to correct the error at
its source.

### Testing

To reproduce the problem, I used the JavaFX application [epd-javafx][2] with the following command:

$ java --add-modules=javafx.graphics \
  --module-path=$HOME/lib/armv6hf-sdk/lib \
  -Dglass.platform=Monocle -Dmonocle.platform=VNC -Dprism.order=sw \
  -jar dist/epd-javafx.jar --pattern=3 --loops=0

You can run the command on an actual ARM device or on a QEMU *armhf* virtual machine running on an *amd64* Linux host.
I connected to the Monocle VNC server with the Remmina VNC client on port 5901 with 24-bit color and encryption
disabled.

Even without access to an ARM device or virtual machine, you can capture the error on any Linux desktop by adding
similar `assert` statements to the `_uploadPixels` method of `GtkView`, along with calls to `Thread.sleep` to change
the timing of the two threads.

Let me know if you would like information on installing a QEMU *armhf* virtual machine, or details on the assertions
and `sleep` calls that allowed me to captured the error directly on my Dell Linux workstation.

[1]: https://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html#usage
[2]: https://github.com/jgneff/epd-javafx

-------------

PR: https://git.openjdk.java.net/jfx/pull/255