[Rev 02] RFR: 8235627: Blank stages when running JavaFX app in a macOS virtual machine

[Kevin Rushforth]

On Fri, 20 Dec 2019 17:41:00 GMT, Frederic Thevenet <github.com+7450507+fthevenet at openjdk.org> wrote:

>> This is a PR for [JDK-8235627](https://bugs.openjdk.java.net/browse/JDK-8235627).
>> 
>> When running a JavaFX application in macOS guest VM, the main stage is completely blank, with the following errors: CGLChoosePixelFormat error: 10002, CGLCreateContext error: 10002
>> This behavior was observed with a macOS 10.15 guest OS on both VMWare player and VirtualBox, on a Windows 10 host.
>> 
>> This is an old issue that has already been reported several times (notably as JDK-8154852) but is claimed to have been fixed by JDK-8154148.
>> However the fix provided by JDK-8154148 is incomplete; while it does fix a JVM crash, it doesn't prevent an incorrect pixel format to be retrieved, which is the root cause for the stage being empty.
>> 
>> The problematic code is located at #96 in GlassView3D.m:
>> 
>> 095: CGLError err = CGLChoosePixelFormat(attributes, &pix, &npix);
>> 096: if ((err == kCGLNoError) && (npix == 0))
>> 097: {
>> 098: const CGLPixelFormatAttribute attributes2[] =
>> 099: {
>> 100: kCGLPFAAllowOfflineRenderers,
>> 101: (CGLPixelFormatAttribute)0
>> 102: };
>> 103: err = CGLChoosePixelFormat(attributes2, &pix, &npix);
>> 104: }
>> 
>> 
>> In a comment in JDK-8154148, the following claim is made: "[...] an error happens is when something bad or invalid has occurred. An unsatisfied pixel format request is indicated by npix is 0 and pix is NULL. Hence I feel it is a cleaner and clearer logic to request a different format when npix is 0 and err is kCGLNoError." (https://bugs.openjdk.java.net/browse/JDK-8154148?focusedCommentId=13980465&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13980465)
>> 
>> From what I could observe, this is not necessarily true; in this case where a pixel format satisfying the initial conditions could not be found, the return code is kCGLBadPixelFormat (10002); which means we don't try to get a pixel format with less restrictive condition.
>> This suggests that a better condition for the second call to CGLChoosePixelFormat (line #103) should be "if (pix == NULL)" instead of "if ((err == kCGLNoError) && (npix == 0))"
>> This is further supported by a sample in Apple developer's documentation on how to choose a pixel format: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/OpenGL-MacProgGuide/opengl_pixelformats/opengl_pixelformats.html
> 
> The pull request has been updated with 1 additional commit.

Looks good.

Once @arapte approves,you can /integrate this change, and then he can sponsor it.

-------------

Marked as reviewed by kcr (Lead).

PR: https://git.openjdk.java.net/jfx/pull/65