Blank stages when running JavaFX app in a macOS virtual machine

[thevenet.fred at free.fr]

Hello,

When running a JavaFX application in macOS guest VM, the main stage is completely blank, with the following errors: CGLChoosePixelFormat error: 10002, CGLCreateContext error: 10002
This behavior was observed with a macOS 10.15 guest OS on both VMWare player and VirtualBox, on a Windows 10 host.

This is an old issue that has already been reported several times (notably as JDK-8154852) but is claimed to have been fixed by JDK-8154148.
However the fix provided by JDK-8154148 is incomplete; while it does fix a JVM crash, it doesn't prevent an incorrect pixel format to be retrieved, which is the root cause for the stage being empty.

The problematic code is located at #96 in GlassView3D.m:

095: CGLError err = CGLChoosePixelFormat(attributes, &pix, &npix);
096: if ((err == kCGLNoError) && (npix == 0))
097: {
098:   const CGLPixelFormatAttribute attributes2[] =
099:   {
100:     kCGLPFAAllowOfflineRenderers,
101:     (CGLPixelFormatAttribute)0
102:   };
103:   err = CGLChoosePixelFormat(attributes2, &pix, &npix);
104: }


In a comment in JDK-8154148, the following claim is made: "[...] an error happens is when something bad or invalid has occurred. An unsatisfied pixel format request is indicated by npix is 0 and pix is NULL. Hence I feel it is a cleaner and clearer logic to request a different format when npix is 0 and err is kCGLNoError." (https://bugs.openjdk.java.net/browse/JDK-8154148?focusedCommentId=13980465&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13980465)

>From what I could observe, this is not necessarily true; in this case where a pixel format satisfying the initial conditions could not be found, the return code is kCGLBadPixelFormat (10002); which means we don't try to get a pixel format with less restrictive condition.
This suggests that a better condition for the second call to CGLChoosePixelFormat (line #103) should be "if (pix == NULL)" instead of "if ((err == kCGLNoError) && (npix == 0))"
This is further supported by a sample in Apple developer's documentation on how to choose a pixel format: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/OpenGL-MacProgGuide/opengl_pixelformats/opengl_pixelformats.html

I have verified that this change produces the expected behavior and I propose to submit a PR with this change (I have already signed the OCA)

A quick question before I do, however; should I file a new issue prior to submitting a PR or is it better that someone with privileges reopens JDK-8154148?

Regards,

Frederic Thevenet