RFR: 8271024: Implement macOS Metal Rendering Pipeline

[Jayathirth D V]

On Mon, 16 Jun 2025 15:57:31 GMT, Martin Fox <mfox at openjdk.org> wrote:

>> ### Description
>> This is the implementation of new graphics rendering pipeline for JavaFX using Metal APIs on MacOS.
>> We released two Early Access (EA) builds and have reached a stage where it is ready to be integrated.
>> Default rendering pipeline on macOS has not been changed by this PR. OpenGL still stays as the default rendering pipeline and Metal rendering pipeline is optional to choose (by providing  `-Dprism.order=mtl`)
>> The `-Dprism.verbose=true` option can be used to verify the rendering pipeline in use.
>> 
>> ### Details about the changes
>> 
>> **Shader changes**
>> - MSLBackend class: This is the primary class that parses (Prism and Decora) jsl shaders into Metal shaders(msl)
>> - There are a few additional Metal shader files added under directory : modules/javafx.graphics/src/main/native-prism-mtl/msl
>> 
>> **Build changes** - There are new tasks added to build.gradle for
>> - Generation/ Compilation/ linking of Metal shaders
>> - Compilation of Prism Java and Objective C files
>> 
>> **Prism** - Prism is the rendering engine of JavaFX.
>> - Added Metal specific Java classes and respective native implementation which use Metal APIs
>> - Java side changes:
>>   - New Metal specific classes: Classes prefixed with MTL, are added here : modules/javafx.graphics/src/main/java/com/sun/prism/mtl 
>>   - Modification to Prism common classes: A few limited changes were required in Prism common classes to support Metal classes. 
>> - Native side changes:
>>   - New Metal specific Objective C implementation is added here: modules/javafx.graphics/src/main/native-prism-mtl
>> 
>> **Glass** - Glass is the windowing toolkit of JavaFX
>> - Existing Glass classes are refactored to support both OpenGL and Metal pipelines
>> - Added Metal specific Glass implementation.
>> 
>> ### Testing
>> - Testing performed on different hardware and macOS versions.
>>   - HW - macBooks with Intel, Intel with discrete graphics card, Apple Silicon (M1, M2 and M4)
>>   - macOS versions - macOS 13, macOS 14 and macOS 15
>> - Functional Testing:
>>   - All headful tests pass with Metal pipeline.
>>   - Verified samples applications: Ensemble and toys
>> - Performance Testing:
>>   - Tested with RenderPerfTest: Almost all tests match/exceed es2 performance except a few that fall a little short on different hardwares. These will be addressed separately (there are open bugs already filed)
>> 
>> ### Note to reviewers :
>> - Providing `-Dprism.order=mtl` option is a must for testing the Metal pipeline
>> - It woul...
>
> Metal is here! Yes!
> 
> It took me a while to sort out how the GlassView and GlassLayer classes work in this PR. It’s not clear based on the naming that the classes which handle drawing (GlassViewCGL3D and GlassViewMTL3D ) are not subclasses of the class that handles events (GlassView3D). The naming conventions could be clearer or at the very least there should be some comments in the code.
> 
> I think some of this structure is being dictated by the way GlassViewCGL3D is derived from NSOpenGLView since that prevents us from making it a subclass of, say, GlassView3D. But I’m almost certain the NSOpenGLView reference is just cruft. All of the OpenGL processing happens in a CAOpenGLLayer which has nothing to do with the (unused) NSOpenGLView machinery. In the current master and this PR I can derive from NSView instead of NSOpenGLView and everything works fine.
> 
> There is some code in Prism (MacOSXWindowSystemInterface.m) that can associate an NSOpenGLContext with an NSView but I don’t think a view is ever passed in. It would be nice to clean that code up since the view-related calls are deprecated and generating compiler warnings.
> 
> I think the NSOpenGLView reference should be removed. Beyond that you could keep the current class structure and view layout since there’s something to be said for separating drawing and event processing into different NSViews.
> 
> I would rather not see the 3D terminology carried over into the new classes. It's there to make a distinction that's no longer relevant (according to the comments there used to be a GlassView2D). But that is a personal pet peeve so feel free to ignore it.
> 
> Look like it’s not possible to add comments to unchanged code in GitHub (?!) so I guess I have to write the following issues up here.
> 
> In GlassWindow.m line 807 there’s this bit of mysterious code:
> 
> 
> CALayer *layer = [window->view layer];
> if (([layer isKindOfClass:[CAOpenGLLayer class]] == YES) &&
>     (([window->nsWindow styleMask] & NSWindowStyleMaskTexturedBackground) == NO))
> {
>     [((CAOpenGLLayer*)layer) setOpaque:[window->nsWindow isOpaque]];
> }
> 
> 
> It was put there to resolve [JDK-8095004](https://bugs.openjdk.org/browse/JDK-8095004) and [JDK-8095040](https://bugs.openjdk.org/browse/JDK-8095040). In this PR the layer is nil at this point so this code isn’t doing anything. I haven’t verified whether this is causing a regression. In the master branch that setOpaque: call will always happen since the NSWindowStyleMaskTexturedBackground bit is never set (it’s ass...

@beldenfox Thanks for your inputs.

1.  Regarding adding comments in GlassView/Layer classes about how they are not subclasses but subViews/subLayers : Sure i will go ahead and add comments in the class. Also as you mentioned we can name them better:

- GlassView3D ->  GlassViewEvent(Since it is common class which handles events for both OpenGL and Metal pipeline) Or GlassSubView(Since it is added as subView in GlassView). Please suggest if you have better names that we can use here.
- GlassViewCGL3D -> GlassViewCGL(With comment mentioning how it is not subclass but a subView of GlassView3D)
- GlassViewMTL3D -> GlassViewMTL(With comment mentioning how it is not subclass but a subView of GlassView3D)
- At other places also i will remove 3D in names and appropriate comment in CGL/MTL layer class.

2. Regarding NSOpenGLView : Since both CGL and MTL views are not subclass of same NSView we have current structure with Event handling in common class GlassView3D and CGL & MTL Views as subViews of GlassView3D. I also tried removing NSOpenGLView reference and then running Ensemble8 in ES2 pipeline and things work fine. But we want to minimize changes related to already present default ES2 pipeline in this PR as part of refactoring effort. We can work on it in as a follow-up issue.

3. Regarding change in MacOSXWindowSystemInterface.m : Same here we want to minimize changes related to ES2 in this PR. We can work on it in as a follow-up issue.

4. Regarding setOpaque() in GlassWindow.m : While refactoring Glass for Metal pipeline under [JDK-8325379](https://bugs.openjdk.org/browse/JDK-8325379) this was identified as follow up issue : [JDK-8356758](https://bugs.openjdk.org/browse/JDK-8356758). We were aware of how this code related to [JDK-8095004](https://bugs.openjdk.org/browse/JDK-8095004) but thanks for pointing to [JDK-8095040](https://bugs.openjdk.org/browse/JDK-8095040) which was not verified. There are no regression seen with this refactoring, PickTest3D works properly in both ES2 and MTL pipeline of this PR and matches output seen with mainline code. I verified that test mentioned in [JDK-8095040](https://bugs.openjdk.org/browse/JDK-8095040) on latest mainline code and i still see that black filling is drawn. Same behaviour is seen with ES2 and MTL pipeline with this PR. But it works properly in 8u431, looks like the issue has resurfaced so i have created new bug : [JDK-8359904](https://bugs.openjdk.org/browse/JDK-8
 359904)

5. Regarding removal of JNI method getNativeLayer() from GlassView.m : Yes this is not used anywhere and we can remove it. I removed this code, built and tested Ensemble8, started CI headful test run. If everything is green will go ahead and remove it and its corresponding code in MacView.java

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1824#issuecomment-2983606142