[JavaFX 3D ( | Feature Request)] Setting Texture Nearest-Sampling on PhongMaterial

Fri Aug 25 22:17:16 UTC 2023

I've done a preliminary study and assessment for changes in regards to
textures.

Texture metadata (addressing, filtering, wrapping, blending...) is
supported by both OpenGL and D3D in a similar enough way that makes a
uniform API possible and not convoluted. This metadata can be added to
Texture, or perhaps better, TextureMap.
Unfortunately, there is no ideal way of creating the public API because the
textures in PhongMaterial are represented as Images. Interestingly, there
are comments on the map properties in PhongMaterial that touch on this, "//
TODO: 3D - Texture or Image? For Media it might be better to have it as a
Texture", but I think that the question didn't get resolved in time and
Image stayed.

Because we can't change the type, we have the following options:
1. Duplicate the methods in PhongMaterial that use ObjectProperty<Image>
and have a set that uses ObjectProperty<Texture> (or TextureMap) with it
taking precedence. I think that this is a bad option as it creates API
noise and is confusing to use.
2. Add the metadata to Image and enhance its public API. I don't think that
this is a good option either because it will bloat Image with specific data
that is not widely used.
3. Create a new Material, placeholder name PhongMaterial2, that uses
textures instead. We might want to look at other breaking enhancements for
PhongMaterial so we can plan ahead better this time. I think that this is a
reasonable option whose eligibility depends on how much we can't update the
current PhongMaterial. Mipmaping, which requires a set of images, might be
one of them.
4. Create a separate set of methods inside PhongMaterial that deal with the
texture metadata only. These will be something like
`ObjectProperty<TextureData> diffuseTextureDataProperty()` and similarly
for the other maps. I think that this option is also reasonable, although
it also creates API noise (these only take effect if the maps are used to
begin with).

I didn't look deeply into texture aspects outside of the discussed
filtering, but I think we should take the opportunity to do so. For
filtering, I have the following ideas:

For min/mag filters: an enum with NEAREST (nearest-point/neighbor sampling)
and LINEAR ((bi)linear sampling).
For mipmap filters: an enum with NEAREST (nearest mipmap level) and LINEAR
(linear interpolation between levels). While some APIs have a NONE mode to
disable mipmaping, the inexistence of mipmaps can be used as a disabled
mode.
Even though they use the same constant names, the meaning is different, so
I think the enums should be separate.

For mipmaps: a List<Image>. Autogeneration of mipmaps can also be
considered.

Anisotropy is supported in D3D via a filter type (ANISOTROPIC) plus a
strength factor, and in OpenGL since version 4.6 (which version does JavaFX
use?). Metal and Vulkan just use a strength factor to turn it on regardless
of the filter. I think that this is a more flexible approach and we could
add a single int/double property to control anisotropy.

References:
Metal specs [1] (section 2.10)
Vulkan specs [2] (Section 16.8.3)
D3D docs [3][4][5]
OpenGL (external resource) [6][7]

[1]
https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf
[2]
https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html
[3]
https://learn.microsoft.com/en-us/windows/win32/direct3d9/texture-filtering
[4]
https://learn.microsoft.com/en-us/windows/win32/direct3d9/d3dtexturefiltertype
[5]
https://learn.microsoft.com/en-us/windows/win32/api/d3d9helper/nf-d3d9helper-idirect3ddevice9-setsamplerstate
[6] https://learnopengl.com/Getting-started/Textures
[7]
https://gregs-blog.com/2008/01/17/opengl-texture-filter-parameters-explained/

On Sun, Jun 25, 2023 at 9:03 PM Nir Lisker <nlisker at gmail.com> wrote:

> Perhaps Jay and Kevin can weigh in on this.
>
> By the way, I see that the com.sun.prism.Texture interface already defines
> get/setLinearFiltering methods that specify if the filtering is not linear
> then it uses a nearest neighbor algorithm. It's not used by the 3D side it
> seems.
>
> On Sun, Jun 25, 2023 at 8:54 PM Matija Brown <Matija.Brown at outlook.de>
> wrote:
>
>> This is a good point you’re bringing up Nir!
>>
>>
>>
>> I’m really not sure on that. On the one hand your interpretation seems
>> the most sensible, but on the other hand I can’t see any use case of
>> NEAREST filtering in specular or self-illumination maps.
>>
>> However giving more options is also always good, and the same way there
>> aren’t really many places (that I’m aware of) one would have NEAREST
>> filtered diffuse maps and use any of the others at all.
>>
>>
>>
>> TLDL: Would work excellently your way API wise, but I’m not certain if it
>> makes sense use-case wise.
>>
>>
>>
>> Thanks,
>>
>> Matija.
>>
>>
>>
>> *From: *Nir Lisker <nlisker at gmail.com>
>> *Sent: *Sunday, 25 June 2023 19:49
>> *To: *Matija Brown <Matija.Brown at outlook.de>
>> *Cc: *Jayathirth Rao Daarapuram Venkatesh Murthy
>> <jayathirth.d.v at oracle.com>; Kevin Rushforth <kevin.rushforth at oracle.com>;
>> openjfx-dev at openjdk.org
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>>
>>
>> I think that the first question to answer is whether the filtering method
>> is applied to all maps/textures of the material (diffuse, specular,
>> self-illumination) or to each individually. I would imagine that the former
>> makes more sense. If that's the case, the texture filtering parameter will
>> be just one more property in PhongMaterial (perhaps in the Material
>> supertype if it also makes sense for other possible materials - that can be
>> figured out later in any case).
>>
>>
>>
>> On Sun, Jun 25, 2023 at 3:20 PM Matija Brown <Matija.Brown at outlook.de>
>> wrote:
>>
>> Unfortunately it took a little longer (got busy), but have just had a
>> quick look at the
>>
>> OpenGL side of things.
>>
>>
>>
>> The ES2Texture class appears to handle everything there. There are two
>> create-Methods, but the other
>>
>> one is responsible for MediaFrame stuff and it doesn’t really? make sense
>> to have non-linear filtering there?
>>
>> What’s your opinion?
>>
>>
>>
>> The create method seems to parse in all texture parameters simply as
>> arguments (
>> https://github.com/openjdk/jfx/blob/0d9dcf38275528e1b621d71631aac5bdb9452110/modules/javafx.graphics/src/main/java/com/sun/prism/es2/ES2Texture.java#L98
>> ),
>>
>> it would probably be best to just add the filtering in there as well.
>>
>>
>>
>> API-wise it’s probably best to have a Texture or DiffuseMap class that
>> the PhongMaterial uses and stores metadata about the Image.
>>
>> However this might be slightly overkill considering there is only this
>> one single parameter we’re trying to add,
>>
>> thus maybe just a flag in PhongMaterial would suffice?
>>
>>
>>
>> Have a nice day,
>>
>> Matija.
>>
>> *From: *Matija Brown <Matija.Brown at outlook.de>
>> *Sent: *Thursday, 22 June 2023 18:37
>> *To: *Jayathirth Rao Daarapuram Venkatesh Murthy
>> <jayathirth.d.v at oracle.com>; Nir Lisker <nlisker at gmail.com>; Kevin
>> Rushforth <kevin.rushforth at oracle.com>
>> *Cc: *openjfx-dev at openjdk.org
>> *Subject: *RE: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>>
>>
>> I’ll do the OpenGL – have done quite a bit with Vulkan and GL in the past
>> so there’s no problem there.
>>
>> Certainly usefull if you would do some reviewing – if you finish the
>> Metal texture maps someone will surely find
>>
>> themselves to expand it with sampling.
>>
>>
>>
>> Tomorrow I’ll give a short summary on what would be to do for OpenGL, as
>> we have that for D3D already.
>>
>> Then do a little example probably – doesn’t really seem too much work to
>> me?
>>
>>
>>
>> Thanks,
>>
>> Matija.
>>
>>
>>
>> *From: *Jayathirth Rao Daarapuram Venkatesh Murthy
>> <jayathirth.d.v at oracle.com>
>> *Sent: *Thursday, 22 June 2023 06:28
>> *To: *Nir Lisker <nlisker at gmail.com>; Kevin Rushforth
>> <kevin.rushforth at oracle.com>
>> *Cc: *openjfx-dev at openjdk.org
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>>
>>
>> Correcting myself:
>>
>> Currently I am working on Metal implementation of Texture maps in JavaFX
>> 3D : https://bugs.openjdk.org/browse/JDK-8310109 and not on adding
>> nearest sampling(which last mail can imply).
>>
>>
>>
>> Thanks,
>>
>> Jay
>>
>>
>>
>> *From: *openjfx-dev <openjfx-dev-retn at openjdk.org> on behalf of
>> Jayathirth Rao Daarapuram Venkatesh Murthy <jayathirth.d.v at oracle.com>
>> *Date: *Thursday, 22 June 2023 at 9:22 AM
>> *To: *Nir Lisker <nlisker at gmail.com>, Kevin Rushforth <
>> kevin.rushforth at oracle.com>
>> *Cc: *openjfx-dev at openjdk.org <openjfx-dev at openjdk.org>
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>> Currently I am working on Metal implementation of the same and has no
>> bandwidth to work on additional OpenGL thing.
>>
>> But I can help in reviewing the code if we come up with addition of
>> nearest filtering.
>>
>>
>>
>> Thanks,
>>
>> Jay
>>
>>
>>
>> *From: *openjfx-dev <openjfx-dev-retn at openjdk.org> on behalf of Nir
>> Lisker <nlisker at gmail.com>
>> *Date: *Wednesday, 21 June 2023 at 9:40 PM
>> *To: *Kevin Rushforth <kevin.rushforth at oracle.com>
>> *Cc: *openjfx-dev at openjdk.org <openjfx-dev at openjdk.org>
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>> If I remember correctly, in OpenGL you parse the texture filter when the
>> texture is being created, while in DirectX as it is bound
>>
>> to the sampler (it is a sampler state after all) it would have to be set
>> before every render call. However it shouldn’t make any
>>
>> API difference really, as we can just have a field somewhere and the
>> parse it along when needed.
>>
>>
>>
>> Yes, the JBS ticket mentions this difference as well.
>>
>>
>>
>> Since Kevin approved this feature and the API seems to converge nicely
>> between the pipelines, we can start the work. I'm somewhat busy with other
>> tasks as of late, but I will try to formulate an API. Matija or Jay, if one
>> of you can start investigating the changes to the OpenGL pipeline we could
>> create a branch in the sandbox repo and work there.
>>
>>
>>
>> On Wed, Jun 21, 2023 at 4:03 PM Kevin Rushforth <
>> kevin.rushforth at oracle.com> wrote:
>>
>> My preference would be to add support only for Linear and Nearest in any
>> case.
>>
>> -- Kevin
>>
>> On 6/21/2023 4:48 AM, Matija Brown wrote:
>>
>> As Jayathrith said, in OpenGL as well as Metal only NEAREST and LINEAR
>> filters are available.
>>
>>
>>
>> There might be a way of getting around it by implementing some own
>> algorithm for OpenGL and Metal but that seem slightly
>>
>> over the top. So either one would have to keep the options limited to the
>> two supported everywhere
>>
>> or go with the “conditional” features.
>>
>>
>>
>> Having had a look at DirectX the APIs seem not to differ too much.
>>
>> If I remember correctly, in OpenGL you parse the texture filter when the
>> texture is being created, while in DirectX as it is bound
>>
>> to the sampler (it is a sampler state after all) it would have to be set
>> before every render call. However it shouldn’t make any
>>
>> API difference really, as we can just have a field somewhere and the
>> parse it along when needed.
>>
>>
>>
>> Cheers,
>>
>> Matija.
>>
>>
>>
>> *From: *Jayathirth Rao Daarapuram Venkatesh Murthy
>> <jayathirth.d.v at oracle.com>
>> *Sent: *Wednesday, 21 June 2023 13:09
>> *To: *Nir Lisker <nlisker at gmail.com>; Matija Brown
>> <Matija.Brown at outlook.de>
>> *Cc: *openjfx-dev at openjdk.org
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>>
>>
>> In OpenGL we set GL_LINEAR by default at :
>> https://github.com/openjdk/jfx/blob/0d9dcf38275528e1b621d71631aac5bdb9452110/modules/javafx.graphics/src/main/java/com/sun/prism/es2/ES2Texture.java#L221
>>
>>
>>
>> And Metal supports only two types of Min/Max filters : Nearest(default)
>> and Linear. So even if D3D supports multiple types we might be limited to
>> support only these 2 filters for all platforms.
>>
>>
>>
>> Thanks,
>>
>> Jay
>>
>>
>>
>> *From: *openjfx-dev <openjfx-dev-retn at openjdk.org>
>> <openjfx-dev-retn at openjdk.org> on behalf of Nir Lisker
>> <nlisker at gmail.com> <nlisker at gmail.com>
>> *Date: *Wednesday, 21 June 2023 at 3:57 PM
>> *To: *Matija Brown <Matija.Brown at outlook.de> <Matija.Brown at outlook.de>
>> *Cc: *openjfx-dev at openjdk.org <openjfx-dev at openjdk.org>
>> <openjfx-dev at openjdk.org>
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>> First of all, please excuse directly e-mailing you earlier today. Working
>> with mailing lists is quite new for me and apparently I forgot to add
>>
>> the mailing list to cc.
>>
>>
>>
>> I didn't get any private email, so you seem to be doing better than you
>> thought with the mailing list :)
>>
>>
>>
>> With the OpenGL-side I do have some experience. Concerning the D3D-side
>> of things it would probably mean convincing
>>
>> Somebody that it is a good idea to finally apply the suggested change.
>>
>>
>>
>> There would be some API change required of course. As a basic concept it
>> would probably be sensible to add this as a parameter to the
>> PhonMaterial-class.
>>
>> Alternatively it might make sense to add a “Texture” class that is used a
>> a DiffuseMap in the PhonMaterial. But that seems slightly overkill.
>>
>>
>>
>> Where to put the new methods in the Java side is not the concerning part,
>> it's how to create methods that match all the pipelines. In D3D, the method
>> for setting the filter is detailed in [1], and its possible parameters in
>> [2][3][4]. So suppose that I'm looking at the list of available filter
>> types there:
>>
>> D3DTEXF_NONE, D3DTEXF_POINT, D3DTEXF_LINEAR, D3DTEXF_ANISOTROPIC,
>> D3DTEXF_PYRAMIDALQUAD, D3DTEXF_GAUSSIANQUAD, D3DTEXF_CONVOLUTIONMONO,
>>
>> if OpenGL supports a somewhat different set of filters, we will have some
>> clashes in the Java API side.
>>
>>
>>
>> Generally, we would like to give as much flexibility to the user as
>> possible, but need to be careful with platform-specific functionality. We
>> could round *some* corners. For example, I think that if we have an enum
>> for the filter types above that is a union of the ones available in the
>> different pipelines, and if a few are supported by only one of the
>> pipelines, we could note it in the docs and get away with it (something
>> similar to a conditional feature). However, if the whole native pipeline
>> setup for texture filtering is different, which means a different set of
>> Java API methods per pipeline, then that's too much.
>>
>>
>>
>> All this means is that to continue we need to figure out what the API for
>> each pipeline looks like, what's the most functionality we can have for
>> each pipeline, and then how we can unite them into a single Java API with
>> the hopes of being able to reconcile the differences "well enough"
>> (whatever that will mean).
>>
>>
>>
>> [1]
>> https://learn.microsoft.com/en-us/windows/win32/api/d3d9helper/nf-d3d9helper-idirect3ddevice9-setsamplerstate
>>
>> [2]
>> https://learn.microsoft.com/en-us/windows/win32/direct3d9/d3dtexturefiltertype
>>
>> [3]
>> https://learn.microsoft.com/en-us/windows/win32/direct3d9/d3dsamplerstatetype
>>
>> [4]
>> https://learn.microsoft.com/en-us/windows/win32/direct3d9/vertex-textures-in-vs-3-0
>>
>>
>>
>> On Wed, Jun 21, 2023 at 12:45 PM Matija Brown <Matija.Brown at outlook.de>
>> wrote:
>>
>> First of all, please excuse directly e-mailing you earlier today. Working
>> with mailing lists is quite new for me and apparently I forgot to add
>>
>> the mailing list to cc.
>>
>>
>>
>> With the OpenGL-side I do have some experience. Concerning the D3D-side
>> of things it would probably mean convincing
>>
>> Somebody that it is a good idea to finally apply the suggested change.
>>
>>
>>
>> There would be some API change required of course. As a basic concept it
>> would probably be sensible to add this as a parameter to the
>> PhonMaterial-class.
>>
>> Alternatively it might make sense to add a “Texture” class that is used a
>> a DiffuseMap in the PhonMaterial. But that seems slightly overkill.
>>
>>
>>
>> As I am not very well acquainted with current design principles of this
>> library, these things should lie in more experience contributors hands.
>>
>>
>>
>> *From: *Nir Lisker <nlisker at gmail.com>
>> *Sent: *Tuesday, 20 June 2023 20:50
>> *To: *Matija Brown <Matija.Brown at outlook.de>
>> *Cc: *openjfx-dev at openjdk.org
>> *Subject: *Re: [JavaFX 3D ( | Feature Request)] Setting Texture
>> Nearest-Sampling on PhongMaterial
>>
>>
>>
>> Which leads to the question; Does there, in JavaFX exist something
>> comparable to setting the texture-sampler to NEAREST
>>
>> instead of LINEAR sampling?
>>
>>
>>
>> There is no API to set the texture filter. If you would like to
>> contribute and add it, I can help. It needs to be compatible with both
>> Direct3D and OpenGL (not sure how the work on Metal is going), so this can
>> be a challenge.
>>
>>
>>
>> For the D3D side, see this issue in JBS [1]. The relevant code is at [2].
>> I didn't look at the OpenGL side.
>>
>>
>>
>> - Nir
>>
>>
>>
>> [1] https://bugs.openjdk.org/browse/JDK-8092272
>>
>> [2]
>> https://github.com/openjdk/jfx/blob/0d9dcf38275528e1b621d71631aac5bdb9452110/modules/javafx.graphics/src/main/native-prism-d3d/D3DContext.cc#L621
>>
>>
>>
>> On Mon, Jun 19, 2023 at 10:15 PM Matija Brown <Matija.Brown at outlook.de>
>> wrote:
>>
>> On my never ending journey of building a Minecraft-clone in every
>> graphics-framework available,
>>
>> I have come across JavaFX for the next attempt.
>>
>>
>>
>>
>>
>> However a minor inconvenience has crossed my path in the process:
>>
>>
>>
>> Using the (very well developed!) 2D-Graphics displaying pixel-art style
>> images is no trouble whatsoever.
>>
>> Simply rendering it to a canvas and disabling smoothing does the job just
>> fine. Unfortunately, I have been
>>
>> unable to figure out how to achieve a similar thing using the 3D-Graphics
>> engine and the PhongMaterial that comes with it.
>>
>>
>>
>> Which leads to the question; Does there, in JavaFX exist something
>> comparable to setting the texture-sampler to NEAREST
>>
>> instead of LINEAR sampling?
>>
>> Unfortunately the latest information I could find online was from about
>> 2013 and much has (probably)
>>
>> changed since then. Thus the question is being posed once again.
>>
>>
>>
>>
>>
>> I whish to excuse myself for probably repeating a fairly common question,
>>
>>
>>
>> Kind regards,
>> Matija Brown.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20230826/4ba0a98b/attachment-0001.htm>