[Feature Proposal] Vertex Colors on TriangleMesh

[Knee Snap]

In addition to my last email, here's the public API proposal.

*Mesh.java:*
buildGeometry(...) now accepts a vertex color array.

*NGTriangleMesh.java:*
+ void setUserDefinedColors(boolean)
+ boolean isUserDefinedColors()
+ void syncColors(IntegerArraySyncer)

*TriangleMesh.java:*
+ ObservableFloatArray getColors()
*NOTE*
Originally I wanted an ARGB8888 ObservableIntegerArray since that's the
standard color format and I didn't want to waste memory.
Upon further thought this doesn't make sense. In order to put a color into
the buffer it must be converted to floating point numbers anyways, since
"float[4]" is the recommended data type to represent colors in shaders.
Then I also considered how 0-255 is actually a bad way to represent the
range 0.0 to 2.0, since the max value of 255 is odd and cannot be divided
by two. For my personal situation this is fine, but I think in terms of a
general solution for everyone else a floating point number is more flexible.
Finally, when I started considering how others might use it, I realized
floating points had the additional advantage of not being capped at 2.0, so
someone could put in a vertex color modulation value of 3.0, where-as in
ARGB8888 they could not.
So I've concluded that ObservableFloatArray with four elements (R, G, B, A)
is preferable to ObservableIntegerArray with a single ARGB8888 element.

*VertexFormat.java:*
+ static final VertexFormat POINT_TEXCOORD_COLOR
+ static final VertexFormat POINT_NORMAL_TEXCOORD_COLOR
+ int getColorElementSize()
+ int getColorIndexOffset()

Any thoughts on this? My next goal is to create a draft PR with this
implemented, but I'm still looking for feedback.

On Tue, Aug 20, 2024 at 8:41 PM Knee Snap <kneester77 at gmail.com> wrote:

> Thanks for the reply!
>
> *> If you can give links to the API and docs of vertex coloring in D3D9,
> OpenGL, Metal and possibly Vulkan and D3D11 for future-proofing, we could
> see what APIs they support exactly and come up with a common one for
> JavaFX. Sometimes different pipelines support the same features in
> different ways, which makes it harder to provide a single API abstraction.
> D3D9, OpenGL, Metal and possibly Vulkan and D3D11*
> I've laid out a very basic explanation of the API changes I imagine in the
> first email, but I'll follow up in a bit with a more in-depth proposal once
> the aforementioned Kevin or Johan give the OK on the proposal to continue
> to the API discussion stage detailed in the contribution guide.
>
>
> Fixed Rendering Pipeline Documentation Examples (PLEASE READ BELOW FOR
> MORE CONTEXT FIRST):
>  - Enable/Disable Per-Vertex Coloring
> <https://learn.microsoft.com/en-us/windows/win32/direct3d9/per-vertex-color-state>
>  (
> https://learn.microsoft.com/en-us/windows/win32/direct3d9/per-vertex-color-state
> )
>  - Shows D3DFVF_DIFFUSE as the flag to use to enable vertex colors in the
> vertex format.
> <https://learn.microsoft.com/en-us/windows/win32/direct3d9/d3dfvf> (
> https://learn.microsoft.com/en-us/windows/win32/direct3d9/d3dfvf)
>
> Providing documentation is a little hard to do, and I think I need to
> explain/give a little more context , since JavaFX doesn't appear to use any
> fixed rendering pipelines.
> When I said "DirectX / OpenGL supports vertex colors", what I'm referring
> to is two things: the fixed rendering pipeline which explicitly supports
> vertex colors, and the shader-based pipeline capable of supporting
> vertex colors.
> The fixed rendering pipeline is the one where there would be a specific
> library function call relating to vertex colors. I've included some links
> above as examples.
> However, fixed rendering pipelines aren't often used these days because
> they're less flexible because shaders are actually akin to small embedded
> programs that run in parallel, which the developer can write themselves.
> Since JavaFX doesn't use fixed rendering pipelines and it wants to
> anticipate the future, I don't see any reason to focus on the fixed
> rendering pipeline, which makes it more difficult to give documentation on
> how to do this.
>
> I think I can still answer the spirit of the question though.
> The reason there's no explicit API call when using shaders is because
> these APIs aren't actually managing the color for us directly anymore.
> Instead, JavaFX is responsible for creating/ordering/organizing
> arrays/buffers itself, and giving the GPU a shader program configuring how
> to render the contents of the buffer.
> So in order to make vertex colors work under this paradigm, it's one of
> the most simple operations out there, a simple multiplication. We'll put
> the color in the buffer, then when drawing a vertex, the GPU shader will
> take the color out of the buffer and perform a multiplication operation.
> That's all it takes to make vertex colors work with the shader paradigm,
> so it's very hard to imagine a future potential shader-based system which
> cannot do this, as it's hard to imagine a shader which doesn't both have
> even a single one of these three things:
>  1) Access to arbitrary buffer data per-vertex supplied by the CPU
> (otherwise how could it render anything?)
>  2) Multiplication support (How could it do anything useful without this?)
>  3) Ability to control the output color on a per-pixel basis,
> using interpolated data from the vertiex buffer(s) used for the given
> polygon. (This is the entire purpose of the pixel/fragment shader so it's
> unfathomable for this to not exist, there would be almost no point to
> shaders if this didn't exist.)
>
> These are all super critical to shaders in general being useful, so it's
> very difficult to imagine a scenario where JavaFX would be looking to
> support a shader-based backend that didn't have even a single one of those
> things.
> I think this should hopefully answer that question, even if I didn't do
> exactly the way you wanted.
>
> But wait, if I'm not actually making major changes that involve any
> Direct3D / OpenGL function calls, what would even get changed for this
> feature?
> Firstly, the shader(s) themselves. We've got to make it so the shaders
> properly describe that they should receive vertex color data in a buffer,
> and multiply that vertex color data as described previously.
> (modules\src\main\resources\prism.es2.glsl\ and
> modules\src\main\resources\native-prism-d3d\hlsl)
> Secondly, JavaFX's buffer management code needs to be changed to include
> vertex color data in the buffers, when applicable. (BaseMesh.java?)
> Lastly, all of the code sitting between the previous steps and the public
> user-facing API additions will need to be filled in. I think this is mostly
> just the bridge between TriangleMesh.java and NGTriangleMesh.java.
>
> *> 1. Are there other closely related APIs that can come with this
> feature? This is required to be able to plan ahead. For example, if an API
> provides setting one of the modes A, B or C, and a user proposes an API
> that only deals with A, you'd end up with a single method that sets or
> unsets A. However, if you then want to evolve the API to support B and C,
> you would have liked to model it in a different way that allows you set any
> of the modes, and not end up with a single method for each. So, if vertex
> color is part of a larger API context, we will need to plan accordingly.*
> I've thought of a few features which could be related, but I don't think
> vertex colors impact these features in a way which adds difficulties which
> aren't already present:
>  - The ability to use custom shaders supplied by the user alongside
> arbitrary buffers, textures, and globals. (This is probably never coming,
> but if it does, good to plan ahead of time)
>  - Deferred rendering to allow efficient implementation of more than 3
> lights at a time. (Could even be thousands of lights)
>  - Allowing the user to choose 3D scene axis orientations instead of
> always defaulting to Y- being "up". (Perhaps this can be done just by
> messing with the camera without a JavaFX feature, not sure.)
>  - Allowing the user to choose between clockwise/counter-clockwise winding
> order.
>  - Allowing the user to choose whether mip-maps are enabled.
>  - Allowing the user to choose the texture wrapping mode for a
> PhongMaterial.
>
> If anyone can think of any other features which it could conflict with, or
> ways it could complicate the features I've mentioned, do share the concerns.
>
>
> *> 2. The performance of 3D is already problematic in some aspects. Small
> additions can drop the FPS considerably. More research should be done to
> find out why. We will need to make sure performance isn't going to drop for
> cases where this feature isn't used. Seeing as this is only vertex  shader
> data, which tends to be cheap and the number of registers available for it
> large, I don't foresee problems off the top of my head.*
> The biggest thing for performance is going to be how the underlying
> implementations work for each backend. If done using best practices, this
> should have no meaningful performance impact. After all, this technique has
> been used since before 3D hardware acceleration, so it should be very
> performance friendly.
> As for when the feature is disabled, this feature should be possible to
> completely disable when not in use, ensuring it would have zero performance
> impact. There's already code I see to allow compiling different versions of
> shaders, we could compile a version without the vertex colors enabled. I'm
> not sure if this would even be necessary, but the option is there.
>
> *> Implementing this will require simultaneous implementations for D3D9
> and OpenGL. Metal is still in the works, so I'm not sure about that one.*
> I see, is there someone/a group working on the Metal backend? If so, it'd
> be nice to hear if they have any thoughts on this.
>
>
> On Tue, Aug 20, 2024 at 1:08 PM Nir Lisker <nlisker at gmail.com> wrote:
>
>> When modern APIs agree on a feature, it makes sense to consider it for
>> JavaFX (that lacks a lot of standard features). If you can give links to
>> the API and docs of vertex coloring in D3D9, OpenGL, Metal and possibly
>> Vulkan and D3D11 for future-proofing, we could see what APIs they support
>> exactly and come up with a common one for JavaFX. Sometimes different
>> pipelines support the same features in different ways, which makes it
>> harder to provide a single API abstraction.
>>
>> There are a couple of points I would look at when designing this feature:
>> 1. Are there other closely related APIs that can come with this feature?
>> This is required to be able to plan ahead. For example, if an API provides
>> setting one of the modes A, B or C, and a user proposes an API that only
>> deals with A, you'd end up with a single method that sets or unsets A.
>> However, if you then want to evolve the API to support B and C, you would
>> have liked to model it in a different way that allows you set any of the
>> modes, and not end up with a single method for each. So, if vertex color is
>> part of a larger API context, we will need to plan accordingly.
>> 2. The performance of 3D is already problematic in some aspects. Small
>> additions can drop the FPS considerably. More research should be done to
>> find out why. We will need to make sure performance isn't going to drop for
>> cases where this feature isn't used. Seeing as this is only vertex  shader
>> data, which tends to be cheap and the number of registers available for it
>> large, I don't foresee problems off the top of my head.
>>
>> Implementing this will require simultaneous implementations for D3D9 and
>> OpenGL. Metal is still in the works, so I'm not sure about that one.
>>
>> It will need the approval of Kevin or Johan.
>>
>> - Nir
>>
>> On Tue, Aug 20, 2024 at 9:08 AM Knee Snap <kneester77 at gmail.com> wrote:
>>
>>> This is my first time attempting to submit a feature request so bear
>>> with me if I mess up the instructions, I'll do my best to rectify any
>>> issues if they occur and are pointed out.
>>>
>>> *Feature Proposal:* Vertex Colors on TriangleMesh
>>> Add a new ObservableIntegerArray to TriangleMesh which contains vertex
>>> colors in ARGB8888 integer format. (I'm open to the pixel format being
>>> flexible too, but this may be unnecessary).
>>> Operation of this array should be the same as the other mesh arrays, in
>>> the sense that the VertexFormat object configures how they should operate.
>>> Similarly to how normals currently work, the array is optional, and may
>>> not necessarily need to be included.
>>> Additionally, the face array will also work the same as usual, where the
>>> new color indices will be specified on a per-face-vertex basis. (Each
>>> vertex of each face will have an index specified)
>>> If the array is present/valid, the colors should be multiplied by 2x the
>>> diffuse/selfIllumination/specular textures when calculating the FragColor.
>>> (Clamp the color to the maximum color value of 1.0 when done)
>>> The reason we multiply by two is to allow the vertex colors to create
>>> color components larger than what was seen in the texture. So for example,
>>> a blue component of .75 will result in 1.5x the blue value from the
>>> material texture.
>>> This is the standard way of calculating vertex colors when there is a
>>> texture as I understand. If there were no texture, it would not be
>>> multiplied by 2.0
>>> The rasterization process should be responsible for interpolation
>>> colors, as seen in my examples below.
>>>
>>> Example without feature:
>>> https://i.imgur.com/Hg81LtU.png
>>>
>>>
>>> Example with feature:
>>> Notice how the lava stream & pipes are the same texture repeated many
>>> times, but once the vertex coloring feature is used it becomes
>>> significantly less apparent.
>>> It can also be used to add details like the rust on various
>>> metal surfaces, or how the lava fades to black towards the edges. JavaFX
>>> doesn't contain the appropriate tools to render this scene, but pretty much
>>> any other 3D library does.
>>>
>>>
>>>
>>>
>>> *Why adding this API to the core of JavaFX is a positive/useful addition
>>> for different applications.*Vertex coloring is a core 3D rendering
>>> feature supported by virtually every other 3D rendering system on the
>>> planet. It works on low-end systems, and has been part of 3D rendering
>>> since the beginning.
>>> This is true whether the 3D rendering library is high-level (SGI
>>> OpenInventor, Java3D, jmonkeyengine, etc), low-level (DirectX, Metal,
>>> Vulkan, OpenGL), or somewhere in-between (Unreal Engine, Unity, Blender,
>>> etc).
>>> The only library I could even find which did not support this was JavaFX.
>>>
>>> But, just because everyone else does it doesn't mean JavaFX should,
>>> right?
>>> Well, I'd actually like to argue that it should. If JavaFX wants to be
>>> reasonably compatible/interoperable with virtually any other 3D format,
>>> application, or tool, it needs to support this.
>>> Any tool with the capability of importing/exporting 3D models has only
>>> one for displaying vertex coloring right now: to generate their own texture
>>> sheet with vertex color shading baked in on a per-face basis.
>>> This can be a pretty slow process when there are a lot of different
>>> faces, since every single face with even a single polygon with a slightly
>>> different texture coordinate will need a new texture on the texture sheet.
>>> The above screenshot I've shown requires a 4096x4096 texture sheet in
>>> order to render in JavaFX, and it's a pretty small scene! I've got
>>> significantly larger scenes which I'm completely unable to render with
>>> shading because I need to generate a texture sheet more than a gigabyte
>>> large in order to shade it properly.
>>> And forget about any kind of common texCoord/UV animations. The CPU
>>> needs to be constantly redrawing shading textures onto the texture sheet in
>>> order to ensure animations have the correct shading.
>>> For the above example, trying to animate the lava to flow down the
>>> screen as it's supposed to brings my FPS below 30.
>>> This is despite the texture drawing process having run in parallel and
>>> extensive performance profiling on my part. I've concluded there's no way
>>> to get around having proper vertex color support as the main bottleneck is
>>> just sending such a massively large image to the GPU so frequently as well
>>> as having to actually shade the images still taking a pretty decent amount
>>> of time even when done in parallel. And remember, these performance issues
>>> are happening on one of my small scenes.
>>>
>>> Normally drawing vertex colors like this is extremely fast (even on 30
>>> year old hardware!) when done by the GPU/dedicated hardware, but JavaFX
>>> just doesn't provide a way to do it with the GPU.
>>> JavaFX is also the only mainstream 3D library I could find which lacks
>>> this feature.
>>> There are 3D environments I cannot even display with shading right now
>>> at all due to the sheer amount of large textures that need to have shading
>>> manually drawn onto them.
>>> This cannot be solved by using a project like Fxyz
>>> <https://github.com/FXyz/FXyz>, since it just comes down to the texture
>>> sheets being too large.
>>> These scenes aren't even particularly large either, having been made for
>>> game consoles in the range of 20-30 years ago.
>>>
>>> This will happen to anyone who needs to use vertex coloring with a
>>> significant number of textures, and requires writing a lot of code to
>>> manage texture sheets to ultimately just for it to only partially suffice.
>>>
>>> There are other reasons to want vertex coloring outside of compatibility
>>> with other software too.
>>> Vertex coloring can often be used to add significant mesh details.
>>> In the above example, I've already demonstrated how this can be used to
>>> add details such as rust, fade-outs, etc. But there are many more ways to
>>> include it too, such as using art styles with flat textures, such as for
>>> clouds, cartoon art-styles, etc. It's even possible to create shadows by
>>> using darker colors in areas obscured by others. There are so many clever
>>> ways this can be used, and I'd spend too much time on individual examples.
>>> The reason vertex colors have been so prevalent is specifically because of
>>> the payoff between how easy they are to add and how much can be achieved
>>> with them.
>>> I'm not enough of an artist to explain everything with words, but I can
>>> certainly give a lot of examples via screenshot, if desired.
>>>
>>> Since the guideline was to tailor this pitch to the JavaFX community, I
>>> thought I'd include mention of a few JavaFX projects that I thought would
>>> benefit significantly. There were others that would benefit in less
>>> significant ways, so I tried to keep it brief:
>>>  - Fxyz <https://github.com/FXyz/FXyz>: Probably the most popular
>>> JavaFX 3D project.
>>>  - GameExtractor <https://github.com/wattostudios/GameExtractor/>:
>>> Capable of showing 3D objects in many common formats (.obj, .3ds, Unity
>>> Format, Unreal Engine 2/3 Format, etc), but is unable to show vertex colors
>>> due to JavaFX's lack of support.
>>> -  FrogLord <https://github.com/Kneesnap/FrogLord>: This is my project,
>>> but every single game I support desperately needs this feature in order to
>>> accurately preview the custom levels as they are edited, so the user can
>>> see how they will appear in their game.
>>>
>>> Pretty much any project which creates texture sheets containing
>>> interpolated colors will benefit from this change as in many cases they
>>> won't need to do that anymore.
>>> Due to the reduced time and memory usage it will also become
>>> significantly more feasible/performant to use meshes which change over time
>>> as well, since fewer changes to the texture sheet will be required, if any
>>> at all.
>>>
>>> I also imagine this fitting in very easily with the existing API, as
>>> pretty much all of the public facing APIs could just have a few new methods
>>> in them, without needing to change any pre-existing APIs at all.
>>>
>>> In summary,
>>> Vertex colors are a feature as ubiquitous in the 3D world as texture
>>> coordinates, lighting, and materials, yet JavaFX doesn't have them.
>>> Adding them will allow for a significant amount of detail for virtually
>>> no performance or maintenance cost, and for some projects it will even
>>> improve performance & simplify texture sheets.
>>>
>>> Additional Note:
>>> https://bugs.openjdk.org/browse/JDK-6586559
>>> It looks like this feature may have already been planned at one point,
>>> but was forgotten due to being low-priority? I don't know.
>>>
>>> What does everyone think about this proposal? I eagerly await responses!
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20240820/71708727/attachment-0001.htm>