[Feature Proposal] Vertex Colors on TriangleMesh

[Knee Snap]

This is my first time attempting to submit a feature request so bear with
me if I mess up the instructions, I'll do my best to rectify any issues if
they occur and are pointed out.

*Feature Proposal:* Vertex Colors on TriangleMesh
Add a new ObservableIntegerArray to TriangleMesh which contains vertex
colors in ARGB8888 integer format. (I'm open to the pixel format being
flexible too, but this may be unnecessary).
Operation of this array should be the same as the other mesh arrays, in the
sense that the VertexFormat object configures how they should operate.
Similarly to how normals currently work, the array is optional, and may not
necessarily need to be included.
Additionally, the face array will also work the same as usual, where the
new color indices will be specified on a per-face-vertex basis. (Each
vertex of each face will have an index specified)
If the array is present/valid, the colors should be multiplied by 2x the
diffuse/selfIllumination/specular textures when calculating the FragColor.
(Clamp the color to the maximum color value of 1.0 when done)
The reason we multiply by two is to allow the vertex colors to create color
components larger than what was seen in the texture. So for example, a blue
component of .75 will result in 1.5x the blue value from the material
texture.
This is the standard way of calculating vertex colors when there is a
texture as I understand. If there were no texture, it would not be
multiplied by 2.0
The rasterization process should be responsible for interpolation colors,
as seen in my examples below.

Example without feature:
https://i.imgur.com/Hg81LtU.png


Example with feature:
Notice how the lava stream & pipes are the same texture repeated many
times, but once the vertex coloring feature is used it becomes
significantly less apparent.
It can also be used to add details like the rust on various
metal surfaces, or how the lava fades to black towards the edges. JavaFX
doesn't contain the appropriate tools to render this scene, but pretty much
any other 3D library does.




*Why adding this API to the core of JavaFX is a positive/useful addition
for different applications.*Vertex coloring is a core 3D rendering feature
supported by virtually every other 3D rendering system on the planet. It
works on low-end systems, and has been part of 3D rendering since the
beginning.
This is true whether the 3D rendering library is high-level (SGI
OpenInventor, Java3D, jmonkeyengine, etc), low-level (DirectX, Metal,
Vulkan, OpenGL), or somewhere in-between (Unreal Engine, Unity, Blender,
etc).
The only library I could even find which did not support this was JavaFX.

But, just because everyone else does it doesn't mean JavaFX should, right?
Well, I'd actually like to argue that it should. If JavaFX wants to be
reasonably compatible/interoperable with virtually any other 3D format,
application, or tool, it needs to support this.
Any tool with the capability of importing/exporting 3D models has only one
for displaying vertex coloring right now: to generate their own texture
sheet with vertex color shading baked in on a per-face basis.
This can be a pretty slow process when there are a lot of different faces,
since every single face with even a single polygon with a slightly
different texture coordinate will need a new texture on the texture sheet.
The above screenshot I've shown requires a 4096x4096 texture sheet in order
to render in JavaFX, and it's a pretty small scene! I've got significantly
larger scenes which I'm completely unable to render with shading because I
need to generate a texture sheet more than a gigabyte large in order to
shade it properly.
And forget about any kind of common texCoord/UV animations. The CPU needs
to be constantly redrawing shading textures onto the texture sheet in order
to ensure animations have the correct shading.
For the above example, trying to animate the lava to flow down the screen
as it's supposed to brings my FPS below 30.
This is despite the texture drawing process having run in parallel and
extensive performance profiling on my part. I've concluded there's no way
to get around having proper vertex color support as the main bottleneck is
just sending such a massively large image to the GPU so frequently as well
as having to actually shade the images still taking a pretty decent amount
of time even when done in parallel. And remember, these performance issues
are happening on one of my small scenes.

Normally drawing vertex colors like this is extremely fast (even on 30 year
old hardware!) when done by the GPU/dedicated hardware, but JavaFX just
doesn't provide a way to do it with the GPU.
JavaFX is also the only mainstream 3D library I could find which lacks this
feature.
There are 3D environments I cannot even display with shading right now at
all due to the sheer amount of large textures that need to have shading
manually drawn onto them.
This cannot be solved by using a project like Fxyz
<https://github.com/FXyz/FXyz>, since it just comes down to the texture
sheets being too large.
These scenes aren't even particularly large either, having been made for
game consoles in the range of 20-30 years ago.

This will happen to anyone who needs to use vertex coloring with a
significant number of textures, and requires writing a lot of code to
manage texture sheets to ultimately just for it to only partially suffice.

There are other reasons to want vertex coloring outside of compatibility
with other software too.
Vertex coloring can often be used to add significant mesh details.
In the above example, I've already demonstrated how this can be used to add
details such as rust, fade-outs, etc. But there are many more ways to
include it too, such as using art styles with flat textures, such as for
clouds, cartoon art-styles, etc. It's even possible to create shadows by
using darker colors in areas obscured by others. There are so many clever
ways this can be used, and I'd spend too much time on individual examples.
The reason vertex colors have been so prevalent is specifically because of
the payoff between how easy they are to add and how much can be achieved
with them.
I'm not enough of an artist to explain everything with words, but I can
certainly give a lot of examples via screenshot, if desired.

Since the guideline was to tailor this pitch to the JavaFX community, I
thought I'd include mention of a few JavaFX projects that I thought would
benefit significantly. There were others that would benefit in less
significant ways, so I tried to keep it brief:
 - Fxyz <https://github.com/FXyz/FXyz>: Probably the most popular JavaFX 3D
project.
 - GameExtractor <https://github.com/wattostudios/GameExtractor/>: Capable
of showing 3D objects in many common formats (.obj, .3ds, Unity Format,
Unreal Engine 2/3 Format, etc), but is unable to show vertex colors due to
JavaFX's lack of support.
-  FrogLord <https://github.com/Kneesnap/FrogLord>: This is my project, but
every single game I support desperately needs this feature in order to
accurately preview the custom levels as they are edited, so the user can
see how they will appear in their game.

Pretty much any project which creates texture sheets containing
interpolated colors will benefit from this change as in many cases they
won't need to do that anymore.
Due to the reduced time and memory usage it will also become significantly
more feasible/performant to use meshes which change over time as well,
since fewer changes to the texture sheet will be required, if any at all.

I also imagine this fitting in very easily with the existing API, as pretty
much all of the public facing APIs could just have a few new methods in
them, without needing to change any pre-existing APIs at all.

In summary,
Vertex colors are a feature as ubiquitous in the 3D world as texture
coordinates, lighting, and materials, yet JavaFX doesn't have them.
Adding them will allow for a significant amount of detail for virtually no
performance or maintenance cost, and for some projects it will even improve
performance & simplify texture sheets.

Additional Note:
https://bugs.openjdk.org/browse/JDK-6586559
It looks like this feature may have already been planned at one point, but
was forgotten due to being low-priority? I don't know.

What does everyone think about this proposal? I eagerly await responses!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20240819/921091b8/attachment-0001.htm>