[Feature Proposal] Vertex Colors on TriangleMesh

Tue Aug 20 20:08:13 UTC 2024

When modern APIs agree on a feature, it makes sense to consider it for
JavaFX (that lacks a lot of standard features). If you can give links to
the API and docs of vertex coloring in D3D9, OpenGL, Metal and possibly
Vulkan and D3D11 for future-proofing, we could see what APIs they support
exactly and come up with a common one for JavaFX. Sometimes different
pipelines support the same features in different ways, which makes it
harder to provide a single API abstraction.

There are a couple of points I would look at when designing this feature:
1. Are there other closely related APIs that can come with this feature?
This is required to be able to plan ahead. For example, if an API provides
setting one of the modes A, B or C, and a user proposes an API that only
deals with A, you'd end up with a single method that sets or unsets A.
However, if you then want to evolve the API to support B and C, you would
have liked to model it in a different way that allows you set any of the
modes, and not end up with a single method for each. So, if vertex color is
part of a larger API context, we will need to plan accordingly.
2. The performance of 3D is already problematic in some aspects. Small
additions can drop the FPS considerably. More research should be done to
find out why. We will need to make sure performance isn't going to drop for
cases where this feature isn't used. Seeing as this is only vertex  shader
data, which tends to be cheap and the number of registers available for it
large, I don't foresee problems off the top of my head.

Implementing this will require simultaneous implementations for D3D9 and
OpenGL. Metal is still in the works, so I'm not sure about that one.

It will need the approval of Kevin or Johan.

- Nir

On Tue, Aug 20, 2024 at 9:08 AM Knee Snap <kneester77 at gmail.com> wrote:

> This is my first time attempting to submit a feature request so bear with
> me if I mess up the instructions, I'll do my best to rectify any issues if
> they occur and are pointed out.
>
> *Feature Proposal:* Vertex Colors on TriangleMesh
> Add a new ObservableIntegerArray to TriangleMesh which contains vertex
> colors in ARGB8888 integer format. (I'm open to the pixel format being
> flexible too, but this may be unnecessary).
> Operation of this array should be the same as the other mesh arrays, in
> the sense that the VertexFormat object configures how they should operate.
> Similarly to how normals currently work, the array is optional, and may
> not necessarily need to be included.
> Additionally, the face array will also work the same as usual, where the
> new color indices will be specified on a per-face-vertex basis. (Each
> vertex of each face will have an index specified)
> If the array is present/valid, the colors should be multiplied by 2x the
> diffuse/selfIllumination/specular textures when calculating the FragColor.
> (Clamp the color to the maximum color value of 1.0 when done)
> The reason we multiply by two is to allow the vertex colors to create
> color components larger than what was seen in the texture. So for example,
> a blue component of .75 will result in 1.5x the blue value from the
> material texture.
> This is the standard way of calculating vertex colors when there is a
> texture as I understand. If there were no texture, it would not be
> multiplied by 2.0
> The rasterization process should be responsible for interpolation colors,
> as seen in my examples below.
>
> Example without feature:
> https://i.imgur.com/Hg81LtU.png
>
>
> Example with feature:
> Notice how the lava stream & pipes are the same texture repeated many
> times, but once the vertex coloring feature is used it becomes
> significantly less apparent.
> It can also be used to add details like the rust on various
> metal surfaces, or how the lava fades to black towards the edges. JavaFX
> doesn't contain the appropriate tools to render this scene, but pretty much
> any other 3D library does.
>
>
>
>
> *Why adding this API to the core of JavaFX is a positive/useful addition
> for different applications.*Vertex coloring is a core 3D rendering
> feature supported by virtually every other 3D rendering system on the
> planet. It works on low-end systems, and has been part of 3D rendering
> since the beginning.
> This is true whether the 3D rendering library is high-level (SGI
> OpenInventor, Java3D, jmonkeyengine, etc), low-level (DirectX, Metal,
> Vulkan, OpenGL), or somewhere in-between (Unreal Engine, Unity, Blender,
> etc).
> The only library I could even find which did not support this was JavaFX.
>
> But, just because everyone else does it doesn't mean JavaFX should, right?
> Well, I'd actually like to argue that it should. If JavaFX wants to be
> reasonably compatible/interoperable with virtually any other 3D format,
> application, or tool, it needs to support this.
> Any tool with the capability of importing/exporting 3D models has only one
> for displaying vertex coloring right now: to generate their own texture
> sheet with vertex color shading baked in on a per-face basis.
> This can be a pretty slow process when there are a lot of different faces,
> since every single face with even a single polygon with a slightly
> different texture coordinate will need a new texture on the texture sheet.
> The above screenshot I've shown requires a 4096x4096 texture sheet in
> order to render in JavaFX, and it's a pretty small scene! I've got
> significantly larger scenes which I'm completely unable to render with
> shading because I need to generate a texture sheet more than a gigabyte
> large in order to shade it properly.
> And forget about any kind of common texCoord/UV animations. The CPU needs
> to be constantly redrawing shading textures onto the texture sheet in order
> to ensure animations have the correct shading.
> For the above example, trying to animate the lava to flow down the screen
> as it's supposed to brings my FPS below 30.
> This is despite the texture drawing process having run in parallel and
> extensive performance profiling on my part. I've concluded there's no way
> to get around having proper vertex color support as the main bottleneck is
> just sending such a massively large image to the GPU so frequently as well
> as having to actually shade the images still taking a pretty decent amount
> of time even when done in parallel. And remember, these performance issues
> are happening on one of my small scenes.
>
> Normally drawing vertex colors like this is extremely fast (even on 30
> year old hardware!) when done by the GPU/dedicated hardware, but JavaFX
> just doesn't provide a way to do it with the GPU.
> JavaFX is also the only mainstream 3D library I could find which lacks
> this feature.
> There are 3D environments I cannot even display with shading right now at
> all due to the sheer amount of large textures that need to have shading
> manually drawn onto them.
> This cannot be solved by using a project like Fxyz
> <https://github.com/FXyz/FXyz>, since it just comes down to the texture
> sheets being too large.
> These scenes aren't even particularly large either, having been made for
> game consoles in the range of 20-30 years ago.
>
> This will happen to anyone who needs to use vertex coloring with a
> significant number of textures, and requires writing a lot of code to
> manage texture sheets to ultimately just for it to only partially suffice.
>
> There are other reasons to want vertex coloring outside of compatibility
> with other software too.
> Vertex coloring can often be used to add significant mesh details.
> In the above example, I've already demonstrated how this can be used to
> add details such as rust, fade-outs, etc. But there are many more ways to
> include it too, such as using art styles with flat textures, such as for
> clouds, cartoon art-styles, etc. It's even possible to create shadows by
> using darker colors in areas obscured by others. There are so many clever
> ways this can be used, and I'd spend too much time on individual examples.
> The reason vertex colors have been so prevalent is specifically because of
> the payoff between how easy they are to add and how much can be achieved
> with them.
> I'm not enough of an artist to explain everything with words, but I can
> certainly give a lot of examples via screenshot, if desired.
>
> Since the guideline was to tailor this pitch to the JavaFX community, I
> thought I'd include mention of a few JavaFX projects that I thought would
> benefit significantly. There were others that would benefit in less
> significant ways, so I tried to keep it brief:
>  - Fxyz <https://github.com/FXyz/FXyz>: Probably the most popular JavaFX
> 3D project.
>  - GameExtractor <https://github.com/wattostudios/GameExtractor/>:
> Capable of showing 3D objects in many common formats (.obj, .3ds, Unity
> Format, Unreal Engine 2/3 Format, etc), but is unable to show vertex colors
> due to JavaFX's lack of support.
> -  FrogLord <https://github.com/Kneesnap/FrogLord>: This is my project,
> but every single game I support desperately needs this feature in order to
> accurately preview the custom levels as they are edited, so the user can
> see how they will appear in their game.
>
> Pretty much any project which creates texture sheets containing
> interpolated colors will benefit from this change as in many cases they
> won't need to do that anymore.
> Due to the reduced time and memory usage it will also become significantly
> more feasible/performant to use meshes which change over time as well,
> since fewer changes to the texture sheet will be required, if any at all.
>
> I also imagine this fitting in very easily with the existing API, as
> pretty much all of the public facing APIs could just have a few new methods
> in them, without needing to change any pre-existing APIs at all.
>
> In summary,
> Vertex colors are a feature as ubiquitous in the 3D world as texture
> coordinates, lighting, and materials, yet JavaFX doesn't have them.
> Adding them will allow for a significant amount of detail for virtually no
> performance or maintenance cost, and for some projects it will even improve
> performance & simplify texture sheets.
>
> Additional Note:
> https://bugs.openjdk.org/browse/JDK-6586559
> It looks like this feature may have already been planned at one point, but
> was forgotten due to being low-priority? I don't know.
>
> What does everyone think about this proposal? I eagerly await responses!
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20240820/c0ea3be1/attachment-0001.htm>