[Feature Proposal]: TriangleMesh - Vertex Color Support

[Nir Lisker]

Finally getting to this. To add to what Michael said about the development
process, there are many large ongoing efforts in various stages (interest
gathering, proposals, discussions, PRs...) that require a lot of investment
from the maintainers. From the top of my head: various CSS changes +
reimplementation idea, Rich Text control, Behavior/Input/Skin
decomposition, Focus traversal policies, listeners reimplementation and a
few others. People work in parallel on whichever of these topics they want.
Usually there is a quickish response (a few days tops) that amounts to
"makes sense" or "doesn't make sense", and then the  discussion and
research starts. Colored Vertices is definitely a "makes sense", but *how*
to do the API for a backwards-compatible library is not trivial.

I'll start by answering your questions and then give some thoughts on the
API and the larger 3D picture.

### Q&A

*1) Eg: I either want an explanation of the workflow for Metal. How I will
> be including vertex colors (a feature which hasn't officially been merged
> yet!!) in Metal, or will I wait until vertex colors are officially merged
> before adding support to Metal? I'm fine with either, I just want the
> process laid out for me.*


The details about Metal will have to wait until Kevin or one of the
engineers that work on it will give the answers. This feature is not
hindered by Metal not being done, it just means that when Metal is
sufficiently done, it will need to implement the additional API here.


> *2) Where can I get contact info for those who are working on Metal
> support? Looking for a lead or someone prominent.*


Here, but it can take some time, especially with very long discussions like
this one.


> 3) Can I get some clarification on expectations for contribution?


The feature, from *my* assessment, gained enough public support to show
that it's relevant (I, you, Michael, Risto and the original JBS issue
author). I don't think Kevin objects to the idea either. In your proposal
you also say "I can point out various JavaFX projects which... explicitly
said they'd like vertex color support", so while I don't know who they are,
it could strengthen your case if required.

The proposal and draft PR are in a good state. Now you need to wait to get
feedback, which is what I have been looking at on and off since the
weekend. The API is the critical part here and I suspect this is where
contention will come from (myself included, see below).
Don't worry about JBS tickets right now. It's part of the formal process
and they can be changed at any time when needed. You will also need a CSR
(a JBS issue for detailing the API changes), which you should wait with
anyway until we know what the API is.

* -> 3a) How long does it generally take to get responses on low priority
> feature proposals? I have no reference for what to temper my expectations
> against, which would help me a lot. Should I expect it to take a week
> before Kevin weighs in? Two weeks? One month? 6 months? One year? I've seen
> projects with response times covering the whole spectrum, and I just don't
> know where to set my expectations for when it is appropriate to follow-up
> on an email which hasn't gotten a reply. *


Depends on the current load, which is currently high (see my introduction
:) ). Give it another week.


> * -> 3b) Every single other recent active PR besides mine has had a review
> comment from at least one other person, including ones created after mine.
> How should I interpret this besides me having done something wrong? At what
> point should I ask for a follow-up and for people to review it if I don't
> hear back from anyone?*


Probably because these PRs have had their discussion finished already, or
are not new features that require discussions. Regular maintenance like bug
fixes is faster. There is no FIFO/LIFO queue.

*4) How long should I expect before my OCA signature is reviewed? I
> submitted it around 2 weeks ago at this point, and I don't think it's been
> reviewed yet, which is significantly longer than the website stated it
> would take.*


This is not up to OpenJFX. Usually it takes up to 2 weeks. In any case, it
needs to be completed by the time of PR integration, so it's far from a
blocker.

*5) **Why does the Direct3D 3D pipeline appear to use fixed vertex buffer
> definitions in the D3D calls, as opposed to making a more modern vertex
> buffer definition?*


I don't know. This was done over 15 years ago. Possibly because it's easier
(the pipeline does the work for you). If changing to the dynamic way offers
enough benefits and someone wants to take that work on, I would assume they
could.


> *6) Why was your PR for #1281 not labelled as a draft PR or WIP? If I
> follow the contribution guidelines, I think it should be labelled as a
> draft? Hoping for some clarification.*


Because it was ready to be reviewed. It does not have any new API and it
was mostly alignment of D3D with ES2.

*7) Do you mind sending out a reply to the original thread so I am able to
> reply to it instead of violating the contribution guidelines? Thanks!*


OK


> *8) Where am I currently in the contribution guide? What is the next
> steps? What is expected from me?*


See 3.

*9) What is some reasonable criteria for me to use.*


See my first paragraph. There are no strict timings. I would say about a
week with no response, or after someone says "I'll get to it on ___" but
doesn't, is reasonable.


### Feedback on the API

I'm not yet convinced that new vertex formats are the way to go. The reason
is the telescopic expansion we will face if we want to introduce another
per-vertex data. I don't know what addition it could be because I haven't
gone over all the source material I want to, but it's enough that there's 1
more "foo" feature and now the vertex format grows to 8 variants(!). And
the complexity of yes/no normals, yes/no colors, yes/no foo in terms of how
the arrays are treated is going to be too large. Unless I'm convinced that
there will be no more such additions, I would think about another way of
doing this. I found this 3D library (that I've not heard of before) that
has many vertex format combinations [0].

Subclassing falls into a reciprocal problem. Suppose we define a
ColoredTriangleMesh where the current vertex formats act the same way, but
the color data is required (otherwise don't use this subclass). Now if we
have a "foo" vertex property, what do we subclass? If we subclass
TriangleMesh, it means that we can't have both a Colored and a Foo'd
TriangleMesh. If we subclass ColoredTriangleMesh, we force all
FooTriangleMesh to also be colored. And then we can ask why not invert the
inheritance. So instead of having too much data in one class, we have too
little.

Sometimes these types of problems are solved with a builder pattern, though
I don't know if there is a suitable parallel here. As for aligning with the
current class design, I'm not sure that it's a good design to begin with.
Having data (normals and face smoothing groups) that is only relevant
depending on another parameter (vertex format) isn't known to be good
design nowadays. This dual mode class reminds me of Scanner [1]. I wouldn't
be surprised if the designers of the class thought that this is fine as
long as the triangle mesh doesn't get more attributes later, and we might
be repeating sins of the past.

I'll need to play around with some local prototypes to see if I can come up
with something more appetizing.


### On the topic of 3D in JavaFX

As someone who has done a lot of work in this area, I would be one of the
first proponents of developing it further. As Risto Vaher said in this
thread, it could be a huge boon for JavaFX if modernized. I don't mean that
it should be a competitor to a bonafide gaming engine with immersive
graphics (usually requiring volumetric lighting or ray tracing), but there
are many other genres that aren't *that* graphically demanding that JavaFX
could certainly be used for with proper updates. Scientific applications
also have a decent need for 3D visualizations that JavaFX could serve,
these tend to require more in the way of performance than fanciness; think
many-particle systems, anatomical modeling etc.

Oracle has been investing in creating a Metal pipeline seeing as OpenGL is
being deprecated in MacOS. It's a large investment. I've laid out
foundations in #1281 for common missing texture features like
sampling/filtering, wrapping, and mipmaps. It also needs to be done
carefully, though #1281 is all internal so it can afford some misses. There
are other common missing features, like stenciling techniques such as shape
outlining (usually used for highlight on hover) and basic shadows; other
primitives such as points and lines [2] that are used in particle effects;
HDR lighting and so on. If contributors want to take these on, I think that
would help bring 3D to a more prominent state. Discussion on this topic
should be done separately.

- Nir

[0] https://cesium.com/learn/cesiumjs/ref-doc/VertexFormat.html
[1]
https://stuartmarks.wordpress.com/2020/04/14/scanner-is-a-weird-but-useful-beast/
[2] https://bugs.openjdk.org/browse/JDK-8316398

On Fri, Sep 13, 2024 at 11:40 AM Knee Snap <kneester77 at gmail.com> wrote:

> Gottttcha, thank you Michael, that really helps a lot.
>
> I read the contribution guide and I definitely took away the wrong thing.
> I expected a very bearaucratic process, so I've largely interpreted the
> conversations this far with that mindset. It makes a lot more sense if I
> consider the previous conversations as a democratic process.
>
> The 3D system being niche therefore getting less attention makes sense. I
> hope some of these changes can make it become less niche since I think a
> handful of missing features will make a very large impact on its
> usability/how appealing it is to different projects. Unfortunately that's
> pretty hard to measure, but I hope that can be made up for by showing how
> the rest of the 3D world support these features without fail. That won't
> make up for a lack of overwhelming enthusiasm, but I'm hopeful that because
> my planned proposals all have very mature specifications for how they work
> across the broader 3D world with consistent operations across pretty much
> all applications (except JavaFX) that it counts for something. Perhaps I
> should start following other conversations more closely to get a better
> feel on how these discussions tend to go.
>
> The bottom line is that your message has helped me reconceive how I should
> be thinking about this and I feel a lot more confident in my ability to be
> a helpful contributor from that.
>
> Thank you (and everyone else) for being so understanding, contributing to
> an open source project so mature/stable/widely used is a new experience for
> me I must say, so I really do appreciate helping me acclimate like this.
> It's nice to not need to accompdate business pressures/goals and just make
> decisions based on what provides the most utility to yourself and others.
>
> On Thu, Sep 12, 2024, 9:23 PM Michael Strauß <michaelstrau2 at gmail.com>
> wrote:
>
>> Just a comment about the culture of the OpenJFX project, which you
>> seem to be struggling with:
>>
>> From my experience, this project is usually quite democratic in its
>> processes. The features and enhancements that successfully make it
>> into JFX start with a discussion on the mailing list. This is where
>> you gather buy-in from other developers (if no one is interested in
>> your proposed enhancement, you're in a tough spot). Don't worry too
>> much about formal processes, it's about community support at this
>> stage. Get other people to be excited about your enhancement, and
>> hopefully some will find the time to look at the design and your
>> implementation. Some proposals build up inertia and move forward,
>> others end up dying due to a lack of interest.
>>
>> 3D graphics in JavaFX is a bit of a niche feature (and vertex colors
>> is even more niche), which is probably why the response to your
>> proposal is a bit slow. That being said, I'll take a closer look at
>> your proposal during the weekend.
>>
>>
>>
>> On Fri, Sep 13, 2024 at 4:59 AM Knee Snap <kneester77 at gmail.com> wrote:
>> >
>> > Thanks Nir! No worries about being busy, my concern was when I didn't
>> get replies to my questions about what I should be doing and procedural
>> questions. But, it seems maybe I wasn't super clear about my questions
>> since you aren't sure what they were.
>> > I also went through many days waiting for responses about expectations
>> all while he was replying quickly to other emails, seemingly ignoring mine.
>> I'm not suggesting he's done anything wrong, I can imagine perfectly good
>> reasons for not having replied, but my questions were related to the
>> contribution guidelines and expectations. So I've felt like I've been going
>> foward without clear expectations of both myself and contributors, and like
>> I wasn't even guaranteed a response at all. The only reference I had for
>> how long I should expect to wait was based on the other emails I saw, which
>> had fast replies. I'm hoping to clarify my responsibilities and
>> expectations, and what I should expect from contributors, since there's a
>> lot of stuff to get familiar with in the contribution guide, and it's my
>> first time contributing. Due to that, it felt like those questions went
>> mostly ignored and it wasn't a matter of being busy, as other emails were
>> getting quick replies, though perhaps I didn't phrase them clearly. Hoping
>> to now clear up my confusions about responsibilities. Your email has helped
>> reassure me that I was not getting ignored, but instead just had a lower
>> priority or something else that could cause it to take longer to get a
>> reply.
>> >
>> > Overall, I'd like to give some insight into what it has felt like for
>> me to contribute to JavaFX for my first time. The contribution guide is
>> good, but it hasn't been enough.
>> > This isn't here to point blame or to criticize, but instead give some
>> perspective into my experience here and why this has been such a
>> frustrating experience. Perhaps this will help improve the contribution
>> experience for newcomers like me going forward keeping this in mind, but
>> ideally it'll help set the stage for why I am so confused and help you guys
>> resolve my confusions.
>> >  -> The contribution guide misled me in many ways here, and there are
>> many questions I still do not have an answer to.
>> >  -> To start off, it said to include a feature proposal, and it gave
>> suggestions for various things to include.
>> >  -> So, I wrote a long feature proposal email and sent it off.  For a
>> week I had a discussion with a contributor where we hashed out different
>> design choices, I felt like we had appropriately discussed everything which
>> had been discussed adequately, and was ready to get my feature proposal
>> approved by Kevin.
>> >  -> Everything was going great until one week in when Kevin replied and
>> said he "would need to be convinced that multiple applications would
>> benefit from such a feature, and that your proposed solution -- as
>> documented and exposed by the public API -- is the best way to go."
>> >  -> He also asked if I had read the contribution guide, implying he
>> didn't think what I had previously sent wasn't me 1:1 following the
>> contribution guide.
>> >  -> I still don't understand why he sent that, I believe I presented
>> exactly what the contribution guide wanted. There were a handful of things
>> that got missed, but each were discussed in the email thread.
>> >  -> Did he not read the proposal and assume I did it wrong? Did he read
>> it and think it wasn't convincing? Did he want a new proposal which
>> included the things which had been discussed in the email thread?
>> >  -> So, with this confusion I asked for clarification as well as other
>> questions relating to Metal like who was working on it, if I could have a
>> contact for them, etc.
>> >  -> Never got a reply! Okay, so what the heck am I supposed to do here?
>> Kevin asked for new a feature proposal, and I guess for some inexplicable
>> reason this one wasn't good enough? It doesn't seem like he even realized
>> this was a feature proposal, but either way, he asked for a new one, so
>> I'll make a new one.
>> >  -> After looking at other feature proposals, I retooled my proposal
>> and come up with something I was happy with, which I felt went above/beyond
>> what was necessary. I've even prototyped an implementation which works that
>> I can use to get a screenshot for the proposal, and to ensure my opinions
>> of the design didn't change even after seeing it in practice.
>> >  -> During this time Kevin even helped me sort out an issue building
>> JavaFX, so here I am thinking that I've been given the green light to
>> present a final feature proposal with all of my changes, and Kevin will
>> review it.
>> >  -> So, after finalizing a new proposal, I'm feeling good. I've
>> expanded the proposal significantly, and it's now in a Git repository,
>> which is what other contributors have been doing.
>> >  -> I'd like to give everyone a chance to review the updated proposal,
>> who knows maybe there could be something I missed. Then, I'll pass it off
>> to Kevin.
>> >  -> There's some discussion and Kevin even chimes in at one point.
>> After a few days when there's no more discussion, I send out an email
>> saying I'm reading to pass it off to Kevin for review.
>> >  -> Then, I don't get a reply. After a little bit, I start to get very
>> concerned since I see him actively replying to other emails, and he's still
>> never touched the original email I sent asking him for clarification, and
>> follow-up questions.
>> >  -> At this point it's been almost 3 weeks since his confusing email,
>> and I still have no clarification, and I've spent quite a lot of effort on
>> this. The questions from before have come back. Did he even read my initial
>> feature proposal? What if all this time he had just not read the original
>> one? I've got 3 weeks since an email which is still waiting for a reply
>> from him and I've just not heard anything. Not even an indication he plans
>> to reply but is busy. I have no idea what's normal/what I should expect,
>> but I can't even ask for clarification on that.
>> >
>> > So, at this point I've lost track. I have no idea where I am in the
>> contribution process, none of my questions have been answered, and the only
>> outcome I see happening right now is that this proposal is lost & forgotten.
>> > I don't know what else on the guide I can do, nor does it feel like
>> Kevin is bothering to answer important questions, let alone review it.
>> > I completely understand the slowness, it's an open source project after
>> all, no less a very mature one which tries to avoid all changes to public
>> APIs. My frustration has mainly been that I believe I've been asking
>> questions that take little effort to reply to, but are getting ignored
>> while I see him actively replying to other threads. Perhaps my emails are
>> too long and contain so much detail that the questions don't immediately
>> pop out, and thus they get missed. But at this point it feels more like I'm
>> getting ignored for an unknown reason.
>> > If I had some sort of confirmation that it's normal for low-priority
>> emails to sit a bit, but that Kevin WILL eventually get to it, this would
>> go a LONG way.
>> >
>> > But from this I've learned that I need to be extremely explicit with my
>> questions. So here are the questions I'd like answered (Some of these I
>> hadn't asked earlier, others have gotten lost):
>> > 1) Eg: I either want an explanation of the workflow for Metal. How I
>> will be including vertex colors (a feature which hasn't officially been
>> merged yet!!) in Metal, or will I wait until vertex colors are officially
>> merged before adding support to Metal? I'm fine with either, I just want
>> the process laid out for me.
>> >  -> I know Metal support is getting done by Oracle, but I think you
>> guys can still provide input to this question. (I will also be asking them,
>> if I can get contact info for them)
>> >  -> The Metal Pipeline doesn't strike me as being very close to being
>> finished, so it feels like vertex colors can be merged before them. If that
>> is the case, I'd like to have vertex colors merged without Metal support,
>> then at that point I can add vertex color support to Metal, as this is the
>> clearest path that I see to avoiding conflicts between 3 different
>> repositories.
>> >  -> I see this as a very conservative approach though, and it's my
>> proposal because I'm not familiar with JavaFX contribution workflow yet.
>> I'd like to hear feedback from others on how they suggest I handle this.
>> > 2) Where can I get contact info for those who are working on Metal
>> support? Looking for a lead or someone prominent.
>> >  -> I mentioned several questions regarding Metal support, but I think
>> the best people to answer them will be the people actually working on the
>> Metal support. So a contact should be enough.
>> > 3) Can I get some clarification on expectations for contribution?
>> >  -> From the contribution guide, my understanding is that the current
>> thing I'm waiting on is for design approval (on the feature proposal) from
>> Kevin.
>> >  -> But, I think I'm OK to seek out feedback on my PR as well, which
>> I've recently realized already has a JBS Bug ID (or I think that's what it
>> is? It's got an entry on at least one of the multiple web portals I've seen
>> which track issues.)
>> >  -> 3a) How long does it generally take to get responses on low
>> priority feature proposals? I have no reference for what to temper my
>> expectations against, which would help me a lot. Should I expect it to take
>> a week before Kevin weighs in? Two weeks? One month? 6 months? One year?
>> I've seen projects with response times covering the whole spectrum, and I
>> just don't know where to set my expectations for when it is appropriate to
>> follow-up on an email which hasn't gotten a reply.
>> >  -> 3b) Every single other recent active PR besides mine has had a
>> review comment from at least one other person, including ones created after
>> mine. How should I interpret this besides me having done something wrong?
>> At what point should I ask for a follow-up and for people to review it if I
>> don't hear back from anyone?
>> > 4) How long should I expect before my OCA signature is reviewed? I
>> submitted it around 2 weeks ago at this point, and I don't think it's been
>> reviewed yet, which is significantly longer than the website stated it
>> would take.
>> >  -> We're also beyond the timeframe that the JFX repository said it
>> would take.
>> >  -> Also, nobody has commented on my PR to run the expected commands
>> whereas other PRs uploaded later have ~25 comments. Why is this? Is it due
>> to low priority? Is it due to the 3D rendering system having fewer people
>> working on it?
>> > 5) Why does the Direct3D 3D pipeline appear to use fixed vertex buffer
>> definitions in the D3D calls, as opposed to making a more modern vertex
>> buffer definition?
>> >  -> The 2D pipeline uses the more modern vertex buffer definitions, but
>> the 3D pipeline uses an FVF definition, which means many non-texCoord
>> values are marked as texCoords in the shaders. (I don't know if that's
>> necessary, but it seems weird to me).
>> >  -> My best guess for this is there was some kind of compatibility
>> consideration, but I haven't been able to find any information on it. Does
>> anyone (maybe you) know why it's done as an FVF declaration instead of a
>> vertex declaration?
>> > 6) Why was your PR for #1281 not labelled as a draft PR or WIP? If I
>> follow the contribution guidelines, I think it should be labelled as a
>> draft? Hoping for some clarification.
>> > 7) Do you mind sending out a reply to the original thread so I am able
>> to reply to it instead of violating the contribution guidelines? Thanks!
>> >  -> What I was hoping for PR #1281 was that since I joined the mailing
>> list recently, while I can read the previous emails in the archive, I have
>> no way to reply to pre-existing threads, as they're not in my inbox. I'd
>> have to start a new email thread which the contributing guidelines say
>> explicitly not to do.
>> > 8) Where am I currently in the contribution guide? What is the next
>> steps? What is expected from me?
>> > 9) What is some reasonable criteria for me to use.
>> >  -> 9a) If I send out a proposal for design discussion, how long should
>> I wait from the last email before sending it to Kevin
>> >  -> 9b) If I send out a feature proposal to Kevin for approval, how
>> long should I wait before following up with Kevin if I don't hear back?
>> >  -> 9c) If I send out a PR/Draft PR, how long after receiving no
>> feedback should I wait before following up?
>> >
>> > Thanks again! I really appreciate JavaFX contributors and I want to
>> make it clear that I'm not taking any of this for granted. I just am very
>> confused and feel like the documentation has left me in a situation where
>> I'm just utterly confused and don't know what's going on, and haven't been
>> getting my questions answered.
>> > In retrospect, I definitely didn't make these questions anywhere near
>> clear enough, so I can understand why they were missed. So ideally this
>> email can serve as both a way for me to clear up my confusion, but also to
>> improve the experience for future contributors who can be spun around by
>> having a mismatch between their expectations from the contribution guide
>> and the demonstrated behavior of contributors up to the project lead Kevin.
>> I want to be fully clear again that I'm not trying to argue or start a war,
>> just neutrally explain my perspective, and express the desire to work with
>> everyone to get everything sorted.
>> > I look forward to your feedback when you have a chance to review the PR.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20240919/4913bea4/attachment-0001.htm>