[External] : Re: Proof of concept pull request for Behavior API (PR 1265)

[Andy Goryachev]

Thank you, John.

-andy

From: John Hendrikx <john.hendrikx at gmail.com>
Date: Tuesday, October 31, 2023 at 16:13
To: Andy Goryachev <andy.goryachev at oracle.com>, openjfx-dev at openjdk.org <openjfx-dev at openjdk.org>, Kevin Rushforth <kevin.rushforth at oracle.com>
Subject: Re: [External] : Re: Proof of concept pull request for Behavior API (PR 1265)

Hi Andy,

I'm sorry if you feel I blocked your proposal.  Discussions better happen at a stage far before a PR is being reviewed, and instead of having these discussions here, we could have had them on the PR.
I'll try and formalize my proposals.

--John


On 30/10/2023 20:32, Andy Goryachev wrote:
Dear John:

  I'm unsure what the JEP format would contribute here

The benefit of JEP format is
a) to explicitly state the problem being solved (in the Motivation section)
b) to enumerate the public APIs


I have several proposals, and some are too big for a single proposal (IMHO) which would you like me to focus on?

The main reason I am asking you to go through the process is because you effectively blocked by InputMap proposal.  I keep hearing that my proposal is not good for various reasons, while there is a much better way for doing things, so let’s see it in full detail.  It would be nice to have proof-of-concept code based on a complex control rather than a simple Button, but it is not a requirement, at least initially.

I still want to see code examples (pseudo code is fine) for my questions:

Q1. Changing an existing key binding from one key combination to another.
Q2. Remapping an existing key binding to a different function.
Q3. Unmapping an existing key binding.
Q4. Adding a new key binding mapped to a new function.
Q5. (Q1...Q4) scenarios, at run time.
Q6. How the set behavior handles a change from the default skin to a custom skin with some visual elements that expects input removed, and some added.
Q7. Once the key binding has been modified, is it possible to invoke the default functionality?
Q8. How are the platform-specific key bindings created?
Q9. How are the skin-specific (see Q6) handlers removed when changing the skins?
Q10. When a key press happens, does it cause a linear search through listeners or just a map lookup?

I think it is important to answer all questions during the discussion, as it helps all sides to understand how things work, and possibly make corrections.  Since the bulk of my proposal deals with key bindings and user-/skin- installed handlers, I would like to see how you propose to deal with these problems.

Another reason I asked you for a JEP is that you seem to brush aside my objections.  For example, my objection to the stateless behavior was dealt with by inventing BehaviorContext, which I basically take as an acknowledgement that behaviors are not stateless.  So let’s see exactly you envision things by describing the public API and perhaps the proof of concept code should also have two non-trivial controls, just to see whether BehaviorContext depends on the control class or not.

How key bindings are done is IMHO more of an implementation detail of **specific** behaviors,

I categorically disagree with this statement.  As an application developer, I want to be able to set/modify/unmap key bindings via common public mechanism, make sure that the user mappings and handlers always take precedence over the skin ones, and make sure that skin changes leave the user mappings and handlers in place.

It is certainly not an implementation detail and not a property of any specific behavior.

Thank you, and I am looking forward to seeing answers to the questions posted earlier.
-andy





From: John Hendrikx <john.hendrikx at gmail.com><mailto:john.hendrikx at gmail.com>
Date: Saturday, October 28, 2023 at 12:55
To: Andy Goryachev <andy.goryachev at oracle.com><mailto:andy.goryachev at oracle.com>, openjfx-dev at openjdk.org<mailto:openjfx-dev at openjdk.org> <openjfx-dev at openjdk.org><mailto:openjfx-dev at openjdk.org>, Kevin Rushforth <kevin.rushforth at oracle.com><mailto:kevin.rushforth at oracle.com>
Subject: Re: [External] : Re: Proof of concept pull request for Behavior API (PR 1265)

Hi Andy,
On 27/10/2023 19:10, Andy Goryachev wrote:


Dear John:

I think our goal is for all of us to agree on a solution which solves the problems.  We are still talking, right?

I have to say - it is very difficult to have a meaningful conversation when questions are not being answered.  It is even more difficult to do over email and time zones, as the context can often be lost.

I'm really a bit surprised, as I think I responded quite quickly to a lot of the posts surrounding the proposals, and also answered quite a lot of questions.  The mailinglist format seems to have served Java quite well for this purpose for years now, and if memory serves, earlier FX proposals also were discussed here.  I'm unsure what the JEP format would contribute here, given that it does not allow for inline comments or threads, but I'm not unwilling to try my hand at one.




To reiterate, a proposal in a JEP format would be nice, so we can see the public API.

Alright, this will take a bit of time.  I have several proposals, and some are too big for a single proposal (IMHO) which would you like me to focus on?

1. A public Behavior API with the initial focus on being able to reuse and replace default behaviors

This proposal would like to achieve a clear definition of a Behavior and clear separation (to aid in reusability and creation).  It would define a Behavior interface, and a clean way of installing/uninstalling a behavior on controls.  This would be primarily the Behavior/BehaviorContext part of my sample PR https://github.com/openjdk/jfx/pull/1265<https://urldefense.com/v3/__https:/github.com/openjdk/jfx/pull/1265__;!!ACWV5N9M2RV99hQ!LvWBWlmjB-JYcKnpZTbvjYwd1CItXTmycfx2D-BYmkbS7lxG6f4kwt2bmokpBwh63Gg_MNX5JzP747J1wgPuPY6Rq1H1$>, leaving the more controversial event definition parts out for now

1.1. A possible extension of the above Behavior API to allow changing high level behavior

I'm not sure yet what would be a good approach here.  I would probably either lean towards reusing the Event system for this, or doing this with overridable methods; ie. in order to override a function, listen for its event and call different code or trigger a different event; or, ensure there is an overridable method so it can be overriden directly in code.

1.2. An extension to the above behavior API to allow for more user friendly key rebinding

The idea here would be to create a custom behavior (allowed by 1.), call into a behavior you want to modify to install its defaults, and then make changes after.  There is more than one possible approach here.  One I raised earlier was offering more specific methods on BehaviorContext.  Another possiblity is to make it specific to certain behaviors only (MappableBehavior), or behaviors that can somehow provide an InputMap (without entangling everything).

2. An improved event handling system (Michael Strauss already did some work there) that would allow users to override or disable default event processing

This would be a low-level improvement that would allow 3rd parties to override large parts of JavaFX in a supported manner.  It would open the way to a 3rd party behavior system or navigation system, but also simpler things like just changing a key mapping, even ones claimed by behaviors currently.  As it is a low level API, this would be somewhat cumbersome for seemingly simple tasks, and the various platforms would need to be supported manually.

In essence the above proposal would allow user installed event handlers to receive any event before a default handler can get to it, so that say remapping the LEFT_ARROW key is actualy possible and doesn't magically disappear (many new users, including me at the time were/are somewhat surprised that an event handler installed on the control is not receiving all events).  The root cause of this is the sharing of the event handling lists on the control with (magically) installed behaviors.  The above proposal would change this (in a backwards compatible way) to work more like how default exception handlers work -- only unconsumed events that bubble up to the root level are considered for default behavior actions.

------

As you can see, one my problems with answering the key rebinding questions is that IMHO this more of a later extension on a Behavior API; this means to get to the key remapping design, there first would need to be a Behavior API design.  How key bindings are done is IMHO more of an implementation detail of **specific** behaviors, as there are probably more ways to do this.  So for my proposals, a somewhat fleshed out Behavior API design is an important prerequisite before offering key rebinding.

If we can advance this design far enough, we may see a way to do this without needing a Behavior API first; for example, we could have a Behavior interface, and a subtype, MappableBehavior; only behaviors of that sub type support key remapping, while general behaviors don't have to.  This would make the key rebinding just something that **some** behaviors support, and not a general feature if it instead was added to say Control, Behavior or BehaviorContext.  This may be a short cut that we could take to get to key rebinding quickly, without closing off a future behavior API.

Again, thanks for reading, I look forward to some feedback, and as said, I will try put some time towards writing a JEP.  I'm also happy to collaborate on this once a design direction becomes  a bit more clear.

--John




Please, if you have time, answer these questions.  A short pseudo-code example will be fine.

Q1. Changing an existing key binding from one key combination to another.
Q2. Remapping an existing key binding to a different function.
Q3. Unmapping an existing key binding.
Q4. Adding a new key binding mapped to a new function.
Q5. (Q1...Q4) scenarios, at run time.
Q6. How the set behavior handles a change from the default skin to a custom skin with some visual elements that expects input removed, and some added.
Q7. Once the key binding has been modified, is it possible to invoke the default functionality?
Q8. How are the platform-specific key bindings created?
Q9. How are the skin-specific (see Q6) handlers removed when changing the skins?
Q10. When a key press happens, does it cause a linear search through listeners or just a map lookup?

Lastly, I do think prototyping the alternative proposal using simple control like Button is insufficient.  TextArea would be much better, as it has a ton of key bindings, platform-specific logic, various handlers that do and do not consume events by default.

Thank you
-andy

P.S. I noticed that I switched my PR to Open by mistake.  Sorry, it’s back in Draft.




From: John Hendrikx <john.hendrikx at gmail.com><mailto:john.hendrikx at gmail.com>
Date: Thursday, October 26, 2023 at 02:15
To: Andy Goryachev <andy.goryachev at oracle.com><mailto:andy.goryachev at oracle.com>, openjfx-dev at openjdk.org<mailto:openjfx-dev at openjdk.org> <openjfx-dev at openjdk.org><mailto:openjfx-dev at openjdk.org>, Kevin Rushforth <kevin.rushforth at oracle.com><mailto:kevin.rushforth at oracle.com>
Subject: Re: [External] : Re: Proof of concept pull request for Behavior API (PR 1265)

The normal procedure I think is also to first provide a JEP for review, before starting on the implementation...

Given the doubts raised, feedback given and potential alternatives proposed, I don't see why you are still moving forward with your own proposal. The critiques I've given have been mostly hand waved with arguments that have no place in JEP evaluation (time restrictions, existing code already works this way, false equivalency with MVC pattern), and therefore have IMHO not been taken serious at all.

This leaves me in the position of putting in a lot of work that will essentially be ignored as I feel an (internal) decision has already been reached, regardless of the feedback on the mailinglist.

The (partial) proposal I've made, and also simpler proposals so that 3rd parties could do a keybinding implementation, should be sufficient to reconsider the current proposal that is being moved forward.

I'll reiterate my problems with your proposal:

- Introduces a lot of API for what is essentially the configuration of internal event handlers
- The proposed API partially overlaps with the existing event handler API, meaning that some keys could be changed with just event handlers, while some can only be changed with the BaseBehavior API; it also provides for creating new functions and assigning them to keys, essentially a new (very limited) API for what was already possible in the much more flexible event handling API
- Introduces the term "Behavior" in public API without clearly specifying what that is, nor showing enough forethought that may make it possible in the future to have public Behaviors
- Introduces the term "InputMap" in public API, which is just an implementation detail of the internal event handlers
- Doesn't address the real issue IMHO, which is that JavaFX Skins/Behaviors install their Event Handlers directly on Controls, mixing them with user event handlers leading to all sorts of unpredictable behavior due to call order and internal handlers essentially stealing and consuming events before the user has a chance to look at them (and thus blocking any 3rd party key alterations) which leads to the (false) need to change key bindings and Behaviors directly...

So if you want me to work on such a proposal, fully fleshing it out, I would like to know if it will be given consideration. I would also like some more feedback on what is already there, as I think it is sufficient to decide if a full proposal is worth it or not.

My proposals in short:

1.

- Fix the issues with Events being stolen before users can get a them
    - Users should be able to have priority on Events, Michael Strauss already has a PR that fixes the issue in part
    - Events should not be consumed when not used (navigation does this) as this precludes the user being able to change their meaning
    - Even better would be if internal event handlers were isolated and did not mix themselves with user event handlers at all

The above can be done separately, and should already make it possible to do a lot of things that were close to impossible before when it comes to changing key handling, but certainly not everything.

- Building on top of the improved event handling system, introduce a flag to indicate an event is not to be consumed by internal event handlers

These two together can form the basis for a 3rd party Behavior implementation as standard behavior can be prevented from occurring.  It leaves platform dependent behavior to be addressed by such a 3rd party / user implementation as it is a very low level API.  Any key remapping logic would be provided by the 3rd party API.

2.

I also have a more fleshed out alternative proposal that attempts to introduce Behaviors into JavaFX as a first class concept, instead of a potential 3rd party add-on.  Recap:

- Introduce a Behavior interface with a single method "install" to be called by a Control
- The "install" method is provided a context object, BehaviorContext.  This indirects any changes the Behavior can make to a Control, so the Control is fully aware of all changes and can uninstall them without further co-operation from the behavior.
- The BehaviorContext provides low level functions to add/remove event handlers and listeners, but can also provide higher level functions (in perhap a later PR) to allow for some kind of control provided input map system
- Standard Behaviors can be made public and can be easily subclassed or composed as they need not have any state.  State is tracked inside the behavorial installed listeners and handlers themselves (either directly or by referring to some shared State object).
- Clear separation of concerns; Behaviors, a resuable concept that can be applied to a control; BehaviorContext, manages behavior lifecycle by abstracting away direct Control access; behavior state management left up to the implementation and created (on demand and as needed) when "install" is called.
- Indirection from key mapping to semantic meaning is provided by introducing control specific events. These semantic events can be handled, filtered and consumed like all other events, allowing for changing/remapping/blocking or ignoring; this part can be left out from an initial implementation to further evaluate how such events might interact with Skins that need specific events (there is nothing stopping us from having some of these semantic events be handled by the Control directly, and some by the specific needs of the Skin)

To get at the internal key mappings, you'd need to subclass or compose a Behavior.  The Behaviors are setup to allow this easily.  To modify the bindings of a Control, one would install such a modified Behavior as a whole; overkill perhaps for one binding change, but convenient when multiple bindings are changed, and reusable accross controls (the Behavior only need to be created once).

The proposal also includes an indirection between Key/Mouse event and its semantic meaning.  This is achieved by firing higher level more meaningful events, but that's not the only option; it could also be done with overridable methods on the Behavior, or a behavior specific interface if the Event based proposal is seen as too audacious.

This proposal advocates a clear seperation of the Behavior from the Skin, essentially making them Controller and View, where the View has no knowledge of the Controller. I see no reason why this wouldn't be possible, given that it is a standard pattern.  That existing controls may be difficult to untangle is IMHO irrelevant, especially when this can be done one at a time.  I realize that Controllers (Behaviors) may have functions that are sort of View (Skin) specific; this is not an issue, as it should be fine to trigger a behavior without it being consumed; unconsumed behaviorial events just bubble up.  This allows Behaviors to have events specific to a Skin without them interfering if they're unused by an alternative Skin.  An alternative Skin that also needs new behavior will also need to create a new behavior to go along with it (or when paired with the standard one, accept that those new behaviors won't be triggered).

Thanks for reading.

--John




On 26/10/2023 00:59, Andy Goryachev wrote:
Dear John:

It is difficult to review the alternative proposal for a number of reasons.  A prototype is a good start, but for any proposal to go forward we need a bit more work.  Let me enumerate the steps that we expect:

1. Provide an overview of the proposal following a JEP outline:

Summary
Goals
Non-Goals
Motivation
Description
Alternatives
Risks and Assumptions
Dependencies

2. A draft PR that provides a proof of concept, using, in this case, a few complex controls like TextArea, TableView, ComboBox.

3. Address the question raised earlier, perhaps by providing code examples (pseudo code is acceptable, I think).

More specifically, I’d like to know how the following concerns will be addressed by the new proposal:

Q1. Changing an existing key binding from one key combination to another.
Q2. Remapping an existing key binding to a different function.
Q3. Unmapping an existing key binding.
Q4. Adding a new key binding mapped to a new function.
Q5. (Q1...Q4) scenarios, at run time.
Q6. How the set behavior handles a change from the default skin to a custom skin with some visual elements that expects input removed, and some added.
Q7. Once the key binding has been modified, is it possible to invoke the default functionality?
Q8. How are the platform-specific key bindings created?
Q9. How are the skin-specific (see Q6) handlers removed when changing the skins?
Q10. When a key press happens, does it cause a linear search or a map lookup?

Thank you
-andy




From: John Hendrikx <john.hendrikx at gmail.com><mailto:john.hendrikx at gmail.com>
Date: Tuesday, October 24, 2023 at 04:58
To: Andy Goryachev <andy.goryachev at oracle.com><mailto:andy.goryachev at oracle.com>, openjfx-dev at openjdk.org<mailto:openjfx-dev at openjdk.org> <openjfx-dev at openjdk.org><mailto:openjfx-dev at openjdk.org>
Subject: Re: [External] : Re: Proof of concept pull request for Behavior API (PR 1265)


On 23/10/2023 23:57, Andy Goryachev wrote:

You'd create a new class, `MyBehavior`,

By “customizing” I also mean at run time.  Creating new classes wouldn’t work.

This would also work at runtime, as the class you create can be instantiated with parameters that control its key binding behavior.  Even though the standard Behaviors should probably be singletons (so they can be reused and composed) or have public well documented constructors, a custom behavior created by the user has no such re-usability restrictions.

coupling

I don’t think it is our choice - it is up to the skin designed.  If they add a node that needs to take input, or if the behavior is drastically different, it is almost impossible to create a common interface.  So skin and behaviors are coupled, besides we have to design for the worst case (of a totally different skin).  The division between S and B comes mostly from the division between V and C in MVC.  From a distance, the user does not see it at all - all they see is a control.

JavaFX is not doing MVC.
In MVC, the 3 components are not entangled; Model refers View, Controller refers View and Model, View refers nothing; in JavaFX the View (Skin) creates the Controller (Behavior); the View especially normally can be created without any dependencies, and can be tested as such; with Skins being tightly coupled to both Behaviors and Controls, that doesn't even come close.

For it to be MVC you'd need to:

- Remove reference from Skin to Control
- Do not let Skins create Behaviors
- Instantation order should be, create a Skin first (with no Control reference), then create the Control (with Skin as parameter or setter), then create a Behavior (with Control as parameter, and one or more Views (Skins))

What JavaFX is exactly,  I don't know. It doesn't follow MVC (even though it claims to) because in the current setup the Skin is both V and C; that's not MVC.  At most it is MS (Model Skin), and so there is no reason to expose anything beyond the Skin then, as that would just be pretending to be something that it is not.

This suggest another metric at judging the usefulness of a design - how easy it is to understand and perform 80% of most common tasks.
Now that I explained how key remappings would work, I don't see how this would disqualify the alternative proposal.





There are more interesting ideas at the end of the message I am replying to - fxml, css, global changes - these go far beyond the simple input map improvement.  I did mention this already, but neither open source community, nor my employer might have the resources to make such drastic changes.
I didn't mention FXML, but yes, I gave some other things to think about.  As for how drastic any of those are, that remains to be seen.  Certainly the global changes would not be that hard at all.  The CSS proposal would need some research if there is some will to go there; it assumes that the information needed can be transported in a reasonable manner to the key binding system using the existing CSS infrastructure.





So we have to be realistic, I think.  We are travelling to a different planet in a small spaceship and we only have so much material and oxygen to play with.  A simple improvement that helps 80% of use cases might be better than a major redesign (I still think the event proposal involves major redesign).

I think that if that's the case that we'd better focus on making it possible for 3rd parties to deliver these features, and do the simplest thing that would allow them to do so. That would be prioritized event handlers (so a 3rd party can always intercept before the Skin/Behavior gets to it) + a flag to skip system event handlers (ala consumed) to allow bubbling up.

On top of that any key remapping or behavior change system can be constructed already.

--John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20231101/66796e6d/attachment-0001.htm>