From 0863159262d5846518b310bbab30bf30212a2531 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Sat, 6 Nov 2021 13:48:41 +0100 Subject: [PATCH 01/33] Add ui navigation RFC --- rfcs/ui-navigation.md | 394 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 394 insertions(+) create mode 100644 rfcs/ui-navigation.md diff --git a/rfcs/ui-navigation.md b/rfcs/ui-navigation.md new file mode 100644 index 00000000..0d15bd32 --- /dev/null +++ b/rfcs/ui-navigation.md @@ -0,0 +1,394 @@ +# Feature Name: Generic UI Navigation API + +## Summary + +A set of components and a new system to define navigation nodes for the ECS UI. + +## Motivation + +On top of making it easy to implement controller navigation of +in-game UIs, it enables keyboard navigations and answers accessibility concerns +such as the ones +[raised in this issue](https://github.com/bevyengine/bevy/issues/254). + +## User-facing explanation + +We add a _navigation tree_ to the UI system, the tree will be used to enable +key-based navigation of the specified UIs. + +Currently, the developer must themselves write the relationship between +various buttons in the menus and specify which button press leads to other menu +items, etc. Or more simply, limit themselves to pointer-based navigation to +avoid having to think about menu navigation. + +By including navigation-related components such as `Navigable` and `Container`, +the developer can quickly get a controller-navigable menu up and running. The +`Container` accepts various `Plane`s to fine-tune more precisely which keys +trigger which kind of navigations. + +To interact in real-time with the UI, the developer emits `NavCommand`s events. +The navigation system responds with `NavEvent` events, such as +`NavEvent::FocusChanged` or `NavEvent::Caught`. + +The navigation system doesn't handle itself the state of the UI, but instead +emits the `FocusChanged` etc. events that the developer will need to read to, +for example, play an animation showing the change of focus in the game. + +Rather than directly reacting to input, we opt to make our system reactive to +`NavCommand`, this way the developer is free to chose which button does what in +the game. + +What I want is being able to specify how to navigate a menu (however complex it is) +without headaches. A simple menu should be dead easy to specify, while a +complex menu should be possible to specify. Let's start by the simple one. + +### Example + +Let's start with a practical example, and try figure out how we could specify +navigation in it: + +![Typical RPG tabbed menu](https://user-images.githubusercontent.com/26321040/140542742-0618a428-6841-4e64-94f0-d64f2d03d119.png) + +You can move UP/DOWN in the submenu, but you see you can also use LT/RT to +go from one submenu to the other (the _soul_, _body_ etc tabs). + +Here is a draft of how I could specify in-code how to navigate this menu. +`build_ui!` is a theoretical macro that builds a bevy UI without the verbosity +we currently are accustomed to. + +Think of `horizontal`, `vertical`, `panel` etc. as a `NodeBundle`s presets, the +elements between `{}` are additional `Component`s added to the `NodeBundle` and +elements within parenthesis following a keyword are children `Entity` added with +`.with_children(|xyz| {...})`. + +```rust +build_ui! { + vertical {:container "menu"} ( + horizontal( + tab {:navigable} ("soul") + tab {:navigable} ("body") + tab {:navigable} ("mind") + tab {:navigable} ("all") + ) + horizontal {:container "soul menu"} ( + vertical( + panel {:navigable} ( + colored_circle(Color::BLUE) + title_label("gdk") + ) + panel {:navigable} ( + colored_circle(Color::GREEN) + title_label("kfc") + ) + panel {:navigable} ( + colored_circle(Color::BEIGE) + title_label("abc") + ) + ) + vertical {:container "abc menu"} ( + title_label("ABC") + grid( + circle_letter {:navigable} ("a") circle_letter {:navigable} ("A") + circle_letter {:navigable} ("b") circle_letter {:navigable} ("B") + circle_letter {:navigable} ("c") circle_letter {:navigable} ("C") + ) + ) + ) + ) +} +fn react_events(events: ResMut>, game: Res) { + for event in events.iter() { + if matches!(event, NavEvent::Caught(NavCommand::Action, game.ui.start_button)) { + // start game + } + //etc. + } +} +``` + + +## Implementation strategy + +### Basic physical navigation + +Let's drop the idea to change submenu with LT/RT for a while. We would change +submenus by navigating upward with the Dpad or arrow keys to the tab bar and +selecting the tab with the LEFT and RIGHT keys. + +Typical game engine implementation of menu navigation requires the developer to +specify the "next", "left", "right", "previous" etc.[^1] relationships between +focusable elements. This is just lame! Our game engine not only already knows +the position of our UI elements, but it also has knowledge of the logical +organization of elements through the `Parent`/`Children` relationship. (in this +case, children are specified in the parenthesis following the element name, +look at `grid(...)`) + +Therefore, with the little information we provided through this hand wavy +specification, we should already have everything setup for the navigation to +just work™. + + +### Dimensionality + +Let's go back to our ambition of using LT/RT for navigation now. + +UI navigation is not just 2D, it's N-dimensional. The +LT/RT to change tabs is like navigating in a 3rd orthogonal dimension to the +dimensions you navigate with UP DOWN LEFT RIGHT inside the menus. + +I'll call those dimensions `Plane`s because it's easier to type than +"Dimension", I'll limit the implementation to 3 planes. We can imagine truly +exotic menu navigation systems with more than 3 dimensions, but I'll not +worry about that. Our `Plane`s are: +* `Plane::Menu`: Use LT/RT to move from one element to the next/previous +* `Plane::Select`: Instead of emitting an `Action` event when left-clicking or + pressing A/B, go up-down that direction +* `Plane::Physical`: Use the `Transform` positioning and navigate with Dpad or + arrow keys + +Each `NavCommand` moves the focus on a specific `Plane` as follow: +* `Plane::Menu`: `Previous`, `Next` +* `Plane::Select`: `Action`, `Cancel` +* `Plane::Physical`: `MoveUp`, `MoveDown`, `MoveLeft`, `MoveRight` + +We should also be able to "loop" the menu, for example going LEFT of "soul" +loops us back to "all" at the other side of the tab bar. + +### Specifying navigation dimensions + +We posited we could easily infer the physical layout and bake a navigation map +automatically based on the `Transform` positions of the elements. This is not +the case for the dimensions of our navigation. We can't magically infer the +intent of the developer: They need to specify the dimensionality of their menus. + +Let's go back to the menu. In the example code, I refer to `:container` and +`:navigable`. I've not explained yet what those are. Let's clear things up. + +A _navigable_ is an element that can be focused. A _container_ is a node entity +that contains _navigables_ and 0 or 1 other _container_. + +![The previous tabbed menu example, but with _containers_ and _navigables_ highlighted](https://user-images.githubusercontent.com/26321040/140542768-4fdd5f23-2c2e-43c1-9fa4-cc11fe67c619.png) + +The _containers_ are represented as semi-transparent squares; the _navigables_ +are circles. + +In rust, it might look like this: +```rust +struct Container { + inner: Option>, + siblings: NonEmptyVec, + active: SiblingIndex, + plane: Plane, +} +``` +Note: this is the data structure for the navigation algorithm, the ECS +componenet called `Container` will probably look like this: +```rust +#[derive(Component)] +struct Container { + plane: Plane, + loops: bool, +} +``` + +For now, let's focus on navigation within a single _container_. The +_navigables_ are collected by walking through the bevy `Parent`/`Children` +hierarchy until we find a `Parent` with the `Container` component. Transitive +children[^2] `Entity`s marked with `Navigable` are all _navigable siblings_. +When collecting sibling, we do not traverse _container_ boundaries (ie: the +_navigables_ of a contain**ed** _container_ are not the _navigables_ of the +contain**ing** _container_) + +A `Container`'s plane field specifies how to navigate between it's contained +_navigables_. In the case of our menu, it would be `Plane::Menu`. By default it is +`Plane::Physical`. + +So how does that play with `NavCommand`s? For example: You are focused on "B" +navigable in the "abc menu" container, and you issue a `Next` `NavCommand` (press +RT). What happens is this: +1. What is the `plane` of "abc menu"? It is `Physical`, I must look up the + containing `Container`'s plane. +2. What is the `plane` of "soul menu"? It is `Physical`, I must look up the + containing `Container`'s plane. +3. What is the `plane` of "menu"? It is `Menu`! +4. Let's take it's current active _navigable_ and look which other _navigable_ + we can reach with our `NavCommand` + +In short, if your focused element is inside a given _container_ and you emit a `NavCommand`, +you climb up the container hierarchy until you find one in the plane of your +`NavCommand`, then lookup the sibling you can reach with the given `NavCommand`. + +This algorithm results in three different outcomes: +1. We find a container with a matching plane and execute a focus change +2. We find a container with a matching plane, but there is no focus change, + for example when we try to go left when we are focused on a leftmost + element +3. We bubble up the container tree without ever finding a matching plane + + +### Navigation boundaries and navigation tree + +The navigation tree is the entire container hierarchy, including all nodes (which +are always `Container`s) and leaves (`Navigable`s). For the previous example, +it looks like this: + +![A diagram of the navigation tree for the tabbed menu example](https://user-images.githubusercontent.com/26321040/140542937-e28eed5e-70d5-4899-9c41-fb89b222469e.png) + +Important: The navigation tree is linear: it doesn't have "dead end" branches, +it has as many nodes as there are depth levels. + +The algorithm for finding the next focused element based on a `NavCommand` is: +```python +def change_focus( + focused: Navigable, + cmd: NavCommand, + child_stack: List[ChildIndex], + traversal_stack: List[Navigable], +) -> FocusResult: + container = focused.parent + if container is None: + first_focused = traversal_stack.first() or focused + return FocusResult.Uncaught(first_focused, cmd) + + next_focused = container.contained_focus_change(focused, cmd) + if next_focused.is_caught: + first_focused = traversal_stack.first() or focused + return FocusResult.Caught(first_focused, container, cmd) + + elif next_focused.is_open: + parent_sibling_focused = child_stack.pop() + traversal_stack.push(focused) + + return change_focus(parent_sibling_focused, cmd, child_stack, traversal_stack) + + elif next_focused.is_sibling: + first_focused = traversal_stack.first() or focused + traversal_stack.remove_first() + + return FocusResult.FocusChanged( + leaf_from= first_focused, + leaf_to= next_focused.sibling, + branch_from= traversal_stack, + ) + else: + print("This branch should be unreachable!") +``` + +### UI Benchmarks discussion + +As mentioned in [this +RFC](https://github.com/alice-i-cecile/rfcs/blob/ui-central-dogma/rfcs/ui-central-dogma.md), +we should benchmark our design against actual UI examples. I think that the +one used in this RFC as driving example (the RPG tabbed menu) illustrates well +enough how we could implement Benchmark 2 (main menu) using our system. +Benchmark 1 is irrelevant and Benchmark 3 seems to only require a `Physical` +layer. + + +## Prior art + +I've only had a cursory glance at what already exists in the domain. Beside the +godot example, I found the [Microsoft +`FocusManager` documentation](https://docs.microsoft.com/en-us/windows/apps/design/input/focus-navigation-programmatic) +to be very close to what I am currently doing ([additional Microsoft +resources](https://docs.microsoft.com/en-us/windows/apps/design/input/focus-navigation)) + +Our design differs from Microsoft's classical UI navigation in a few key ways. +We have game-specific assumptions. For example: we assume a "trail" of active elements +that leads to the sub-sub-submenu we are currently focused in, we rely on that +trail to navigate containing menus from the focused element in the submenu. + +Otherwise, given my little experience in game programming, I am probably overlooking some +standard practices. My go-to source ("Game Engine Architecture" by Jason +Gregory) doesn't mention ui navigation systems. + +## Rationale and alternatives + +### More holistic approach + +This system relieves the game developer from implementing themselves an often +tedious UI navigation system. However, it still asks them to manage the +graphical aspect of UI such as visual highlights, a more holistic approach +probably solves this, at the cost of being more complex and less flexible. Such +a system might be explored in the future, but for the sake of +having an implementation to work with, I think aiming for a smaller scope is a +better idea. + +### As plugin or integrated in the engine? + +I want to first draft this proposition as a standalone plugin ([see on +github](https://github.com/nicopap/ui-navigation)). The current +design doesn't require any specific insight into the engine that cannot be +accessed by a 3rd party plugin. However as I already mentioned, this solves +[bevy-specific concerns](https://github.com/bevyengine/bevy/issues/254), and +bevy would benefit from integrating a good ui navigation system. + +### Naming + +I'm not comfortable with the names I chose for the various concepts I +introduced. (such a `Container`, `Plane`, `Physical`) Do not be afraid to +suggest name changes. Naming is important and I'm already not satisfied with +the nomenclature I came up with. + +## Drawbacks and design limitations + +### Unique hierarchy + +If we implementation this with a cached navigation tree, it's going to be +easy to assume accidentally that we have a single fully reachable navigation +tree. This is not obvious, it's easy to just add the `Container` or `Navigable` +`Component` to an entity that doesn't have itself a `Container` parent more +than once. + +This needs to be solved, I'm thinking that an error message would be enough in +this case. But it's worth considering design changes that makes this situation +a non-issue. + +### No concurrent submenus + +You have to "build up" reactively the submenus as you navigate through them, +since **there can only be a single path through the hierarchy**. + +We expect that on `FocusChanged`, the developer adds the components needed if +the focus change leads to a new sub-menu. + +I'm implementing first to see how cumbersome this is, and maybe revise the +design with learning from that. + +Maybe it's possible to integrate more than one `Container` within other +containers. The only motivation for this currently is that it makes it easier to +implement and reason about the system. + +### Inflexible navigation + +The developer might want to add arbitrary "jump" relations between various +menus and buttons. This is not possible with this design. Maybe we could add a +```rust + bridges: HashMap, +``` +field to the `Container` to reroute `NavCommand`s that failed to change focus +within the menu. I think this might enable cycling references within the +hierarchy and break useful assumptions. + +## Unresolved questions + +Building the physical navigation tree seems not-so trivial, I'm not sure how +feasible it is, although I really think it's not that difficult. + + +## Isn't this overkill? + +Idk. My first design involved even more concepts, such as `Fence`s and `Bubble` +where each `Container` could be open or closed to specific types of `NavCommand`s. +After formalizing my thought, I came up with the current design because it +solved issues with the previous ones, and on top of it, required way less +concepts. + +For the use-case I considered, I think it's a perfect solution. It seems to +be a very useful solution to less complex use cases too. Is it overkill for +that? I think not, because we don't need to define relations at all, especially +for a menu exclusively in the physical plane. On top of that, I'm not creative +enough to imagine more complex use-cases. + +[^1]: See [godot documentation](https://github.com/godotengine/godot-docs/blob/master/tutorials/ui/gui_navigation.rst) +[^2]: ie: includes children, grand-children, grand-grand-children etc. From 30cba860155bcf5ccd06aad1455ad8dbf2150221 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Sat, 6 Nov 2021 14:03:27 +0100 Subject: [PATCH 02/33] Rename rfc file --- rfcs/{ui-navigation.md => 41-ui-navigation.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename rfcs/{ui-navigation.md => 41-ui-navigation.md} (100%) diff --git a/rfcs/ui-navigation.md b/rfcs/41-ui-navigation.md similarity index 100% rename from rfcs/ui-navigation.md rename to rfcs/41-ui-navigation.md From e2c1187cdc8a9f6807033562694a80ccc3850260 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Mon, 8 Nov 2021 10:44:06 +0100 Subject: [PATCH 03/33] Revise ui-navigation RFC tldr: * Get rid of concept of `Plane` * A lot of renaming: Navigable => Focusable, Container => NavFence * Put more emphasis on "single navigation path" limitations * Add a `Terminology` section * Improve illustrations to highlight the concept of `Active` * Elude UI specification from example * Use rust code for the algorithm * Remove plain-text English explanation of the algorithm --- rfcs/41-ui-navigation.md | 423 ++++++++++++++++++--------------------- 1 file changed, 199 insertions(+), 224 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 0d15bd32..0b18501f 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -13,95 +13,99 @@ such as the ones ## User-facing explanation +### Terminology + +* A **`NavRequest`** is the sum of all possible ui-related user inputs. It's what + the algorithm takes as input, on top of the _navigation tree_ to figure out + the next focused element. +* A **`Focusable`** is an Entity that can be navigated-to and highlighted (focused) + with a controller or keyboard +* A **`NavFence`** adds "scope" to `Focusable`s. All `Focusable`s children of a + `NavFence` will be isolated from other `Focusable` and will require special + rules to access and leave. +* The **navigation tree** is the logical relationship between all `NavFence`s. + `Focusables` are not really part of the tree, as you never break out of the + `NavFence` without a special `NavRequest` +* There is a single `Focused` Entity, which is the currently highlighted/used + Entity. +* There can be several **`Active`** elements, they represent the "path" through the + _navigation tree_ to the current `Focused` element (there isn't an exact 1 + to 1 relationship, see following examples) +* The navigation algorithm **resolves** a `NavRequest` by finding which `Entity` + to next focus to +* The **active trail** is all the `Active` elements from the outermost + `NavFence` up to the innermost `NavFence` containing the `Focused` element. + +### Game developer interactions with the system + We add a _navigation tree_ to the UI system, the tree will be used to enable -key-based navigation of the specified UIs. +key-based (and joystick controllable) navigation of UI, by allowing users to +rotate between different UI elements. Currently, the developer must themselves write the relationship between various buttons in the menus and specify which button press leads to other menu items, etc. Or more simply, limit themselves to pointer-based navigation to avoid having to think about menu navigation. -By including navigation-related components such as `Navigable` and `Container`, -the developer can quickly get a controller-navigable menu up and running. The -`Container` accepts various `Plane`s to fine-tune more precisely which keys -trigger which kind of navigations. +By including navigation-related components such as `Focusable` and `NavFence`, +the developer can quickly get a controller-navigable menu up and running. + +We could also add the `Focusable` component to standard UI widget bundles such +as `ButtonBundle` and a potential `TextField`. This way, the developer _doesn't +even need to specify the `Focusable` elements_. -To interact in real-time with the UI, the developer emits `NavCommand`s events. +To interact in real-time with the UI, the developer emits `NavRequest`s events. The navigation system responds with `NavEvent` events, such as -`NavEvent::FocusChanged` or `NavEvent::Caught`. +`NavEvent::FocusChanged` or `NavEvent::Caught`. The navigation system doesn't handle itself the state of the UI, but instead emits the `FocusChanged` etc. events that the developer will need to read to, for example, play an animation showing the change of focus in the game. -Rather than directly reacting to input, we opt to make our system reactive to -`NavCommand`, this way the developer is free to chose which button does what in -the game. +The game developer may also chose to listen to `Changed` and read +the `Focusable.is_active` and `Focusable.is_focused` fields. -What I want is being able to specify how to navigate a menu (however complex it is) -without headaches. A simple menu should be dead easy to specify, while a -complex menu should be possible to specify. Let's start by the simple one. +Rather than directly reacting to input, we opt to make our system reactive to +`NavRequest`, this way the developer is free to chose which button does what in +the game. The developer also has the complete freedom to swap at runtime what +sends or how those `NavRequest` are generated. ### Example Let's start with a practical example, and try figure out how we could specify navigation in it: -![Typical RPG tabbed menu](https://user-images.githubusercontent.com/26321040/140542742-0618a428-6841-4e64-94f0-d64f2d03d119.png) - -You can move UP/DOWN in the submenu, but you see you can also use LT/RT to -go from one submenu to the other (the _soul_, _body_ etc tabs). +![Typical RPG tabbed menu](https://user-images.githubusercontent.com/26321040/140716885-eb089626-21d2-4711-a0c9-cf185bc0f19a.png) -Here is a draft of how I could specify in-code how to navigate this menu. -`build_ui!` is a theoretical macro that builds a bevy UI without the verbosity -we currently are accustomed to. - -Think of `horizontal`, `vertical`, `panel` etc. as a `NodeBundle`s presets, the -elements between `{}` are additional `Component`s added to the `NodeBundle` and -elements within parenthesis following a keyword are children `Entity` added with -`.with_children(|xyz| {...})`. +The tab "soul" and the panel "abc" are highlighted to show the player what menu +they are down, they are **active** elements. The circle "B" is highlighted +differently to show it is the **focused** element. +You can move `UP`/`DOWN` in the submenu, but you can also use `LT`/`RT` to +go from one submenu to the other (the _soul_, _body_ etc tabs). This seems to +be a reasonable API to interact with the focus change events: ```rust -build_ui! { - vertical {:container "menu"} ( - horizontal( - tab {:navigable} ("soul") - tab {:navigable} ("body") - tab {:navigable} ("mind") - tab {:navigable} ("all") - ) - horizontal {:container "soul menu"} ( - vertical( - panel {:navigable} ( - colored_circle(Color::BLUE) - title_label("gdk") - ) - panel {:navigable} ( - colored_circle(Color::GREEN) - title_label("kfc") - ) - panel {:navigable} ( - colored_circle(Color::BEIGE) - title_label("abc") - ) - ) - vertical {:container "abc menu"} ( - title_label("ABC") - grid( - circle_letter {:navigable} ("a") circle_letter {:navigable} ("A") - circle_letter {:navigable} ("b") circle_letter {:navigable} ("B") - circle_letter {:navigable} ("c") circle_letter {:navigable} ("C") - ) - ) - ) - ) +fn setup_ui() { + // TODO: add an example in the ui-navigation repo and link to it +} +fn ui_events_system(mut ui_events: EventReader, game: Res) { + for event in ui_events.iter() { + match event { + NavEvent::Unresolved(NavRequest::Action, button) if button == game.ui.start_game_button => { + // Start the game + } + _ => {} + } + } } -fn react_events(events: ResMut>, game: Res) { - for event in events.iter() { - if matches!(event, NavEvent::Caught(NavCommand::Action, game.ui.start_button)) { - // start game +fn ui_draw_system(focus_changes: Query<(Entity, &Focusable), Changed>) { + for (button, focus_state) in focus_changes.iter() { + if focus_state.is_focused { + // Draw the button as being focused + } else if focus_state.is_active { + // etc. } - //etc. + // etc. } } ``` @@ -111,17 +115,14 @@ fn react_events(events: ResMut>, game: Res) { ### Basic physical navigation -Let's drop the idea to change submenu with LT/RT for a while. We would change -submenus by navigating upward with the Dpad or arrow keys to the tab bar and -selecting the tab with the LEFT and RIGHT keys. +Let's drop the idea to change submenu with `LT`/`RT` for a while. Let's focus on +navigating the "ABC menu". Typical game engine implementation of menu navigation requires the developer to specify the "next", "left", "right", "previous" etc.[^1] relationships between focusable elements. This is just lame! Our game engine not only already knows the position of our UI elements, but it also has knowledge of the logical -organization of elements through the `Parent`/`Children` relationship. (in this -case, children are specified in the parenthesis following the element name, -look at `grid(...)`) +organization of elements through the `Parent`/`Children` relationship. Therefore, with the little information we provided through this hand wavy specification, we should already have everything setup for the navigation to @@ -130,159 +131,121 @@ just work™. ### Dimensionality -Let's go back to our ambition of using LT/RT for navigation now. +Let's go back to our ambition of using `LT`/`RT` for navigation now. UI navigation is not just 2D, it's N-dimensional. The -LT/RT to change tabs is like navigating in a 3rd orthogonal dimension to the -dimensions you navigate with UP DOWN LEFT RIGHT inside the menus. - -I'll call those dimensions `Plane`s because it's easier to type than -"Dimension", I'll limit the implementation to 3 planes. We can imagine truly -exotic menu navigation systems with more than 3 dimensions, but I'll not -worry about that. Our `Plane`s are: -* `Plane::Menu`: Use LT/RT to move from one element to the next/previous -* `Plane::Select`: Instead of emitting an `Action` event when left-clicking or - pressing A/B, go up-down that direction -* `Plane::Physical`: Use the `Transform` positioning and navigate with Dpad or - arrow keys - -Each `NavCommand` moves the focus on a specific `Plane` as follow: -* `Plane::Menu`: `Previous`, `Next` -* `Plane::Select`: `Action`, `Cancel` -* `Plane::Physical`: `MoveUp`, `MoveDown`, `MoveLeft`, `MoveRight` - -We should also be able to "loop" the menu, for example going LEFT of "soul" +`LT`/`RT` to change tabs is like navigating in a 3rd orthogonal dimension to the +dimensions you navigate with `UP` `DOWN` `LEFT` `RIGHT` inside the menus. And to go +from one menu to another you most often press `A`/`B`. + +We should also be able to "loop" the menu, for example going `LEFT` of "soul" loops us back to "all" at the other side of the tab bar. -### Specifying navigation dimensions +### Specifying navigation between menus -We posited we could easily infer the physical layout and bake a navigation map -automatically based on the `Transform` positions of the elements. This is not -the case for the dimensions of our navigation. We can't magically infer the +We posited we could easily navigate with directional inputs the menu based on +the position on-screen of each ui elements. This is not +the case for the other dimensions of navigation. We can't magically infer the intent of the developer: They need to specify the dimensionality of their menus. -Let's go back to the menu. In the example code, I refer to `:container` and -`:navigable`. I've not explained yet what those are. Let's clear things up. +This is where `NavFence`s become useful. We could specify a fully navigable +menu with any arbitrary **graphical layout** but without any **navigation +layout** by not bothering to insert any `NavFence` component. But a serious™ +game oftentimes has deep nested menus with arbitrary limitations on the +navigation layout to help the player navigate easily the menu. +The tabbed menu example could be from such a game. + +In our example, we want to be able to go from "soul menu" to "ABC menu" with +`A`/`B` (Action, Cancel). `Action` let you enter the related submenu, while +`Cancel` does the opposite. -A _navigable_ is an element that can be focused. A _container_ is a node entity -that contains _navigables_ and 0 or 1 other _container_. +`NavFence` specify the level of nestedness of each menu elements. -![The previous tabbed menu example, but with _containers_ and _navigables_ highlighted](https://user-images.githubusercontent.com/26321040/140542768-4fdd5f23-2c2e-43c1-9fa4-cc11fe67c619.png) +![The previous tabbed menu example, but with _fences_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/140716902-bd579243-9cfa-4bdf-a633-572344e15242.png) -The _containers_ are represented as semi-transparent squares; the _navigables_ +The _fences_ are represented as semi-transparent squares; the _focusables_ are circles. -In rust, it might look like this: +How this relates to dimensionality becomes obvious when drawing the +relationship between menus as a graph. Here we see that `Action` goes down the +navigation graph, while `Cancel` goes up: + +![Graph menu layout](https://user-images.githubusercontent.com/26321040/140716920-fd298afb-093b-47f9-8309-c4c354c3d40f.png) +(orange represents `Focused` elements, gold are `Active` elements) + +In bevy, a `NavFence` might be nothing else than a component: ```rust -struct Container { - inner: Option>, - siblings: NonEmptyVec, - active: SiblingIndex, - plane: Plane, -} +#[derive(Component)] +struct NavFence; +``` + +### Navigating the tab bar + +However, it's not enough to be able to go up and down the menu hierarchy, we +also want to be able to directly "speak" with the tab menu and switch from the +"soul menu" to the "body menu" with a simple press of `RT` without having to +press `B` twice to get to the tab menu. + +To solve this, we add a field to our `NavFence`, +```rust + sequence_menu: bool, ``` -Note: this is the data structure for the navigation algorithm, the ECS -componenet called `Container` will probably look like this: +With this, when we traverse upward the navigation tree, for the `Next` and +`Previous` `NavRequest`, we check for this field and try to move within that +`NavFence`. + + +### Loops + +TODO: specify how we solve and deduce loops + +### Algorithm + +This is a lot of words to describe something that is actually quite simple. An +implementation is available [in this +repo](https://github.com/nicopap/ui-navigation), but the focus resolution +algorithm can be copied here: ```rust -#[derive(Component)] -struct Container { - plane: Plane, - loops: bool, +/// Change focus within provided set of `siblings`, `None` if impossible. +fn resolve_within( + focused: Entity, + request: NavRequest, + siblings: &[Entity], + transform: &Query<&GlobalTransform>, +) -> Option; + +fn resolve( + focused: Entity, + request: NavRequest, + queries: &NavQueries, + mut from: Vec, +) -> NavEvent { + let nav_fence = match parent_nav_fence(focused, queries) { + Some(entity) => entity, + None => return NavEvent::Uncaught { request, focused }, + }; + let siblings = children_focusables(nav_fence, queries); + let focused = get_active(&siblings, &queries.actives); + from.push(focused); + match resolve_within(focused, request, &siblings, &queries.transform) { + Some(to) => NavEvent::FocusChanged { to, from }, + None => resolve(nav_fence, request, queries, from), + } } ``` +The `NavEvent` can then be used to tell bevy to modify the `Focusable`, +`Active` and `Focused` component as needed. -For now, let's focus on navigation within a single _container_. The -_navigables_ are collected by walking through the bevy `Parent`/`Children` -hierarchy until we find a `Parent` with the `Container` component. Transitive -children[^2] `Entity`s marked with `Navigable` are all _navigable siblings_. -When collecting sibling, we do not traverse _container_ boundaries (ie: the -_navigables_ of a contain**ed** _container_ are not the _navigables_ of the -contain**ing** _container_) - -A `Container`'s plane field specifies how to navigate between it's contained -_navigables_. In the case of our menu, it would be `Plane::Menu`. By default it is -`Plane::Physical`. - -So how does that play with `NavCommand`s? For example: You are focused on "B" -navigable in the "abc menu" container, and you issue a `Next` `NavCommand` (press -RT). What happens is this: -1. What is the `plane` of "abc menu"? It is `Physical`, I must look up the - containing `Container`'s plane. -2. What is the `plane` of "soul menu"? It is `Physical`, I must look up the - containing `Container`'s plane. -3. What is the `plane` of "menu"? It is `Menu`! -4. Let's take it's current active _navigable_ and look which other _navigable_ - we can reach with our `NavCommand` - -In short, if your focused element is inside a given _container_ and you emit a `NavCommand`, -you climb up the container hierarchy until you find one in the plane of your -`NavCommand`, then lookup the sibling you can reach with the given `NavCommand`. - -This algorithm results in three different outcomes: -1. We find a container with a matching plane and execute a focus change -2. We find a container with a matching plane, but there is no focus change, - for example when we try to go left when we are focused on a leftmost - element -3. We bubble up the container tree without ever finding a matching plane - - -### Navigation boundaries and navigation tree - -The navigation tree is the entire container hierarchy, including all nodes (which -are always `Container`s) and leaves (`Navigable`s). For the previous example, -it looks like this: - -![A diagram of the navigation tree for the tabbed menu example](https://user-images.githubusercontent.com/26321040/140542937-e28eed5e-70d5-4899-9c41-fb89b222469e.png) - -Important: The navigation tree is linear: it doesn't have "dead end" branches, -it has as many nodes as there are depth levels. - -The algorithm for finding the next focused element based on a `NavCommand` is: -```python -def change_focus( - focused: Navigable, - cmd: NavCommand, - child_stack: List[ChildIndex], - traversal_stack: List[Navigable], -) -> FocusResult: - container = focused.parent - if container is None: - first_focused = traversal_stack.first() or focused - return FocusResult.Uncaught(first_focused, cmd) - - next_focused = container.contained_focus_change(focused, cmd) - if next_focused.is_caught: - first_focused = traversal_stack.first() or focused - return FocusResult.Caught(first_focused, container, cmd) - - elif next_focused.is_open: - parent_sibling_focused = child_stack.pop() - traversal_stack.push(focused) - - return change_focus(parent_sibling_focused, cmd, child_stack, traversal_stack) - - elif next_focused.is_sibling: - first_focused = traversal_stack.first() or focused - traversal_stack.remove_first() - - return FocusResult.FocusChanged( - leaf_from= first_focused, - leaf_to= next_focused.sibling, - branch_from= traversal_stack, - ) - else: - print("This branch should be unreachable!") -``` ### UI Benchmarks discussion -As mentioned in [this -RFC](https://github.com/alice-i-cecile/rfcs/blob/ui-central-dogma/rfcs/ui-central-dogma.md), +As mentioned in [this RFC](https://github.com/alice-i-cecile/rfcs/blob/ui-central-dogma/rfcs/ui-central-dogma.md), we should benchmark our design against actual UI examples. I think that the one used in this RFC as driving example (the RPG tabbed menu) illustrates well enough how we could implement Benchmark 2 (main menu) using our system. -Benchmark 1 is irrelevant and Benchmark 3 seems to only require a `Physical` -layer. +Benchmark 1 is irrelevant. Benchmark 3 could potentially be solved by just +marking the interactive element with the `Focusable` component and nothing +else, so that every element can be reached from anywhere. ## Prior art @@ -323,26 +286,33 @@ accessed by a 3rd party plugin. However as I already mentioned, this solves [bevy-specific concerns](https://github.com/bevyengine/bevy/issues/254), and bevy would benefit from integrating a good ui navigation system. -### Naming - -I'm not comfortable with the names I chose for the various concepts I -introduced. (such a `Container`, `Plane`, `Physical`) Do not be afraid to -suggest name changes. Naming is important and I'm already not satisfied with -the nomenclature I came up with. - ## Drawbacks and design limitations ### Unique hierarchy -If we implementation this with a cached navigation tree, it's going to be -easy to assume accidentally that we have a single fully reachable navigation -tree. This is not obvious, it's easy to just add the `Container` or `Navigable` -`Component` to an entity that doesn't have itself a `Container` parent more -than once. +The proposed navigation tree has a surprising design. The navigation tree only +manages the path to the current active element and the siblings of the +fences you have to traverse from the root to get to the focused element. + +This has several drawbacks: +* The user must be aware and understand the expectations of the navigation + algorithm. It's easy to detect violations of the expectations, communicating + the fix to the user less so. Potentially we could use a runtime error or + warning. +* We don't have any memory beside the activation trail. A good example of focus + memory is a classical text editor with multiple tabs. When you go from Tab 2 + to Tab 1, and back from Tab 1 to Tab 2, you will find the text cursor at the + same location you left it. This isn't possible with this system, as the only + memory is from the root of the window to the current cursor. + -This needs to be solved, I'm thinking that an error message would be enough in -this case. But it's worth considering design changes that makes this situation -a non-issue. +I think this is fixable. It's possible to store the entire navigation tree. We +could simply keep track of the active trail as boolean flags on each branches. +The focus algorithm only cares about the active path, so there would be very +little to change on that side. + +The focus memory use case is not important for most games. However it will +be necessary for any complex editor. ### No concurrent submenus @@ -355,31 +325,36 @@ the focus change leads to a new sub-menu. I'm implementing first to see how cumbersome this is, and maybe revise the design with learning from that. -Maybe it's possible to integrate more than one `Container` within other -containers. The only motivation for this currently is that it makes it easier to -implement and reason about the system. +The only motivation for this is that it makes it easier to +implement and reason about the system. It's far easier to implement. In fact +the current implementation doesn't even use a tree data type, but a vector of +vectors where each row represents a layer of _focusables_. + +It's possible to keep more than one `NavFence` within other fences. +The solution is that of [the previous section](#unique-hierarchy). + ### Inflexible navigation The developer might want to add arbitrary "jump" relations between various menus and buttons. This is not possible with this design. Maybe we could add a ```rust - bridges: HashMap, + bridges: HashMap, ``` -field to the `Container` to reroute `NavCommand`s that failed to change focus +field to the `NavFence` to reroute `NavRequest`s that failed to change focus within the menu. I think this might enable cycling references within the hierarchy and break useful assumptions. -## Unresolved questions - -Building the physical navigation tree seems not-so trivial, I'm not sure how -feasible it is, although I really think it's not that difficult. +### Mouse support +We should add mouse support (pointer devices). Though it is not at all +mentioned in this RFC, it is self-evident that integrating the navigation-based +focus with pointer-based focus is the way forward. ## Isn't this overkill? Idk. My first design involved even more concepts, such as `Fence`s and `Bubble` -where each `Container` could be open or closed to specific types of `NavCommand`s. +where each `NavFence` could be open or closed to specific types of `NavRequest`s. After formalizing my thought, I came up with the current design because it solved issues with the previous ones, and on top of it, required way less concepts. @@ -387,7 +362,7 @@ concepts. For the use-case I considered, I think it's a perfect solution. It seems to be a very useful solution to less complex use cases too. Is it overkill for that? I think not, because we don't need to define relations at all, especially -for a menu exclusively in the physical plane. On top of that, I'm not creative +for a menu without any hierarchy. On top of that, I'm not creative enough to imagine more complex use-cases. [^1]: See [godot documentation](https://github.com/godotengine/godot-docs/blob/master/tutorials/ui/gui_navigation.rst) From ee4edc44ffdf842378be68e8883d609c1c4de401 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Sun, 14 Nov 2021 10:37:24 +0100 Subject: [PATCH 04/33] Complete UI navigation RFC Very close to a full rewrite. tldr: * Completely remove the idea of "Dimensionality" * Design reflects implementation: it's now completely "menu oriented" * Short mention of the bevy `Interactive` component * Add more links to the implementation rather than describe in the RFC the implementation itself * More context on what is the tree and how useful it is --- rfcs/41-ui-navigation.md | 473 +++++++++++++++++++++------------------ 1 file changed, 252 insertions(+), 221 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 0b18501f..09d4bf9c 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -2,45 +2,51 @@ ## Summary -A set of components and a new system to define navigation nodes for the ECS UI. +Modification of the `Interaction` component to support keyboard & controller +based navigation of UI on top of mouse-oriented navigation. In the rest of this +RFC, consider the already existing "`Interaction`" bevy component as being a good +candidate for being replaced by what we describe as "`Focusable`". -## Motivation - -On top of making it easy to implement controller navigation of -in-game UIs, it enables keyboard navigations and answers accessibility concerns -such as the ones -[raised in this issue](https://github.com/bevyengine/bevy/issues/254). +The new navigation system by default enables unrestricted navigation between +all `Focusable` entities. But the developer may chose to opt in to a more +complex navigation scheme by adding `NavFence`, encapsulating groups of +`Focusable`s into their own menus. The `NavFence` component can also express +the navigation scheme within the menus. -## User-facing explanation +The goal is to make the easiest navigation scheme work out of the box and the +more complex schemes easy or at least possible to implement. -### Terminology +## Terminology * A **`NavRequest`** is the sum of all possible ui-related user inputs. It's what the algorithm takes as input, on top of the _navigation tree_ to figure out the next focused element. * A **`Focusable`** is an Entity that can be navigated-to and highlighted (focused) with a controller or keyboard -* A **`NavFence`** adds "scope" to `Focusable`s. All `Focusable`s children of a - `NavFence` will be isolated from other `Focusable` and will require special - rules to access and leave. -* The **navigation tree** is the logical relationship between all `NavFence`s. - `Focusables` are not really part of the tree, as you never break out of the - `NavFence` without a special `NavRequest` -* There is a single `Focused` Entity, which is the currently highlighted/used +* A **`NavFence`** adds "scope" to `Focusable`s. Navigation between `Focusable` + entities will be constrained to other `Focusable` that are children of the + same `NavFence`. It creates a self-contained menu. **A `NavFence` may therefore + be better described as a _menu_**. Note that the `Focusable`s do not need to + be direct children of the `NavFence`, only transitive children[^1] +* The **menu tree** (sometimes _navigation tree_) is the logical relationship + between all `NavFence`s. +* There is a single **focused** Entity, which is the currently highlighted/used Entity. -* There can be several **`Active`** elements, they represent the "path" through the - _navigation tree_ to the current `Focused` element (there isn't an exact 1 - to 1 relationship, see following examples) +* There can be several **active** elements, they represent the "path" through the + _navigation tree_ to the current _focused_ element. Which focusable in the + previous menus were activated in order to reach the menu we are in currently. +* A **dormant** element is a previously active element from a branch of the + menu tree that is currently not focused. * The navigation algorithm **resolves** a `NavRequest` by finding which `Entity` to next focus to -* The **active trail** is all the `Active` elements from the outermost - `NavFence` up to the innermost `NavFence` containing the `Focused` element. - -### Game developer interactions with the system +* **focus memory** is what, for example, makes it possible in your text editor + to tab to a different file, edit it, and tab back to the file you were + previously editing and find yourself at the exact location where you left it. + It is the ability for the software to keep track where you were at. + **dormant** elements are necessary to track what was previously focused and + come back to it when needed -We add a _navigation tree_ to the UI system, the tree will be used to enable -key-based (and joystick controllable) navigation of UI, by allowing users to -rotate between different UI elements. +## Motivation Currently, the developer must themselves write the relationship between various buttons in the menus and specify which button press leads to other menu @@ -50,20 +56,29 @@ avoid having to think about menu navigation. By including navigation-related components such as `Focusable` and `NavFence`, the developer can quickly get a controller-navigable menu up and running. -We could also add the `Focusable` component to standard UI widget bundles such -as `ButtonBundle` and a potential `TextField`. This way, the developer _doesn't -even need to specify the `Focusable` elements_. +There already exists components with similar purposes in Bevy: +[`Interaction` and `FocusPolicy`](https://github.com/bevyengine/bevy/blob/main/crates/bevy_ui/src/focus.rs#L14). +However, it only supports a barebone mouse-oriented navigation scheme. And the +focus handling is not even implemented. + +My proposal, on top of making it easy to implement controller navigation of +in-game UIs, enables keyboard navigation and answers accessibility concerns +such as the ones +[raised in this issue](https://github.com/bevyengine/bevy/issues/254). + +## User-facing explanation + +Examples and an implementation-oriented description is available in the +[ui-navigation plugin README](https://github.com/nicopap/ui-navigation/blob/master/Readme.md). To interact in real-time with the UI, the developer emits `NavRequest`s events. The navigation system responds with `NavEvent` events, such as `NavEvent::FocusChanged` or `NavEvent::Caught`. -The navigation system doesn't handle itself the state of the UI, but instead -emits the `FocusChanged` etc. events that the developer will need to read to, -for example, play an animation showing the change of focus in the game. - -The game developer may also chose to listen to `Changed` and read -the `Focusable.is_active` and `Focusable.is_focused` fields. +The navigation system is the only one to handle the focus state of the UI. The +user interacts with it strictly by sending `NavRequest`s, reading `NavEvent` or +using the `Focusable` methods. Typically in a system with a +`Query<&Focusable, Changed>` parameter. Rather than directly reacting to input, we opt to make our system reactive to `NavRequest`, this way the developer is free to chose which button does what in @@ -82,160 +97,159 @@ they are down, they are **active** elements. The circle "B" is highlighted differently to show it is the **focused** element. You can move `UP`/`DOWN` in the submenu, but you can also use `LT`/`RT` to -go from one submenu to the other (the _soul_, _body_ etc tabs). This seems to -be a reasonable API to interact with the focus change events: +go from one submenu to the other (the _soul_, _body_ etc tabs). In code, we +might react to focus changes (for example to change the color of +focused/unfocused elements) as follow: ```rust fn setup_ui() { - // TODO: add an example in the ui-navigation repo and link to it + // setup UI like you would setup UI in bevy today } -fn ui_events_system(mut ui_events: EventReader, game: Res) { - for event in ui_events.iter() { - match event { - NavEvent::Unresolved(NavRequest::Action, button) if button == game.ui.start_game_button => { - // Start the game - } - _ => {} +fn handle_nav_events(mut events: EventReader, game: Res) { + use bevy_ui_navigation::{NavEvent::Caught, NavRequest::Action}; + for event in events.iter() { + match event { + Caught { from, request: Action } if from.first() == game.start_game_button => { + // Start the game on "A" or "ENTER" button press + } + _ => {} + } } - } } -fn ui_draw_system(focus_changes: Query<(Entity, &Focusable), Changed>) { - for (button, focus_state) in focus_changes.iter() { - if focus_state.is_focused { - // Draw the button as being focused - } else if focus_state.is_active { - // etc. +fn button_system( + materials: Res, + mut focusables: Query<(&Focusable, &mut Handle), Changed>, +) { + for (focus_state, mut material) in focusables.iter_mut() { + if focus_state.is_focused() { + *material = materials.focused.clone(); + } else { + *material = materials.inert.clone(); + } } - // etc. - } } ``` - ## Implementation strategy -### Basic physical navigation +### Multiple menus -Let's drop the idea to change submenu with `LT`/`RT` for a while. Let's focus on -navigating the "ABC menu". +The first intuition is that this screenshot shows three menus: +* The "tabs menu" containing _soul_, _body_, _mind_ and _all_ "tabs" +* The "soul menu" containing _gdk_, _kfc_ and _abc_ "panels" +* The "ABC menu" containing a, b, c, A, B, C "buttons" -Typical game engine implementation of menu navigation requires the developer to -specify the "next", "left", "right", "previous" etc.[^1] relationships between -focusable elements. This is just lame! Our game engine not only already knows -the position of our UI elements, but it also has knowledge of the logical -organization of elements through the `Parent`/`Children` relationship. +We expect to be able to navigate the "tabs menu" **from anywhere** by pressing +`LT`/`RT`. (or `TAB`,`SHIFT+TAB` on keyboard) We expect to be able to move +between buttons within the "ABC menu" with arrow keys, same with the "soul menu". -Therefore, with the little information we provided through this hand wavy -specification, we should already have everything setup for the navigation to -just work™. +Finally, we expect to be able to go from one menu to another by pressing `A` +when the corresponding element in the parent menu is _focused_ or go back to +the parent menu when pressing `B`. (`ENTER` and `BACKSPACE` on keyboard) +### Navigating within a single menu -### Dimensionality +Our game engine already knows the position of our UI elements. We can use this +as information to deduce the next _focused_ element when pressing an arrow key. -Let's go back to our ambition of using `LT`/`RT` for navigation now. - -UI navigation is not just 2D, it's N-dimensional. The -`LT`/`RT` to change tabs is like navigating in a 3rd orthogonal dimension to the -dimensions you navigate with `UP` `DOWN` `LEFT` `RIGHT` inside the menus. And to go -from one menu to another you most often press `A`/`B`. - -We should also be able to "loop" the menu, for example going `LEFT` of "soul" -loops us back to "all" at the other side of the tab bar. +In fact, if we have only a single menu on screen, we could stop at this point, +since there is no restrictions on navigation between `Focusable`s. ### Specifying navigation between menus -We posited we could easily navigate with directional inputs the menu based on -the position on-screen of each ui elements. This is not -the case for the other dimensions of navigation. We can't magically infer the -intent of the developer: They need to specify the dimensionality of their menus. - -This is where `NavFence`s become useful. We could specify a fully navigable -menu with any arbitrary **graphical layout** but without any **navigation -layout** by not bothering to insert any `NavFence` component. But a serious™ -game oftentimes has deep nested menus with arbitrary limitations on the -navigation layout to help the player navigate easily the menu. -The tabbed menu example could be from such a game. +But we have more than one menu. We can't magically infer what the developer +thinks is a menu, this is why we need `NavFence`s. In our example, we want to be able to go from "soul menu" to "ABC menu" with `A`/`B` (Action, Cancel). `Action` let you enter the related submenu, while `Cancel` does the opposite. -`NavFence` specify the level of nestedness of each menu elements. +With _fences_ the menu layout may look like this: -![The previous tabbed menu example, but with _fences_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/140716902-bd579243-9cfa-4bdf-a633-572344e15242.png) +![The previous tabbed menu example, but with _fences_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) The _fences_ are represented as semi-transparent squares; the _focusables_ are circles. -How this relates to dimensionality becomes obvious when drawing the -relationship between menus as a graph. Here we see that `Action` goes down the -navigation graph, while `Cancel` goes up: +A `NavFence` keeps track of the Entity `Focusable` that leads to it (in the +example, the "soul" tab entity is tracked by the "soul menu" `NavFence`). + +The `NavFence` isolates the set of `Focusable`s of a single menu: ![Graph menu layout](https://user-images.githubusercontent.com/26321040/140716920-fd298afb-093b-47f9-8309-c4c354c3d40f.png) -(orange represents `Focused` elements, gold are `Active` elements) +(red-orange represents _focused_ elements, gold are _active_ elements) -In bevy, a `NavFence` might be nothing else than a component: +In bevy, a `NavFence` might be nothing more than a component to add to the +`NodeBundle` parent of the _focusables_ you want in your menu. On top of that, +it needs to track which _focusable_ you _activated_ to enter this menu: ```rust #[derive(Component)] -struct NavFence; +struct NavFence { + parent_focusable: Option, +} ``` -### Navigating the tab bar +### `NavFence` settings + +#### Sequence menu However, it's not enough to be able to go up and down the menu hierarchy, we -also want to be able to directly "speak" with the tab menu and switch from the -"soul menu" to the "body menu" with a simple press of `RT` without having to -press `B` twice to get to the tab menu. +also want to be able to directly "speak" with the tabs menu and switch from the +"soul menu" to the "body menu" with a simple press of `RT`, not having to +press `B` twice to get to the tabs menu. -To solve this, we add a field to our `NavFence`, -```rust - sequence_menu: bool, -``` -With this, when we traverse upward the navigation tree, for the `Next` and -`Previous` `NavRequest`, we check for this field and try to move within that -`NavFence`. +The tabs menu is a special sort of menu, with movement that can be triggered +from another menu reachable from it. We must add a `setting: NavSetting` field +to the `NavFence` component to distinguish this sort of menu. In our +implementation we call them `sequence menu`s. +The `NavRequest::MenuMove` request associated with `RT`/`LT` (or anything the +developer sets) will climb up the _menu tree_ to find the enclosing +_sequence menu_ and move focus within that menu. -### Loops +#### Looping -TODO: specify how we solve and deduce loops +We may also chose to lock or unlock the ability to wrap back right when going +left from the leftmost element of a menu, same with any other directions. For +example pressing `DOWN` when "c" is _focused_ could change focus to "a". We +should use the `setting` field to keep track of this as well. + +### Navigation tree + +All in all, the navigation tree is just the specification of which menu is +reachable from where. Not the relation between each individual focusable. This +drastically simplifies the structure of the tree. Because a game may have many +interactive UI elements, but not that many menus. + +The previous graph diagram may have been a bit misleading. It's not the real +_navigation graph_ (that we introduced as _menu graph_ for good reasons). +The tree that the algorithm deals with and traverse is between menus +rather than `Focusable` elements. The graph as it is used in our code is the +following: + +![RPG menu demo split between menus with pointers leading to the "parent +focusable"](https://user-images.githubusercontent.com/26321040/141672009-e023855e-10a7-4c50-a8dd-eff10acb2208.png) + +We may also imagine an implementation where the non-selected menus are hidden +but loaded into the ECS. In this case, the navigation tree might look like +this: + +![RPG menu graph without focusables, only +menus](https://user-images.githubusercontent.com/26321040/141672033-8bd43660-78af-4521-b367-03c7270c58c5.png) ### Algorithm -This is a lot of words to describe something that is actually quite simple. An -implementation is available [in this -repo](https://github.com/nicopap/ui-navigation), but the focus resolution -algorithm can be copied here: -```rust -/// Change focus within provided set of `siblings`, `None` if impossible. -fn resolve_within( - focused: Entity, - request: NavRequest, - siblings: &[Entity], - transform: &Query<&GlobalTransform>, -) -> Option; - -fn resolve( - focused: Entity, - request: NavRequest, - queries: &NavQueries, - mut from: Vec, -) -> NavEvent { - let nav_fence = match parent_nav_fence(focused, queries) { - Some(entity) => entity, - None => return NavEvent::Uncaught { request, focused }, - }; - let siblings = children_focusables(nav_fence, queries); - let focused = get_active(&siblings, &queries.actives); - from.push(focused); - match resolve_within(focused, request, &siblings, &queries.transform) { - Some(to) => NavEvent::FocusChanged { to, from }, - None => resolve(nav_fence, request, queries, from), - } -} -``` -The `NavEvent` can then be used to tell bevy to modify the `Focusable`, -`Active` and `Focused` component as needed. +The algorithm is the `resolve` function in [ui-navigation](https://github.com/nicopap/ui-navigation/blob/master/src/lib.rs#L340). +The meat of it is 50 lines of code. + +The current implementation has an example demonstrating all features we +discussed in this RFC. For a menu layout as follow: + +![Example layout](https://user-images.githubusercontent.com/26321040/141671978-a6ef0e99-8ee1-4e5f-9d56-26c65f748331.png) +We can navigate it with a controller as follow: + +![Example layout being navigated with a controller, button presses shown as +overlay](https://user-images.githubusercontent.com/26321040/141612751-ba0e62b2-23d6-429a-b5d1-48b09c10d526.gif) ### UI Benchmarks discussion @@ -250,13 +264,14 @@ else, so that every element can be reached from anywhere. ## Prior art -I've only had a cursory glance at what already exists in the domain. Beside the -godot example, I found the [Microsoft +I've only had a cursory glance at what already exists in the domain. + +Beside [godot](https://github.com/godotengine/godot-docs/blob/master/tutorials/ui/gui_navigation.rst), I found the [Microsoft `FocusManager` documentation](https://docs.microsoft.com/en-us/windows/apps/design/input/focus-navigation-programmatic) to be very close to what I am currently doing ([additional Microsoft resources](https://docs.microsoft.com/en-us/windows/apps/design/input/focus-navigation)) -Our design differs from Microsoft's classical UI navigation in a few key ways. +Our design differs from Microsoft's UI navigation in a few key ways. We have game-specific assumptions. For example: we assume a "trail" of active elements that leads to the sub-sub-submenu we are currently focused in, we rely on that trail to navigate containing menus from the focused element in the submenu. @@ -265,105 +280,121 @@ Otherwise, given my little experience in game programming, I am probably overloo standard practices. My go-to source ("Game Engine Architecture" by Jason Gregory) doesn't mention ui navigation systems. -## Rationale and alternatives +## Open questions -### More holistic approach +### Naming -This system relieves the game developer from implementing themselves an often -tedious UI navigation system. However, it still asks them to manage the -graphical aspect of UI such as visual highlights, a more holistic approach -probably solves this, at the cost of being more complex and less flexible. Such -a system might be explored in the future, but for the sake of -having an implementation to work with, I think aiming for a smaller scope is a -better idea. +* `NavFence`: If a `NavFence` is pretty much just an abstraction to describe a + menu, why not call it a `NavMenu` instead? +* `sequence menu`: doesn't express it's a menu that can be changed with special + `NavRequest::MenuMove` even when focusing in another menu. +* `NavRequest::MenuMove`: bad naming, given everything is moving within a menu +* `NavEvent::Caught`: doesn't really express that it's just a `NavRequest` that + led to no change in focus -### As plugin or integrated in the engine? +### `NavRequest` system -I want to first draft this proposition as a standalone plugin ([see on -github](https://github.com/nicopap/ui-navigation)). The current -design doesn't require any specific insight into the engine that cannot be -accessed by a 3rd party plugin. However as I already mentioned, this solves -[bevy-specific concerns](https://github.com/bevyengine/bevy/issues/254), and -bevy would benefit from integrating a good ui navigation system. +I wanted to defer the raw input processing to the user. The current +implementation **requires** the user to specify the input button mapping to the +`NavRequest`s. However, if we want to provide an out-of-the-box working +experience, we should have at least a default mapping that can be modified at +runtime. -## Drawbacks and design limitations +I think it makes sense to have a default mapping, as technically it's already +the case with the `Interaction` component. And having something working +out of the box is a huge positive. -### Unique hierarchy +I am against an inflexible system where it is impossible to change the input +mappings, any respectable game let you change the input mapping. And if not, +your players start very quickly to ask for one! So it is necessary to be able +to remap the buttons. -The proposed navigation tree has a surprising design. The navigation tree only -manages the path to the current active element and the siblings of the -fences you have to traverse from the root to get to the focused element. +How would that work? Is there already examples in bevy where we provide default +button mappings that can be changed? -This has several drawbacks: -* The user must be aware and understand the expectations of the navigation - algorithm. It's easy to detect violations of the expectations, communicating - the fix to the user less so. Potentially we could use a runtime error or - warning. -* We don't have any memory beside the activation trail. A good example of focus - memory is a classical text editor with multiple tabs. When you go from Tab 2 - to Tab 1, and back from Tab 1 to Tab 2, you will find the text cursor at the - same location you left it. This isn't possible with this system, as the only - memory is from the root of the window to the current cursor. +I think we should think deeply about input handling in default plugins. +However, this is out of scope for this RFC. +### Game-oriented assumptions -I think this is fixable. It's possible to store the entire navigation tree. We -could simply keep track of the active trail as boolean flags on each branches. -The focus algorithm only cares about the active path, so there would be very -little to change on that side. +This was not intended during implementation, but in retrospect, the `NavFence` +system is relatively opinionated. -The focus memory use case is not important for most games. However it will -be necessary for any complex editor. +Indeed, we assume we are trying to build a game UI with a series of "menus" and +that it is possible to keep track of a "trail" of buttons from a "root menu" to +the deepest submenu we are currently browsing. -### No concurrent submenus +The example of a fancy editor with a bunch of dockered widgets seems to break +that assumption. However, I think we could make it so each docker has a tab, +and we could navigate between the tabs. I think such an implementation strategy +encourages better mouseless navigation support and it might not be a +limitation. -You have to "build up" reactively the submenus as you navigate through them, -since **there can only be a single path through the hierarchy**. +![Krita UI screenshot](https://user-images.githubusercontent.com/26321040/141671939-24b8a7c3-b296-4fd4-8ae0-3bbe7fe4c9a3.png) -We expect that on `FocusChanged`, the developer adds the components needed if -the focus change leads to a new sub-menu. +In this example, we can imagine the root `NodeBundle` having a `NavFence` +component, and the elements highlighted in yellow have a `Focusable` component. +This way we create a menu to navigate between dockers. This is perfectly +consistent with our design. -I'm implementing first to see how cumbersome this is, and maybe revise the -design with learning from that. -The only motivation for this is that it makes it easier to -implement and reason about the system. It's far easier to implement. In fact -the current implementation doesn't even use a tree data type, but a vector of -vectors where each row represents a layer of _focusables_. +## Drawbacks and design limitations -It's possible to keep more than one `NavFence` within other fences. -The solution is that of [the previous section](#unique-hierarchy). +### User upheld invariants +I think the most problematic aspect of this implementation is that we push on +the user the responsibility of upholding invariants. This is not a safety +issue, but it leads to panics. The invariants are: +1. Never create a cycle of menu connections +2. There must be at most one root `NavFence` +3. Each `NavFence` must at least have one contained `Focusable` -### Inflexible navigation +The upside is that the invariants become only relevant if the user _opts into_ +`NavFence`. Already, this is a choice made by the user, so they can be made +aware of the requirements in the `NavFence` documentation. -The developer might want to add arbitrary "jump" relations between various -menus and buttons. This is not possible with this design. Maybe we could add a -```rust - bridges: HashMap, -``` -field to the `NavFence` to reroute `NavRequest`s that failed to change focus -within the menu. I think this might enable cycling references within the -hierarchy and break useful assumptions. +If the invariants are not upheld, the navigation system will simply panic. I +didn't test enough the current implementation to see if all cases of panics +lead to meaningful error messages. + +It may be possible to be resilient to the violation of the invariants, but I +think it's better to panic, because violation will most likely come from +misunderstanding how the navigation system work or programming errors. + +### Jargon + +The end user will have to deal with new concepts introduced by this RFC: +* `dormant` +* `active` +* `focused` +* `NavRequest` +* `NavEvent` +* `NavFence` +* `Focusable` +* `looping` +* `sequence menu` + +This seems alright, although `dormant` may be misleading, given it's a synonym +to "inactive". It is technically an antonym to "active", but in the current +design, it is not really the case. Beside I have a tendency to type "doormat" +instead. + +On the implementation side. We have `inert` which is synonym to `dormant` and +`inactive` but is just a word to fill the roll of "none of the above". -### Mouse support +I think the one term that can be improved is `NavFence`. Replacing it with +`NavMenu` constructs on pre-existing knowledge and is a good approximation of +what it represents. -We should add mouse support (pointer devices). Though it is not at all -mentioned in this RFC, it is self-evident that integrating the navigation-based -focus with pointer-based focus is the way forward. +### Performance -## Isn't this overkill? +The navigation system might be slow with a very large amount of `Focusable` +elements. **I didn't benchmark it**. -Idk. My first design involved even more concepts, such as `Fence`s and `Bubble` -where each `NavFence` could be open or closed to specific types of `NavRequest`s. -After formalizing my thought, I came up with the current design because it -solved issues with the previous ones, and on top of it, required way less -concepts. +Note that the current implementation only performs computation on `NavRequest` +events. -For the use-case I considered, I think it's a perfect solution. It seems to -be a very useful solution to less complex use cases too. Is it overkill for -that? I think not, because we don't need to define relations at all, especially -for a menu without any hierarchy. On top of that, I'm not creative -enough to imagine more complex use-cases. +We might improve the performance with well designed caches and the addition of +the `child_of_interest` field to `NavFence` (which is planned). -[^1]: See [godot documentation](https://github.com/godotengine/godot-docs/blob/master/tutorials/ui/gui_navigation.rst) -[^2]: ie: includes children, grand-children, grand-grand-children etc. +[^1]: ie: includes children, grand-children, grand-grand-children etc. From ea18c84dad20658708cea2d04fc9a8c214119c0a Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 26 Nov 2021 12:14:16 +0100 Subject: [PATCH 05/33] Integrate latest name changes & mouse discussion --- rfcs/41-ui-navigation.md | 119 ++++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 58 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 09d4bf9c..31902f15 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -9,8 +9,8 @@ candidate for being replaced by what we describe as "`Focusable`". The new navigation system by default enables unrestricted navigation between all `Focusable` entities. But the developer may chose to opt in to a more -complex navigation scheme by adding `NavFence`, encapsulating groups of -`Focusable`s into their own menus. The `NavFence` component can also express +complex navigation scheme by adding `NavMenu`, encapsulating groups of +`Focusable`s into their own menus. The `NavMenu` component can also express the navigation scheme within the menus. The goal is to make the easiest navigation scheme work out of the box and the @@ -23,13 +23,13 @@ more complex schemes easy or at least possible to implement. the next focused element. * A **`Focusable`** is an Entity that can be navigated-to and highlighted (focused) with a controller or keyboard -* A **`NavFence`** adds "scope" to `Focusable`s. Navigation between `Focusable` +* A **`NavMenu`** adds "scope" to `Focusable`s. Navigation between `Focusable` entities will be constrained to other `Focusable` that are children of the - same `NavFence`. It creates a self-contained menu. **A `NavFence` may therefore + same `NavMenu`. It creates a self-contained menu. **A `NavMenu` may therefore be better described as a _menu_**. Note that the `Focusable`s do not need to - be direct children of the `NavFence`, only transitive children[^1] + be direct children of the `NavMenu`, only transitive children[^1] * The **menu tree** (sometimes _navigation tree_) is the logical relationship - between all `NavFence`s. + between all `NavMenu`s. * There is a single **focused** Entity, which is the currently highlighted/used Entity. * There can be several **active** elements, they represent the "path" through the @@ -53,7 +53,7 @@ various buttons in the menus and specify which button press leads to other menu items, etc. Or more simply, limit themselves to pointer-based navigation to avoid having to think about menu navigation. -By including navigation-related components such as `Focusable` and `NavFence`, +By including navigation-related components such as `Focusable` and `NavMenu`, the developer can quickly get a controller-navigable menu up and running. There already exists components with similar purposes in Bevy: @@ -157,40 +157,40 @@ since there is no restrictions on navigation between `Focusable`s. ### Specifying navigation between menus But we have more than one menu. We can't magically infer what the developer -thinks is a menu, this is why we need `NavFence`s. +thinks is a menu, this is why we need `NavMenu`s. In our example, we want to be able to go from "soul menu" to "ABC menu" with `A`/`B` (Action, Cancel). `Action` let you enter the related submenu, while `Cancel` does the opposite. -With _fences_ the menu layout may look like this: +With `NavMenu`s the menu layout may look like this: -![The previous tabbed menu example, but with _fences_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) +![The previous tabbed menu example, but with _menus_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) -The _fences_ are represented as semi-transparent squares; the _focusables_ +The _menus_ are represented as semi-transparent squares; the _focusables_ are circles. -A `NavFence` keeps track of the Entity `Focusable` that leads to it (in the -example, the "soul" tab entity is tracked by the "soul menu" `NavFence`). +A `NavMenu` keeps track of the Entity `Focusable` that leads to it (in the +example, the "soul" tab entity is tracked by the "soul menu" `NavMenu`). -The `NavFence` isolates the set of `Focusable`s of a single menu: +The `NavMenu` isolates the set of `Focusable`s of a single menu: ![Graph menu layout](https://user-images.githubusercontent.com/26321040/140716920-fd298afb-093b-47f9-8309-c4c354c3d40f.png) (red-orange represents _focused_ elements, gold are _active_ elements) -In bevy, a `NavFence` might be nothing more than a component to add to the +In bevy, a `NavMenu` might be nothing more than a component to add to the `NodeBundle` parent of the _focusables_ you want in your menu. On top of that, it needs to track which _focusable_ you _activated_ to enter this menu: ```rust #[derive(Component)] -struct NavFence { +struct NavMenu { parent_focusable: Option, } ``` -### `NavFence` settings +### `NavMenu` settings -#### Sequence menu +#### Scope menu However, it's not enough to be able to go up and down the menu hierarchy, we also want to be able to directly "speak" with the tabs menu and switch from the @@ -199,12 +199,12 @@ press `B` twice to get to the tabs menu. The tabs menu is a special sort of menu, with movement that can be triggered from another menu reachable from it. We must add a `setting: NavSetting` field -to the `NavFence` component to distinguish this sort of menu. In our -implementation we call them `sequence menu`s. +to the `NavMenu` component to distinguish this sort of menu. In our +implementation we call them `scope menu`s. -The `NavRequest::MenuMove` request associated with `RT`/`LT` (or anything the +The `NavRequest::ScopeMove` request associated with `RT`/`LT` (or anything the developer sets) will climb up the _menu tree_ to find the enclosing -_sequence menu_ and move focus within that menu. +_scope menu_ and move focus within that menu. #### Looping @@ -238,7 +238,7 @@ menus](https://user-images.githubusercontent.com/26321040/141672033-8bd43660-78a ### Algorithm -The algorithm is the `resolve` function in [ui-navigation](https://github.com/nicopap/ui-navigation/blob/master/src/lib.rs#L340). +The algorithm is the `resolve` function in [ui-navigation](https://github.com/nicopap/ui-navigation/blob/master/src/lib.rs#L414). The meat of it is 50 lines of code. The current implementation has an example demonstrating all features we @@ -282,32 +282,20 @@ Gregory) doesn't mention ui navigation systems. ## Open questions -### Naming - -* `NavFence`: If a `NavFence` is pretty much just an abstraction to describe a - menu, why not call it a `NavMenu` instead? -* `sequence menu`: doesn't express it's a menu that can be changed with special - `NavRequest::MenuMove` even when focusing in another menu. -* `NavRequest::MenuMove`: bad naming, given everything is moving within a menu -* `NavEvent::Caught`: doesn't really express that it's just a `NavRequest` that - led to no change in focus - ### `NavRequest` system I wanted to defer the raw input processing to the user. The current -implementation **requires** the user to specify the input button mapping to the -`NavRequest`s. However, if we want to provide an out-of-the-box working -experience, we should have at least a default mapping that can be modified at -runtime. +implementation provides default systems to manage ui input, with a +`InputMapping` resource to customize a little bit the system's behaviors. -I think it makes sense to have a default mapping, as technically it's already -the case with the `Interaction` component. And having something working -out of the box is a huge positive. +We **requires** the user to explicitly add the input systems to their `App` +rather than adding them to our `NavigationPlugin` because we **need** the +flexibility to swap out the default input handling after a certain point in the +game development phase (as my personal practical experience shows). -I am against an inflexible system where it is impossible to change the input -mappings, any respectable game let you change the input mapping. And if not, -your players start very quickly to ask for one! So it is necessary to be able -to remap the buttons. +Though this is a very basic solution that screams "temporary", and should +probably in the future be swapper in favor of something more thought out and +ergonomic. How would that work? Is there already examples in bevy where we provide default button mappings that can be changed? @@ -315,9 +303,18 @@ button mappings that can be changed? I think we should think deeply about input handling in default plugins. However, this is out of scope for this RFC. + +### Mouse hover behavior + +I went the easy route when implementing mouse support, of having the focus +follow the mouse. In certain circumstances this is not wishable, but currently +I don't see an alternative that doesn't need to completely reinvent a new +system on top of the current one. + + ### Game-oriented assumptions -This was not intended during implementation, but in retrospect, the `NavFence` +This was not intended during implementation, but in retrospect, the `NavMenu` system is relatively opinionated. Indeed, we assume we are trying to build a game UI with a series of "menus" and @@ -332,7 +329,7 @@ limitation. ![Krita UI screenshot](https://user-images.githubusercontent.com/26321040/141671939-24b8a7c3-b296-4fd4-8ae0-3bbe7fe4c9a3.png) -In this example, we can imagine the root `NodeBundle` having a `NavFence` +In this example, we can imagine the root `NodeBundle` having a `NavMenu` component, and the elements highlighted in yellow have a `Focusable` component. This way we create a menu to navigate between dockers. This is perfectly consistent with our design. @@ -346,12 +343,12 @@ I think the most problematic aspect of this implementation is that we push on the user the responsibility of upholding invariants. This is not a safety issue, but it leads to panics. The invariants are: 1. Never create a cycle of menu connections -2. There must be at most one root `NavFence` -3. Each `NavFence` must at least have one contained `Focusable` +2. There must be at most one root `NavMenu` +3. Each `NavMenu` must at least have one contained `Focusable` The upside is that the invariants become only relevant if the user _opts into_ -`NavFence`. Already, this is a choice made by the user, so they can be made -aware of the requirements in the `NavFence` documentation. +`NavMenu`. Already, this is a choice made by the user, so they can be made +aware of the requirements in the `NavMenu` documentation. If the invariants are not upheld, the navigation system will simply panic. I didn't test enough the current implementation to see if all cases of panics @@ -369,10 +366,10 @@ The end user will have to deal with new concepts introduced by this RFC: * `focused` * `NavRequest` * `NavEvent` -* `NavFence` +* `NavMenu` * `Focusable` -* `looping` -* `sequence menu` +* `cycling` +* `scope menu` This seems alright, although `dormant` may be misleading, given it's a synonym to "inactive". It is technically an antonym to "active", but in the current @@ -382,19 +379,25 @@ instead. On the implementation side. We have `inert` which is synonym to `dormant` and `inactive` but is just a word to fill the roll of "none of the above". -I think the one term that can be improved is `NavFence`. Replacing it with +I think the one term that can be improved is `NavMenu`. Replacing it with `NavMenu` constructs on pre-existing knowledge and is a good approximation of what it represents. ### Performance The navigation system might be slow with a very large amount of `Focusable` -elements. **I didn't benchmark it**. +elements. **I didn't benchmark it**. In fact the `ultimate_menu_navigation.rs` +example is laggy. Especially visible with mouse focus. I'm not sure why. It +seems to be because of the large number of `Focusable` and doing geometry +computation on all of them (which is necessary to find the next element to +focus or check which element is under the cursor) Note that the current implementation only performs computation on `NavRequest` -events. +events. The input systems might add their own processing. I tried to minimise +as much as possible the `default_mouse_input` runtime, but it still seem quite +laggy. -We might improve the performance with well designed caches and the addition of -the `child_of_interest` field to `NavFence` (which is planned). +We might improve the performance with well designed 2d navigation caches. But +i'm not sure how this would look like. [^1]: ie: includes children, grand-children, grand-grand-children etc. From 73a8c2ad4b5d6b4b89d9f09d500ce31550c85e74 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 12:04:53 +0200 Subject: [PATCH 06/33] Rework ui navigation RFC --- rfcs/41-ui-navigation.md | 510 +++++++++++++++++---------------------- 1 file changed, 219 insertions(+), 291 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 31902f15..c149c883 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -2,264 +2,248 @@ ## Summary -Modification of the `Interaction` component to support keyboard & controller -based navigation of UI on top of mouse-oriented navigation. In the rest of this -RFC, consider the already existing "`Interaction`" bevy component as being a good -candidate for being replaced by what we describe as "`Focusable`". +Introduce [`bevy-ui-navigation`] into the bevy tree. -The new navigation system by default enables unrestricted navigation between -all `Focusable` entities. But the developer may chose to opt in to a more -complex navigation scheme by adding `NavMenu`, encapsulating groups of -`Focusable`s into their own menus. The `NavMenu` component can also express -the navigation scheme within the menus. - -The goal is to make the easiest navigation scheme work out of the box and the -more complex schemes easy or at least possible to implement. +By default this, amounts to replacing `Interaction` with `Focusable` in `bevy_ui`. +On top of the current behavior, this adds the following capabilities: +- Focus management for **gamepads** +- Decouples input from focus management +- Decouples focus management from `bevy_ui` +- User-specified input handling for UI interaction +- An optional menu tree management system ## Terminology -* A **`NavRequest`** is the sum of all possible ui-related user inputs. It's what - the algorithm takes as input, on top of the _navigation tree_ to figure out - the next focused element. -* A **`Focusable`** is an Entity that can be navigated-to and highlighted (focused) - with a controller or keyboard -* A **`NavMenu`** adds "scope" to `Focusable`s. Navigation between `Focusable` - entities will be constrained to other `Focusable` that are children of the - same `NavMenu`. It creates a self-contained menu. **A `NavMenu` may therefore - be better described as a _menu_**. Note that the `Focusable`s do not need to - be direct children of the `NavMenu`, only transitive children[^1] -* The **menu tree** (sometimes _navigation tree_) is the logical relationship - between all `NavMenu`s. -* There is a single **focused** Entity, which is the currently highlighted/used - Entity. -* There can be several **active** elements, they represent the "path" through the - _navigation tree_ to the current _focused_ element. Which focusable in the - previous menus were activated in order to reach the menu we are in currently. -* A **dormant** element is a previously active element from a branch of the - menu tree that is currently not focused. -* The navigation algorithm **resolves** a `NavRequest` by finding which `Entity` - to next focus to -* **focus memory** is what, for example, makes it possible in your text editor - to tab to a different file, edit it, and tab back to the file you were - previously editing and find yourself at the exact location where you left it. - It is the ability for the software to keep track where you were at. - **dormant** elements are necessary to track what was previously focused and - come back to it when needed +- _menu tree_: the access hierarchy of menus. +TODO: illustration +- _activating_ a focusable: Sending the `Action` event while this focusable is focused. +- _focused menu_: The menu in which the currently focused `Focusable` is. +- _child menu_ (of a focusable): The menu accessed from this focusable. ## Motivation -Currently, the developer must themselves write the relationship between -various buttons in the menus and specify which button press leads to other menu -items, etc. Or more simply, limit themselves to pointer-based navigation to -avoid having to think about menu navigation. +`bevy_ui` interaction story is crude. +The only way to interact with the UI currently +is through the `Interaction` and `FocusPolicy` components. +`Interaction` only supports mouse or touch-based input. +We want gamepad supports and more game-oriented functionalities, +such as menu trees and movement between menus. -By including navigation-related components such as `Focusable` and `NavMenu`, -the developer can quickly get a controller-navigable menu up and running. +The current UI focus system is limited to `bevy_ui` and +can't be swapped out for something the user deems more appropriate. -There already exists components with similar purposes in Bevy: -[`Interaction` and `FocusPolicy`](https://github.com/bevyengine/bevy/blob/main/crates/bevy_ui/src/focus.rs#L14). -However, it only supports a barebone mouse-oriented navigation scheme. And the -focus handling is not even implemented. +This is in contradiction with the general philosophy of bevy, +which is of providing to the user all the tools necessary to create something +that fits their needs optimally. -My proposal, on top of making it easy to implement controller navigation of -in-game UIs, enables keyboard navigation and answers accessibility concerns -such as the ones -[raised in this issue](https://github.com/bevyengine/bevy/issues/254). +This RFC proposes to resolve all those questions. +[`bevy-ui-navigation`] provides a highly customizable ECS-based navigation engine. ## User-facing explanation -Examples and an implementation-oriented description is available in the -[ui-navigation plugin README](https://github.com/nicopap/ui-navigation/blob/master/Readme.md). - -To interact in real-time with the UI, the developer emits `NavRequest`s events. -The navigation system responds with `NavEvent` events, such as -`NavEvent::FocusChanged` or `NavEvent::Caught`. - -The navigation system is the only one to handle the focus state of the UI. The -user interacts with it strictly by sending `NavRequest`s, reading `NavEvent` or -using the `Focusable` methods. Typically in a system with a -`Query<&Focusable, Changed>` parameter. - -Rather than directly reacting to input, we opt to make our system reactive to -`NavRequest`, this way the developer is free to chose which button does what in -the game. The developer also has the complete freedom to swap at runtime what -sends or how those `NavRequest` are generated. - -### Example +The [bevy-ui-navigation README][`bevy-ui-navigation`] is a good start. +To summarize, here are the headlines: +- The navigation system can only be interacted with with events: + `NavRequest` as input and `NavEvent` as output. +- `NavEvent::FocusChanged` contains the list of entities that are traversed + to go from the last focus position to the new one. +- ui-navigation provides default input systems, + but it's possible to disable them and instead use custom ones. + This is why we restrict interactions with navigation to events. +- ui-navigation provides a system to declare buttons as "leading to menu X" + in order to create menu trees. +- ui-navigation defines [custom `SystemParam`][event_helpers]s + to ease reaction to events (such a button press or menu change). + The most naive usage results into [elm architecture][elm_architecture]-like code. + Systems updating the model (ECS hierarchy) and systems reacting to events. +- It is possible to create isolated menus by using the `NavMenu` component. +- All `Focusable` children in the `NavMenu` entity's tree will be part + of this menu, and moving using a gamepad from one focusable in a menu + can only lead to another focusable within the same menu. + You can specify a focusable in any other menu that will lead to this + one once activated. + +## Implementation + +Since `bevy-ui-navigation` already exists, +we will discuss its design, it's functionalities and their uses. + +Implementation details are already covered by the implementation, +which btw is fully documented (including internal architecture). + +### Exposed API + +The crux of `bevy-ui-navigation` are the following types: +- `Focusable` component + - `lock`: activating it will lock the navigation engine and discard all `NavRequest`. + - `cancel`: activating it will do the equivalent of receiving a `Cancel` request. + - `dormant`: this will be the preferred entity to focus when entering a menu. +- `NavMenu` enum + - `Wrapping**`/`Bound**`: navigation can wrap + if directional sticks are pressed in a direction there are no focusables in + - `**Scope`/`**2d`: A scopped menu is tab menu navigable with special hotkeys. +- `NavRequest` event + - `Action`: equivalent to left click or `A` button on controller + - `Cancel`: backspace or `B` on controller + - `FocusOn`: request to move the focus to a specific `Entity` + - `Move(Direction)`: directional gamepad or keyboard input + - `ScopeMove(ScopeDirection)`: tab menu hotkey +- `NavEvent` event + - `InitiallyFocused`: We went from 0 focused to 1 focused element + - `NoChanges`: The `NavRequest` did not change the focus state. + - `FocusChanged`: The `NavRequest` changed the focus state, + has the list of items that were made inactive + and items made active. + - `Locked`: The `NavRequest` caused the focus system to lock. + - `Unlocked`: Something unlocked the focus system. +- `MoveParam` trait + - Used to pick a focusable in a provided direction + - The `NavigationPlugin` is generic over the `MoveParam` + - `DefaultNavigationPlugins` provides a default implementation of `MoveParam` + +### `Focusable` + +A `Focusable` is an entity that can be focused. +For gamepad-based navigation, +this requires a way to pick a focusable in a provided direction. +This is delegated to the UI library implementation +through the `MoveParam` trait. +In the case of bevy, `bevy_ui` is the UI library implementation, +and the `GlobalTransform` is used to locate focusables in regard to each-other. + +The `Focusable` component holds state about what happens when it is activated +and what focus state it is in. (focused, active, inactive, etc.) + +#### Note on hovering state + +Since navigation is completely decoupled from input and ui lib, +it is impossible for it to tell whether a focusable is hovered. +To keep the hovering state functionality, +it will be necessary to add it as an independent component +separate from the navigation library. + +### Input customization + +`ui-navigation` can be added to the app in two ways: +1. With `NavigationPlugin` +2. With `DefaultNavigationPlugins` + +(1) only inserts the navigation systems and resources, +while (2) also adds the input systems generating the `NavRequest`s +for gamepads, mouse and keyboard. + +This enables convenient defaults +while letting users insert their own custom input logic. + + +### What to focus first? + +Gamepad input assumes an initially focused element +for navigating "move in direction" or "press action". +At the very beginning of execution, there are no focused element, +so we need fallbacks. +By default, it is any `Focusable` when none are focused yet. +But the user can spawn a `Focusable` as `dormant` +and the algorithm will prefer it when no focused nodes exist yet. -Let's start with a practical example, and try figure out how we could specify -navigation in it: - -![Typical RPG tabbed menu](https://user-images.githubusercontent.com/26321040/140716885-eb089626-21d2-4711-a0c9-cf185bc0f19a.png) - -The tab "soul" and the panel "abc" are highlighted to show the player what menu -they are down, they are **active** elements. The circle "B" is highlighted -differently to show it is the **focused** element. - -You can move `UP`/`DOWN` in the submenu, but you can also use `LT`/`RT` to -go from one submenu to the other (the _soul_, _body_ etc tabs). In code, we -might react to focus changes (for example to change the color of -focused/unfocused elements) as follow: ```rust -fn setup_ui() { - // setup UI like you would setup UI in bevy today -} -fn handle_nav_events(mut events: EventReader, game: Res) { - use bevy_ui_navigation::{NavEvent::Caught, NavRequest::Action}; - for event in events.iter() { - match event { - Caught { from, request: Action } if from.first() == game.start_game_button => { - // Start the game on "A" or "ENTER" button press - } - _ => {} - } - } -} -fn button_system( - materials: Res, - mut focusables: Query<(&Focusable, &mut Handle), Changed>, -) { - for (focus_state, mut material) in focusables.iter_mut() { - if focus_state.is_focused() { - *material = materials.focused.clone(); - } else { - *material = materials.inert.clone(); - } - } -} +self.focusables + .iter() + // The focused focusable. + .find_map(|(e, focus)| (focus.state == Focused).then(|| e)) + // Dormant focusable within the root menu (if it exists) + .or_else(root_dormant) + // Any dormant focusable + .or_else(any_dormant) + // Any focusable + .or_else(fallback) ``` -## Implementation strategy - -### Multiple menus - -The first intuition is that this screenshot shows three menus: -* The "tabs menu" containing _soul_, _body_, _mind_ and _all_ "tabs" -* The "soul menu" containing _gdk_, _kfc_ and _abc_ "panels" -* The "ABC menu" containing a, b, c, A, B, C "buttons" - -We expect to be able to navigate the "tabs menu" **from anywhere** by pressing -`LT`/`RT`. (or `TAB`,`SHIFT+TAB` on keyboard) We expect to be able to move -between buttons within the "ABC menu" with arrow keys, same with the "soul menu". - -Finally, we expect to be able to go from one menu to another by pressing `A` -when the corresponding element in the parent menu is _focused_ or go back to -the parent menu when pressing `B`. (`ENTER` and `BACKSPACE` on keyboard) - -### Navigating within a single menu - -Our game engine already knows the position of our UI elements. We can use this -as information to deduce the next _focused_ element when pressing an arrow key. - -In fact, if we have only a single menu on screen, we could stop at this point, -since there is no restrictions on navigation between `Focusable`s. - -### Specifying navigation between menus - -But we have more than one menu. We can't magically infer what the developer -thinks is a menu, this is why we need `NavMenu`s. - -In our example, we want to be able to go from "soul menu" to "ABC menu" with -`A`/`B` (Action, Cancel). `Action` let you enter the related submenu, while -`Cancel` does the opposite. -With `NavMenu`s the menu layout may look like this: +### Menu navigation handling -![The previous tabbed menu example, but with _menus_ and _focusables_ highlighted](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) +The public API has `NavMenu`, but we use internally `TreeMenu`, +this prevents end users from breaking assumptions about the menu trees. -The _menus_ are represented as semi-transparent squares; the _focusables_ -are circles. +We define the `TreeMenu` component. +All `Focusable` direct or indirect children of a `TreeMenu` are part of that menu. +A `TreeMenu` has a `Focusable` focus parent, +when that `Focusable` is focused and the `Action` request is received, +the focus changes to the `TreeMenu` -A `NavMenu` keeps track of the Entity `Focusable` that leads to it (in the -example, the "soul" tab entity is tracked by the "soul menu" `NavMenu`). +In short, the navigation tree is built on two elements: +- The `reachable_from` that a `NavMenu` was spawned with +- The `Focusable` within the hierarchy of a `NavMenu` -The `NavMenu` isolates the set of `Focusable`s of a single menu: +This is a tree in the ECS. -![Graph menu layout](https://user-images.githubusercontent.com/26321040/140716920-fd298afb-093b-47f9-8309-c4c354c3d40f.png) -(red-orange represents _focused_ elements, gold are _active_ elements) +In the following screenshot, `Focusable`s are represented as circles, and menus +as rectangles. -In bevy, a `NavMenu` might be nothing more than a component to add to the -`NodeBundle` parent of the _focusables_ you want in your menu. On top of that, -it needs to track which _focusable_ you _activated_ to enter this menu: -```rust -#[derive(Component)] -struct NavMenu { - parent_focusable: Option, -} -``` - -### `NavMenu` settings - -#### Scope menu - -However, it's not enough to be able to go up and down the menu hierarchy, we -also want to be able to directly "speak" with the tabs menu and switch from the -"soul menu" to the "body menu" with a simple press of `RT`, not having to -press `B` twice to get to the tabs menu. +![A screenshot of a RPG-style menu with the navigation tree overlayed](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) -The tabs menu is a special sort of menu, with movement that can be triggered -from another menu reachable from it. We must add a `setting: NavSetting` field -to the `NavMenu` component to distinguish this sort of menu. In our -implementation we call them `scope menu`s. +A `Focusable` used to go from the root menu +to the menu in which the currently focused entity is is _dormant_. +The current focus position is represented as a trail of breadcrumb dormant +focusables and the currently focused focusable. -The `NavRequest::ScopeMove` request associated with `RT`/`LT` (or anything the -developer sets) will climb up the _menu tree_ to find the enclosing -_scope menu_ and move focus within that menu. +This is reflected into the `NavEvent::FocusChanged` event. +When going from a focusable to another, +the navigation system emits a `FocusChanged` event, +the event contains two lists of entities: +- The list of entities that after the navigation event are no more focused + or dormant +- The list of entities that are newly dormant or focused -#### Looping +For the simplest case of navigating from one focusable to another +in the same menu, the two list have a single element, +so it is capital that it is easy to access those elements. +This is why we use a `NonEmpty`, from the `non_empty_vec` crate. +A `NonEmpty` has a `first` method that returns a `T`, +which in our case would be the last and new focused elements. +I chose this crate over similar ones, +because the code is dead simple and easy to vet. -We may also chose to lock or unlock the ability to wrap back right when going -left from the leftmost element of a menu, same with any other directions. For -example pressing `DOWN` when "c" is _focused_ could change focus to "a". We -should use the `setting` field to keep track of this as well. -### Navigation tree +#### Spawning menus -All in all, the navigation tree is just the specification of which menu is -reachable from where. Not the relation between each individual focusable. This -drastically simplifies the structure of the tree. Because a game may have many -interactive UI elements, but not that many menus. +Menus store a reference to their parent, +but that parent is not necessarily their hierarchical parent. +The parent is just a button in another menu. +It is inconvenient to have to pre-spawn each button to acquire their `Entity` id +just to be able to spawn the menu you'll get to from that button. +`ui-navigation` uses a proxy component holding both the parent and the menu state info. +That proxy component is `MenuSeed`, +you can initialize it either with the `Name` or `Entity` of the parent focusable of the menu. +A system will take all `MenuSeed` components and replace them with a `TreeMenu`. -The previous graph diagram may have been a bit misleading. It's not the real -_navigation graph_ (that we introduced as _menu graph_ for good reasons). -The tree that the algorithm deals with and traverse is between menus -rather than `Focusable` elements. The graph as it is used in our code is the -following: +This introduces a single frame lag on menu spawn, +but it improves ergonomics to users. +I thought it was a pretty good trade-off, +since spawning menus is not time critical, unlike input. -![RPG menu demo split between menus with pointers leading to the "parent -focusable"](https://user-images.githubusercontent.com/26321040/141672009-e023855e-10a7-4c50-a8dd-eff10acb2208.png) -We may also imagine an implementation where the non-selected menus are hidden -but loaded into the ECS. In this case, the navigation tree might look like -this: +#### cancel events -![RPG menu graph without focusables, only -menus](https://user-images.githubusercontent.com/26321040/141672033-8bd43660-78af-4521-b367-03c7270c58c5.png) +When entering a menu, you want a way to leave it and return to the previous menu. +To do this, we have two options: +- The `Cancel` request +- The `cancel` focusable -### Algorithm +This will change the focus to the parent button of the focused menu. -The algorithm is the `resolve` function in [ui-navigation](https://github.com/nicopap/ui-navigation/blob/master/src/lib.rs#L414). -The meat of it is 50 lines of code. +#### Moving into a menu -The current implementation has an example demonstrating all features we -discussed in this RFC. For a menu layout as follow: +When the `Action` request is sent while a `Focusable` with a child menu is focused, +the focus will move to the child menu, +selecting the focusable to focus +with a heuristic similar to the initial focused selection. -![Example layout](https://user-images.githubusercontent.com/26321040/141671978-a6ef0e99-8ee1-4e5f-9d56-26c65f748331.png) +### Locking -We can navigate it with a controller as follow: - -![Example layout being navigated with a controller, button presses shown as -overlay](https://user-images.githubusercontent.com/26321040/141612751-ba0e62b2-23d6-429a-b5d1-48b09c10d526.gif) - -### UI Benchmarks discussion - -As mentioned in [this RFC](https://github.com/alice-i-cecile/rfcs/blob/ui-central-dogma/rfcs/ui-central-dogma.md), -we should benchmark our design against actual UI examples. I think that the -one used in this RFC as driving example (the RPG tabbed menu) illustrates well -enough how we could implement Benchmark 2 (main menu) using our system. -Benchmark 1 is irrelevant. Benchmark 3 could potentially be solved by just -marking the interactive element with the `Focusable` component and nothing -else, so that every element can be reached from anywhere. +To ease implementation of widgets, such as sliders, +some focusables can "lock" the UI, preventing all other form of interactions. +Another system will be in charge of unlocking the UI by sending a `NavRequest::Free`. ## Prior art @@ -288,30 +272,6 @@ I wanted to defer the raw input processing to the user. The current implementation provides default systems to manage ui input, with a `InputMapping` resource to customize a little bit the system's behaviors. -We **requires** the user to explicitly add the input systems to their `App` -rather than adding them to our `NavigationPlugin` because we **need** the -flexibility to swap out the default input handling after a certain point in the -game development phase (as my personal practical experience shows). - -Though this is a very basic solution that screams "temporary", and should -probably in the future be swapper in favor of something more thought out and -ergonomic. - -How would that work? Is there already examples in bevy where we provide default -button mappings that can be changed? - -I think we should think deeply about input handling in default plugins. -However, this is out of scope for this RFC. - - -### Mouse hover behavior - -I went the easy route when implementing mouse support, of having the focus -follow the mouse. In certain circumstances this is not wishable, but currently -I don't see an alternative that doesn't need to completely reinvent a new -system on top of the current one. - - ### Game-oriented assumptions This was not intended during implementation, but in retrospect, the `NavMenu` @@ -334,6 +294,16 @@ component, and the elements highlighted in yellow have a `Focusable` component. This way we create a menu to navigate between dockers. This is perfectly consistent with our design. +### (Solved, but worth asking) Moving UI camera & 3d UI + +The completely decoupled implementation of the navigation system +enables user to implement their own UI. +It is perfectly possible to add a `Focusable` component to a 3d object +and, for example, provide a [mouse picking] based input system +sending the `NavRequest`, +while using `world_to_viewport` in `MoveParam` for gamepad navigation. +(or completely omitting gamepad support by providing a stub `MoveParam`) + ## Drawbacks and design limitations @@ -343,61 +313,19 @@ I think the most problematic aspect of this implementation is that we push on the user the responsibility of upholding invariants. This is not a safety issue, but it leads to panics. The invariants are: 1. Never create a cycle of menu connections -2. There must be at most one root `NavMenu` -3. Each `NavMenu` must at least have one contained `Focusable` +2. Each `NavMenu` must at least have one contained `Focusable` The upside is that the invariants become only relevant if the user _opts into_ `NavMenu`. Already, this is a choice made by the user, so they can be made aware of the requirements in the `NavMenu` documentation. -If the invariants are not upheld, the navigation system will simply panic. I -didn't test enough the current implementation to see if all cases of panics -lead to meaningful error messages. +If the invariants are not upheld, the navigation system will simply panic. It may be possible to be resilient to the violation of the invariants, but I think it's better to panic, because violation will most likely come from misunderstanding how the navigation system work or programming errors. -### Jargon - -The end user will have to deal with new concepts introduced by this RFC: -* `dormant` -* `active` -* `focused` -* `NavRequest` -* `NavEvent` -* `NavMenu` -* `Focusable` -* `cycling` -* `scope menu` - -This seems alright, although `dormant` may be misleading, given it's a synonym -to "inactive". It is technically an antonym to "active", but in the current -design, it is not really the case. Beside I have a tendency to type "doormat" -instead. - -On the implementation side. We have `inert` which is synonym to `dormant` and -`inactive` but is just a word to fill the roll of "none of the above". - -I think the one term that can be improved is `NavMenu`. Replacing it with -`NavMenu` constructs on pre-existing knowledge and is a good approximation of -what it represents. - -### Performance - -The navigation system might be slow with a very large amount of `Focusable` -elements. **I didn't benchmark it**. In fact the `ultimate_menu_navigation.rs` -example is laggy. Especially visible with mouse focus. I'm not sure why. It -seems to be because of the large number of `Focusable` and doing geometry -computation on all of them (which is necessary to find the next element to -focus or check which element is under the cursor) - -Note that the current implementation only performs computation on `NavRequest` -events. The input systems might add their own processing. I tried to minimise -as much as possible the `default_mouse_input` runtime, but it still seem quite -laggy. - -We might improve the performance with well designed 2d navigation caches. But -i'm not sure how this would look like. - -[^1]: ie: includes children, grand-children, grand-grand-children etc. +[mouse picking]: https://github.com/aevyrie/bevy_mod_picking/ +[`bevy-ui-navigation`]: https://github.com/nicopap/ui-navigation +[event_helpers]: https://docs.rs/bevy-ui-navigation/0.18.0/bevy_ui_navigation/event_helpers/index.html#examples +[elm_architecture]: https://guide.elm-lang.org/architecture/ From b8d858b9d919d8239b8712f4a76d60e617857988 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 12:06:10 +0200 Subject: [PATCH 07/33] Remove TODO --- rfcs/41-ui-navigation.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index c149c883..70eb9d1c 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -15,7 +15,6 @@ On top of the current behavior, this adds the following capabilities: ## Terminology - _menu tree_: the access hierarchy of menus. -TODO: illustration - _activating_ a focusable: Sending the `Action` event while this focusable is focused. - _focused menu_: The menu in which the currently focused `Focusable` is. - _child menu_ (of a focusable): The menu accessed from this focusable. From d2d2ee9a6782df0a1f88c02decfb993fb200b536 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 13:25:24 +0200 Subject: [PATCH 08/33] Correct dormant/active mixup in navigation handling chapter --- rfcs/41-ui-navigation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 70eb9d1c..f3b0108a 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -182,8 +182,8 @@ as rectangles. ![A screenshot of a RPG-style menu with the navigation tree overlayed](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) A `Focusable` used to go from the root menu -to the menu in which the currently focused entity is is _dormant_. -The current focus position is represented as a trail of breadcrumb dormant +to the menu in which the currently focused entity is is _active_. +The current focus position is represented as a trail of breadcrumb active focusables and the currently focused focusable. This is reflected into the `NavEvent::FocusChanged` event. @@ -191,8 +191,8 @@ When going from a focusable to another, the navigation system emits a `FocusChanged` event, the event contains two lists of entities: - The list of entities that after the navigation event are no more focused - or dormant -- The list of entities that are newly dormant or focused + or active +- The list of entities that are newly active or focused For the simplest case of navigating from one focusable to another in the same menu, the two list have a single element, From c736e489aebb010d0c2bb3555b42ea9cadf319db Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 13:32:09 +0200 Subject: [PATCH 09/33] Add limitation about spawn/despawn menus --- rfcs/41-ui-navigation.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index f3b0108a..7bb6c30f 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -306,6 +306,29 @@ while using `world_to_viewport` in `MoveParam` for gamepad navigation. ## Drawbacks and design limitations +### How does this work with the spawn/despawn workflow? + +The current design imposes to the user that all UI nodes such a buttons and +menus must be already loaded, even if not visible on screen. +This is to support re-focusing the last focused element when moving between +menus. +The design of this RFC _hinges on_ the whole tree been loaded in ECS while +executing navigation requests, and would work poorly if menus were spawned and +despawned dynamically. +From what I gather from questions asked in the `#help` discord channel, +I think the design most people come up with naturally is to spawn and despawn +dynamically the menus as they are traversed, which is incompatible with +this design. +However, in practice, in my own games, I worked around it by expanding the +whole UI tree beyond the UI camera view and moving around the camera. +An alternative would simply to set the `style.display` of non-active menus to +`None` and change the style when focus enters them. +This also fixes the already-existing 1 frame latency issue with the +spawn/despawn design. + +Something that automates that could be a natural extension of the exposed API. +It may also help end users go with the best design first. + ### User upheld invariants I think the most problematic aspect of this implementation is that we push on From bb400a4e09ade59c12ae79b8486ee10ba3de0555 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 13:47:43 +0200 Subject: [PATCH 10/33] Add precisions about dormant focusables --- rfcs/41-ui-navigation.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 7bb6c30f..5a006f9a 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -203,6 +203,14 @@ which in our case would be the last and new focused elements. I chose this crate over similar ones, because the code is dead simple and easy to vet. +#### Going back to a previous menus + +`Focusable`s have a `dormant` state that is set when they go from +active/focused to not active. +This is a form of memory that allows focus to go back to the last focused +element within a menu when it is re-visited. +This is also the mechanism used to let the user decide which focusable to focus +when none are focused yet. #### Spawning menus From 0bbc2bcdfe4469e1ce0d0606bf34db0bc5681710 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Tue, 19 Jul 2022 15:54:40 +0200 Subject: [PATCH 11/33] Add precision about locking --- rfcs/41-ui-navigation.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 5a006f9a..6fc5f1ed 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -252,6 +252,9 @@ To ease implementation of widgets, such as sliders, some focusables can "lock" the UI, preventing all other form of interactions. Another system will be in charge of unlocking the UI by sending a `NavRequest::Free`. +This may be better served by keeping the `FocusPolicy` component, +in situations where a global lock prevents other uses. + ## Prior art From 3b3ffb70fa4b6b436b6052b35c80de9a08647798 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Thu, 21 Jul 2022 20:48:18 +0200 Subject: [PATCH 12/33] Remove reference to event_helpers --- rfcs/41-ui-navigation.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 6fc5f1ed..bfa536fc 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -51,10 +51,6 @@ To summarize, here are the headlines: This is why we restrict interactions with navigation to events. - ui-navigation provides a system to declare buttons as "leading to menu X" in order to create menu trees. -- ui-navigation defines [custom `SystemParam`][event_helpers]s - to ease reaction to events (such a button press or menu change). - The most naive usage results into [elm architecture][elm_architecture]-like code. - Systems updating the model (ECS hierarchy) and systems reacting to events. - It is possible to create isolated menus by using the `NavMenu` component. - All `Focusable` children in the `NavMenu` entity's tree will be part of this menu, and moving using a gamepad from one focusable in a menu @@ -360,5 +356,3 @@ misunderstanding how the navigation system work or programming errors. [mouse picking]: https://github.com/aevyrie/bevy_mod_picking/ [`bevy-ui-navigation`]: https://github.com/nicopap/ui-navigation -[event_helpers]: https://docs.rs/bevy-ui-navigation/0.18.0/bevy_ui_navigation/event_helpers/index.html#examples -[elm_architecture]: https://guide.elm-lang.org/architecture/ From 608371c11f0445584ed20ed4fa705fae094cde02 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 12:27:35 +0200 Subject: [PATCH 13/33] Add more precision about use and implementation --- rfcs/41-ui-navigation.md | 552 ++++++++++++++++++++++++++++++++------- 1 file changed, 451 insertions(+), 101 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index bfa536fc..9806ae41 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -18,6 +18,10 @@ On top of the current behavior, this adds the following capabilities: - _activating_ a focusable: Sending the `Action` event while this focusable is focused. - _focused menu_: The menu in which the currently focused `Focusable` is. - _child menu_ (of a focusable): The menu accessed from this focusable. +- _transitive_: If `A` is _foo_ of `B` (swap _foo_ with anything such as + _child_, _left_ etc.) + Then `A` is _transitive foo_ of `B` if there is any chain of `C`, `D`, + `E` etc. such as `A` is _foo_ `C`, `C` is _foo_ `D`, ... , `E` is _foo_ `B`. ## Motivation @@ -31,16 +35,24 @@ such as menu trees and movement between menus. The current UI focus system is limited to `bevy_ui` and can't be swapped out for something the user deems more appropriate. -This is in contradiction with the general philosophy of bevy, -which is of providing to the user all the tools necessary to create something -that fits their needs optimally. - -This RFC proposes to resolve all those questions. -[`bevy-ui-navigation`] provides a highly customizable ECS-based navigation engine. - ## User-facing explanation The [bevy-ui-navigation README][`bevy-ui-navigation`] is a good start. + +The API exposed in this RFC are relatively "low level". +Ideally, we provide default wrapping systems to simplify using the focus +system. + +However, it's completely open for the user to plug their own input +and focus handling. +[`bevy-ui-navigation`] provides its own wrapper around `NavEvent`, +called [`event_helpers`]. +However, [`event_helpers`] is not included in this RFC, +because it is a very opinionated API. +It is better left to a different PR where the contentious points +can be solved independently of the rest of the navigation system. + + To summarize, here are the headlines: - The navigation system can only be interacted with with events: `NavRequest` as input and `NavEvent` as output. @@ -49,8 +61,6 @@ To summarize, here are the headlines: - ui-navigation provides default input systems, but it's possible to disable them and instead use custom ones. This is why we restrict interactions with navigation to events. -- ui-navigation provides a system to declare buttons as "leading to menu X" - in order to create menu trees. - It is possible to create isolated menus by using the `NavMenu` component. - All `Focusable` children in the `NavMenu` entity's tree will be part of this menu, and moving using a gamepad from one focusable in a menu @@ -58,107 +68,313 @@ To summarize, here are the headlines: You can specify a focusable in any other menu that will lead to this one once activated. -## Implementation +Following, a detailed tour. -Since `bevy-ui-navigation` already exists, -we will discuss its design, it's functionalities and their uses. +### Using a `Focusable` -Implementation details are already covered by the implementation, -which btw is fully documented (including internal architecture). +See `bevy-ui-navigation` [`Focusable`] doc. -### Exposed API +The `Interaction` component in `ButtonBundle` is replaced by `Focusable`. -The crux of `bevy-ui-navigation` are the following types: -- `Focusable` component - - `lock`: activating it will lock the navigation engine and discard all `NavRequest`. - - `cancel`: activating it will do the equivalent of receiving a `Cancel` request. - - `dormant`: this will be the preferred entity to focus when entering a menu. -- `NavMenu` enum - - `Wrapping**`/`Bound**`: navigation can wrap - if directional sticks are pressed in a direction there are no focusables in - - `**Scope`/`**2d`: A scopped menu is tab menu navigable with special hotkeys. -- `NavRequest` event - - `Action`: equivalent to left click or `A` button on controller - - `Cancel`: backspace or `B` on controller - - `FocusOn`: request to move the focus to a specific `Entity` - - `Move(Direction)`: directional gamepad or keyboard input - - `ScopeMove(ScopeDirection)`: tab menu hotkey -- `NavEvent` event - - `InitiallyFocused`: We went from 0 focused to 1 focused element - - `NoChanges`: The `NavRequest` did not change the focus state. - - `FocusChanged`: The `NavRequest` changed the focus state, - has the list of items that were made inactive - and items made active. - - `Locked`: The `NavRequest` caused the focus system to lock. - - `Unlocked`: Something unlocked the focus system. -- `MoveParam` trait - - Used to pick a focusable in a provided direction - - The `NavigationPlugin` is generic over the `MoveParam` - - `DefaultNavigationPlugins` provides a default implementation of `MoveParam` +"Using" a `Focusable` is as easy as simply spawning a button using +`ButtonBundle`. +- All navigation input systems are added to the default plugins. + Including mouse and gamepad. +- The resolution algorithm takes care of picking up an initial focused element + if none exist yet. + \ + To specify which focusable to focus initially, spawn a `ButtonBundle` with + `focusable: Focusable::dormant()`. + \ + See [implementation section about initial focus](#what-to-focus-first). -### `Focusable` +### Reacting to button activation & cursor changes -A `Focusable` is an entity that can be focused. -For gamepad-based navigation, -this requires a way to pick a focusable in a provided direction. -This is delegated to the UI library implementation -through the `MoveParam` trait. -In the case of bevy, `bevy_ui` is the UI library implementation, -and the `GlobalTransform` is used to locate focusables in regard to each-other. +See `bevy-ui-navigation` [`NavEvent`] doc, +[and example](https://github.com/nicopap/ui-navigation#simple-case). -The `Focusable` component holds state about what happens when it is activated -and what focus state it is in. (focused, active, inactive, etc.) +The basic "change color of button based on focus state" is very similar to the +current focus management: -#### Note on hovering state +```rust +fn focus_color(mut interaction_query: Query<(&Focusable, &mut UiColor), Changed>) { + for (focusable, mut material) in interaction_query.iter_mut() { + if let FocusState::Focused = focusable.state() { + *material = Color::ORANGE_RED.into(); + } else { + *material = Color::DARK_GRAY.into(); + } + } +} +``` -Since navigation is completely decoupled from input and ui lib, -it is impossible for it to tell whether a focusable is hovered. -To keep the hovering state functionality, -it will be necessary to add it as an independent component -separate from the navigation library. +Note that the `focus_color` system should be added `.after(NavRequestSystem)` +for same-frame updates. -### Input customization +For more advanced behaviors, the user should listen to the [`NavEvent`] event +reader. -`ui-navigation` can be added to the app in two ways: -1. With `NavigationPlugin` -2. With `DefaultNavigationPlugins` +The default way of hooking your UI to your code in [`bevy-ui-navigation`] is +a bit rough: +* Mark your UI elements with either marker components or an enum +* listen for [`NavEvent`] in a UI handling system, +* filter for the ones you care about (typically `NavEvent::NoChanges`), +* check what [`NavRequest`] triggered it, +* retrieve the focused entity from the event, +* check against your own queries what entity it is, +* write code for each case you want to handle -(1) only inserts the navigation systems and resources, -while (2) also adds the input systems generating the `NavRequest`s -for gamepads, mouse and keyboard. +```rust +use bevy::prelude::*; +use bevy_ui_navigation::events::{NavEvent, NavRequest}; + +#[derive(Component)] +enum MainMenuButton { Start, Options, Exit } +/// Marker component +#[derive(Component)] struct ActiveButton; + +fn handle_ui( + mut events: EventReader, + buttons: Query<&MainMenuButton, With>, +) { + // iterate NavEvents + for event in events.iter() { + // Check for a `NoChanges` event with Action + if let NavEvent::NoChanges { from, request: NavRequest::Action } = event { + // Get the focused entity (from.first()), check the button for it. + match buttons.get(*from.first()) { + // Do things when specific button is activated + Ok(MainMenuButton::Start) => {} + Ok(MainMenuButton::Options) => {} + Ok(MainMenuButton::Exit) => {} + Err(_) => {} + } + } + } +} +``` -This enables convenient defaults -while letting users insert their own custom input logic. +It could be useful to provide a higher level API on top of [`NavEvent`]. +[`bevy-ui-navigation`] has a module dedicated to it: [`event_helpers`]. +Using [`event_helpers`] would reduce the previous code to: +```rust +fn handle_ui(mut button_events: NavEventQuery<&MainMenuButton, With>) { + // NOTE: this will silently ignore multiple navigation event at the same frame. + // It should be a very rare occurrence. + match button_events.single_activated().ignore_remaining() { + // Do things when specific button is activated + Some(MainMenuButton::Start) => {} + Some(MainMenuButton::Options) => {} + Some(MainMenuButton::Exit) => {} + None => {} + } +} +``` -### What to focus first? +### Moving the cursor -Gamepad input assumes an initially focused element -for navigating "move in direction" or "press action". -At the very beginning of execution, there are no focused element, -so we need fallbacks. -By default, it is any `Focusable` when none are focused yet. -But the user can spawn a `Focusable` as `dormant` -and the algorithm will prefer it when no focused nodes exist yet. +See `bevy-ui-navigation` [`NavRequest`] doc. +All navigation input systems are added to the default plugins. +Including mouse and gamepad. + +However, if the user's game needs it, +the user can change how input interacts with navigation. +It is easy to remove the default input systems +to replace them with your own custom ones. + +Use `NavRequest` to control the navigation state. +- For mouse picking, use `NavRequest::FocusOn(entity)` to move focus + to the exact entity you want. +- For gamepad or any other kind of directional input, use the + `NavRequest::Move` event. +- Standard "action" and "cancel", see the [`NavRequest`] doc for details + +An input system should run before the request handling system using the +system label mechanism: ```rust -self.focusables - .iter() - // The focused focusable. - .find_map(|(e, focus)| (focus.state == Focused).then(|| e)) - // Dormant focusable within the root menu (if it exists) - .or_else(root_dormant) - // Any dormant focusable - .or_else(any_dormant) - // Any focusable - .or_else(fallback) + .add_system(my_input.before(NavRequestSystem)) +``` + +The exact mechanism for disabling default input is still to be designed. +[See relevant implementation section](#input-customization). + + +### Creating a menu + +See the `bevy-ui-navigation` [`NavMenu`] doc, +[and example](https://github.com/nicopap/ui-navigation/blob/master/examples/menu_navigation.rs). + +To create a menu, use the newly added `MenuBundle`, +this collects all [`Focusable`] children of that node into a single menu. + +Note that this counts for **transitive** children of the menu entity. +Meaning you don't have to make your [`Focusable`]s direct children of the menu. + +```rust +#[derive(Bundle)] +struct MenuBundle { + #[bundle] + pub node_bundle: NodeBundle, + pub menu: MenuSeed, +} ``` +See [relevant implementation section](#spawning-menus) for details on `MenuSeed`. + +[`bevy-ui-navigation`] supports menus and inter-menu navigation out-of-the-box. +Users will have to write code to hide/show menus, +but not to move the cursor between them. + +A menu is: +- An isolated piece of UI where you can navigate between focusables + within it. +- It's either + - a _root_ menu, the default menu with focus, + - or _reachable from_ a [`Focusable`]. +- To enter a menu, you have to either + - _activate_ the [`Focusable`] it is _reachable from_, + - _cancel_ while focus is in one of its sub-menus. + +Menus have two parameters: +- Whether they are a "scope" menu: a "scope" menu is like a + browser tab and can be directly navigated through hotkeys when + focus is within a transitive submenu. +- Whether directional navigation is wrapping (e.g. going leftward from the + leftmost `Focusable` focuses the rightmost `Focusable`) + +Here again, a higher-level API could benefit users, +by automating the process of hiding and showing menus +that are focused and unfocused. + + +### Creating a custom widget + +This is a case study of how [Warlock's Gambit implemented +sliders][gambit-slider] using [`bevy-ui-navigation`]. + +Warlock's Gambit has audio sliders, built on top of `bevy_ui` +and [`bevy-ui-navigation`]. +They are not optimal, as they were made for a 7 days jam. + +We used the [locking mechanism](#locking) to disable navigation when starting to +drag the slider's knob, when the player pressed down the left mouse button. +This is to prevent other [`Focusable`]s from stealing focus while dragging +around the knob with the mouse. + +We then use mouse movement to move around the knob, +and update the audio level based on the knob's position. + +We send a `NavRequest::Free` when the player release's the left mouse button. + + +### Custom directional navigation + +Beyond just changing the inputs that generate [`NavRequest`]s, +it's possible to customize gamepad-style directional input generated by +`NavRequest::Direction`. +The `NavigationPlugin` is generic over directional input. +It accepts a `M: MenuNavigationStrategy` type parameter. + +The default implementation, the one for `bevy_ui`, uses `GlobalTransform` +to resolve position. + +See [relevant implementation section](#MenuNavigationStrategy). + + +## Implementation + +Since `bevy-ui-navigation` already exists, +we will discuss its design, it's functionalities and their uses. + +The nitty-gritty code… is already available! +The [`bevy-ui-navigation`] repo contains the code. +[this is a good start for people interested in the architecture][ui-nav-arch]. -### Menu navigation handling +This section therefore will not discuss the detailed implementation, +but rather the overall design and the decisions that led to them. + +### Exposed API + +The crux of `bevy-ui-navigation` are the following types: +- [`Focusable`](#Focusable) component +- [`NavMenu`](#NavMenu) enum +- `NavRequest` event [on docs.rs][`NavRequest`] +- `NavEvent` event [on docs.rs][`NavEvent`] +- [`MenuNavigationStrategy`](#MenuNavigationStrategy) trait + +### `Focusable` + +See `bevy-ui-navigation` [`Focusable`] doc + +The `Focusable` component holds state about what happens when it is activated +and what focus state it is in. (focused, active, inactive, etc.) + + +#### Focusable state + +See `bevy-ui-navigation` [`FocusState`] doc. + +The focus state encodes roughly the equivalent of `Interaction`, +it can be accessed with the `state` method on `Focusable`. + +**Hovering state is not specified here**, since it is orthogonal to +a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) + +* **Dormant**: + An entity that was previously `Active` from a branch of the menu tree that is + currently not focused. When focus comes back to the `NavMenu` containing this + `Focusable`, the `Dormant` element will be the `Focused` entity. +* **Focused**: + The currently highlighted/used entity, there is only a single focused entity. + \ + All navigation requests start from it. + \ + To set an arbitrary `Focusable` to focused, you should send a `NavRequest::FocusOn` + request. +* **Active**: + This Focusable is on the path in the menu tree to the current Focused entity. + \ + `FocusState::Active` focusables are the `Focusable`s from previous menus that + were activated in order to reach the `NavMenu` containing the currently focused + element. + \ + It is one of the "breadcrumb" of buttons to reach the current focused + element. +* **Inert**: + None of the above: This Focusable is neither Dormant, Focused or Active. + +#### Focusable action types + +See ui-navigation [`FocusAction`] doc. + +A `Focusable` can execute a variety of `FocusAction` when receiving +`NavRequest::Action`, the default one is `FocusAction::Normal`. + +* **Normal**: + Acts like a standard navigation node. + \ + Goes into relevant menu if any [`NavMenu`] is + `reachable_from` this `Focusable`. +* **Cancel**: + If we receive `NavRequest::Action` while this `Focusable` is + focused, it will act as a `NavRequest::Cancel` (leaving submenu to + enter the parent one). +* **Lock**: + If we receive `NavRequest::Action` while this `Focusable` is + focused, the navigation system will freeze until `NavRequest::Free` + is received, sending a `NavEvent::Unlocked`. + \ + This is useful to implement widgets with complex controls you don't + want to accidentally unfocus, or suspending the navigation system while + in-game. + +### `NavMenu` The public API has `NavMenu`, but we use internally `TreeMenu`, this prevents end users from breaking assumptions about the menu trees. +[More details on this decision](#spawning-menus). We define the `TreeMenu` component. All `Focusable` direct or indirect children of a `TreeMenu` are part of that menu. @@ -170,8 +386,6 @@ In short, the navigation tree is built on two elements: - The `reachable_from` that a `NavMenu` was spawned with - The `Focusable` within the hierarchy of a `NavMenu` -This is a tree in the ECS. - In the following screenshot, `Focusable`s are represented as circles, and menus as rectangles. @@ -182,7 +396,7 @@ to the menu in which the currently focused entity is is _active_. The current focus position is represented as a trail of breadcrumb active focusables and the currently focused focusable. -This is reflected into the `NavEvent::FocusChanged` event. +This is visible in the `NavEvent::FocusChanged` event. When going from a focusable to another, the navigation system emits a `FocusChanged` event, the event contains two lists of entities: @@ -213,13 +427,28 @@ when none are focused yet. Menus store a reference to their parent, but that parent is not necessarily their hierarchical parent. The parent is just a button in another menu. + It is inconvenient to have to pre-spawn each button to acquire their `Entity` id just to be able to spawn the menu you'll get to from that button. -`ui-navigation` uses a proxy component holding both the parent and the menu state info. +[`bevy-ui-navigation`] uses a proxy component holding both the parent +and the menu state info. + That proxy component is `MenuSeed`, you can initialize it either with the `Name` or `Entity` of the parent focusable of the menu. A system will take all `MenuSeed` components and replace them with a `TreeMenu`. +This also enables front-loading the check of whether the parent focusable +`Entity` is indeed a `Focusable`. +Before turning the `MenuSeed` into a `TreeMenu`, +we check that the alleged parent is indeed `Focusable`, and panic otherwise. + +If we couldn't assume parents to be focusable, +the focus resolution algorithm would trip. + +This also completely hides the `TreeMenu` component from end-users. +This allows safely changing internals of the navigation system without +breaking user code. + This introduces a single frame lag on menu spawn, but it improves ergonomics to users. I thought it was a pretty good trade-off, @@ -235,6 +464,10 @@ To do this, we have two options: This will change the focus to the parent button of the focused menu. +Since "cancel" or "go back to previous menu" buttons are a very common +occurrence, I thought it sensible to integrate it fully in the navigation +algorithm. + #### Moving into a menu When the `Action` request is sent while a `Focusable` with a child menu is focused, @@ -242,14 +475,100 @@ the focus will move to the child menu, selecting the focusable to focus with a heuristic similar to the initial focused selection. +### `MenuNavigationStrategy` + +This trait is used to pick a focusable in a provided direction. +- The `NavigationPlugin` is generic over a `T` that implement this trait +- `DefaultNavigationPlugins` provides a default implementation of this trait + +This decouples the UI from the navigation system. +For example, this allows using the navigation system with 3D elements. + +```rust +/// System parameter used to resolve movement and cycling focus updates. +/// +/// This is useful if you don't want to depend on bevy's [`GlobalTransform`] +/// for your UI, or want to implement your own navigation algorithm. For example, +/// if you want your ui to be 3d elements in the world. +/// +/// See the [`UiProjectionQuery`] source code for implementation hints. +pub trait MenuNavigationStrategy { + /// Which `Entity` in `siblings` can be reached from `focused` in + /// `direction` if any, otherwise `None`. + /// + /// * `focused`: The currently focused entity in the menu + /// * `direction`: The direction in which the focus should move + /// * `cycles`: Whether the navigation should loop + /// * `siblings`: All the other focusable entities in this menu + /// + /// Note that `focused` appears once in `siblings`. + fn resolve_2d<'a>( + &self, + focused: Entity, + direction: events::Direction, + cycles: bool, + siblings: &'a [Entity], + ) -> Option<&'a Entity>; +} +``` + + ### Locking To ease implementation of widgets, such as sliders, some focusables can "lock" the UI, preventing all other form of interactions. Another system will be in charge of unlocking the UI by sending a `NavRequest::Free`. -This may be better served by keeping the `FocusPolicy` component, -in situations where a global lock prevents other uses. +The default input sets the escape key to send a `NavRequest::Free`, +but the user may chose to `Free` through any other mean. + +This might be better served to a locking component that block navigation +of specific menu trees, +but I didn't find the need for such fine-grained controls. + + +### Input customization + +[See the ui-navigation README example](https://github.com/nicopap/ui-navigation#simple-case). + +`ui-navigation` can be added to the app in two ways: +1. With `NavigationPlugin` +2. With `DefaultNavigationPlugins` + +(1) only inserts the navigation systems and resources, +while (2) also adds the input systems generating the `NavRequest`s +for gamepads, mouse and keyboard. + +This enables convenient defaults +while letting users insert their own custom input logic. + +More customization is available: +[`MenuNavigationStrategy`](#MenuNavigationStrategy) allows customizing +the spacial resolution between focusables within a menu. + + +### What to focus first? + +Gamepad input assumes an initially focused element +for navigating "move in direction" or "press action". +At the very beginning of execution, there are no focused element, +so we need fallbacks. +By default, it is any `Focusable` when none are focused yet. +But the user can spawn a `Focusable` as `dormant` +and the algorithm will prefer it when no focused nodes exist yet. + +```rust +self.focusables + .iter() + // The focused focusable. + .find_map(|(e, focus)| (focus.state == Focused).then(|| e)) + // Dormant focusable within the root menu (if it exists) + .or_else(root_dormant) + // Any dormant focusable + .or_else(any_dormant) + // Any focusable + .or_else(fallback) +``` ## Prior art @@ -270,13 +589,35 @@ Otherwise, given my little experience in game programming, I am probably overloo standard practices. My go-to source ("Game Engine Architecture" by Jason Gregory) doesn't mention ui navigation systems. -## Open questions +## Major differences with the current focus implementation -### `NavRequest` system +### `Hovered` is not covered -I wanted to defer the raw input processing to the user. The current -implementation provides default systems to manage ui input, with a -`InputMapping` resource to customize a little bit the system's behaviors. +Since navigation is completely decoupled from input and ui lib, +it is impossible for it to tell whether a focusable is hovered. +To keep the hovering state functionality, +it will be necessary to add it as an independent component +separate from the navigation library. + +A fully generic navigation API cannot handle hover state. +This doesn't mean we have to entirely give up hovering. +It can simply be added back as a component independent +from the focus system. + +### Interaction tree + +In this design, interaction is not "bubble up" as in the current `bevy_ui` +design. + +This is because there is a clear and exact distinction +in what can be "focused", what is a "menu", and everything else. +Focused element do not share their state with their parent. +The breadcrumb of menus defined [in the `NavMenu` section](#NavMenu) +is a very different concept from marking parents as focused. + +As such, `FocusPolicy` becomes useless and is removed. + +## Open questions ### Game-oriented assumptions @@ -326,10 +667,9 @@ From what I gather from questions asked in the `#help` discord channel, I think the design most people come up with naturally is to spawn and despawn dynamically the menus as they are traversed, which is incompatible with this design. -However, in practice, in my own games, I worked around it by expanding the -whole UI tree beyond the UI camera view and moving around the camera. -An alternative would simply to set the `style.display` of non-active menus to -`None` and change the style when focus enters them. + +In practice, this should be done by setting the `style.display` of +non-active menus to `None` and change the style when focus enters them. This also fixes the already-existing 1 frame latency issue with the spawn/despawn design. @@ -356,3 +696,13 @@ misunderstanding how the navigation system work or programming errors. [mouse picking]: https://github.com/aevyrie/bevy_mod_picking/ [`bevy-ui-navigation`]: https://github.com/nicopap/ui-navigation +[ui-nav-arch]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L1-L33 +[`event_helpers`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/event_helpers/index.html +[gambit-slider]: https://github.com/team-plover/warlocks-gambit/blob/3f8132fbbd6cce124a1cd102755dad228035b2dc/src/ui/main_menu.rs#L59-L95 +[`NavRequest`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/events/enum.NavRequest.html +[`NavEvent`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/events/enum.NavEvent.html +[`Focusable`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/struct.Focusable.html +[`FocusState`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.FocusState.html +[`FocusAction`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.FocusAction.html +[`NavMenu`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.NavMenu.html +[`UiProjectionQuery`]: https://github.com/nicopap/ui-navigation/blob/17c771b7f752cfd604f21056f9d4ca6772529c6f/src/resolve.rs#L378-L429 From 5372fbe32953c0b90c593cae5103b72e0a5083d4 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 12:39:26 +0200 Subject: [PATCH 14/33] Phrasing, consistency --- rfcs/41-ui-navigation.md | 22 +++++----------------- 1 file changed, 5 insertions(+), 17 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 9806ae41..90a1de3a 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -39,19 +39,8 @@ can't be swapped out for something the user deems more appropriate. The [bevy-ui-navigation README][`bevy-ui-navigation`] is a good start. -The API exposed in this RFC are relatively "low level". -Ideally, we provide default wrapping systems to simplify using the focus -system. - -However, it's completely open for the user to plug their own input -and focus handling. -[`bevy-ui-navigation`] provides its own wrapper around `NavEvent`, -called [`event_helpers`]. -However, [`event_helpers`] is not included in this RFC, -because it is a very opinionated API. -It is better left to a different PR where the contentious points -can be solved independently of the rest of the navigation system. - +The API exposed in this RFC is relatively "low level". +Ideally, we provide default wrapping APIs to simplify using the focus system. To summarize, here are the headlines: - The navigation system can only be interacted with with events: @@ -256,7 +245,6 @@ sliders][gambit-slider] using [`bevy-ui-navigation`]. Warlock's Gambit has audio sliders, built on top of `bevy_ui` and [`bevy-ui-navigation`]. -They are not optimal, as they were made for a 7 days jam. We used the [locking mechanism](#locking) to disable navigation when starting to drag the slider's knob, when the player pressed down the left mouse button. @@ -347,7 +335,7 @@ a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) #### Focusable action types -See ui-navigation [`FocusAction`] doc. +See `bevy-ui-navigation` [`FocusAction`] doc. A `Focusable` can execute a variety of `FocusAction` when receiving `NavRequest::Action`, the default one is `FocusAction::Normal`. @@ -529,9 +517,9 @@ but I didn't find the need for such fine-grained controls. ### Input customization -[See the ui-navigation README example](https://github.com/nicopap/ui-navigation#simple-case). +[See the `bevy-ui-navigation` README example](https://github.com/nicopap/ui-navigation#simple-case). -`ui-navigation` can be added to the app in two ways: +The `NavigationPlugin` can be added to the app in two ways: 1. With `NavigationPlugin` 2. With `DefaultNavigationPlugins` From f02cd9f7674eaa0318bce27871bb58bbc5edea1b Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 12:42:18 +0200 Subject: [PATCH 15/33] =?UTF-8?q?dormant=20=E2=86=92=20prioritized?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- rfcs/41-ui-navigation.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 90a1de3a..a2e6ec97 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -73,7 +73,7 @@ The `Interaction` component in `ButtonBundle` is replaced by `Focusable`. if none exist yet. \ To specify which focusable to focus initially, spawn a `ButtonBundle` with - `focusable: Focusable::dormant()`. + `focusable: Focusable::prioritized()`. \ See [implementation section about initial focus](#what-to-focus-first). @@ -310,10 +310,10 @@ it can be accessed with the `state` method on `Focusable`. **Hovering state is not specified here**, since it is orthogonal to a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) -* **Dormant**: +* **Prioritized**: An entity that was previously `Active` from a branch of the menu tree that is currently not focused. When focus comes back to the `NavMenu` containing this - `Focusable`, the `Dormant` element will be the `Focused` entity. + `Focusable`, the `Prioritized` element will be the `Focused` entity. * **Focused**: The currently highlighted/used entity, there is only a single focused entity. \ @@ -331,7 +331,7 @@ a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) It is one of the "breadcrumb" of buttons to reach the current focused element. * **Inert**: - None of the above: This Focusable is neither Dormant, Focused or Active. + None of the above: This Focusable is neither Prioritized, Focused or Active. #### Focusable action types @@ -403,7 +403,7 @@ because the code is dead simple and easy to vet. #### Going back to a previous menus -`Focusable`s have a `dormant` state that is set when they go from +`Focusable`s have a `prioritized` state that is set when they go from active/focused to not active. This is a form of memory that allows focus to go back to the last focused element within a menu when it is re-visited. @@ -542,7 +542,7 @@ for navigating "move in direction" or "press action". At the very beginning of execution, there are no focused element, so we need fallbacks. By default, it is any `Focusable` when none are focused yet. -But the user can spawn a `Focusable` as `dormant` +But the user can spawn a `Focusable` as `prioritized` and the algorithm will prefer it when no focused nodes exist yet. ```rust @@ -550,10 +550,10 @@ self.focusables .iter() // The focused focusable. .find_map(|(e, focus)| (focus.state == Focused).then(|| e)) - // Dormant focusable within the root menu (if it exists) - .or_else(root_dormant) - // Any dormant focusable - .or_else(any_dormant) + // Prioritized focusable within the root menu (if it exists) + .or_else(root_prioritized) + // Any prioritized focusable + .or_else(any_prioritized) // Any focusable .or_else(fallback) ``` From 44c193ceb2964d98b53f4ec7b9cdc428308e81d5 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 13:35:21 +0200 Subject: [PATCH 16/33] Add Internal representation section + copy doc Internal representation + copy the documentation from bevy-ui-navigation for a few more enums, so that they can be clearly explained in this RFC without external references. --- rfcs/41-ui-navigation.md | 211 ++++++++++++++++++++++++++++++--------- 1 file changed, 166 insertions(+), 45 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index a2e6ec97..a490a813 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -280,16 +280,87 @@ The nitty-gritty code… is already available! The [`bevy-ui-navigation`] repo contains the code. [this is a good start for people interested in the architecture][ui-nav-arch]. -This section therefore will not discuss the detailed implementation, -but rather the overall design and the decisions that led to them. +### Internal representation + +The navigation tree is a set of relationships between entities with either the +[`Focusable`] component or the [`TreeMenu`] component. + +The hierarchy is specified by both the native bevy hierarchy +and the `focus_parent` of the [`TreeMenu`] component. + +In the following screenshot, `Focusable`s are represented as circles, menus +as rectangles, and the `focus_parent` by blue arrows +(menu points to its parent). + +![A screenshot of a RPG-style menu with the navigation tree overlayed](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) + +Note that gold buttons are `FocusState::Active` while the orange-red button +("B") is `FocusState::Focused`. + +Since the "tabs menu" doesn't have an outgoing arrow, it is the root menu. + +The active buttons are a breadcrumb of buttons to activate to reach +the current menu with the `Focused` focusable from the root menu. + +To move the focus from the "soul menu" to the "ABC menu", +you need to send `NavRequest::Action` while the button "abc" is focused +(i.e.: _"activating"_ the button). + +Such a move would generate the following `NavEvent`: +```rust +NavEvent::FocusChanged { + to: [, ], + from: [], +} +``` + +If there were a "KFC menu" (currently hidden) [child of](#Terminology) +the "kfc" button, then [activating](#Terminology) +the "kfc" button would send the focus to +the prioritized focusable within the "KFC menu". + +To navigate from "ABC menu" back to "soul menu", +you would send `NavRequest::Cancel`. +Such a move would generate the following `NavEvent`: +```rust +NavEvent::FocusChanged { + to: [], + from: [, ], +} +``` +The "tabs menu" is defined as a "scope menu", which means that +by default, the `LT` and `RT` gamepad buttons will navigate +the "tabs menu" regardless of the current focus position. + +Pressing `RT` while "B" if focused, would generate the following `NavEvent`: +```rust +NavEvent::FocusChanged { + to: [, ], + from: [, , ], +} +``` + +There is no [child menus](#Terminology) of the "B" button +and "B" is of type `FocusAction::Normal`, +therefore, sending `NavRequest::Action` while "B" is highlighted +will do nothing and generate the following `NavEvent`: +```rust +NavEvent::NoChanges { + request: NavRequest::Action, + from: [, , ], +} +``` + +See [relevant implementation section for details on `NavEvent`](#NavEvent) + ### Exposed API The crux of `bevy-ui-navigation` are the following types: -- [`Focusable`](#Focusable) component -- [`NavMenu`](#NavMenu) enum +- [`Focusable`](#Focusable) component [on docs.rs][`Focusable`] +- [`NavMenu`](#NavMenu) enum [on docs.rs][`NavMenu`] - `NavRequest` event [on docs.rs][`NavRequest`] -- `NavEvent` event [on docs.rs][`NavEvent`] +- [`NavEvent`](#NavEvent) event [on docs.rs][`NavEvent`] - [`MenuNavigationStrategy`](#MenuNavigationStrategy) trait ### `Focusable` @@ -360,46 +431,71 @@ A `Focusable` can execute a variety of `FocusAction` when receiving ### `NavMenu` -The public API has `NavMenu`, but we use internally `TreeMenu`, +See `bevy-ui-navigation` [`NavMenu`] doc. + +The public API has `NavMenu`, but we use internally [`TreeMenu`], this prevents end users from breaking assumptions about the menu trees. [More details on this decision](#spawning-menus). -We define the `TreeMenu` component. -All `Focusable` direct or indirect children of a `TreeMenu` are part of that menu. -A `TreeMenu` has a `Focusable` focus parent, -when that `Focusable` is focused and the `Action` request is received, -the focus changes to the `TreeMenu` - -In short, the navigation tree is built on two elements: -- The `reachable_from` that a `NavMenu` was spawned with -- The `Focusable` within the hierarchy of a `NavMenu` - -In the following screenshot, `Focusable`s are represented as circles, and menus -as rectangles. - -![A screenshot of a RPG-style menu with the navigation tree overlayed](https://user-images.githubusercontent.com/26321040/141671969-ea4a461d-0949-4b98-b060-c492d7b896bd.png) - -A `Focusable` used to go from the root menu -to the menu in which the currently focused entity is is _active_. -The current focus position is represented as a trail of breadcrumb active -focusables and the currently focused focusable. - -This is visible in the `NavEvent::FocusChanged` event. -When going from a focusable to another, -the navigation system emits a `FocusChanged` event, -the event contains two lists of entities: -- The list of entities that after the navigation event are no more focused - or active -- The list of entities that are newly active or focused - -For the simplest case of navigating from one focusable to another -in the same menu, the two list have a single element, -so it is capital that it is easy to access those elements. -This is why we use a `NonEmpty`, from the `non_empty_vec` crate. -A `NonEmpty` has a `first` method that returns a `T`, -which in our case would be the last and new focused elements. -I chose this crate over similar ones, -because the code is dead simple and easy to vet. +A menu that isolate children [`Focusable`]s from other +focusables and specify navigation method within itself. + +A [`NavMenu`] can be used to: +* Prevent navigation from one specific submenu to another +* Specify if 2d navigation wraps around the screen. +* Specify "scope menus" such that a + `NavRequest::ScopeMove` emitted when + the focused element is a [`Focusable`] nested within this `NavMenu` + will navigate this menu. +* Specify _submenus_ and specify from where those submenus are reachable. +* Specify which entity will be the parents of this [`NavMenu`], see + `NavMenu::reachable_from` or `NavMenu::reachable_from_named` if you don't + have access to the `Entity` + for the parent [`Focusable`] + +If you want to specify which [`Focusable`] should be +focused first when entering a menu, you should mark one of the children of +this menu with `Focusable::prioritized`. + +#### Invariants + +**You need to follow those rules to avoid panics**: +1. A menu must have **at least one** [`Focusable`] child in the UI + hierarchy. +2. There must not be a menu loop. I.e.: a way to go from menu A to menu B and + then from menu B to menu A while never going back. + +#### Panics + +Thankfully, programming errors are caught early and you'll probably get a +panic fairly quickly if you don't follow the invariants. +* Invariant (1) panics as soon as you add the menu without focusable + children. +* Invariant (2) panics if the focus goes into a menu loop. + +#### Variants + +* **Bound2d**: Non-wrapping menu with 2d navigation. + It is possible to move around this menu in all cardinal directions, the + focus changes according to the physical position of the + [`Focusable`] in it. + \ + If the player moves to a direction where there aren't any focusables, + nothing will happen. +* **Wrapping2d**: Wrapping menu with 2d navigation. + It is possible to move around this menu in all cardinal directions, the + focus changes according to the physical position of the + [`Focusable`] in it. + \ + If the player moves to a direction where there aren't any focusables, + the focus will "wrap" to the other direction of the screen. +* **BoundScope**: Non-wrapping scope menu + Controlled with `NavRequest::ScopeMove` + even when the focused element is not in this menu, but in a submenu + reachable from this one. +* **WrappingScope**: Wrapping scope menu + Controlled with `NavRequest::ScopeMove` even + when the focused element is not in this menu, but in a submenu reachable from this one. #### Going back to a previous menus @@ -443,12 +539,36 @@ I thought it was a pretty good trade-off, since spawning menus is not time critical, unlike input. +### `NavEvent` + +See `bevy-ui-navigation` [`NavEvent`] doc. + +Events emitted by the navigation system. + +(Please see the docs.rs page, the rest of this section refers to terms +explained in those pages) + +Note the usage of `NonEmpty` from the `non_empty_vec` crate. + +In most case, we only care about a single focusable in `NoChanges` and +`FocusChanged` (the `.first()` one). +Furthermore, we _know_ those lists will never be empty (since there is always +one focusable at all time). +We also really don't want to provide a clunky API where you have to `unwrap` or +adapt to `Option`s when it is not needed. +So we provide the list of changed focusable as an `NonEmpty`. + +I vetted several "non empty vec" crates, and `non_empty_vec` +had the cleanest and easiest to understand implementation. +This is why, I chose it. + + #### cancel events When entering a menu, you want a way to leave it and return to the previous menu. To do this, we have two options: -- The `Cancel` request -- The `cancel` focusable +- A `NavRequest::Cancel` +- _activating_ a `FocusAction::Cancel` focusable. This will change the focus to the parent button of the focused menu. @@ -607,7 +727,7 @@ As such, `FocusPolicy` becomes useless and is removed. ## Open questions -### Game-oriented assumptions +### (Solved, but worth asking) Game-oriented assumptions This was not intended during implementation, but in retrospect, the `NavMenu` system is relatively opinionated. @@ -694,3 +814,4 @@ misunderstanding how the navigation system work or programming errors. [`FocusAction`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.FocusAction.html [`NavMenu`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.NavMenu.html [`UiProjectionQuery`]: https://github.com/nicopap/ui-navigation/blob/17c771b7f752cfd604f21056f9d4ca6772529c6f/src/resolve.rs#L378-L429 +[`TreeMenu`]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L201-L216 From ecde6edb68f0f192541bb501cabf05b6433a8c75 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 13:38:55 +0200 Subject: [PATCH 17/33] Minor grammatical correction --- rfcs/41-ui-navigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index a490a813..e3446bc6 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -546,7 +546,7 @@ See `bevy-ui-navigation` [`NavEvent`] doc. Events emitted by the navigation system. (Please see the docs.rs page, the rest of this section refers to terms -explained in those pages) +explained in it) Note the usage of `NonEmpty` from the `non_empty_vec` crate. From 3d3df58d75159623f1cc771e0a8878be749bdcd6 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 13:42:23 +0200 Subject: [PATCH 18/33] Add comment on accessibility --- rfcs/41-ui-navigation.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index e3446bc6..04653a55 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -35,6 +35,9 @@ such as menu trees and movement between menus. The current UI focus system is limited to `bevy_ui` and can't be swapped out for something the user deems more appropriate. +Decoupling navigation from input and graphics also provides a handy API +for third party integration such as accessibility tools. + ## User-facing explanation The [bevy-ui-navigation README][`bevy-ui-navigation`] is a good start. From ed26995a0a41b146b88c4da9778ccf4369366103 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 13:54:04 +0200 Subject: [PATCH 19/33] Grammar fix --- rfcs/41-ui-navigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 04653a55..a3bd0d1e 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -739,7 +739,7 @@ Indeed, we assume we are trying to build a game UI with a series of "menus" and that it is possible to keep track of a "trail" of buttons from a "root menu" to the deepest submenu we are currently browsing. -The example of a fancy editor with a bunch of dockered widgets seems to break +The example of a fancy editor with a bunch of docked widgets seems to break that assumption. However, I think we could make it so each docker has a tab, and we could navigate between the tabs. I think such an implementation strategy encourages better mouseless navigation support and it might not be a From 67d7db1709af5e1fc2487332d61070f52f04d2c5 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:01:18 +0200 Subject: [PATCH 20/33] Remove unnecessary paragraph --- rfcs/41-ui-navigation.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index a3bd0d1e..eb2522bd 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -276,9 +276,6 @@ See [relevant implementation section](#MenuNavigationStrategy). ## Implementation -Since `bevy-ui-navigation` already exists, -we will discuss its design, it's functionalities and their uses. - The nitty-gritty code… is already available! The [`bevy-ui-navigation`] repo contains the code. [this is a good start for people interested in the architecture][ui-nav-arch]. From 3bef235a62f52b56629528712039bc9348cddc36 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:03:24 +0200 Subject: [PATCH 21/33] Fix broken link --- rfcs/41-ui-navigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index eb2522bd..e3c5aa95 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -351,7 +351,7 @@ NavEvent::NoChanges { } ``` -See [relevant implementation section for details on `NavEvent`](#NavEvent) +See [relevant implementation section](#NavEvent) for details on `NavEvent`. ### Exposed API From b75ecf1236d32893c3e446883d06c94f796caf97 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:05:18 +0200 Subject: [PATCH 22/33] Add dot to sentence --- rfcs/41-ui-navigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index e3c5aa95..653a5e4e 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -365,7 +365,7 @@ The crux of `bevy-ui-navigation` are the following types: ### `Focusable` -See `bevy-ui-navigation` [`Focusable`] doc +See `bevy-ui-navigation` [`Focusable`] doc. The `Focusable` component holds state about what happens when it is activated and what focus state it is in. (focused, active, inactive, etc.) From 2389127d90caebd4d184d8af0a23fc4e33e0de9a Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:13:34 +0200 Subject: [PATCH 23/33] Add 'Future possibilities' section --- rfcs/41-ui-navigation.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 653a5e4e..0271c6e1 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -759,6 +759,29 @@ sending the `NavRequest`, while using `world_to_viewport` in `MoveParam` for gamepad navigation. (or completely omitting gamepad support by providing a stub `MoveParam`) +## Future possibilities + +### New and better mouse picking! + +Since the focus algorithm does not handle input, it is not relevant +to how we chose which entity is focused. + +The input methods can be interchanged without having to change or update +the navigation plugins. + +### Multiple cursors + +A non-so-uncommon pattern of video games (especially split-screen) +is to have two different players handle two different cursors on screen. + +The current design assumes a unique cursor navigating the menus +and can't account for multi-cursor. +However, it seems not fundamentally impossible to add multiple cursors. + +It would require modifying the `NavRequest` to include a sort of ID +to identify which cursor originated the request, +and keeping track in the `FocusState` which cursor the `Focused`, `Activated` +and `Prioritized` variants are for. ## Drawbacks and design limitations From 70aa04c1d07bb8c526f16227d1b000f722e9bf92 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:15:27 +0200 Subject: [PATCH 24/33] Move around the last sections --- rfcs/41-ui-navigation.md | 62 ++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 0271c6e1..94f1f162 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -697,37 +697,9 @@ Otherwise, given my little experience in game programming, I am probably overloo standard practices. My go-to source ("Game Engine Architecture" by Jason Gregory) doesn't mention ui navigation systems. -## Major differences with the current focus implementation +## Closed, interesting questions -### `Hovered` is not covered - -Since navigation is completely decoupled from input and ui lib, -it is impossible for it to tell whether a focusable is hovered. -To keep the hovering state functionality, -it will be necessary to add it as an independent component -separate from the navigation library. - -A fully generic navigation API cannot handle hover state. -This doesn't mean we have to entirely give up hovering. -It can simply be added back as a component independent -from the focus system. - -### Interaction tree - -In this design, interaction is not "bubble up" as in the current `bevy_ui` -design. - -This is because there is a clear and exact distinction -in what can be "focused", what is a "menu", and everything else. -Focused element do not share their state with their parent. -The breadcrumb of menus defined [in the `NavMenu` section](#NavMenu) -is a very different concept from marking parents as focused. - -As such, `FocusPolicy` becomes useless and is removed. - -## Open questions - -### (Solved, but worth asking) Game-oriented assumptions +### Game-oriented assumptions This was not intended during implementation, but in retrospect, the `NavMenu` system is relatively opinionated. @@ -749,7 +721,7 @@ component, and the elements highlighted in yellow have a `Focusable` component. This way we create a menu to navigate between dockers. This is perfectly consistent with our design. -### (Solved, but worth asking) Moving UI camera & 3d UI +### Moving UI camera & 3d UI The completely decoupled implementation of the navigation system enables user to implement their own UI. @@ -785,6 +757,34 @@ and `Prioritized` variants are for. ## Drawbacks and design limitations +### Major differences with the current focus implementation + +#### `Hovered` is not covered + +Since navigation is completely decoupled from input and ui lib, +it is impossible for it to tell whether a focusable is hovered. +To keep the hovering state functionality, +it will be necessary to add it as an independent component +separate from the navigation library. + +A fully generic navigation API cannot handle hover state. +This doesn't mean we have to entirely give up hovering. +It can simply be added back as a component independent +from the focus system. + +#### Interaction tree + +In this design, interaction is not "bubble up" as in the current `bevy_ui` +design. + +This is because there is a clear and exact distinction +in what can be "focused", what is a "menu", and everything else. +Focused element do not share their state with their parent. +The breadcrumb of menus defined [in the `NavMenu` section](#NavMenu) +is a very different concept from marking parents as focused. + +As such, `FocusPolicy` becomes useless and is removed. + ### How does this work with the spawn/despawn workflow? The current design imposes to the user that all UI nodes such a buttons and From c387c7101bca10f0428cdb18b50fa4e9a3c74220 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:22:27 +0200 Subject: [PATCH 25/33] Add higher level wrappers to future possibilities --- rfcs/41-ui-navigation.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 94f1f162..ebf4724a 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -755,6 +755,14 @@ to identify which cursor originated the request, and keeping track in the `FocusState` which cursor the `Focused`, `Activated` and `Prioritized` variants are for. +### Higher level wrappers + +In this RFC we discussed the possibility of improved API in a few places: +- The `bevy-ui-navigation` [`event_helpers`] module, to simplify interacting + with `NavEvent`s +- A visual wrapper for menu navigation that handles `display` `Style`s of menus + when they are entered and left + ## Drawbacks and design limitations ### Major differences with the current focus implementation From 8e788bf1bff42008868c5f2f917860d34f784784 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 14:34:58 +0200 Subject: [PATCH 26/33] Fix order of examples --- rfcs/41-ui-navigation.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index ebf4724a..e8142dfa 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -309,7 +309,7 @@ you need to send `NavRequest::Action` while the button "abc" is focused Such a move would generate the following `NavEvent`: ```rust NavEvent::FocusChanged { - to: [, ], + to: [, ], from: [], } ``` @@ -325,7 +325,7 @@ Such a move would generate the following `NavEvent`: ```rust NavEvent::FocusChanged { to: [], - from: [, ], + from: [, ], } ``` The "tabs menu" is defined as a "scope menu", which means that @@ -335,8 +335,8 @@ the "tabs menu" regardless of the current focus position. Pressing `RT` while "B" if focused, would generate the following `NavEvent`: ```rust NavEvent::FocusChanged { - to: [, ], - from: [, , ], + to: [, ], + from: [, , ], } ``` @@ -347,7 +347,7 @@ will do nothing and generate the following `NavEvent`: ```rust NavEvent::NoChanges { request: NavRequest::Action, - from: [, , ], + from: [ , , ], } ``` From c33f2b19bf32a7cf1d28258e29c9e9da5d460288 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 18:15:53 +0200 Subject: [PATCH 27/33] Rename NavMenu into MenuSetting --- rfcs/41-ui-navigation.md | 46 +++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index e8142dfa..39b9328c 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -53,8 +53,8 @@ To summarize, here are the headlines: - ui-navigation provides default input systems, but it's possible to disable them and instead use custom ones. This is why we restrict interactions with navigation to events. -- It is possible to create isolated menus by using the `NavMenu` component. -- All `Focusable` children in the `NavMenu` entity's tree will be part +- It is possible to create isolated menus by using the `MenuSetting` component. +- All `Focusable` children in the `MenuSetting` entity's tree will be part of this menu, and moving using a gamepad from one focusable in a menu can only lead to another focusable within the same menu. You can specify a focusable in any other menu that will lead to this @@ -195,7 +195,7 @@ The exact mechanism for disabling default input is still to be designed. ### Creating a menu -See the `bevy-ui-navigation` [`NavMenu`] doc, +See the `bevy-ui-navigation` [`MenuSetting`] doc, [and example](https://github.com/nicopap/ui-navigation/blob/master/examples/menu_navigation.rs). To create a menu, use the newly added `MenuBundle`, @@ -358,7 +358,7 @@ See [relevant implementation section](#NavEvent) for details on `NavEvent`. The crux of `bevy-ui-navigation` are the following types: - [`Focusable`](#Focusable) component [on docs.rs][`Focusable`] -- [`NavMenu`](#NavMenu) enum [on docs.rs][`NavMenu`] +- [`MenuSetting`](#MenuSetting) enum [on docs.rs][`MenuSetting`] - `NavRequest` event [on docs.rs][`NavRequest`] - [`NavEvent`](#NavEvent) event [on docs.rs][`NavEvent`] - [`MenuNavigationStrategy`](#MenuNavigationStrategy) trait @@ -383,7 +383,7 @@ a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) * **Prioritized**: An entity that was previously `Active` from a branch of the menu tree that is - currently not focused. When focus comes back to the `NavMenu` containing this + currently not focused. When focus comes back to the `MenuSetting` containing this `Focusable`, the `Prioritized` element will be the `Focused` entity. * **Focused**: The currently highlighted/used entity, there is only a single focused entity. @@ -396,7 +396,7 @@ a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) This Focusable is on the path in the menu tree to the current Focused entity. \ `FocusState::Active` focusables are the `Focusable`s from previous menus that - were activated in order to reach the `NavMenu` containing the currently focused + were activated in order to reach the `MenuSetting` containing the currently focused element. \ It is one of the "breadcrumb" of buttons to reach the current focused @@ -414,7 +414,7 @@ A `Focusable` can execute a variety of `FocusAction` when receiving * **Normal**: Acts like a standard navigation node. \ - Goes into relevant menu if any [`NavMenu`] is + Goes into relevant menu if any [`MenuSetting`] is `reachable_from` this `Focusable`. * **Cancel**: If we receive `NavRequest::Action` while this `Focusable` is @@ -429,27 +429,27 @@ A `Focusable` can execute a variety of `FocusAction` when receiving want to accidentally unfocus, or suspending the navigation system while in-game. -### `NavMenu` +### `MenuSetting` -See `bevy-ui-navigation` [`NavMenu`] doc. +See `bevy-ui-navigation` [`MenuSetting`] doc. -The public API has `NavMenu`, but we use internally [`TreeMenu`], +The public API has `MenuSetting`, but we use internally [`TreeMenu`], this prevents end users from breaking assumptions about the menu trees. [More details on this decision](#spawning-menus). A menu that isolate children [`Focusable`]s from other focusables and specify navigation method within itself. -A [`NavMenu`] can be used to: +A [`MenuSetting`] can be used to: * Prevent navigation from one specific submenu to another * Specify if 2d navigation wraps around the screen. * Specify "scope menus" such that a `NavRequest::ScopeMove` emitted when - the focused element is a [`Focusable`] nested within this `NavMenu` + the focused element is a [`Focusable`] nested within this `MenuSetting` will navigate this menu. * Specify _submenus_ and specify from where those submenus are reachable. -* Specify which entity will be the parents of this [`NavMenu`], see - `NavMenu::reachable_from` or `NavMenu::reachable_from_named` if you don't +* Specify which entity will be the parents of this [`MenuSetting`], see + `MenuSetting::reachable_from` or `MenuSetting::reachable_from_named` if you don't have access to the `Entity` for the parent [`Focusable`] @@ -514,9 +514,11 @@ The parent is just a button in another menu. It is inconvenient to have to pre-spawn each button to acquire their `Entity` id just to be able to spawn the menu you'll get to from that button. -[`bevy-ui-navigation`] uses a proxy component holding both the parent +[`bevy-ui-navigation`] uses a proxy holding both the parent and the menu state info. + + That proxy component is `MenuSeed`, you can initialize it either with the `Name` or `Entity` of the parent focusable of the menu. A system will take all `MenuSeed` components and replace them with a `TreeMenu`. @@ -701,7 +703,7 @@ Gregory) doesn't mention ui navigation systems. ### Game-oriented assumptions -This was not intended during implementation, but in retrospect, the `NavMenu` +This was not intended during implementation, but in retrospect, the `MenuSetting` system is relatively opinionated. Indeed, we assume we are trying to build a game UI with a series of "menus" and @@ -716,7 +718,7 @@ limitation. ![Krita UI screenshot](https://user-images.githubusercontent.com/26321040/141671939-24b8a7c3-b296-4fd4-8ae0-3bbe7fe4c9a3.png) -In this example, we can imagine the root `NodeBundle` having a `NavMenu` +In this example, we can imagine the root `NodeBundle` having a `MenuSetting` component, and the elements highlighted in yellow have a `Focusable` component. This way we create a menu to navigate between dockers. This is perfectly consistent with our design. @@ -788,7 +790,7 @@ design. This is because there is a clear and exact distinction in what can be "focused", what is a "menu", and everything else. Focused element do not share their state with their parent. -The breadcrumb of menus defined [in the `NavMenu` section](#NavMenu) +The breadcrumb of menus defined [in the `MenuSetting` section](#MenuSetting) is a very different concept from marking parents as focused. As such, `FocusPolicy` becomes useless and is removed. @@ -821,11 +823,11 @@ I think the most problematic aspect of this implementation is that we push on the user the responsibility of upholding invariants. This is not a safety issue, but it leads to panics. The invariants are: 1. Never create a cycle of menu connections -2. Each `NavMenu` must at least have one contained `Focusable` +2. Each menu must at least have one contained `Focusable` The upside is that the invariants become only relevant if the user _opts into_ -`NavMenu`. Already, this is a choice made by the user, so they can be made -aware of the requirements in the `NavMenu` documentation. +`MenuSetting`. Already, this is a choice made by the user, so they can be made +aware of the requirements in the `MenuSetting` documentation. If the invariants are not upheld, the navigation system will simply panic. @@ -843,6 +845,6 @@ misunderstanding how the navigation system work or programming errors. [`Focusable`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/struct.Focusable.html [`FocusState`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.FocusState.html [`FocusAction`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.FocusAction.html -[`NavMenu`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.NavMenu.html +[`MenuSetting`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.NavMenu.html [`UiProjectionQuery`]: https://github.com/nicopap/ui-navigation/blob/17c771b7f752cfd604f21056f9d4ca6772529c6f/src/resolve.rs#L378-L429 [`TreeMenu`]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L201-L216 From f1c6ee6198020b5199b0744867ad07594640a1fe Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 22 Jul 2022 18:19:35 +0200 Subject: [PATCH 28/33] Update MenuSetting to reflect current implementation --- rfcs/41-ui-navigation.md | 40 ++++++++++++++++------------------------ 1 file changed, 16 insertions(+), 24 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 39b9328c..a1e7608f 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -473,29 +473,23 @@ panic fairly quickly if you don't follow the invariants. children. * Invariant (2) panics if the focus goes into a menu loop. -#### Variants +#### Fields -* **Bound2d**: Non-wrapping menu with 2d navigation. - It is possible to move around this menu in all cardinal directions, the - focus changes according to the physical position of the - [`Focusable`] in it. - \ - If the player moves to a direction where there aren't any focusables, - nothing will happen. -* **Wrapping2d**: Wrapping menu with 2d navigation. - It is possible to move around this menu in all cardinal directions, the - focus changes according to the physical position of the - [`Focusable`] in it. - \ - If the player moves to a direction where there aren't any focusables, - the focus will "wrap" to the other direction of the screen. -* **BoundScope**: Non-wrapping scope menu - Controlled with `NavRequest::ScopeMove` - even when the focused element is not in this menu, but in a submenu - reachable from this one. -* **WrappingScope**: Wrapping scope menu - Controlled with `NavRequest::ScopeMove` even - when the focused element is not in this menu, but in a submenu reachable from this one. +```rust +pub struct MenuSetting { + /// Whether to wrap navigation. + /// + /// When the player moves to a direction where there aren't any focusables, + /// if this is true, the focus will "wrap" to the other direction of the screen. + pub wrapping: bool, + /// Whether this is a scope menu. + /// + /// A scope menu is controlled with [`NavRequest::ScopeMove`] + /// even when the focused element is not in this menu, but in a submenu + /// reachable from this one. + pub scope: bool, +} +``` #### Going back to a previous menus @@ -517,8 +511,6 @@ just to be able to spawn the menu you'll get to from that button. [`bevy-ui-navigation`] uses a proxy holding both the parent and the menu state info. - - That proxy component is `MenuSeed`, you can initialize it either with the `Name` or `Entity` of the parent focusable of the menu. A system will take all `MenuSeed` components and replace them with a `TreeMenu`. From fda1e081437159d8b72e4933db2cfeea98bd2841 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Sat, 23 Jul 2022 10:47:31 +0200 Subject: [PATCH 29/33] Add more caveat on migration based on implementation --- rfcs/41-ui-navigation.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index a1e7608f..fa7180c0 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -4,6 +4,8 @@ Introduce [`bevy-ui-navigation`] into the bevy tree. +[PR Status][this RFC's PR]. + By default this, amounts to replacing `Interaction` with `Focusable` in `bevy_ui`. On top of the current behavior, this adds the following capabilities: - Focus management for **gamepads** @@ -774,6 +776,20 @@ This doesn't mean we have to entirely give up hovering. It can simply be added back as a component independent from the focus system. +#### `Clicked` is not a focus state + +Compared to the current `Interaction` system, the new focus system +separates semantics of "action" events and "UI focus state". + +Meaning that `Interaction::Clicked`'s equivalent in the new navigation system +is the `NavEvent::NoChanges` event. + +This is important, as code relying on the "Clicked" focus state +will need to be replace with reaction to a `NavEvent`. + +Looking at the code changes in the existing UI examples in [this RFC's PR] +is a good indicator at how the code will need to be changed. + #### Interaction tree In this design, interaction is not "bubble up" as in the current `bevy_ui` @@ -840,3 +856,4 @@ misunderstanding how the navigation system work or programming errors. [`MenuSetting`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/enum.NavMenu.html [`UiProjectionQuery`]: https://github.com/nicopap/ui-navigation/blob/17c771b7f752cfd604f21056f9d4ca6772529c6f/src/resolve.rs#L378-L429 [`TreeMenu`]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L201-L216 +[this RFC's PR]: https://github.com/bevyengine/bevy/pull/5378 From a0210bcc2e87aa7547045520f2c6b010b7f44458 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 29 Jul 2022 19:01:02 +0200 Subject: [PATCH 30/33] Fix initial focus with multiple menus --- rfcs/41-ui-navigation.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index fa7180c0..fbd3e77e 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -670,6 +670,8 @@ self.focusables .or_else(root_prioritized) // Any prioritized focusable .or_else(any_prioritized) + // Any focusable within the root menu (if it exists) + .or_else(in_root_menu) // Any focusable .or_else(fallback) ``` From ca9d822aa7792fac75ea14bdcd833470e29b5e43 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Fri, 29 Jul 2022 19:06:53 +0200 Subject: [PATCH 31/33] Update details on changes to existing components --- rfcs/41-ui-navigation.md | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index fbd3e77e..66065cf3 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -729,6 +729,27 @@ sending the `NavRequest`, while using `world_to_viewport` in `MoveParam` for gamepad navigation. (or completely omitting gamepad support by providing a stub `MoveParam`) +#### How does this work with `FocusPolicy`? + +`FocusPolicy` is a way to prevent or allow mouse pointer "hover" +to go to an UI element below the one in front of the camera. +Think of it as a way to make "transparent" or "opaque" to mouse cursor cast +queries selectively UI elements. + +Since navigation is orthogonal to cursor pointing, we keep the `FocusPolicy`. +If the policy is set to `Capture`, the element will be focused by hover. +If the policy is set to `Pass`, the pointing algorithm tries to find another +element behind it. + +There is a major difference though: +There could be multiple objects under the cursor, +therefore several could be "active" at a time. +The navigation design described in this RFC disallows +multiple focused elements at a time. +This is a semantic change to `Interaction`, which is removed, +the user would have needed to migrate to `Focusable` already, +so the impact of this change is nil. + ## Future possibilities ### New and better mouse picking! @@ -775,9 +796,12 @@ separate from the navigation library. A fully generic navigation API cannot handle hover state. This doesn't mean we have to entirely give up hovering. -It can simply be added back as a component independent +It is simply added back as a component independent from the focus system. +Note that the new `Hover` component does not comply with `FocusPolicy`. +This allows reducing the complexity of the focus code. + #### `Clicked` is not a focus state Compared to the current `Interaction` system, the new focus system @@ -792,19 +816,6 @@ will need to be replace with reaction to a `NavEvent`. Looking at the code changes in the existing UI examples in [this RFC's PR] is a good indicator at how the code will need to be changed. -#### Interaction tree - -In this design, interaction is not "bubble up" as in the current `bevy_ui` -design. - -This is because there is a clear and exact distinction -in what can be "focused", what is a "menu", and everything else. -Focused element do not share their state with their parent. -The breadcrumb of menus defined [in the `MenuSetting` section](#MenuSetting) -is a very different concept from marking parents as focused. - -As such, `FocusPolicy` becomes useless and is removed. - ### How does this work with the spawn/despawn workflow? The current design imposes to the user that all UI nodes such a buttons and From d82c888b488b04f0971beb71acb557ed8099db6c Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Mon, 8 Aug 2022 18:16:04 +0200 Subject: [PATCH 32/33] Update to represent latest changes to ui-navigation --- rfcs/41-ui-navigation.md | 327 ++++++++++++++++++++++++++++----------- 1 file changed, 239 insertions(+), 88 deletions(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index 66065cf3..b1bf2151 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -108,15 +108,16 @@ for same-frame updates. For more advanced behaviors, the user should listen to the [`NavEvent`] event reader. -The default way of hooking your UI to your code in [`bevy-ui-navigation`] is +The default way of hooking the UI to code in [`bevy-ui-navigation`] is a bit rough: -* Mark your UI elements with either marker components or an enum + +* Mark the UI elements with either marker components or an enum * listen for [`NavEvent`] in a UI handling system, -* filter for the ones you care about (typically `NavEvent::NoChanges`), +* filter for the ones cared about (typically `NavEvent::NoChanges`), * check what [`NavRequest`] triggered it, * retrieve the focused entity from the event, -* check against your own queries what entity it is, -* write code for each case you want to handle +* check against other queries what entity it is, +* write code for each case to handle ```rust use bevy::prelude::*; @@ -148,23 +149,31 @@ fn handle_ui( } ``` -It could be useful to provide a higher level API on top of [`NavEvent`]. -[`bevy-ui-navigation`] has a module dedicated to it: [`event_helpers`]. -Using [`event_helpers`] would reduce the previous code to: +To make it easier to react to "activation" events, we provide a trait extension +to `EventReader`. This adds the `nav_iter` method, which returns a +wrapper struct that understands the concept of "activation": + ```rust -fn handle_ui(mut button_events: NavEventQuery<&MainMenuButton, With>) { - // NOTE: this will silently ignore multiple navigation event at the same frame. - // It should be a very rare occurrence. - match button_events.single_activated().ignore_remaining() { - // Do things when specific button is activated - Some(MainMenuButton::Start) => {} - Some(MainMenuButton::Options) => {} - Some(MainMenuButton::Exit) => {} - None => {} +fn handle_ui( + mut events: EventReader, + buttons: Query<&MainMenuButton, With>, +) { + for button in events.nav_iter().activated_in_query(&buttons) { + match button { + // Do things when specific button is activated + MainMenuButton::Start => {} + MainMenuButton::Options => {} + MainMenuButton::Exit => {} + } } } ``` +The current implementation is missing a method to handle mutable queries. + +A previous design added a custom `SystemParam` that the user could call +directly. The trait extension allows the user to seamlessly go from the wrapper +struct to the actual `EventReader` and vis-versa, which is a plus. ### Moving the cursor @@ -197,6 +206,15 @@ The exact mechanism for disabling default input is still to be designed. ### Creating a menu +[`bevy-ui-navigation`]'s menu system is purely optional, +and it is likely that a large section of users will ignore it +and content themselves with `Focusable`s, which is capable enough. + +The menu system is also the source of most of the complexity and pitfalls. + +However, having such a system available out of the box integrating seamlessly +with the existing focus system is just great user experience. + See the `bevy-ui-navigation` [`MenuSetting`] doc, [and example](https://github.com/nicopap/ui-navigation/blob/master/examples/menu_navigation.rs). @@ -211,11 +229,12 @@ Meaning you don't have to make your [`Focusable`]s direct children of the menu. struct MenuBundle { #[bundle] pub node_bundle: NodeBundle, - pub menu: MenuSeed, + pub menu: MenuBuilder, } ``` -See [relevant implementation section](#spawning-menus) for details on `MenuSeed`. +See [relevant implementation section](#spawning-menus) for details on +`MenuBuilder`. [`bevy-ui-navigation`] supports menus and inter-menu navigation out-of-the-box. Users will have to write code to hide/show menus, @@ -259,7 +278,7 @@ around the knob with the mouse. We then use mouse movement to move around the knob, and update the audio level based on the knob's position. -We send a `NavRequest::Free` when the player release's the left mouse button. +We send a `NavRequest::Unlock` when the player release's the left mouse button. ### Custom directional navigation @@ -284,11 +303,26 @@ The [`bevy-ui-navigation`] repo contains the code. ### Internal representation +`Focusable`s are just components, +if the user doesn't ever spawn a `MenuBundle`, +then there is really not that much to describe. + +Outside of menus, +most of the complexity resides in handling input and the event responses. +Which is already implemented and will be re-used by [`bevy-ui-navigation`]. + +For menus, to work, however, we design around a _navigation tree_. + The navigation tree is a set of relationships between entities with either the [`Focusable`] component or the [`TreeMenu`] component. -The hierarchy is specified by both the native bevy hierarchy -and the `focus_parent` of the [`TreeMenu`] component. +There are two relationships backed in UI navigation: + +1. A fully private one defining the "parent" of a menu in [`TreeMenu`] +2. A fully public one dependent on bevy_hierarchy Parent/Children + used to tell which focusables is in a menu. + A focusable is in a menu if it is a descendent of a [`TreeMenu`] + (with children of `Focusable`s and `TreeMenu`s within that menu culled) In the following screenshot, `Focusable`s are represented as circles, menus as rectangles, and the `focus_parent` by blue arrows @@ -356,6 +390,33 @@ NavEvent::NoChanges { See [relevant implementation section](#NavEvent) for details on `NavEvent`. +### The resolve algorithm + +This is just a plain-english description of the `resolve` function in +`resolve.rs`: + +The `TreeMenu::focus_parent` (the private relationship) +allows to define "links" between menus. + +Menus usually are not children of other menus, +so it can't use the Parent/Child relation + +Internally, what happens is: + +- `FocusOn(Entity)`: pick the current focused and the target entity. + Build the breadcrumbs from them to the root (this is easy with focus_parent), + then trim the common tail, update the relevant focusable's state. +- `Action`, check if any menu has its field `focus_parent` = current focused, + then set focused to that menu's `active_child`. +- `Cancel`, find this focusable's parent menu, set the new focused to its focus_parent +- `Move(Direction)` just call the `MenuNavigationStrategy`'s resolve_2d + with a list of all focusables within the focused's menu + +The navigation tree is "just there" in the ECS, +and traversed based on various entry points +depending on the `NavRequest` it received. +The direction of tree traversal is basically always from leaf to root. + ### Exposed API The crux of `bevy-ui-navigation` are the following types: @@ -405,6 +466,11 @@ a generic navigation system. (see [dedicated section](#hovered-is-not-covered)) element. * **Inert**: None of the above: This Focusable is neither Prioritized, Focused or Active. +* **Blocked**: + Prevents all interactions with this Focusable. + \ + This is equivalent to removing the Focusable component from the entity, + but without the latency. #### Focusable action types @@ -424,7 +490,7 @@ A `Focusable` can execute a variety of `FocusAction` when receiving enter the parent one). * **Lock**: If we receive `NavRequest::Action` while this `Focusable` is - focused, the navigation system will freeze until `NavRequest::Free` + focused, the navigation system will freeze until `NavRequest::Unlock` is received, sending a `NavEvent::Unlocked`. \ This is useful to implement widgets with complex controls you don't @@ -459,21 +525,27 @@ If you want to specify which [`Focusable`] should be focused first when entering a menu, you should mark one of the children of this menu with `Focusable::prioritized`. -#### Invariants +#### Limitations + +Menu navigation relies heavily on the bevy hierarchy being consistent. +You might get inconsistent state under the folowing conditions: -**You need to follow those rules to avoid panics**: -1. A menu must have **at least one** [`Focusable`] child in the UI - hierarchy. -2. There must not be a menu loop. I.e.: a way to go from menu A to menu B and - then from menu B to menu A while never going back. +- You despawned an entity in the `FocusState::Active` state +- You changed the parent of a [`Focusable`] member of a menu to a new menu. + +The navigation system might still work as expected, +however, `Focusable::state` may be missleading +for the length of one frame. #### Panics -Thankfully, programming errors are caught early and you'll probably get a -panic fairly quickly if you don't follow the invariants. -* Invariant (1) panics as soon as you add the menu without focusable - children. -* Invariant (2) panics if the focus goes into a menu loop. +**Menu loops will cause a panic**. +A menu loop is a way to go from menu A to menu B and +then from menu B to menu A while never going back. + +Don't worry though, menu loops are really hard to make by accident, +and it will only panic if you use a `NavRequest::FocusOn(entity)` +where `entity` is inside a menu loop. #### Fields @@ -513,13 +585,13 @@ just to be able to spawn the menu you'll get to from that button. [`bevy-ui-navigation`] uses a proxy holding both the parent and the menu state info. -That proxy component is `MenuSeed`, +That proxy component is `MenuBuilder`, you can initialize it either with the `Name` or `Entity` of the parent focusable of the menu. -A system will take all `MenuSeed` components and replace them with a `TreeMenu`. +A system will take all `MenuBuilder` components and replace them with a `TreeMenu`. This also enables front-loading the check of whether the parent focusable `Entity` is indeed a `Focusable`. -Before turning the `MenuSeed` into a `TreeMenu`, +Before turning the `MenuBuilder` into a `TreeMenu`, we check that the alleged parent is indeed `Focusable`, and panic otherwise. If we couldn't assume parents to be focusable, @@ -621,10 +693,12 @@ pub trait MenuNavigationStrategy { To ease implementation of widgets, such as sliders, some focusables can "lock" the UI, preventing all other form of interactions. -Another system will be in charge of unlocking the UI by sending a `NavRequest::Free`. +Another system will be in charge of unlocking the UI by sending a `NavRequest::Unlock`. + +It is also possible to send lock requests with `NavRequest::Lock`. -The default input sets the escape key to send a `NavRequest::Free`, -but the user may chose to `Free` through any other mean. +The default input sets the escape key to send a `NavRequest::Unlock`, +but the user may chose to `Unlock` through any other mean. This might be better served to a locking component that block navigation of specific menu trees, @@ -633,46 +707,65 @@ but I didn't find the need for such fine-grained controls. ### Input customization -[See the `bevy-ui-navigation` README example](https://github.com/nicopap/ui-navigation#simple-case). - -The `NavigationPlugin` can be added to the app in two ways: -1. With `NavigationPlugin` -2. With `DefaultNavigationPlugins` - -(1) only inserts the navigation systems and resources, -while (2) also adds the input systems generating the `NavRequest`s -for gamepads, mouse and keyboard. - -This enables convenient defaults -while letting users insert their own custom input logic. - -More customization is available: -[`MenuNavigationStrategy`](#MenuNavigationStrategy) allows customizing -the spacial resolution between focusables within a menu. +As a default bevy plugin, letting user change defaults is a bit more tricky, +here is how it is done is the current implementation: +```rust +App::new() + .add_plugins_with(DefaultPlugins, |group| { + group + // Add your own cursor navigation system + // by using `NavigationPlugin::::new()` + // See the [`bevy_ui_navigation::MenuNavigationStrategy`] trait. + // + // You can use a custom gamepad directional handling system if you want to. + // This could be useful if you want such navigation in 3d space + // to take into consideration the 3d camera perspective. + // + // Here we use the default one provided by `bevy_ui` because + // it is already capable of handling navigation in 2d space + // (even using `Sprite` over UI `Node`) + .add(BevyUiNavigationPlugin::new()) + // Prevent `UiPlugin` from adding the default input systems for navigation. + // We want to add our own mouse input system (mouse_pointer_system). + .add(UiPlugin { + default_navigation: false, + }) + }) + // Since gamepad input already works for Sprite-based menus, + // we add back the default gamepad input handling from `bevy_ui`. + // default_gamepad_input depends on NavigationInputMapping so we + // need to also add this resource back. + .init_resource::() + // can manually add back the systems defined in bevy_ui_navigation + .add_system(default_gamepad_input.before(NavRequestSystem)) + // And add user-defined one as well. In this example, + // we removed the default mouse input system and replaced it with our own. + .add_system(mouse_pointer_system.before(NavRequestSystem)) +``` ### What to focus first? Gamepad input assumes an initially focused element for navigating "move in direction" or "press action". -At the very beginning of execution, there are no focused element, -so we need fallbacks. +At the very beginning of execution, +or after having despawned the focused element, +there are no focused element, so we need fallbacks. + By default, it is any `Focusable` when none are focused yet. But the user can spawn a `Focusable` as `prioritized` and the algorithm will prefer it when no focused nodes exist yet. ```rust -self.focusables - .iter() - // The focused focusable. - .find_map(|(e, focus)| (focus.state == Focused).then(|| e)) - // Prioritized focusable within the root menu (if it exists) - .or_else(root_prioritized) - // Any prioritized focusable +// The focused focusable if it exists. +focused + // Any focusable in the "active" menu, if there is such a thing + .or_else(any_in_active) + // Any focusable marked as prioritized if there is one .or_else(any_prioritized) - // Any focusable within the root menu (if it exists) - .or_else(in_root_menu) - // Any focusable + // Any focusable in the root menu, if there is one + .or_else(any_in_root) + // Just any focusable at all, if there is one .or_else(fallback) ``` @@ -750,6 +843,18 @@ This is a semantic change to `Interaction`, which is removed, the user would have needed to migrate to `Focusable` already, so the impact of this change is nil. +### Panics on menu loop + +It is possible to define a cycle of menu connections, +and this will cause a panic at run time. + +However, it's hard to define a menu cycle by accident. +Because the `Entity` used to access a menu is a property of the menu itself, +So it can only have a single parent. Which makes it harder to define a loop. + +It is still possible to define a loop, and this will cause a panic the moment +the loop is entered. + ## Future possibilities ### New and better mouse picking! @@ -774,13 +879,45 @@ to identify which cursor originated the request, and keeping track in the `FocusState` which cursor the `Focused`, `Activated` and `Prioritized` variants are for. -### Higher level wrappers +### Higher level menu wrapper + +A visual wrapper for menu navigation that handles `display` `Style`s of menus +when they are entered and left. + +### Tab navigation + +We should add an optional navigation system +that allows navigating through `Focusable`s using keyboard, such as tab. + +### User placement of the `TreeMenu` insertion systems + +It should be possible for the user to add the insertion system wherever they +want, this would allow lower latency in spawning the UI. + +However, those systems currently depend on private types, +this requires a design rethinking to allow to expose those systems without +exposing internals that will blow up in the user's face. + +### Optimization + +I took care to avoid quadratic behaviors in the navigation algorithm, +but the linear constant factor is fairly high: + +- Multiple iteration of the list of `Focusable`s at times +- a few allocations per `NavRequest` +- multiple recursive exploration of hierarchy tree at times + +### More robustness + +As mentioned in [a future section](#Menu-hierarchy-invariants), +the menu system is frail to changes to the hierarchy, +mostly affecting `Active` menus. + +However, I'm not sure how bad the hierarchy changes affect navigation, +and it would be a mistake to "not panic at all cost." +A panic is an opportunity to teach the user a better way of doing things. +Not panicking might result in making ui-navigation harder to use. -In this RFC we discussed the possibility of improved API in a few places: -- The `bevy-ui-navigation` [`event_helpers`] module, to simplify interacting - with `NavEvent`s -- A visual wrapper for menu navigation that handles `display` `Style`s of menus - when they are entered and left ## Drawbacks and design limitations @@ -805,7 +942,7 @@ This allows reducing the complexity of the focus code. #### `Clicked` is not a focus state Compared to the current `Interaction` system, the new focus system -separates semantics of "action" events and "UI focus state". +separates semantics of "action" events and "UI focus state." Meaning that `Interaction::Clicked`'s equivalent in the new navigation system is the `NavEvent::NoChanges` event. @@ -813,6 +950,17 @@ is the `NavEvent::NoChanges` event. This is important, as code relying on the "Clicked" focus state will need to be replace with reaction to a `NavEvent`. +`Clicked` also allowed to change the button style +when the mouse button is held down. + +We somewhat emulate that by first providing focus on pressing down +and sending `NavRequest::Action` on mouse up. + +However, the corresponding `NavEvent` for button activation will only be +sent after the `NavRequest::Action` is sent. +Which means, activation happens when before `Interaction` +went from `Clicked` to `Hovered`. + Looking at the code changes in the existing UI examples in [this RFC's PR] is a good indicator at how the code will need to be changed. @@ -838,23 +986,26 @@ spawn/despawn design. Something that automates that could be a natural extension of the exposed API. It may also help end users go with the best design first. -### User upheld invariants +### Menu hierarchy invariants + +For menus to be optimally ergonomic to declare and use, +we chose which `Focusable` is in a menu by looking at the bevy hierarchy. + +This has downsides, mainly, that the user can at any time completely change the +hierarchy: despawn entities, change their parent, remove their parents etc. -I think the most problematic aspect of this implementation is that we push on -the user the responsibility of upholding invariants. This is not a safety -issue, but it leads to panics. The invariants are: -1. Never create a cycle of menu connections -2. Each menu must at least have one contained `Focusable` +It is impossible to anticipate how the user will change the menu hierarchy, +yet we rely on it. -The upside is that the invariants become only relevant if the user _opts into_ -`MenuSetting`. Already, this is a choice made by the user, so they can be made -aware of the requirements in the `MenuSetting` documentation. +This problem only shows up when using menus, therefore, +it will likely only be a concern for more complex games with navigable menus. -If the invariants are not upheld, the navigation system will simply panic. +In addition, because it's an _opt in_ feature, +we can advertise the dangers of manipulating the hierarchy of menus +in the documentation for the components and bundles used to create menus. -It may be possible to be resilient to the violation of the invariants, but I -think it's better to panic, because violation will most likely come from -misunderstanding how the navigation system work or programming errors. +The failure mods are unclear, +the implementation might in fact be resilient to hierarchy changes. [mouse picking]: https://github.com/aevyrie/bevy_mod_picking/ [`bevy-ui-navigation`]: https://github.com/nicopap/ui-navigation From 542dbd72a389e12decb386c0e0bb56a689271da8 Mon Sep 17 00:00:00 2001 From: Nicola Papale Date: Wed, 31 Aug 2022 13:04:06 +0200 Subject: [PATCH 33/33] Add note on ergonomic event wrapper --- rfcs/41-ui-navigation.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/rfcs/41-ui-navigation.md b/rfcs/41-ui-navigation.md index b1bf2151..10420f14 100644 --- a/rfcs/41-ui-navigation.md +++ b/rfcs/41-ui-navigation.md @@ -918,6 +918,27 @@ and it would be a mistake to "not panic at all cost." A panic is an opportunity to teach the user a better way of doing things. Not panicking might result in making ui-navigation harder to use. +### SystemParams for better ergonomics + +Previously, [`bevy-ui-navigation`] had a [`event_helpers`] module, +to help smooth out usage, it adds two `SystemParam` to simplify +combining focus events with ECS state. + +The proposed design replaces the module by a single trait +implemented on `EventReader`: [`NavEventReaderExt`]. + +However, it's not clear which design to chose. + +`SystemParam`: +- Difficult to discover, you need to know about the types +- Add more API surface +- Hides the internals of handling events + +`NavEventReaderExt` extension trait: +- Not fluent rust +- Need to use an additional method in the body of the system `.nav_iter()` +- Even more difficult to discover. +- Can easily go from a specialized method to just calling `EventReader::iter` ## Drawbacks and design limitations @@ -1010,7 +1031,7 @@ the implementation might in fact be resilient to hierarchy changes. [mouse picking]: https://github.com/aevyrie/bevy_mod_picking/ [`bevy-ui-navigation`]: https://github.com/nicopap/ui-navigation [ui-nav-arch]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L1-L33 -[`event_helpers`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/event_helpers/index.html +[`event_helpers`]: https://docs.rs/bevy-ui-navigation/0.18.0/bevy_ui_navigation/event_helpers/index.html [gambit-slider]: https://github.com/team-plover/warlocks-gambit/blob/3f8132fbbd6cce124a1cd102755dad228035b2dc/src/ui/main_menu.rs#L59-L95 [`NavRequest`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/events/enum.NavRequest.html [`NavEvent`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/events/enum.NavEvent.html @@ -1021,3 +1042,4 @@ the implementation might in fact be resilient to hierarchy changes. [`UiProjectionQuery`]: https://github.com/nicopap/ui-navigation/blob/17c771b7f752cfd604f21056f9d4ca6772529c6f/src/resolve.rs#L378-L429 [`TreeMenu`]: https://github.com/nicopap/ui-navigation/blob/30c828a465f9d4c440d0ce6e97051a5f7fafa425/src/resolve.rs#L201-L216 [this RFC's PR]: https://github.com/bevyengine/bevy/pull/5378 +[`NavEventReaderExt`]: https://docs.rs/bevy-ui-navigation/latest/bevy_ui_navigation/events/trait.NavEventReaderExt.html