Win32 a11y bridge and platform node delegates #29829

cbracken · 2021-11-19T02:24:10Z

This is the third in a series of patches adding accessibility support
for the Windows embedder. This patch wires in the Accessibility bridge,
and lands the core structure of the Windows FlutterPlatformNodeDelegate
and AccessibilityBridgeDelegate classes, including:

Instantiating the AccessibilityBridge when the semantics tree is
enabled.
Creating FlutterPlatformNodeDelegate wrappers for semantics tree
nodes.
Building and updating the accessibility tree on semantics updates.
Returning the native IAccessible objects when queried.
The scaffolding for an eventual UWP implementation. This is low
priority, but required if we don't want to break the UWP build.

Breaking this out so that the follow-up patches can be reviewed and
landed in smaller, independent chunks.

Issue: flutter/flutter#77838
Issue: flutter/flutter#93928

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

shell/platform/windows/flutter_windows_engine.cc

shell/platform/windows/flutter_windows_view.cc

third_party/accessibility/ax/ax_node.h

chunhtai · 2021-11-19T20:56:49Z

shell/platform/windows/window_win32.cc

+    gfx::NativeViewAccessible root_iaccessible = GetNativeViewAccessible();
+    if (root_iaccessible) {
+      LresultFromObject(IID_IAccessible, wparam, root_iaccessible);
+    }


is there a way to actively attach the root accessibility node to the view?

The first time this method is called root_iaccessible will be null because the update will only come after one frame delay.

btw, Is this also how chromium listens to the accessibility services?

This is how the root accessibility node is attached to the view.

When Windows wants the accessibility node associated with the HWND it fires a WM_GETOBJECT event into the main event loop, which we then handle in this method. We look up the root AXPlatformNodeWin (which implements IAccessible) above in GetNativeViewAccessible() and return it via LresultFromObject() here.

shell/platform/windows/flutter_windows_engine.h

chunhtai · 2021-11-19T20:57:50Z

shell/platform/windows/flutter_windows_engine.cc

+      [](const FlutterSemanticsCustomAction* action, void* user_data) {
+        auto host = static_cast<FlutterWindowsEngine*>(user_data);
+        if (!action || action->id == kFlutterSemanticsNodeIdBatchEnd) {
+          host->accessibility_bridge_->CommitUpdates();


We probably need to attach root accessibility node to the view as well, this will be the first time the accessibility tree is avaliable

Replied above about how attaching the root node is done via WM_GETOBJECT.

sounds good

chunhtai · 2021-11-19T21:03:36Z

I understand this is only part of the patch, but since it is already hooked up to the flutter engine, we should have a simple test to just make sure the thing doesn't crash.

Maybe just some test to make sure you can turn semantics on and off and receive some update without crashing the test

This is the third in a series of patches adding accessibility support for the Windows embedder. This patch wires in the Accessibility bridge, and lands the core structure of the Windows FlutterPlatformNodeDelegate and AccessibilityBridgeDelegate classes, including: * Instantiating the AccessibilityBridge when the semantics tree is enabled. * Creating FlutterPlatformNodeDelegate wrappers for semantics tree nodes. * Building and updating the accessibility tree on semantics updates. * Returning the native IAccessible objects when queried. Breaking this out so that the follow-up patches can be reviewed and landed in smaller, independent chunks. Issue: flutter/flutter#77838

There's nothing to do here.

Verify that a windows IAccessible COM object is generated.

cbracken · 2021-11-29T19:21:51Z

Maybe just some test to make sure you can turn semantics on and off and receive some update without crashing the test.

Added.

chunhtai

LGTM, I didn't test it locally since i don't have a device, but the wiring logic looks good

chunhtai · 2021-11-29T20:42:25Z

shell/platform/windows/flutter_platform_node_delegate_win32.cc

+void FlutterPlatformNodeDelegateWin32::Init(std::weak_ptr<OwnerBridge> bridge,
+                                            ui::AXNode* node) {
+  FlutterPlatformNodeDelegate::Init(bridge, node);
+  ax_platform_node_ = ui::AXPlatformNode::Create(this);


Where is the fragment root created? Is it in a later patch?

Fragment roots are only needed if we implement UIA, which is the newer a11y API. Talked to @clarkezone who mentioned we're safe with sticking with MSAA for the time being.

chunhtai · 2021-11-29T20:44:35Z

shell/platform/windows/flutter_windows_engine.cc

+      [](const FlutterSemanticsCustomAction* action, void* user_data) {
+        auto host = static_cast<FlutterWindowsEngine*>(user_data);
+        if (!action || action->id == kFlutterSemanticsNodeIdBatchEnd) {
+          host->accessibility_bridge_->CommitUpdates();


sounds good

…#29829)

cbracken requested a review from chunhtai November 19, 2021 02:24

flutter-dashboard bot added the platform-windows label Nov 19, 2021

google-cla bot added the cla: yes label Nov 19, 2021

cbracken added the Work in progress (WIP) Not ready (yet) for review! label Nov 19, 2021

cbracken changed the title ~~Wire in for Win32 AccessibilityBridge~~ Win32 a11y bridge and platform node delegates Nov 19, 2021

chunhtai reviewed Nov 19, 2021

View reviewed changes

cbracken added 9 commits November 28, 2021 20:41

review: return weak_ptr

512013c

Add custom action update

b23e78b

Implement FlutterPlatformNodeDelegateWin32::GetBoundsRect

3ab733c

Add UIA COM object lookup

384f0a0

Formatting

d58c4c7

Eliminate ctor TODOs

1d04059

There's nothing to do here.

Formatting

c405bf4

Add unittest to verify axtree construction

c687518

Verify that a windows IAccessible COM object is generated.

cbracken added 3 commits November 29, 2021 11:33

Test value as well

2c8b6cc

Check role

147081e

Minor cleanup/clarification for readability

51b18c2

chunhtai approved these changes Nov 29, 2021

View reviewed changes

cbracken merged commit 57ac30a into flutter:main Nov 29, 2021

cbracken deleted the a11y-windows-delegates branch November 29, 2021 21:23

engine-flutter-autoroll mentioned this pull request Nov 30, 2021

Roll Engine from 7ba215c2c6a0 to f37334353f48 (25 revisions) flutter/flutter#94435

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 30, 2021

57ac30a Win32 a11y bridge and platform node delegates (flutter/engine…

d3e1d96

…#29829)

Win32 a11y bridge and platform node delegates #29829

Win32 a11y bridge and platform node delegates #29829

Uh oh!

Conversation

cbracken commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chunhtai commented Nov 19, 2021

Uh oh!

cbracken commented Nov 29, 2021

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cbracken commented Nov 19, 2021 •

edited

Loading