From dfed2fbae3c48cc37288eeca75f4015ab3270860 Mon Sep 17 00:00:00 2001 From: Chris Bracken Date: Tue, 20 Oct 2020 17:47:55 -0700 Subject: [PATCH 1/2] Add support for IME-based text input on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This updates the Win32 desktop embedder to support input method (abbreviated IM or IME) composing regions. In contrast to languages such as English, where keyboard input is managed keystroke-by-keystroke, languages such as Japanese require a multi-step input process wherein the user begins a composing sequence, during which point their keystrokes are captured by a system input method and converted into a text sequence. During composing, the user is able to edit the composing range and manage the conversion from keyboard input to text before eventually committing the text to the underlying text input field. To illustrate this, in Japanese, this sequence might look something like the following: 1. User types 'k'. The character 'k' is added to the composing region. Typically, the text 'k' will be inserted inline into the underlying text field but the composing range will be highlighted in some manner, frequently with a highlight or underline. 2. User types 'a'. The composing range is replaced with the phonetic kana character 'か' (ka). The composing range continues to be highlighted. 3. User types 'k'. The character 'k' is appended to the composing range such that the highlighted text is now 'かｋ' 4. User types 'u'. The trailing 'k' is replaced with the phonetic kana character 'く' (ku) such that the composing range now reads 'かく' The composing range continues to be highlighted. 5. The user presses the space bar to convert the kana characters to kanji. The composing range is replaced with '書く' (kaku: to write). 6. The user presses the space bar again to show other conversions. The user's configured input method (for example, ibus) pops up a completions menu populated with alternatives such as 各 (kaku: every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角 (kaku: angle), etc. 7. The user uses the arrow keys to navigate the completions menu and select the alternative to input. As they do, the inline composing region in the text field is updated. It continues to be highlighted or underlined. 8. The user hits enter to commit the composing region. The text is committed to the underlying text field and the visual highlighting is removed. 9. If the user presses another key, a new composing sequence begins. If a selection is present when composing begins, it is preserved until the first keypress of input is received, at which point the selection is deleted. If a composing sequence is aborted before the first keypress, the selection is preserved. Creating a new selection (with the mouse, for example) aborts composing and the composing region is automatically committed. A composing range and selection, both with an extent, are not permitted to co-exist. During composing, keyboard navigation via the arrow keys, or home and end (or equivalent shortcuts) is restricted to the composing range, as are deletions via backspace and the delete key. This patch adds two new private convenience methods, `editing_range` and `text_range`. The former returns the range for which editing is currently active -- the composing range, if composing, otherwise the full range of the text. The latter, returns a range from position 0 (inclusive) to `text_.length()` exclusive. Windows IME support revolves around two main UI windows: the composition window and the candidate window. The composition window is a system window overlaid within the current window bounds which renders the composing string. Flutter already renders this string itself, so we request that this window be hidden. The candidate window is a system-rendered dropdown that displays all possible conversions for the text in the composing region. Since the contents of this window are specific to the particular IME in use, and because the user may have installed one or more third-party IMEs, Flutter does not attempt to render this as a widget itself, but rather delegates to the system-rendered window. The lifecycle of IME composing begins follows the following event order: 1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to DefWindowProc() in order to hide the composition window, which Flutter already renders itself. 2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new text. We use this event to set Flutter's TextInputModel into composing mode. 3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces, or deletes text in the composing region, navigates with their cursor within the composing region, or selects a new conversion candidate from the candidates list. 4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text in the composing region and decides to commit or abort the composition. Additionally, the following IME-related events are emitted but not yet handled: * WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using the system language selection menu. Since there some language-specific behaviours to IMEs, we may want to make use of this in the future. * WM_IME_NOTIFY: triggered to notify of various status events such as opening or closing the candidate window, setting the conversion mode, etc. None of these are relevant to Flutter at the moment. * WM_IME_REQUEST: triggered to notify of various commands/requests such as triggering reconversion of text, which should begin composition mode, insert the selected text into the composing region, and allow the user to select new alternative candidates for the text in question before re-committing their new selection. This patch doesn't support this feature, but it's an important feature that we should support in future. --- ci/licenses_golden/licenses_flutter | 2 + shell/platform/common/cpp/text_input_model.cc | 12 +- shell/platform/common/cpp/text_input_model.h | 16 +- shell/platform/windows/BUILD.gn | 7 +- .../platform/windows/flutter_windows_view.cc | 32 ++++ shell/platform/windows/flutter_windows_view.h | 27 ++++ shell/platform/windows/key_event_handler.cc | 13 ++ shell/platform/windows/key_event_handler.h | 9 ++ .../platform/windows/keyboard_hook_handler.h | 19 +++ .../windows/testing/mock_win32_window.h | 3 + shell/platform/windows/text_input_manager.cc | 139 ++++++++++++++++++ shell/platform/windows/text_input_manager.h | 103 +++++++++++++ shell/platform/windows/text_input_plugin.cc | 66 +++++++-- shell/platform/windows/text_input_plugin.h | 9 ++ .../platform/windows/win32_flutter_window.cc | 15 +- shell/platform/windows/win32_flutter_window.h | 9 ++ .../windows/win32_flutter_window_unittests.cc | 6 + shell/platform/windows/win32_window.cc | 118 ++++++++++++++- shell/platform/windows/win32_window.h | 42 ++++++ .../windows/window_binding_handler_delegate.h | 18 +++ 20 files changed, 641 insertions(+), 24 deletions(-) create mode 100644 shell/platform/windows/text_input_manager.cc create mode 100644 shell/platform/windows/text_input_manager.h diff --git a/ci/licenses_golden/licenses_flutter b/ci/licenses_golden/licenses_flutter index 40a5df92483d1..a2cc16781ef02 100644 --- a/ci/licenses_golden/licenses_flutter +++ b/ci/licenses_golden/licenses_flutter @@ -1497,6 +1497,8 @@ FILE: ../../../flutter/shell/platform/windows/task_runner_win32.cc FILE: ../../../flutter/shell/platform/windows/task_runner_win32.h FILE: ../../../flutter/shell/platform/windows/task_runner_winuwp.cc FILE: ../../../flutter/shell/platform/windows/task_runner_winuwp.h +FILE: ../../../flutter/shell/platform/windows/text_input_manager.cc +FILE: ../../../flutter/shell/platform/windows/text_input_manager.h FILE: ../../../flutter/shell/platform/windows/text_input_plugin.cc FILE: ../../../flutter/shell/platform/windows/text_input_plugin.h FILE: ../../../flutter/shell/platform/windows/text_input_plugin_delegate.h diff --git a/shell/platform/common/cpp/text_input_model.cc b/shell/platform/common/cpp/text_input_model.cc index 649baede20f51..70d8001c1651d 100644 --- a/shell/platform/common/cpp/text_input_model.cc +++ b/shell/platform/common/cpp/text_input_model.cc @@ -67,11 +67,7 @@ void TextInputModel::BeginComposing() { composing_range_ = TextRange(selection_.start()); } -void TextInputModel::UpdateComposingText(const std::string& composing_text) { - std::wstring_convert, char16_t> - utf16_converter; - std::u16string text = utf16_converter.from_bytes(composing_text); - +void TextInputModel::UpdateComposingText(const std::u16string& text) { // Preserve selection if we get a no-op update to the composing region. if (text.length() == 0 && composing_range_.collapsed()) { return; @@ -82,6 +78,12 @@ void TextInputModel::UpdateComposingText(const std::string& composing_text) { selection_ = TextRange(composing_range_.end()); } +void TextInputModel::UpdateComposingText(const std::string& text) { + std::wstring_convert, char16_t> + utf16_converter; + UpdateComposingText(utf16_converter.from_bytes(text)); +} + void TextInputModel::CommitComposing() { // Preserve selection if no composing text was entered. if (composing_range_.collapsed()) { diff --git a/shell/platform/common/cpp/text_input_model.h b/shell/platform/common/cpp/text_input_model.h index a9560b7b1483d..340a2b62b8738 100644 --- a/shell/platform/common/cpp/text_input_model.h +++ b/shell/platform/common/cpp/text_input_model.h @@ -46,13 +46,21 @@ class TextInputModel { // are restricted to the composing range. void BeginComposing(); - // Replaces the composing range with new text. + // Replaces the composing range with new UTF-16 text. // // If a selection of non-zero length exists, it is deleted if the composing // text is non-empty. The composing range is adjusted to the length of - // |composing_text| and the selection base and offset are set to the end of - // the composing range. - void UpdateComposingText(const std::string& composing_text); + // |text| and the selection base and offset are set to the end of the + // composing range. + void UpdateComposingText(const std::u16string& text); + + // Replaces the composing range with new UTF-8 text. + // + // If a selection of non-zero length exists, it is deleted if the composing + // text is non-empty. The composing range is adjusted to the length of + // |text| and the selection base and offset are set to the end of the + // composing range. + void UpdateComposingText(const std::string& text); // Commits composing range to the string. // diff --git a/shell/platform/windows/BUILD.gn b/shell/platform/windows/BUILD.gn index a1ce9f169c8b8..ad4d4ee8e3135 100644 --- a/shell/platform/windows/BUILD.gn +++ b/shell/platform/windows/BUILD.gn @@ -68,6 +68,8 @@ source_set("flutter_windows_source") { "string_conversion.h", "system_utils.h", "task_runner.h", + "text_input_manager.cc", + "text_input_manager.h", "text_input_plugin.cc", "text_input_plugin.h", "window_binding_handler.h", @@ -103,7 +105,10 @@ source_set("flutter_windows_source") { "win32_window_proc_delegate_manager.h", ] - libs = [ "dwmapi.lib" ] + libs = [ + "dwmapi.lib", + "imm32.lib", + ] } configs += [ diff --git a/shell/platform/windows/flutter_windows_view.cc b/shell/platform/windows/flutter_windows_view.cc index a5c709357305b..2f3b4c282698c 100644 --- a/shell/platform/windows/flutter_windows_view.cc +++ b/shell/platform/windows/flutter_windows_view.cc @@ -137,6 +137,19 @@ bool FlutterWindowsView::OnKey(int key, return SendKey(key, scancode, action, character, extended); } +void FlutterWindowsView::OnComposeBegin() { + SendComposeBegin(); +} + +void FlutterWindowsView::OnComposeEnd() { + SendComposeEnd(); +} + +void FlutterWindowsView::OnComposeChange(const std::u16string& text, + int cursor_pos) { + SendComposeChange(text, cursor_pos); +} + void FlutterWindowsView::OnScroll(double x, double y, double delta_x, @@ -240,6 +253,25 @@ bool FlutterWindowsView::SendKey(int key, return false; } +void FlutterWindowsView::SendComposeBegin() { + for (const auto& handler : keyboard_hook_handlers_) { + handler->ComposeBeginHook(); + } +} + +void FlutterWindowsView::SendComposeEnd() { + for (const auto& handler : keyboard_hook_handlers_) { + handler->ComposeEndHook(); + } +} + +void FlutterWindowsView::SendComposeChange(const std::u16string& text, + int cursor_pos) { + for (const auto& handler : keyboard_hook_handlers_) { + handler->ComposeChangeHook(text, cursor_pos); + } +} + void FlutterWindowsView::SendScroll(double x, double y, double delta_x, diff --git a/shell/platform/windows/flutter_windows_view.h b/shell/platform/windows/flutter_windows_view.h index 0998f04912685..c868a4c7b1ee2 100644 --- a/shell/platform/windows/flutter_windows_view.h +++ b/shell/platform/windows/flutter_windows_view.h @@ -106,6 +106,15 @@ class FlutterWindowsView : public WindowBindingHandlerDelegate, char32_t character, bool extended) override; + // |WindowBindingHandlerDelegate| + void OnComposeBegin() override; + + // |WindowBindingHandlerDelegate| + void OnComposeEnd() override; + + // |WindowBindingHandlerDelegate| + void OnComposeChange(const std::u16string& text, int cursor_pos) override; + // |WindowBindingHandlerDelegate| void OnScroll(double x, double y, @@ -185,6 +194,24 @@ class FlutterWindowsView : public WindowBindingHandlerDelegate, char32_t character, bool extended); + // Reports an IME compose begin event. + // + // Triggered when the user begins editing composing text using a multi-step + // input method such as in CJK text input. + void SendComposeBegin(); + + // Reports an IME compose end event. + // + // Triggered when the user commits the composing text while using a multi-step + // input method such as in CJK text input. + void SendComposeEnd(); + + // Reports an IME composing region change event. + // + // Triggered when the user edits the composing text while using a multi-step + // input method such as in CJK text input. + void SendComposeChange(const std::u16string& text, int cursor_pos); + // Reports scroll wheel events to Flutter engine. void SendScroll(double x, double y, diff --git a/shell/platform/windows/key_event_handler.cc b/shell/platform/windows/key_event_handler.cc index b4b1b8149c3fd..67942e1e12edb 100644 --- a/shell/platform/windows/key_event_handler.cc +++ b/shell/platform/windows/key_event_handler.cc @@ -243,4 +243,17 @@ bool KeyEventHandler::KeyboardHook(FlutterWindowsView* view, return true; } +void KeyEventHandler::ComposeBeginHook() { + // Ignore. +} + +void KeyEventHandler::ComposeEndHook() { + // Ignore. +} + +void KeyEventHandler::ComposeChangeHook(const std::u16string& text, + int cursor_pos) { + // Ignore. +} + } // namespace flutter diff --git a/shell/platform/windows/key_event_handler.h b/shell/platform/windows/key_event_handler.h index 86433e6f6322a..62f1b768afcd8 100644 --- a/shell/platform/windows/key_event_handler.h +++ b/shell/platform/windows/key_event_handler.h @@ -44,6 +44,15 @@ class KeyEventHandler : public KeyboardHookHandler { void TextHook(FlutterWindowsView* window, const std::u16string& text) override; + // |KeyboardHookHandler| + void ComposeBeginHook() override; + + // |KeyboardHookHandler| + void ComposeEndHook() override; + + // |KeyboardHookHandler| + void ComposeChangeHook(const std::u16string& text, int cursor_pos) override; + private: KEYBDINPUT* FindPendingEvent(uint64_t id); void RemovePendingEvent(uint64_t id); diff --git a/shell/platform/windows/keyboard_hook_handler.h b/shell/platform/windows/keyboard_hook_handler.h index 37bcbafbeb2b3..527a83687f6dc 100644 --- a/shell/platform/windows/keyboard_hook_handler.h +++ b/shell/platform/windows/keyboard_hook_handler.h @@ -32,6 +32,25 @@ class KeyboardHookHandler { // A function for hooking into Unicode text input. virtual void TextHook(FlutterWindowsView* view, const std::u16string& text) = 0; + + // Handler for IME compose begin events. + // + // Triggered when the user begins editing composing text using a multi-step + // input method such as in CJK text input. + virtual void ComposeBeginHook() = 0; + + // Handler for IME compose end events. + // + // Triggered when the user commits the composing text while using a multi-step + // input method such as in CJK text input. + virtual void ComposeEndHook() = 0; + + // Handler for IME compose change events. + // + // Triggered when the user edits the composing text while using a multi-step + // input method such as in CJK text input. + virtual void ComposeChangeHook(const std::u16string& text, + int cursor_pos) = 0; }; } // namespace flutter diff --git a/shell/platform/windows/testing/mock_win32_window.h b/shell/platform/windows/testing/mock_win32_window.h index dc7765e95959a..08417e144990f 100644 --- a/shell/platform/windows/testing/mock_win32_window.h +++ b/shell/platform/windows/testing/mock_win32_window.h @@ -41,6 +41,9 @@ class MockWin32Window : public Win32Window { MOCK_METHOD1(OnText, void(const std::u16string&)); MOCK_METHOD5(OnKey, bool(int, int, int, char32_t, bool)); MOCK_METHOD2(OnScroll, void(double, double)); + MOCK_METHOD0(OnComposeBegin, void()); + MOCK_METHOD0(OnComposeEnd, void()); + MOCK_METHOD2(OnComposeChange, void(const std::u16string&, int)); }; } // namespace testing diff --git a/shell/platform/windows/text_input_manager.cc b/shell/platform/windows/text_input_manager.cc new file mode 100644 index 0000000000000..9e9cfb7ae0639 --- /dev/null +++ b/shell/platform/windows/text_input_manager.cc @@ -0,0 +1,139 @@ +// Copyright 2013 The Flutter Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +// Copyright 2013 The Flutter Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +#include "flutter/shell/platform/windows/text_input_manager.h" + +#include + +#include + +namespace flutter { + +void TextInputManager::SetWindowHandle(HWND window_handle) { + window_handle_ = window_handle; +} + +void TextInputManager::CreateImeWindow() { + if (window_handle_ == nullptr) { + return; + } + + // Some IMEs ignore calls to ::ImmSetCandidateWindow() and use the position of + // the current system caret instead via ::GetCaretPos(). In order to behave + // as expected with these IMEs, we create a temporary system caret. + if (!ime_active_) { + ::CreateCaret(window_handle_, nullptr, 1, 1); + } + ime_active_ = true; + + // Set the position of the IME windows. + UpdateImeWindow(); +} + +void TextInputManager::DestroyImeWindow() { + if (window_handle_ == nullptr) { + return; + } + + // Destroy the system caret created in CreateImeWindow(). + if (ime_active_) { + ::DestroyCaret(); + } + ime_active_ = false; +} + +void TextInputManager::UpdateImeWindow() { + if (window_handle_ == nullptr) { + return; + } + + HIMC imm_context = ::ImmGetContext(window_handle_); + if (imm_context) { + MoveImeWindow(imm_context); + ::ImmReleaseContext(window_handle_, imm_context); + } +} + +void TextInputManager::UpdateCaretRect(const Rect& rect) { + caret_rect_ = rect; + + if (window_handle_ == nullptr) { + return; + } + + // TODO(cbracken): wrap these in an RAII container. + HIMC imm_context = ::ImmGetContext(window_handle_); + if (imm_context) { + MoveImeWindow(imm_context); + ::ImmReleaseContext(window_handle_, imm_context); + } +} + +long TextInputManager::GetComposingCursorPosition() const { + if (window_handle_ == nullptr) { + return false; + } + + HIMC imm_context = ::ImmGetContext(window_handle_); + if (imm_context) { + // Read the cursor position within the composing string. + const int pos = + ImmGetCompositionStringW(imm_context, GCS_CURSORPOS, nullptr, 0); + ::ImmReleaseContext(window_handle_, imm_context); + return pos; + } + return -1; +} + +std::optional TextInputManager::GetComposingString() const { + return GetString(GCS_COMPSTR); +} + +std::optional TextInputManager::GetResultString() const { + return GetString(GCS_RESULTSTR); +} + +std::optional TextInputManager::GetString(int type) const { + if (window_handle_ == nullptr || !ime_active_) { + return std::nullopt; + } + HIMC imm_context = ::ImmGetContext(window_handle_); + if (imm_context) { + // Read the composing string length. + const long compose_bytes = + ::ImmGetCompositionString(imm_context, type, nullptr, 0); + const long compose_length = compose_bytes / sizeof(wchar_t); + if (compose_length <= 0) { + ::ImmReleaseContext(window_handle_, imm_context); + return std::nullopt; + } + + std::u16string text(compose_length, '\0'); + ::ImmGetCompositionString(imm_context, type, &text[0], compose_bytes); + ::ImmReleaseContext(window_handle_, imm_context); + return text; + } + return std::nullopt; +} + +void TextInputManager::MoveImeWindow(HIMC imm_context) { + if (GetFocus() != window_handle_ || !ime_active_) { + return; + } + LONG x = caret_rect_.left(); + LONG y = caret_rect_.top(); + ::SetCaretPos(x, y); + + COMPOSITIONFORM cf = {CFS_POINT, {x, y}}; + ::ImmSetCompositionWindow(imm_context, &cf); + + CANDIDATEFORM candidate_form = {0, CFS_CANDIDATEPOS, {x, y}, {0, 0, 0, 0}}; + ::ImmSetCandidateWindow(imm_context, &candidate_form); +} + +} // namespace flutter diff --git a/shell/platform/windows/text_input_manager.h b/shell/platform/windows/text_input_manager.h new file mode 100644 index 0000000000000..6f3a64270cc47 --- /dev/null +++ b/shell/platform/windows/text_input_manager.h @@ -0,0 +1,103 @@ +// Copyright 2013 The Flutter Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +#ifndef FLUTTER_SHELL_PLATFORM_WINDOWS_TEXT_INPUT_MANAGER_H_ +#define FLUTTER_SHELL_PLATFORM_WINDOWS_TEXT_INPUT_MANAGER_H_ + +#include +#include + +#include +#include + +#include "flutter/shell/platform/common/cpp/geometry.h" + +namespace flutter { + +// Management interface for IME-based text input on Windows. +// +// When inputting text in CJK languages, text is entered via a multi-step +// process, where direct keyboard input is buffered into a composing string, +// which is then converted into the desired characters by selecting from a +// candidates list and committing the change to the string. +// +// This implementation wraps the Win32 IMM32 APIs and provides a mechanism for +// creating and positioning the IME window, a system caret, and the candidates +// list as well as for accessing composing and results string contents. +class TextInputManager { + public: + TextInputManager() noexcept = default; + ~TextInputManager() = default; + + TextInputManager(const TextInputManager&) = delete; + TextInputManager& operator=(const TextInputManager&) = delete; + + // Sets the window handle with which the IME is associated. + void SetWindowHandle(HWND window_handle); + + // Creates a new IME window and system caret. + // + // This method should be invoked in response to the WM_IME_SETCONTEXT and + // WM_IME_STARTCOMPOSITION events. + void CreateImeWindow(); + + // Destroys the current IME window and system caret. + // + // This method should be invoked in response to the WM_IME_ENDCOMPOSITION + // event. + void DestroyImeWindow(); + + // Updates the current IME window and system caret position. + // + // This method should be invoked when handling user input via + // WM_IME_COMPOSITION events. + void UpdateImeWindow(); + + // Updates the current IME window and system caret position. + // + // This method should be invoked when handling cursor position/size updates. + void UpdateCaretRect(const Rect& rect); + + // Returns the cursor position within the composing range. + // + // The returned value is relative to the start of the composing range. + long GetComposingCursorPosition() const; + + // Returns the contents of the composing string. + // + // This may be called in response to WM_IME_COMPOSITION events where the + // GCS_COMPSTR flag is set in the lparam. In some IMEs, this string may also + // be set in events where the GCS_RESULTSTR flag is set. This contains the + // in-progress composing string. + std::optional GetComposingString() const; + + // Returns the contents of the result string. + // + // This may be called in response to WM_IME_COMPOSITION events where the + // GCS_RESULTSTR flag is set in the lparam. This contains the final string to + // be committed in the composing region when composition is ended. + std::optional GetResultString() const; + + private: + // Returns either the composing string or result string based on the value of + // the |type| parameter. + std::optional GetString(int type) const; + + // Moves the IME composing and candidates windows to the current caret + // position. + void MoveImeWindow(HIMC imm_context); + + // The window with which the IME windows are associated. + HWND window_handle_ = nullptr; + + // True if IME-based composing is active. + bool ime_active_ = false; + + // The system caret rect. + Rect caret_rect_ = {{0, 0}, {0, 0}}; +}; + +} // namespace flutter + +#endif // FLUTTER_SHELL_PLATFORM_WINDOWS_TEXT_INPUT_MANAGER_H_ diff --git a/shell/platform/windows/text_input_plugin.cc b/shell/platform/windows/text_input_plugin.cc index 69c66de50ba97..59dbe59c63273 100644 --- a/shell/platform/windows/text_input_plugin.cc +++ b/shell/platform/windows/text_input_plugin.cc @@ -102,6 +102,26 @@ TextInputPlugin::TextInputPlugin(flutter::BinaryMessenger* messenger, TextInputPlugin::~TextInputPlugin() = default; +void TextInputPlugin::ComposeBeginHook() { + active_model_->BeginComposing(); + SendStateUpdate(*active_model_); +} + +void TextInputPlugin::ComposeEndHook() { + active_model_->CommitComposing(); + active_model_->EndComposing(); + SendStateUpdate(*active_model_); +} + +void TextInputPlugin::ComposeChangeHook(const std::u16string& text, + int cursor_pos) { + active_model_->AddText(text); + cursor_pos += active_model_->composing_range().base(); + active_model_->UpdateComposingText(text); + active_model_->SetSelection(TextRange(cursor_pos, cursor_pos)); + SendStateUpdate(*active_model_); +} + void TextInputPlugin::HandleMethodCall( const flutter::MethodCall& method_call, std::unique_ptr> result) { @@ -167,23 +187,41 @@ void TextInputPlugin::HandleMethodCall( "Set editing state has been invoked, but without text."); return; } - auto selection_base = args.FindMember(kSelectionBaseKey); - auto selection_extent = args.FindMember(kSelectionExtentKey); - if (selection_base == args.MemberEnd() || selection_base->value.IsNull() || - selection_extent == args.MemberEnd() || - selection_extent->value.IsNull()) { + auto base = args.FindMember(kSelectionBaseKey); + auto extent = args.FindMember(kSelectionExtentKey); + if (base == args.MemberEnd() || base->value.IsNull() || + extent == args.MemberEnd() || extent->value.IsNull()) { result->Error(kInternalConsistencyError, "Selection base/extent values invalid."); return; } // Flutter uses -1/-1 for invalid; translate that to 0/0 for the model. - int base = selection_base->value.GetInt(); - int extent = selection_extent->value.GetInt(); - if (base == -1 && extent == -1) { - base = extent = 0; + int selection_base = base->value.GetInt(); + int selection_extent = extent->value.GetInt(); + if (selection_base == -1 && selection_extent == -1) { + selection_base = selection_extent = 0; } active_model_->SetText(text->value.GetString()); - active_model_->SetSelection(TextRange(base, extent)); + active_model_->SetSelection(TextRange(selection_base, selection_extent)); + + base = args.FindMember(kComposingBaseKey); + extent = args.FindMember(kComposingExtentKey); + if (base == args.MemberEnd() || base->value.IsNull() || + extent == args.MemberEnd() || extent->value.IsNull()) { + result->Error(kInternalConsistencyError, + "Composing base/extent values invalid."); + return; + } + int composing_base = base->value.GetInt(); + int composing_extent = base->value.GetInt(); + if (composing_base == -1 && composing_extent == -1) { + active_model_->EndComposing(); + } else { + int composing_start = std::min(composing_base, composing_extent); + int cursor_offset = selection_base - composing_start; + active_model_->SetComposingRange( + TextRange(composing_base, composing_extent), cursor_offset); + } } else if (method.compare(kSetMarkedTextRect) == 0) { if (!method_call.arguments() || method_call.arguments()->IsNull()) { result->Error(kBadArgumentError, "Method invoked without args"); @@ -259,13 +297,17 @@ void TextInputPlugin::SendStateUpdate(const TextInputModel& model) { TextRange selection = model.selection(); rapidjson::Value editing_state(rapidjson::kObjectType); - editing_state.AddMember(kComposingBaseKey, -1, allocator); - editing_state.AddMember(kComposingExtentKey, -1, allocator); editing_state.AddMember(kSelectionAffinityKey, kAffinityDownstream, allocator); editing_state.AddMember(kSelectionBaseKey, selection.base(), allocator); editing_state.AddMember(kSelectionExtentKey, selection.extent(), allocator); editing_state.AddMember(kSelectionIsDirectionalKey, false, allocator); + + int composing_base = model.composing() ? model.composing_range().base() : -1; + int composing_extent = + model.composing() ? model.composing_range().extent() : -1; + editing_state.AddMember(kComposingBaseKey, composing_base, allocator); + editing_state.AddMember(kComposingExtentKey, composing_extent, allocator); editing_state.AddMember( kTextKey, rapidjson::Value(model.GetText(), allocator).Move(), allocator); args->PushBack(editing_state, allocator); diff --git a/shell/platform/windows/text_input_plugin.h b/shell/platform/windows/text_input_plugin.h index 47415365507bc..a18f025097135 100644 --- a/shell/platform/windows/text_input_plugin.h +++ b/shell/platform/windows/text_input_plugin.h @@ -43,6 +43,15 @@ class TextInputPlugin : public KeyboardHookHandler { // |KeyboardHookHandler| void TextHook(FlutterWindowsView* view, const std::u16string& text) override; + // |KeyboardHookHandler| + void ComposeBeginHook() override; + + // |KeyboardHookHandler| + void ComposeEndHook() override; + + // |KeyboardHookHandler| + void ComposeChangeHook(const std::u16string& text, int cursor_pos) override; + private: // Sends the current state of the given model to the Flutter engine. void SendStateUpdate(const TextInputModel& model); diff --git a/shell/platform/windows/win32_flutter_window.cc b/shell/platform/windows/win32_flutter_window.cc index 5915cd03b7a32..ba4e3880dad1d 100644 --- a/shell/platform/windows/win32_flutter_window.cc +++ b/shell/platform/windows/win32_flutter_window.cc @@ -171,6 +171,19 @@ bool Win32FlutterWindow::OnKey(int key, extended); } +void Win32FlutterWindow::OnComposeBegin() { + binding_handler_delegate_->OnComposeBegin(); +} + +void Win32FlutterWindow::OnComposeEnd() { + binding_handler_delegate_->OnComposeEnd(); +} + +void Win32FlutterWindow::OnComposeChange(const std::u16string& text, + int cursor_pos) { + binding_handler_delegate_->OnComposeChange(text, cursor_pos); +} + void Win32FlutterWindow::OnScroll(double delta_x, double delta_y) { POINT point; GetCursorPos(&point); @@ -181,7 +194,7 @@ void Win32FlutterWindow::OnScroll(double delta_x, double delta_y) { } void Win32FlutterWindow::UpdateCursorRect(const Rect& rect) { - // TODO(cbracken): Implement IMM candidate window positioning. + text_input_manager_.UpdateCaretRect(rect); } } // namespace flutter diff --git a/shell/platform/windows/win32_flutter_window.h b/shell/platform/windows/win32_flutter_window.h index 3e86d9da7c1de..15fd108fb2905 100644 --- a/shell/platform/windows/win32_flutter_window.h +++ b/shell/platform/windows/win32_flutter_window.h @@ -61,6 +61,15 @@ class Win32FlutterWindow : public Win32Window, public WindowBindingHandler { char32_t character, bool extended) override; + // |Win32Window| + void OnComposeBegin() override; + + // |Win32Window| + void OnComposeEnd() override; + + // |Win32Window| + void OnComposeChange(const std::u16string& text, int cursor_pos) override; + // |Win32Window| void OnScroll(double delta_x, double delta_y) override; diff --git a/shell/platform/windows/win32_flutter_window_unittests.cc b/shell/platform/windows/win32_flutter_window_unittests.cc index 8ba4b15f8d425..c53843e9a437e 100644 --- a/shell/platform/windows/win32_flutter_window_unittests.cc +++ b/shell/platform/windows/win32_flutter_window_unittests.cc @@ -69,6 +69,9 @@ class SpyKeyEventHandler : public KeyboardHookHandler { bool extended)); MOCK_METHOD2(TextHook, void(FlutterWindowsView* window, const std::u16string& text)); + MOCK_METHOD0(ComposeBeginHook, void()); + MOCK_METHOD0(ComposeEndHook, void()); + MOCK_METHOD2(ComposeChangeHook, void(const std::u16string& text, int cursor_pos)); private: std::unique_ptr real_implementation_; @@ -98,6 +101,9 @@ class SpyTextInputPlugin : public KeyboardHookHandler, bool extended)); MOCK_METHOD2(TextHook, void(FlutterWindowsView* window, const std::u16string& text)); + MOCK_METHOD0(ComposeBeginHook, void()); + MOCK_METHOD0(ComposeEndHook, void()); + MOCK_METHOD2(ComposeChangeHook, void(const std::u16string& text, int cursor_pos)); virtual void OnCursorRectUpdated(const Rect& rect) {} diff --git a/shell/platform/windows/win32_window.cc b/shell/platform/windows/win32_window.cc index 01432cfb0f6fe..92be90f4dda3a 100644 --- a/shell/platform/windows/win32_window.cc +++ b/shell/platform/windows/win32_window.cc @@ -4,10 +4,17 @@ #include "flutter/shell/platform/windows/win32_window.h" +#include + #include #include "win32_dpi_utils.h" +// KeyCode used to indicate key events to be handled by the IME. These include +// the kana key, fullwidth/halfwidth (zenkaku/hankaku) key, and keypresses when +// the IME is in composing mode. +static constexpr int kImeComposingKeyCode = 229; + namespace flutter { namespace { @@ -91,6 +98,7 @@ LRESULT CALLBACK Win32Window::WndProc(HWND const window, auto that = static_cast(cs->lpCreateParams); that->window_handle_ = window; + that->text_input_manager_.SetWindowHandle(window); } else if (Win32Window* that = GetThisFromHandle(window)) { return that->HandleMessage(message, wparam, lparam); } @@ -109,10 +117,72 @@ void Win32Window::TrackMouseLeaveEvent(HWND hwnd) { } } +void Win32Window::OnImeSetContext(UINT const message, + WPARAM const wparam, + LPARAM const lparam) { + if (wparam != 0) { + text_input_manager_.CreateImeWindow(); + } +} + +void Win32Window::OnImeStartComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam) { + text_input_manager_.CreateImeWindow(); + OnComposeBegin(); +} + +void Win32Window::OnImeComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam) { + // Update the IME window position. + text_input_manager_.UpdateImeWindow(); + + if (lparam & GCS_COMPSTR) { + // Read the in-progress composing string. + long pos = text_input_manager_.GetComposingCursorPosition(); + std::optional text = + text_input_manager_.GetComposingString(); + if (text) { + OnComposeChange(text.value(), pos); + } + } else if (lparam & GCS_RESULTSTR) { + // Read the committed composing string. + long pos = text_input_manager_.GetComposingCursorPosition(); + std::optional text = text_input_manager_.GetResultString(); + if (text) { + OnComposeChange(text.value(), pos); + } + // Next, try reading the composing string. Some Japanese IMEs send a message + // containing both a GCS_RESULTSTR and a GCS_COMPSTR when one composition is + // committed and another immediately started. + text = text_input_manager_.GetResultString(); + if (text) { + OnComposeChange(text.value(), pos); + } + } +} + +void Win32Window::OnImeEndComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam) { + text_input_manager_.DestroyImeWindow(); + OnComposeEnd(); +} + +void Win32Window::OnImeRequest(UINT const message, + WPARAM const wparam, + LPARAM const lparam) { + // TODO(cbracken): Handle IMR_RECONVERTSTRING, IMR_DOCUMENTFEED, + // and IMR_QUERYCHARPOSITION messages. + // https://github.com/flutter/flutter/issues/74547 +} + LRESULT Win32Window::HandleMessage(UINT const message, WPARAM const wparam, LPARAM const lparam) noexcept { + LPARAM result_lparam = lparam; int xPos = 0, yPos = 0; UINT width = 0, height = 0; UINT button_pressed = 0; @@ -152,6 +222,12 @@ Win32Window::HandleMessage(UINT const message, } break; } + case WM_SETFOCUS: + ::CreateCaret(window_handle_, nullptr, 1, 1); + break; + case WM_KILLFOCUS: + ::DestroyCaret(); + break; case WM_LBUTTONDOWN: case WM_RBUTTONDOWN: case WM_MBUTTONDOWN: @@ -198,6 +274,40 @@ Win32Window::HandleMessage(UINT const message, static_cast(WHEEL_DELTA)), 0.0); break; + case WM_INPUTLANGCHANGE: + // TODO(cbracken): pass this to TextInputManager to aid with + // language-specific issues. + break; + case WM_IME_SETCONTEXT: + OnImeSetContext(message, wparam, lparam); + // Strip the ISC_SHOWUICOMPOSITIONWINDOW bit from lparam before passing it + // to DefWindowProc() so that the composition window is hidden since + // Flutter renders the composing string itself. + result_lparam &= ~ISC_SHOWUICOMPOSITIONWINDOW; + break; + case WM_IME_STARTCOMPOSITION: + OnImeStartComposition(message, wparam, lparam); + // Suppress further processing by DefWindowProc() so that the default + // system IME style isn't used, but rather the one set in the + // WM_IME_SETCONTEXT handler. + return TRUE; + case WM_IME_COMPOSITION: + OnImeComposition(message, wparam, lparam); + if (lparam & GCS_RESULTSTR || lparam & GCS_COMPSTR) { + // Suppress further processing by DefWindowProc() since otherwise it + // will emit the result string as WM_CHAR messages on commit. Instead, + // committing the composing text to the EditableText string is handled + // in TextInputModel::CommitComposing, triggered by + // OnImeEndComposition(). + return TRUE; + } + break; + case WM_IME_ENDCOMPOSITION: + OnImeEndComposition(message, wparam, lparam); + return TRUE; + case WM_IME_REQUEST: + OnImeRequest(message, wparam, lparam); + break; case WM_UNICHAR: { // Tell third-pary app, we can support Unicode. if (wparam == UNICODE_NOCHAR) @@ -273,6 +383,12 @@ Win32Window::HandleMessage(UINT const message, break; } unsigned int keyCode(wparam); + if (keyCode == kImeComposingKeyCode) { + // This is an IME composing mode keypress that will be handled via + // WM_IME_* messages, which update the framework via updates to the text + // and composing range in text editing update messages. + break; + } const unsigned int scancode = (lparam >> 16) & 0xff; const bool extended = ((lparam >> 24) & 0x01) == 0x01; // If the key is a modifier, get its side. @@ -286,7 +402,7 @@ Win32Window::HandleMessage(UINT const message, break; } - return DefWindowProc(window_handle_, message, wparam, lparam); + return DefWindowProc(window_handle_, message, wparam, result_lparam); } UINT Win32Window::GetCurrentDPI() { diff --git a/shell/platform/windows/win32_window.h b/shell/platform/windows/win32_window.h index 2d964bb62540f..b1366877cb80d 100644 --- a/shell/platform/windows/win32_window.h +++ b/shell/platform/windows/win32_window.h @@ -11,6 +11,8 @@ #include #include +#include "flutter/shell/platform/windows/text_input_manager.h" + namespace flutter { // A class abstraction for a high DPI aware Win32 Window. Intended to be @@ -101,6 +103,43 @@ class Win32Window { char32_t character, bool extended) = 0; + // Called when IME composing begins. + virtual void OnComposeBegin() = 0; + + // Called when IME composing ends. + virtual void OnComposeEnd() = 0; + + // Called when IME composing text or cursor position changes. + virtual void OnComposeChange(const std::u16string& text, int cursor_pos) = 0; + + // Called when a window is activated in order to configure IME support for + // multi-step text input. + void OnImeSetContext(UINT const message, + WPARAM const wparam, + LPARAM const lparam); + + // Called when multi-step text input begins when using an IME. + void OnImeStartComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam); + + // Called when edits/commit of multi-step text input occurs when using an IME. + void OnImeComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam); + + // Called when multi-step text input ends when using an IME. + void OnImeEndComposition(UINT const message, + WPARAM const wparam, + LPARAM const lparam); + + // Called when the user triggers an IME-specific request such as input + // reconversion, where an existing input sequence is returned to composing + // mode to select an alternative candidate conversion. + void OnImeRequest(UINT const message, + WPARAM const wparam, + LPARAM const lparam); + // Called when mouse scrollwheel input occurs. virtual void OnScroll(double delta_x, double delta_y) = 0; @@ -142,6 +181,9 @@ class Win32Window { // Keeps track of the last key code produced by a WM_KEYDOWN or WM_SYSKEYDOWN // message. int keycode_for_char_message_ = 0; + + protected: + TextInputManager text_input_manager_; }; } // namespace flutter diff --git a/shell/platform/windows/window_binding_handler_delegate.h b/shell/platform/windows/window_binding_handler_delegate.h index 8ea914bde3e9e..8b8b1acc7669e 100644 --- a/shell/platform/windows/window_binding_handler_delegate.h +++ b/shell/platform/windows/window_binding_handler_delegate.h @@ -50,6 +50,24 @@ class WindowBindingHandlerDelegate { char32_t character, bool extended) = 0; + // Notifies the delegate that IME composing mode has begun. + // + // Triggered when the user begins editing composing text using a multi-step + // input method such as in CJK text input. + virtual void OnComposeBegin() = 0; + + // Notifies the delegate that IME composing mode has ended. + // + // Triggered when the user commits the composing text while using a multi-step + // input method such as in CJK text input. + virtual void OnComposeEnd() = 0; + + // Notifies the delegate that IME composing region contents have changed. + // + // Triggered when the user edits the composing text while using a multi-step + // input method such as in CJK text input. + virtual void OnComposeChange(const std::u16string& text, int cursor_pos) = 0; + // Notifies delegate that backing window size has recevied scroll. // Typically called by currently configured WindowBindingHandler virtual void OnScroll(double x, From 19b7117dc8fa783a28ebff877f05e8f70914d61f Mon Sep 17 00:00:00 2001 From: Chris Bracken Date: Sat, 23 Jan 2021 17:34:25 -0800 Subject: [PATCH 2/2] Simplify doc comment. --- shell/platform/windows/text_input_manager.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/shell/platform/windows/text_input_manager.h b/shell/platform/windows/text_input_manager.h index 6f3a64270cc47..cf21809f4e8c2 100644 --- a/shell/platform/windows/text_input_manager.h +++ b/shell/platform/windows/text_input_manager.h @@ -59,9 +59,7 @@ class TextInputManager { // This method should be invoked when handling cursor position/size updates. void UpdateCaretRect(const Rect& rect); - // Returns the cursor position within the composing range. - // - // The returned value is relative to the start of the composing range. + // Returns the cursor position relative to the start of the composing range. long GetComposingCursorPosition() const; // Returns the contents of the composing string.