Tracking issue for IME / composition support #1497

murarth · 2020-03-06T17:56:20Z

chrisduerr · 2020-04-04T17:18:33Z

As a consumer of winit I'd recommend against naming these events CompositionEvent. Applications that deal with display servers will already know that term in connection with window compositing and it could easily be confused with something related to that.

It also does not fall in line with the existing winit naming of functions like set_ime_position.

I think it would be much better to have a more self descriptive name for this event that cannot be confused with anything else related to the winit project. I'd strongly suggest that IME should at least be part of the name.

pickfire · 2020-06-26T09:58:56Z

Oh, I thought CompositionEvent means some kind of compose key thing, I never used those and don't even know how to use compose key, I had only used IME to input characters not in the keyboard.

garasubo · 2020-06-26T13:03:28Z

@pickfire This event naming comes from web specification (Ref: https://developer.mozilla.org/en-US/docs/Web/API/Element/compositionstart_event). And IME will create those composition events.
But as @chrisduerr is pointed out, I agree that calling those events as "composition" is confusing.

garasubo · 2020-07-10T07:36:09Z

Proposed the rename these events to IMEPreeditEvent in #1622

kaiwk · 2020-08-20T15:17:22Z

~~For mac, maybe we can use this API? composedString~~

Seems setMarkedText does the trick:

https://bugzilla.mozilla.org/show_bug.cgi?id=875674

jakerr · 2020-08-21T10:59:58Z

Just FYI here's a similar problem being solved in the glfw project that might provide some hints / inspiration: glfw/glfw#658

kchibisov · 2020-09-27T15:54:30Z

I've decided to take a look on the proposed events wrt compositing, since I've been working on IME handling in winit for Wayland recently.

Right now I can see that the current API that is being proposed looks more like

pub enum CompositionEvent {
    CompositionStart(String),
    CompositionUpdate(String, usize),
    CompositionEnd(String),
}

If I were modelling it from Wayland side it'll be more like.

pub enum CompositionEvent {
    /// IME got enabled on the surface you're on.
    CompositionEnabled
    /// We start a composition, a cursors are offsets into string, which could represent a 'highlight' string. `None` for both means hidden.
    CompositionPreedit(text: String, cursor_start: Option<usize>, cursor_end: Option<usize>)
    /// Deletes surrounding text before and after current cursor.
    CompositionDeleteSurroundingText(before_length: usize, after_length: usize)
    /// Composition is done, and that's the text that should be inserted into widget.
    CompositionCommit(text: String)
    /// Composition is done, user must process all events and apply them atomically
    CompositionDone
    /// IME focus got lost, no need to process any IME stuff anymore.
    CompositionDisabled
}

So those are just plain events that we have straight from a Wayland protocol. The thing is that it works in some kind of atomic commits way, if you think about it. So on CompositionDone you must perform the following:

Replace existing preedit string with the cursor.
Delete requested surrounding text.
Insert commit string with the cursor at its end.
Calculate surrounding text to send.
Insert new preedit text in cursor position.
Place cursor inside preedit text.

Now the question is how to model that with the proposed IME events.

So first of all I don't need CompostionDone event, since this is an event when I'd send things downstream, meaning that I don't need CompositionDeleteSurroundingText. And it seems like I don't need DeleteSurroundingText, since I can model it via CompositionPreedit. CompositionCommit(text: String) is indeed needed so the user will know when to insert text.

So if we now apply that to the proposed composition event it'll look more like.

pub enum CompositionEvent {
    CompositionEnabled,
    CompositionPreedit(String, Option<usize>, Option<usize>),
    CompositionCommit(String),
    CompositionDisabled,
}

I guess it should make it a bit clear, so we removed 1 event for users to carry about when editing takes place.

However what wasn't mentioned is that how user interacts with IME via requests. Right now
in winit we only have fn set_ime_position, which is just some basic stuff. However user may want to
opt for IME handling, via e.g. fn enable_ime, so they will react that way on CompositionEnabled(basically means that user got IME opened). And they could want to hide IME sometimes, when they interact with a keyboard, but it doesn't require IME things, like a game, where you have a chat with IME support, so they will need fn disable_ime.

Also, the user is likely want to set text around the cursor and a cursor position, so they will need set_surrounding_text, and also a content type of the current text being edited, with set_content_type, so IME can provide better suggestions.

In the end Window will get

fn set_ime_surrounding_text(text: String, cursor: usize);
fn enable_ime_handling();
fn disable_ime_handling();
fn set_contents_type(type: ContentType);

The changes shouldn't affect existing work on IME in winit much, and purely about ergonomics and additions.

@garasubo since you've started initial work on updating IME handling in winit, I'm curious whether proposed changes here make sense, and are possible on X11.

@kaiwk As the one who send composition event implementation. Are proposed changes sound reasonable to you and possible on macOS, or macOS could need a bit more things to work, nicely?

@chrisduerr As a winit consumer that deals with text input, and that should implement IME handling, does the proposed API sounds ergonomic to implement? One benefit is that you'll know precisely when you actually have IME, so you won't call to set_ime_position until you've got an event from a winit that IME was enabled for some window.

Also, since winit doesn't have IME handling on a lot of platforms to land IME handling APIs we can only implement it as a concept on platforms that support it, like X11/macOS, and soon™ Wayland. Platforms that don't support IME in particular could just send IME text via ReceivedChar event like IME is working right now.

pickfire · 2020-09-27T17:25:58Z

One benefit is that you'll know precisely when you actually have IME, so you won't call to set_ime_position until you've got an event from a winit that IME was enabled for some window.

Based on my experience with xinput on X11, I believe it is good to be able to call set_ime_position when the IME was enabled rather than updating it for every key press or some weird solution to make some timer to update it once it a while.

garasubo · 2020-09-28T10:20:46Z

However what wasn't mentioned is that how user interacts with IME via requests.

That's a great idea. But for me, adding those APIs sounds out of scope of this issue. My original motivation is to get IME related event from the window system to implement CompositionEvent API in the web browser engine. So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

kaiwk · 2020-09-28T12:19:14Z

Are proposed changes sound reasonable to you and possible on macOS, or macOS could need a bit more things to work, nicely?

Actually, i'm not very familiar about IME, the PR just fix my own problem, but i'll see what i can do for it. Is there any detailed doc about surrounding text and content type here? thanks.

kchibisov · 2020-09-28T14:22:55Z

So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

My motivation was to change enum to those 4 variants, so it's in scope of that issue, since I don't quit like the 3 variants we have right now. We can add window methods later however I'd really like to see enum reworked, so users stop calling IME functions without IME being enabled.

pickfire · 2020-09-28T15:11:43Z

Can it commit partially?

garasubo · 2020-09-29T10:47:00Z

So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

My motivation was to change enum to those 4 variants, so it's in scope of that issue, since I don't quit like the 3 variants we have right now. We can add window methods later however I'd really like to see enum reworked, so users stop calling IME functions without IME being enabled.

I see. I meant that adding API to control IME from window (e.g. enable_ime_handling) could be an independent issue, but either way is fine. I can help x11 implementation anyway.

I'm not sure what is the surrounding text concept. I think in x11 there is no such concept. Could you explain this more specifically? Maybe part of preedit string to convert, like I type "きょうもいいてんきですね" in IME and try to convert "きょう" to "今日" firstly, then, "きょう" is surrounding text and preedit string is "きょうもいいてんきですね"?
Also, when can the second variant of CompositPreedit (cursor_start) be None? In x11, a cursor pos is always defined.

kchibisov · 2020-10-01T08:25:00Z

I see. I meant that adding API to control IME from window (e.g. enable_ime_handling) could be an independent issue, but either way is fine. I can help x11 implementation anyway.

Yeah, I agree, we can do it separately.

I'm not sure what is the surrounding text concept. I think in x11 there is no such concept. Could you explain this more specifically? Maybe part of preedit string to convert, like I type "きょうもいいてんきですね" in IME and try to convert "きょう" to "今日" firstly, then, "きょう" is surrounding text and preedit string is "きょうもいいてんきですね"?

It's for when you already have some text around and start editing, so you tell IME that you have certain things around, and your cursor is at some position. You're not required to send this request on Wayland and if you don't know what is around your cursor you don't send it. So if before starting edition you had きょう and a cursor right behind that text, you send this string and a cursor position right after the last char, so IME will know that it already has きょう, so suggestions are not empty right away, but include your text.

Also, when can the second variant of CompositPreedit (cursor_start) be None? In x11, a cursor pos is always defined.

On Wayland it's None when you should hide a cursor, I'm not sure how to expose that, since Wayland passes negative numbers here, so for rust I've translated that part into None. If you have an idea how to translate that concept it'll be nice.

kchibisov · 2020-10-01T08:28:24Z

Can it commit partially?

The purpose of Commit on IMEs is that you insert text to a widget and move cursor to a new position. You can't commit partially, since it's basically 'apply change'. While you're editing you're getting preedits, which informs about the current string, but once user hit a key that commits its input like Enter(this is handled by IME) you get commit and start new editing.

@kchibisov

Unify `with_app_id` and `with_class` methods Both APIs are used to set application name. This commit unifies the API between Wayland and X11, so downstream applications can remove platform specific code when using `WindowBuilderExtUnix`. Fixes rust-windowing#1739. Unify behavior of `resizable` across platforms This makes X11 and Wayland follow Windows and macOS, so the size of the window could be set even though it has resizable attribute set to false. Fixes rust-windowing#2242. Fix assigning the wrong monitor when receiving Windows move events (rust-windowing#2266) Fix embedded NULs in C wide strings returned from Windows API (rust-windowing#2264) On Wayland, fix hiding cursors on GNOME `wl_pointer::set_cursor` expects a serial number of the last `wl_pointer::enter` event. However other calls expect latest observed pointer serial, so this commit tracks both and use them as required by specification. Fixes rust-windowing#2273. Bump windows-sys version to 0.36 (rust-windowing#2277) Add new `Ime` event for desktop platforms This commit brings new Ime event to account for preedit state of input method, also adding `Window::set_ime_allowed` to toggle IME input on the particular window. This commit implements API as designed in rust-windowing#1497 for desktop platforms. On Wayland, provide option for better CSD While most compositors provide server side decorations, the GNOME does not, and won't provide them. Also Wayland clients must render client side decorations. Winit was already drawing some decorations, however they were bad looking and provided no text rendering, so the title was missing. However this commit makes use of the SCTK external frame similar to GTK's Adwaita theme supporting text rendering and looking similar to other GTK applications. Fixes rust-windowing#1967. Fix warnings on nightly rust (rust-windowing#2295) This was causing CI to fail: https://github.com/rust-windowing/winit/runs/6506026326 On macOS, emit resize event on `frame_did_change` When the window switches mode from normal to tabbed one, it doesn't get resized, however the frame gets resized. This commit makes winit to track resizes when frame changes instead of window. Fixes rust-windowing#2191. Reorganize `EventLoopBuilder::build()` platform documentation Since there's a "Platform-specific" header, it makes sense to put the Linux-specific part under it. On the other hand, "Can only be called on the main thread." is true for all platforms, not just iOS, so there is no reason to call it out for iOS specifically. [Windows] Avoid GetModuleHandle(NULL) (rust-windowing#2301) Use get_instance_handle() over GetModuleHandle(NULL) On Windows, fix reported cursor position. (rust-windowing#2311) When clicking and moving the cursor out of the window negative coordinates were not handled correctly. Revert "On Wayland, fix resize not propagating properly" This reverts commit 78e5a39. It was discovered that in some cases mesa will lock the back buffer, e.g. when making context current, leading to resize missing. Given that applications can restructure their rendering to account for that, and that winit isn't limited to playing nice with mesa reverting the original commit. Set `WindowBuilder` to must_use Add X11 opt-in function for device events Previously on X11, by default all global events were broadcasted to every winit application. This unnecessarily drains battery due to excessive CPU usage when moving the mouse. To resolve this, device events are now ignored by default and users must manually opt into it using `EventLoopWindowTarget::set_filter_device_events`. Fixes (rust-windowing#1634) on Linux. Prevent null dereference on X11 with bad locale Remove old dialog fix that is superseded by rust-windowing#2027 (rust-windowing#2292) This fixes the run_return loop never returning on macos when using multiple windows Migrate from lazy_static to once_cell macOS: Emit LoopDestroyed on CMD+Q (rust-windowing#2073) override applicationWillTerminate: On Android, use `HasRawWindowHandle` directly from the `ndk` crate (rust-windowing#2318) The `ndk` crate now implements [`HasRawWindowHandle` directly on `NativeWindow`], relieving the burden to reimplement it on `winit`. [`HasRawWindowHandle` directly on `NativeWindow`]: rust-mobile/ndk#274 Run clippy on CI Fixes rust-windowing#1402. Make `set_device_event_filter` non-mut Commit f10a984 added `EventLoopWindowTarget::set_device_event_filter` with for a mutable reference, however most winit APIs work with immutable references, so altering API to play nicely with existing APIs. This also disables device event filtering on debug example. Make `WindowAttributes` private (rust-windowing#2134) * Make `WindowAttributes` private, and move its documentation * Reorder WindowAttributes title and fullscreen to match method order Build docs on `docs.rs` for iOS and Android as well (rust-windowing#2324) Remove core-video-sys dependency (rust-windowing#2326) Hasn't been updated in over 2 years - many open PRs, seems abandoned. Is the cause of several duplicate dependencies in our dependency tree! Fix macOS 32bit (rust-windowing#2327) Documentation cleanup (rust-windowing#2328) * Remove redundant documentation links * Add note to README about windows not showing up on Wayland * Fix documentation links * Small documentation fixes * Add note about doing stuff after StartCause::Init on macOS Add `WindowBuilder::transparent` This is required to help hardware accelerated libraries like glutin that accept WindowBuilder instead of RawWindowHandle, since the api to access builder properties directly was removed. Follow up to 44288f6. Refine `Window::set_cursor_grab` API This commit renames `Window::set_cursor_grab` to `Window::set_cursor_grab_mode`. The new API now accepts enumeration to control the way cursor grab is performed. The value could be: `lock`, `confine`, or `none`. This commit also implements `Window::set_cursor_position` for Wayland, since it's tied to locked cursor. Implements API from rust-windowing#1677. examples/window_run_return: Enable on Android (rust-windowing#2321) Android also supports `EventLoopExtRunReturn`. The user will still have to follow the README to turn this example into a `cdylib` and add the `ndk_glue::main()` initialization attribute, though. Fix doubled device events on X11 Fixes rust-windowing#2332 macOS: disallow_highdpi will set explicity the value to avoid the SO value by default (rust-windowing#2339) ci: Disallow warnings in rustdoc and test private items (rust-windowing#2341) Make sure `cargo doc` runs cleanly without any warnings in the CI - some recently introduced but still allowing a PR to get merged. In case someone wishes to add docs on private items, make sure those adhere to the same standards. Bump smithay-client-toolkit to v0.16.0 Disallow multiple EventLoop creation Fix conflict in `WindowFlags` on Windows Map XK_Caps_Lock to VirtualKeyCode::Capital (rust-windowing#1864) This allows applications to handle events for the caps lock key under X11 Less redundancy and improve fullscreen in examples Remove examples/minimize which is redundant Implement From<u64> for WindowId and vise-versa This should help downstream applications to expose WindowId to the end users via e.g. IPC to control particular windows in multi window systems. examples/multiwindow.rs: ignore synthetic key press events Fix infinite recursion in `WindowId` conversion methods Add 'WindowEvent::Occluded(bool)' This commits and an event to track window occlusion state, which could help optimize rendering downstream. Add `refresh_rate_millihertz` for `MonitorHandle` This also alters `VideoMode::refresh_rate` to `VideoMode::refresh_rate_millihertz` which now returns monitor refresh rate in mHz. On Wayland send Focused(false) for new window On Wayland winit will always get an explicit focused event from the system and will transfer it downstream. So send focused false to enforce it. On Wayland, drop wl_surface on window close web: Manually emit focused event on mouse click (rust-windowing#2202) * Manually emit focused event on mouse click * Update CHANGELOG.md Co-authored-by: Markus Røyset <[email protected]> web: Add `EventLoop::spawn` (rust-windowing#2208) * web: Add `EventLoop::spawn` This is the same as `EventLoop::run`, but doesn't throw an exception in order to return `!`. I decided to name it `spawn` rather than `run_web` because I think that's more descriptive, but I'm happy to change it to `run_web`. Resolves rust-windowing#1714 * Update src/platform/web.rs * Fix outdated names Co-authored-by: Markus Røyset <[email protected]> Fix changelog entry for `EventLoopExtWebSys` (rust-windowing#2372) android: Hold `NativeWindow` lock until after notifying the user with `Event::Suspended` (rust-windowing#2307) This applies rust-mobile/ndk#117 on the `winit` side: Android destroys its window/surface as soon as the user returns from [`onNativeWindowDestroyed`], and we "fixed" this on the `ndk-glue` side by sending the `WindowDestroyed` event before locking the window and removing it: this lock has to wait for any user of `ndk-glue` - ie. `winit` - to give up its readlock on the window, which is what we utilize here to give users of `winit` "time" to destroy any resource created on top of a `RawWindowHandle`. since we can't pass the user a `RawWindowHandle` through the `HasRawWindowHandle` trait we have to document this case explicitly and keep the lock alive on the `winit` side instead. [`onNativeWindowDestroyed`]: https://developer.android.com/ndk/reference/struct/a-native-activity-callbacks#onnativewindowdestroyed web: add `with_prevent_default`, `with_focusable` (rust-windowing#2365) * web: add `with_prevent_default`, `with_focusable` `with_prevent_default` controls whether `event.preventDefault` is called `with_focusable` controls whether `tabindex` is added Fixes rust-windowing#1768 * Remove extra space from CHANGELOG windows: Use correct value for mouse wheel delta (rust-windowing#2374) Make winit focus take activity into account on Windows (rust-windowing#2159) winit's notion of "focus" is very simple; you're either focused or not. However, Windows has both notions of focused window and active window and paying attention only to WM_SETFOCUS/WM_KILLFOCUS can cause a window to believe the user is interacting with it when they're not. (this manifests when a user switches to another application between when a winit application starts and it creates its first window) Fix typos (rust-windowing#2375) Bump sctk-adwaita to 0.4.1 This should force the use of system libraries for Fontconfig and freetype instead of building them with cmake if missing. This also fixes compilation failures on nightly. Fixes rust-windowing#2373. Tidy up "platform-specifc" doc sections (rust-windowing#2356) * Tidy up "platform-specific" doc sections * Unrelated grammatical fix * Subjective improvements Android: avoid deadlocks while handling UserEvent (rust-windowing#2343) Replace `Arc<Mutex<VecDeque<T>>` by `mpsc` Update raw-window-handle to v0.5.0 This updates raw-window-handle to v0.5.0. On macOS, fix confirmed character inserted When confirming input in e.g. Korean IME or using characters like `+` winit was sending those twice, once via `Ime::Commit` and the other one via `ReceivedCharacter`, since those events weren't generating any `Ime::Preedit` and were forwarded due to `do_command_by_selector`. Add method to hook xlib error handler This should help glutin to handle errors coming from GLX and offer multithreading support in a safe way. Fixes rust-windowing#2378. Windows: apply skip taskbar state when taskbar is restarted (rust-windowing#2380) Fix hiding a maximized window On Windows (rust-windowing#2336) Bump `ndk` and `ndk-glue` dependencies to stable `0.7.0` release (rust-windowing#2392) Fix type hint reference for xlib hook Consistently deliver a Resumed event on all platforms To be more consistent with mobile platforms this updates the Windows, macOS, Wayland, X11 and Web backends to all emit a Resumed event immediately after the initial `NewEvents(StartCause::Init)` event. The documentation for Suspended and Resumed has also been updated to provide general recommendations for how to handle Suspended and Resumed events in portable applications as well as providing Android and iOS specific details. This consistency makes it possible to write applications that lazily initialize their graphics state when the application resumes without any platform-specific knowledge. Previously, applications that wanted to run on Android and other systems would have to maintain two, mutually-exclusive, initialization paths. Note: This patch does nothing to guarantee that Suspended events will be delivered. It's still reasonable to say that most OSs without a formal lifecycle for applications will simply never "suspend" your application. There are currently no known portability issues caused by not delivering `Suspended` events consistently and technically it's not possible to guarantee the delivery of `Suspended` events if the OS doesn't define an application lifecycle. (app can always be terminated without any kind of clean up notification on most non-mobile OSs) Fixes rust-windowing#2185. ci: manually point ANDROID_NDK_ROOT to latest supplied version It seems the symlink to `ndk-bundle` and this environment variable pointing to it have been removed to prevent the sdkmanager from failing, when finding the SDK setup to be in an "indeterminate" state. It is now up to the users themselves to install an NDK through that tool or point the right variables to a preinstalled "latest" NDK. actions/runner-images#2689 actions/runner-images#5926 Fix changelog entry wrt scrolling The breaking change was put into the wrong release section. Release 0.27.0 version Explicitly specify minimum supported rust version This should help with distributing apps using winit. Fixes rust-windowing#1075. On X11, fix crash when can't disable IME Fixes rust-windowing#2402. Release 0.27.1 version Windows: respect min/max sizes when creating the window (rust-windowing#2393) On X11, fix window hints not persisting This commit fixes the issue with min, max, and resize increments not persisting across the dpi changes. Fix tracking of phase changes for mousewheel on trackpad (rust-windowing#2158) On Windows, add opt-in function for device events (rust-windowing#2409) Add CODEOWNERS file (rust-windowing#2420) * Add CODEOWNERS file This makes it very clear when you're stepping down from the post as a maintainer, and makes it clear for users who is expected to review their PR * Fix grammar * Make @kchibisov receive pings for the X11 platform * Fix typo Implement version 0.4 of the HasRawWindowHandle trait This makes Winit 0.27 compatible with crates like Wgpu 0.13 that are using the raw_window_handle v0.4 crate and aren't able to upgrade to 0.5 until they do a new release (since it requires a semver change). The change is intended to be self-contained (instead of pushing the details into all the platform_impl backends) since this is only intended to be a temporary trait implementation for backwards compatibility that will likely be removed before the next Winit release. Fixes rust-windowing#2415. Fix missleading breaking change on Windows The applications should not rely on not-implemented behavior and should use the right functions for that. Remove redundant steps from CI Tests are already building the entire crate, so no need for a separate builds slowing down the CI. On Wayland, fix `Window::request_redraw` being delayed On Waylnad when asking for redraw before `MainEventsCleared` would result for redraw being send on the next event loop tick, which is not expectable given that it must be delivered on the same event loop tick. Release 0.27.2 version On Windows, improve support for undecorated windows (rust-windowing#2419) Add touchpad magnify and rotate gestures support for macOS (rust-windowing#2157) * Add touchpad magnify support for macOS * Add touchpad rotate support for macOS * Add macOS rotate and magnify gesture cancelled phases * Correct docs for TouchpadRotate event * Fix tracing macros Document `WindowEvent::Moved` as unsupported on Wayland Update `sctk-adwaita` to use `ab_glyph` The crossfont will still be available under the option. Mark new events as breaking change Adding a new enum variant is a breaking change in winit. Co-Authored-By: kas <[email protected]> Co-Authored-By: Artur Kovacs <[email protected]> Co-Authored-By: Markus Siglreithmaier <[email protected]> Co-Authored-By: Murarth <[email protected]> Co-Authored-By: Yusuke Kominami <[email protected]> Co-Authored-By: moko256 <[email protected]> Co-Authored-By: Mads Marquart <[email protected]> Co-Authored-By: Markus Røyset <[email protected]> Co-Authored-By: Marijn Suijten <[email protected]> Co-Authored-By: Kirill Chibisov <[email protected]>

Isopod · 2023-04-04T01:11:28Z

IMEs on Linux still don't work, as far as I can tell. I should say I'm not some kind of IME guru, I only use it for testing. I tested it with Anthy (Japanese IME) on X11. For the record, the IME works in most other programs, which includes Firefox, Chrome and all GTK and Qt applications.

I attempted to enable IME support like this:

window.set_ime_allowed(true);

This makes compose sequences (dead keys, e.g. ´+e = é) work, but not IMEs. In order to support IMEs on Linux, I think winit would need to implement IBus, since this is what most desktop environments (Gnome etc.) use these days.

kchibisov · 2023-04-04T01:16:31Z

Winit supports XIM and text_input_v3, dbus based implementation are out of scope.

GNOME works with both of them out of the box if you enable IME, same with kwin.

Be aware that once you enable ime support, you must handle the Ime events.

Isopod · 2023-04-04T01:33:57Z

At the moment I'm just dumping all events to the console. I'm not receiving any IME events. I just get ReceivedCharacter() events with latin letters.

kchibisov · 2023-04-04T01:38:37Z

if you use alacritty do you have IME input in it?

kchibisov · 2023-04-04T01:40:22Z

Also, make sure that what you're using is using XIM.

Isopod · 2023-04-04T01:52:22Z

I do happen to use alacritty and no, I don't have IME input there, either. When I manually set the input method to XIM by overriding environment variables (e.g. GTK_IM_MODULE=xim), then I don't have any IME input in those applications, either. But that's to be expected, isn't it? After all, I'm using IBus and not XIM. GTK and Qt have their own input modules for IBus, so that's not an issue.

Is IBus really supposed to be completely backward-compatible with XIM? Because that has never been the case for me , ~~the compatibility seems to end at compose sequences/dead keys~~ (*). I don't know if I'm doing anything wrong. I tried launching ibus with --xim, but it doesn't make any difference. I don't use any desktop environment, just i3.

Edit: (*) IBus might not have had any involvement in that at all.

kchibisov · 2023-04-04T01:57:31Z

Is IBus really supposed to be completely backward-compatible with XIM? Because that has never been the case for me, the compatibility seems to end at compose sequences/dead keys. I don't know if I'm doing anything wrong. I tried launching ibus with --xim, but it doesn't make any difference. I don't use any desktop environment, just i3.

I'd suggest to consult the ibus docs, I know that ibus the way setup on GNOME works without any issue with alacritty on X11 using XIM.

There're plenty IMEs supporting XIM (e.g. fcitx) and surely ibus can do that given that it works on stock fedora gnome(x11, wayland is also oob, but it has different story) for me.

notgull · 2023-04-04T02:08:12Z

dbus based implementation are out of scope.

Do you have a link to discussion on this topic? In my opinion, if IME is in scope then ibus should be in scope.

kchibisov · 2023-04-04T03:12:24Z

Do you have a link to discussion on this topic? In my opinion, if IME is in scope then ibus should be in scope.

Yet on Wayland you don't need it and having ibus where other IMEs exist is not an option (you have uim and so on as well, I never had ibus and use IME daily).

Isopod · 2023-04-04T12:43:18Z

Ok, I just tested it on a fresh Fedora 37 VM, can confirm that the IME works there on both Wayland and X11 (tested in alacritty). The UX isn't the best, but it appears to be usable. Not sure what Fedora is doing differently, it might just be an issue on my machine then.

pickfire · 2023-04-04T13:26:09Z

I used alacritty-git alacritty 0.13.0-dev (2df8f860) with IME on both x11 and wayland kde on arch linux and it works, and I have no idea what is XIM even though I heard about it, I don't think I touched that. But I remember on wayland some tweaks needs to be done, like set the virtual keyboard https://fcitx-im.org/wiki/FAQ#Non_Gtk.2FQt_Wayland_Application_.28Alacritty.2C_kitty.2C_etc.29.

Isopod · 2023-04-04T13:31:55Z

The issue I had might be specific to the Arch Linux ibus package. The PKGBUILD does not pass --enable-xim to configure. I recompiled the package with --enable-xim and it works now. Note that you also have to start ibus-daemon with -x or --xim.

lucasmerlin · 2023-07-08T17:43:38Z

I managed to add IME support to android and iOS:
Android input works really well, including autocomplete and suggestions:

Video

Screen_Recording_20230708_144153.mp4

iOS supports basic text entry, but no autocomplete /autocorrect yet:

Video

IMG_0149.MP4

The android support is based on @rib's work on android-activity here: rust-mobile/android-activity#24
I had to add a new Event to winit, basically just passing through the TextInputState to egui, handling the logic there.
Implementation in egui is pretty simple, basically

send the initial TextInputState (text + cursor position/selection) to winit when focusing a text field + open the keyboard
update the text in egui whenever a new TextInputState is received, handling the selection and compose region.

The TextInputState struct looks like this:

Expand me

/// This struct holds a span within a region of text from `start` (inclusive) to
/// `end` (exclusive).
///
/// An empty span or cursor position is specified with `Some(start) == Some(end)`.
///
/// An undefined span is specified with start = end = `None`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct TextSpan {
    /// The start of the span (inclusive)
    pub start: Option<usize>,

    /// The end of the span (exclusive)
    pub end: Option<usize>,
}

#[derive(Debug, Clone, PartialEq, Eq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct TextInputState {
    pub text: String,
    /// A selection defined on the text.
    pub selection: TextSpan,
    /// A composing region defined on the text.
    pub compose_region: TextSpan,
}

For iOS text support, I implemented the UIKeyInput api to receive basic key events but it doesn't support autocomplete / autocorrect / character composition yet. There also is UITextInput to support these features, I think it should be possible to write a implementation in winit that provides the same simple TextInputState api to applications, but I don't have the objective c skills to implement this.

If you want to try these in an egui app, you can add the following to your Cargo.toml to try my branches:

[patch.crates-io]
winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" }
egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}

Or try this example for android: https://github.com/lucasmerlin/rust-android-examples/tree/ime_support_showcase/agdk-eframe.

These are the relevant branches:
https://github.com/lucasmerlin/winit/tree/mobile_ime_support
https://github.com/lucasmerlin/egui/tree/mobile_ime_support
https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)

If there is interest to add this to winit I'm happy to open a draft PR.

kchibisov · 2023-07-08T17:55:03Z

Well, we'd need patches upstream to be merged first, you could open a draft PR if you want to though.

rib · 2023-07-28T20:59:11Z

Sorry for the delay following up here @lucasmerlin - very cool that you got something working here.

I'm hoping to get a chance to look at this soon.

It'll be good to compare the egui / winit changes with the ones I experimented with at the time:

https://github.com/rib/winit/tree/android-activity-ime-events
https://github.com/rib/egui/tree/android-winit-ime-support

It'll be good to test this on GameActivity 2.0.2 once I land this PR: rust-mobile/android-activity#88

phnaharris · 2024-03-02T10:22:07Z

So first of all I don't need CompostionDone event, since this is an event when I'd send things downstream, meaning that I don't need CompositionDeleteSurroundingText. And it seems like I don't need DeleteSurroundingText, since I can model it via CompositionPreedit. CompositionCommit(text: String) is indeed needed so the user will know when to insert text.

Hi @kchibisov, I'm wondering why you said that you don't need CompositionDeleteSurroundingText event and how we can model it via CompositionPreedit. Because, the surrounding text is the text around the cursor, even if user committed it to widget or not, as far as I know. I think what you mean is we can modify the preedit, but the problem became a thing if the IME want to delete the surrounding text via text input v3 after the preedit be committed to widget, I think we didn't have a mechanism for doing that yet.

Now I wanna support this feature about surrounding text. Based on the design of GTK for this feature, I need a way for downstream (I'm using egui) to send the surrounding text back to winit whenever it need, through Ime::RetrieveSurrounding event. It would be nice if you have any idea for doing that.

kchibisov · 2024-03-02T10:49:24Z

You should add delete surrounding text, just keep in mind that it doesn't exist without setting surrounding text. The issue is how to communicate that cross platform, but it's a technical detail.

Part of that was done in #2993 . IME will also be reworked for 0.31.

phnaharris · 2024-03-03T16:30:51Z

it doesn't exist without setting surrounding text

I've taken the work in this pull request and got the idea that how the surrounding text be updated in the event loop. I also implement set_ime_surrounding_text for Wayland and call it whenever it changed, especially when text input enabled (and commit it to compositor). But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

Thank you .

kchibisov · 2024-03-03T17:01:35Z

But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

it's IME dependent, you'll get it only when IME thinks that you should get it. Also, ensure to commit when setting surrounding text.

phnaharris · 2024-03-03T17:17:08Z

I understand. But if I reproduce the same situation in the same environment, I should expect the same result right? I've created a text box with gtk and egui (with winit), input the same content and when I go back to change some character, gtk worked fine with surrounding text while winit receive a new Preedit event.

Have you ever got a DeleteSurroundingText event in winit before? I just wanna know if the bug come from the flow that I'm implemented was wrong or something else?

My environment:

Compositor: Hyprland.
IME: Fcitx5.
Toolkit: GTK and egui + winit.

kchibisov · 2024-03-03T18:16:17Z

The event is not handled by winit unless you've added it. Use WAYLAND_DEBUG=1 to compare patters and use that knowledge to troubleshoot, since usually it's all pretty deterministic.

phnaharris · 2024-03-04T02:30:11Z

Okay, I got the idea. Thank you for your help.

murarth added DS - android DS - ios DS - macos DS - wayland DS - web DS - windows H - help wanted Someone please save us S - api Design and usability S - enhancement Wouldn't this be the coolest? labels Mar 6, 2020

murarth mentioned this issue Mar 6, 2020

Improve IME event handling #1293

Closed

garasubo mentioned this issue Apr 3, 2020

Inform downstream about IME state #1452

Closed

garasubo mentioned this issue Jun 26, 2020

Cannot input japanese characters alacritty/alacritty#1101

Closed

garasubo mentioned this issue Jul 10, 2020

rename CompostionEvent to IMEPreeditEvent #1622

Merged

8 tasks

kaiwk mentioned this issue Aug 18, 2020

add bindings for InputMethodKit servo/core-foundation-rs#405

Closed

kaiwk mentioned this issue Aug 21, 2020

add composition event support for Mac #1669

Closed

9 tasks

cjhowedev mentioned this issue Oct 9, 2022

Text input field UI component bevyengine/bevy#6213

Open

Riey mentioned this issue Oct 25, 2022

Android support Riey/erars#64

Open

2 tasks

oscrim mentioned this issue May 10, 2023

Text input doesn't bring up virtual keyboard on mobile vladbat00/bevy_egui#29

Closed

daxpedda mentioned this issue Jun 2, 2023

Can not input the Chinese character on wasm32 platform #2012

Closed

kchibisov mentioned this issue Jul 28, 2023

Heads Up: Updated The Android-Activities Crate #2905

Closed

furkanmustafa mentioned this issue Jan 30, 2024

Linux Roadmap zed-industries/zed#7015

Closed

12 tasks

ElhamAryanpur mentioned this issue Mar 30, 2024

egui should open soft-keyboard on mobile AryanpurTech/BlueEngineUtilities#3

Open

3 tasks

Tracking issue for IME / composition support #1497

Tracking issue for IME / composition support #1497

Comments

murarth commented Mar 6, 2020 • edited by kchibisov Loading

chrisduerr commented Apr 4, 2020

pickfire commented Jun 26, 2020

garasubo commented Jun 26, 2020

garasubo commented Jul 10, 2020

kaiwk commented Aug 20, 2020 • edited Loading

jakerr commented Aug 21, 2020

kchibisov commented Sep 27, 2020

pickfire commented Sep 27, 2020

garasubo commented Sep 28, 2020

kaiwk commented Sep 28, 2020

kchibisov commented Sep 28, 2020

pickfire commented Sep 28, 2020

garasubo commented Sep 29, 2020

kchibisov commented Oct 1, 2020

kchibisov commented Oct 1, 2020

Isopod commented Apr 4, 2023

kchibisov commented Apr 4, 2023

Isopod commented Apr 4, 2023

kchibisov commented Apr 4, 2023

kchibisov commented Apr 4, 2023

Isopod commented Apr 4, 2023 • edited Loading

kchibisov commented Apr 4, 2023 • edited Loading

notgull commented Apr 4, 2023

kchibisov commented Apr 4, 2023

Isopod commented Apr 4, 2023

pickfire commented Apr 4, 2023 • edited Loading

Isopod commented Apr 4, 2023 • edited Loading

lucasmerlin commented Jul 8, 2023 • edited Loading

kchibisov commented Jul 8, 2023

rib commented Jul 28, 2023

phnaharris commented Mar 2, 2024 • edited Loading

kchibisov commented Mar 2, 2024

phnaharris commented Mar 3, 2024

kchibisov commented Mar 3, 2024

phnaharris commented Mar 3, 2024

kchibisov commented Mar 3, 2024

phnaharris commented Mar 4, 2024

murarth commented Mar 6, 2020 •

edited by kchibisov

Loading

kaiwk commented Aug 20, 2020 •

edited

Loading

Isopod commented Apr 4, 2023 •

edited

Loading

kchibisov commented Apr 4, 2023 •

edited

Loading

pickfire commented Apr 4, 2023 •

edited

Loading

Isopod commented Apr 4, 2023 •

edited

Loading

lucasmerlin commented Jul 8, 2023 •

edited

Loading

phnaharris commented Mar 2, 2024 •

edited

Loading