Tracking issue for MacOS support

[feschber]

Tracking issue for MacOS support

This issue is for tracking MacOS support and the current development status.

TODOs

• Make it compile and get frontend up and running (macos: enable running lan-mouse on macos #42)
• Input Emulation
• Keycode Translation
• Input Capture

[feschber]

#42

[feschber]

Mouse input emulation works, for keyboard emulation the keycodes need to be translated.

[feschber]

#81

[kiwiz]

I took a quick look at Input Capture (I'm primarily interested in the ability to release input). It looks like CGEventTap offers the necessary functionality, but the interface is callback-based. There doesn't seem to be an easy way to marry that with the (polling-based) EventProducer paradigm in the code.

[feschber]

The easy way is to just spawn a thread. And as there doesn't seem to be any non-blocking systemcall for input capturing, this is probably the only way.

What I've been considering, is giving the producer access to the sender_tx channel, which passes the events to the udp_task, directly.
The event-consumer API does not have this problem, since the async consume() function can simply call synchronous code but considering that windows and macos both probably need a different thread for input capture, it might be good to do that at some point.

For now, if you want to work on this, I suggest you just spawn a thread and send the events via a tokio::sync::mpsc::channel to the macos_producer.

[rohitsangwan01]

@feschber really awesome project, just mac is missing, any plan or date to start work on Mac Input capture ?
i am working on a project which might depends on this project, so it would be really great to have Mac full support as well

[feschber]

@feschber really awesome project, just mac is missing, any plan or date to start work on Mac Input capture ?
i am working on a project which might depends on this project, so it would be really great to have Mac full support as well

I can't give any estimate date but encryption and MacOS Input capture are next. Be aware the input capture API is a bit specific to this use case in that it locks the cursor and starts when the screen area is exited (this can not be done manually right now).
I was planning to extract this to a separate crate at some point for use in other projects.

[rohitsangwan01]

@feschber this implementation from Synergy might be helpful or inspiration for Mac support,
Also do you think it would be possible to use lan-mouse as a library for another project ?

[mkdalynn]

Hello,
I'm trying to get this working but am having trouble. I have lan-mouse running on both machines (pop-os 22.04, and mac-os 13). I've added a client to both sides and see it's found the host, but the mouse wont move screens. Input is connected to the pop-os machine.

• tried changing ports to 24800, which previously worked with Barrier
• tried using the local ip of each machine as the hostname

I'm not sure if I have something misconfiguration or if there is something else going on.
Happy to provide any other info.

[feschber]

Hello,
I'm trying to get this working but am having trouble. I have lan-mouse running on both machines (pop-os 22.04, and mac-os 13). I've added a client to both sides and see it's found the host, but the mouse wont move screens. Input is connected to the pop-os machine.

• tried changing ports to 24800, which previously worked with Barrier
• tried using the local ip of each machine as the hostname

I'm not sure if I have something misconfiguration or if there is something else going on.
Happy to provide any other info.

Be aware that Gnome < v45 won't work. I believe pop os does not ship gnome 45 yet so you're out of luck here unfortunately (until the next popos release)

Your log should show an error saying that only the dummy input capture is active.

[mkdalynn]

Be aware that Gnome < v45 won't work. I believe pop os does not ship gnome 45 yet so you're out of luck here unfortunately (until the next popos release)

Your log should show an error saying that only the dummy input capture is active.

Ah, I missed that dependency. Thanks for pointing it out.

[feschber]

Be aware that Gnome < v45 won't work. I believe pop os does not ship gnome 45 yet so you're out of luck here unfortunately (until the next popos release)
Your log should show an error saying that only the dummy input capture is active.

Ah, I missed that dependency. Thanks for pointing it out.

Yeah it might be a good idea to add a hint to the ui as well since this is not entirely obvious.

[feschber]

Also do you think it would be possible to use lan-mouse as a library for another project ?

As I said, I am planning to separate the input emulation and capture into separate crates. There are a few limitations (by design):

• lan-mouse uses scancodes for keyevents. To use them for textinput an additional library is required to translate them to text (and the other way). Probably xkbcommon. But this is out of scope of this project.
• Input Capture does not have a way of being enabled manually. This is partly because of limitations imposed by the input capture portal and partly also because of the way the layer-shell backend is designed (it simply spawns an invisible layershell surface).
• The Event type used internally could use some improvements as it is currently tied pretty closely to the rest of lan-mouse.

So yeah depending on the project it could certainly be used as a library.

[rohitsangwan01]

@feschber ohh that would be great,
Not sure if this is the correct place to describe, Am working on Similar implementation but with Android and IOS and got some success as well, which expands Apple's universal control functionality to Android and IOS as well, here is a demo of older version of this software
but my plan is to build a proper cross platform or atleast support LanMouse as server, if there are any documentation of protocols to implement a third party client app for lanMouse, and if you will split in different crates, then it would be really easy to integrate lanMouse support in Flutter with flutter_rust_bridge

[feschber]

Interesting! I did a prototype android app before (only for Android input capture, not emulation) and it worked quite well.
The network protocol I'm currently using is a custom one and bound to change. Because of that it is also not yet documented.

If you want something to work with, heres a brief overview:

The types are encoded as two bytes - one for the event type (pointer / keyboard, etc.) and one for the event sub-type (e.g. motion / button).
The rest of the data depends on the specific event sub-type.

You can check out https://github.com/feschber/lan-mouse/blob/main/src/event.rs for the current implementation.
It is basically just a type-length-value (TLV) protocol with the length omitted.

Admittedly its a bit ugly at the moment but I plan to rework it anyway so dont spend too much time on it.

In the future, I want to include variable length integer encoding for bandwidth savings and remove some redundant data I was not sure is needed when I started out.
There is also some Event types that are still missing (notably touch input and high precision scroll events but maybe also gamepad events at some point).

[rohitsangwan01]

i see, ok i guess it would be better to invest client integration time after refactoring or atleast initial documentation when protocols will be stable, Thanks for the detailed info

[feschber]

#131

[isleshocky77]

I don't want to make a new issue if this is already known; however, it also may be related to #91 . I'm able to move from the server (Ubuntu 24.04) to the client (mac os); however, I'm unable to double-click the mouse on the client's screen.

Everything else seems to be working so far (minus the ability to come back without using the keyboard shortcuts).

[livexia]

There is a problem when I test with two mac, I found that shift won't work while using another machine's keyboard.

shift + m input just m not M

[feschber]

@livexia this should now be fixed. Would be nice, if you can confirm!

[livexia]

using git repo version, install with cargo build --release, run with cargo run --release. I can connect mac to linux, both can control each other. I current only have one mac, so I can not verify two mac connection. With one linux and one mac, using mac keyboard on linux shift + m still only m not M, and scroll direction is still invert. I also find that it's smooth connect macos from linux, with other way around the mouse is a bit inconsistent.

[bdols]

I've built the latest commit on Fedora and Mac, with linux connecting to a remote mac.
Am I supposed to add the fingerprint for every connection on both sides?
When I did not add the incoming connection's fingerprint, I only saw this error and had no indication that I might need to add another fingerprint:

[2024-11-16T22:11:52Z WARN  lan_mouse::listen] accept: expected and actual verify data does not match

I also see that the mouse scrolling is inverted for the remote to the mac, and things like double-clicking to highlight selection by word does not work on the remote.

on the linux side, I see this warning, not sure if it's necessary:
2024-11-17T00:21:32Z WARN input_emulation] wlroots backend: wayland protocol "wlr-virtual-pointer-unstable-v1" not supported: the requested global was not found in the registry

but it's functional, I'm going to try to run this for a week

[livexia]

I have using this for past few days, I play factorio on mac and reading docs or blueprint on linux tablet. Even with problems, I think it's working ok. I also uisng a config file to avoid everytime config fingerprints.

[livexia]

I think there is a problem with stuck key, sometimes one key will stuck, even after disconnect.
logs: keyA is stuck

[2024-11-18T07:57:06Z INFO  lan_mouse_gtk] frontend exited successfully
[2024-11-18T07:57:06Z INFO  lan_mouse::service] terminating service ...
[2024-11-18T07:57:06Z INFO  lan_mouse::listen] dtls client disconnected 192.168.31.29:47017
[2024-11-18T07:57:06Z INFO  lan_mouse::listen] dtls client disconnected 192.168.31.29:52188
[2024-11-18T07:57:06Z INFO  lan_mouse::listen] dtls client disconnected 192.168.31.29:42928
[2024-11-18T07:57:06Z INFO  lan_mouse::listen] dtls client disconnected 192.168.31.29:60574
[2024-11-18T07:57:06Z WARN  input_emulation] releasing stuck key: KeyA
[2024-11-18T07:57:06Z INFO  lan_mouse::listen] dtls client disconnected 192.168.31.29:45465
[2024-11-18T07:57:06Z INFO  lan_mouse] service exited!

[bdols]

I also have seen a stuck key issue once and I have seen 2-3 times where both sides disconnect and reconnect after about 1 second. I'm using the keyboard remotely and it's perhaps more noticeable for me.

Other than that, I have not had issues with repeat key events (i.e. holding down one of the arrow keys) which is good.

two surprises: 1) I usually can horizontally scroll by holding the left shift key while moving the scroll wheel up/down on my mouse, but that's not working. 2) My linux desktop with K+M will try to lock itself while I'm connected remotely, I don't recall if that's expected behavior that similar software also exhibits. I had one issue where i had to restart both lan-mouse instances after unlocking it to get it working again -- could be a wayland issue that I was having.