Implement keyboard event handling

[bk138]

Implement full keyboard handling so that a remote viewer can send keyboad events and does not have to rely on the Android soft keyboard.

If you'd like to sponsor implementing this feature, you can do so at https://issuehunt.io/r/bk138/droidVNC-NG/issues/4

[meiskam]

fyi alpha vnc lite supports keyboard events

looks like they created their own "keyboard", and routed keypresses through that https://developer.android.com/guide/topics/text/creating-input-method

[sudomain]

I might give this a try but have an implementation question:

... does not have to rely on the Android soft keyboard.

Is there a (non-root) way to send key events in Android without changing the soft keyboard? I know of apps, such as KeyMapper and KeyPassDX, which can input text for an user, but they do via their own soft keyboard implementations.

[bk138]

I think this is called input method extension aka IME.

[meiskam]

creating a keyboard/IME and routing keys through it doesn't require changing the user's soft keyboard

[meiskam]

this could also be implemented with adb shell cmd input

root@barbet:/system/bin # cmd input
Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...]

The sources are:
      dpad
      keyboard
      mouse
      touchpad
      gamepad
      touchnavigation
      joystick
      touchscreen
      stylus
      trackball

-d: specify the display ID.
      (Default: -1 for key event, 0 for motion event if not specified.)
The commands and default sources are:
      text <string> (Default: touchscreen)
      keyevent [--longpress|--doubletap] <key code number or name> ... (Default: keyboard)
      tap <x> <y> (Default: touchscreen)
      swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      press (Default: trackball)
      roll <dx> <dy> (Default: trackball)
      motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen)
      keycombination <key code 1> <key code 2> ... (Default: keyboard)

[ghost]

wifikeyboard (F-Droid) is a simple example

[Holzhauer]

wifikeyboard does not work for me an Android 12 (website available, but "Status: Connection problem"). Seems out of date.

[ghost]

wifikeyboard does not work for me an Android 12 (website available, but "Status: Connection problem"). Seems out of date.
wifikeyboard on Android 9
Have you set it as the default IME?
https://user-images.githubusercontent.com/107785860/174596382-dedb7d70-31f4-4a4d-a7fb-b7d06c103c6a.mp4

[sudomain]

I guess wifikeyboard is technically not unmaintained, but its last real update (not a translation or typo fix) was in 2013. If other solutions are being considered, consider scrcpy

Edit: There's also ADBKeyboard which I've used, but this requires the user to know how to send intents

[bigrck64]

Having keyboard support is a feature I would love to see in droidVNC-NG, virtual keyboard seems to be the standard in other vnc server.

[arminlinder]

Cudos to bk136, the software does work very well on my mobile and my tablet, and, working as a teacher, it gives me the opportunity to show my pupils what I can do with my Android devices via a beamer.

I do strongly assist that remote keyboard support be implemented, as well as true two-way clipboard support.

I do also have ways to spend some real money on the feature, the school has a certain budget I could spend. Unfortunately, AFAIK, the funding campaign above has not raised any funds yet, and I'd hate to sink money into a feature which then does never get implemented because funds are not high enough.

Dear bk136, what is the amount of funds you need to see, before you start working on implementing these two features?

[bigrck64]

Good question to bk136, I can also pays some funds to have keyboard support :)
So what is your money target for that ?

[bk138]

Hi @arminlinder @bigrck64 my usual rate for professional work is 90€/h; this feature might take 3 or more work days to complete. That said, money isn't the blocker here, but time. I'd actually like to work on this in my free time, but currently don't have much :-/. The rough plan at the moment is to finish #98 which will allow closing of a bunch of other issues, then tackle this one here. Nonetheless, some reasonable funding in https://issuehunt.io/r/bk138/droidVNC-NG/issues/4 might be a nice incentive (for me or anyone else) 😉.

[bigrck64]

Thanks for coming back to us bk138 :)
On my side I can give $300, but issuehunt.io only accept credit card payment, and I would like to use Paypal for this.

[bk138]

@bigrck64 thanks for the commitment! Let's do it the mutual trust way: you keep the investment until the work is done, afterwards you can wire it to the implementer (me or maybe someone else who's faster). What do you think?

[bigrck64]

Good idea! I will keep this money and send it to you (or the faster one ;-) when the work will be done.
@bk138: If it's you, could you accept Paypal payment ?

[bk138]

Good idea! I will keep this money and send it to you (or the faster one ;-) when the work will be done. @bk138: If it's you, could you accept Paypal payment ?

Yes.

[bigrck64]

Yes

Perfect :)

@arminlinder: will you be able to join me and help to found this dev ?

[IssueHuntBot]

@panicride has funded $50.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[bk138]

Sooo, basic ASCII keyboard event handling done, finally!

This implements additional keyboard input from VNC viewers in the form of:

• ASCII characters, i.e. basic typing
• Left and Right arrow handling to select text cursor position
• Delete and Backspace handling for editing text
• Enter/Return handling on API level >= 30

All display-specific, i.e. provides one keyboard focus per display.

This will be in v2.4.0.

Please file follow-up issues for extended keyboard functionality requests.

@bigrck64 https://paypal.me/dankechristianbeier if you still feel inclined ;-)

[IssueHuntBot]

@bk138 has rewarded $45.00 to @bk138. See it on IssueHunt

• 💰 Total deposit: $50.00
• 🎉 Repository reward(0%): $0.00
• 🔧 Service fee(10%): $5.00

[bigrck64]

Awesome ! Looking for the next release and yes, I still want to send you paypal money if keyboard is really working :)

[whichusernameISavailable]

Greetings, just found your VNC today. What are the requirements for connecting viewer and Android version to get text keyboard input working? I have Huawei Honor 9X with Android 10 and tried to connect from Windows 10 PC with TightVNC and UltraVNC. In both cases control hotkey combinations work on phone side, but no app gets text input from VNC viewers.

[bk138]

Greetings, just found your VNC today. What are the requirements for connecting viewer and Android version to get text keyboard input working? I have Huawei Honor 9X with Android 10 and tried to connect from Windows 10 PC with TightVNC and UltraVNC. In both cases control hotkey combinations work on phone side, but no app gets text input from VNC viewers.

You basically need to enable the accessibility service, the app is asking for this on startup. If hotkey like Pos1/Home do work but text input does not, please file a bug with as much details as possible.

[whichusernameISavailable]

Greetings, just found your VNC today. What are the requirements for connecting viewer and Android version to get text keyboard input working? I have Huawei Honor 9X with Android 10 and tried to connect from Windows 10 PC with TightVNC and UltraVNC. In both cases control hotkey combinations work on phone side, but no app gets text input from VNC viewers.

You basically need to enable the accessibility service, the app is asking for this on startup. If hotkey like Pos1/Home do work but text input does not, please file a bug with as much details as possible.

The whole separate bug thread? What if it's not a bug but just some weird limitation of Huawei EMUI? There's not too much to say really. Accesibility service of droneVNC-NG is enabled. Yes, hotkeys like Home/Pos1, Esc and third long combo work. Pasting text from VNC viewer clipboard to apps which can get focus of text input area on mouse click and show blinking cursor there (Firefox address bar, Ghost Commander text editor) also works, but native Notepad app can't get it. I googled which way I could see some logs. Zip archive attached to comment contains portion of log that adb command "adb logcat –v threadtime > android-debug.log" from SDK Platform Tools put out via USB connection. DroidVNC-NG was started after 15:20 and stopped after 15:22, so log includes 3 minutes of various records. I don't know what exactly could be useful.
android-log-1520-1522.zip

[Martii]

@bk138

please file a bug with as much details as possible.

Looks like I'll have to file one when I find some time. Android 9 Galaxy S8 (older I know) with TigerVNC doesn't seem to support for the additional keys atm and just confirmed 2.4.0 and droidVNC-NG is already on in Installed services. Will file probably next week as I'm quite busy and just popping in to recheck what you said a few days ago and whoever else replied afterwards.

[mrmstn]

Great to see the keyboard support :)! Thanks a lot @bk138 !

I just experienced something pretty strange on Androi 9. First, the keyboard worked like a charm, but after some time idling in an app, it just didn't work any more…

After a quick log cat, I saw the following errors:

The first issue raises, when I click on a focused input box, the second one appears as soon as I press a key in the focused input box

06-10 10:23:03.251  6317  6317 D InputService: onAccessibilityEvent: EventType: TYPE_VIEW_CLICKED; EventTime: 2042995; PackageName: com.android.launcher3; MovementGranularity: 0; Action: 0; ContentChangeTypes: []; WindowChangeTypes: [] [ ClassName: android.widget.EditText; Text: [bio]; ContentDescription: null; ItemCount: -1; CurrentItemIndex: -1; Enabled: true; Password: false; Checked: false; FullScreen: false; Scrollable: false; BeforeText: null; FromIndex: -1; ToIndex: -1; ScrollX: -1; ScrollY: -1; MaxScrollX: -1; MaxScrollY: -1; AddedCount: -1; RemovedCount: -1; ParcelableData: null ]; recordCount: 0
06-10 10:23:03.252  6317  6317 E InputService: onAccessibilityEvent: java.lang.IllegalStateException: Already in the pool!
06-10 10:23:03.252  6317  6317 E InputService: 	at android.util.Pools$SimplePool.release(Pools.java:112)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.util.Pools$SynchronizedPool.release(Pools.java:168)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.view.accessibility.AccessibilityNodeInfo.recycle(AccessibilityNodeInfo.java:3156)
06-10 10:23:03.252  6317  6317 E InputService: 	at net.christianbeier.droidvnc_ng.InputService.onAccessibilityEvent(InputService.java:143)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.accessibilityservice.AccessibilityService$2.onAccessibilityEvent(AccessibilityService.java:1538)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.executeMessage(AccessibilityService.java:1724)
06-10 10:23:03.252  6317  6317 E InputService: 	at com.android.internal.os.HandlerCaller$MyHandler.handleMessage(HandlerCaller.java:37)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.os.Handler.dispatchMessage(Handler.java:106)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.os.Looper.loop(Looper.java:193)
06-10 10:23:03.252  6317  6317 E InputService: 	at android.app.ActivityThread.main(ActivityThread.java:6718)
06-10 10:23:03.252  6317  6317 E InputService: 	at java.lang.reflect.Method.invoke(Native Method)
06-10 10:23:03.252  6317  6317 E InputService: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
06-10 10:23:03.252  6317  6317 E InputService: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
06-10 10:23:13.897  6317  7210 D InputService: onKeyEvent: keysym 100 down 1 by client 477161971328
06-10 10:23:13.897  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:13.985  6317  7210 D InputService: onKeyEvent: keysym 100 down 0 by client 477161971328
06-10 10:23:13.985  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.108  6317  7210 D InputService: onKeyEvent: keysym 100 down 1 by client 477161971328
06-10 10:23:14.108  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.185  6317  7210 D InputService: onKeyEvent: keysym 100 down 0 by client 477161971328
06-10 10:23:14.185  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.276  6317  7210 D InputService: onKeyEvent: keysym 100 down 1 by client 477161971328
06-10 10:23:14.276  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.340  6317  7210 D InputService: onKeyEvent: keysym 100 down 0 by client 477161971328
06-10 10:23:14.340  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.419  6317  7210 D InputService: onKeyEvent: keysym 100 down 1 by client 477161971328
06-10 10:23:14.419  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.
06-10 10:23:14.516  6317  7210 D InputService: onKeyEvent: keysym 100 down 0 by client 477161971328
06-10 10:23:14.516  6317  7210 E InputService: onKeyEvent: failed: java.lang.IllegalStateException: Cannot perform this action on a not sealed instance.

@bk138 Should I file a new issue, or is this the correct place for the report?

[bk138]

@mrmstn That's very probably nothing we can do anything about with the a11y current API. #208 might help though.

Does the server crash?

Are you able to input after clicking somewhere else and then the text input again? This worked for me.

[Martii]

... on Androi 9. ...

That's very probably nothing we can do anything about with the a11y current API.

Uhmm...

"Android Pie (codenamed Android P during development), also known as Android 9 (API 28) is the ninth major release and the 16th version of the Android mobile operating system. It was first released as a developer preview on March 7, 2018" * wikipedia

How is the statement of eleven (11) years old in relation to the six (6) real years it's been out? i.e. What am I missing here?

[bk138]

@Martii I think there's a misunderstanding. I'm referring to the pre-API-level-33 a11y API which is apparently implemented in a suboptimal way on some devices. The newer API-level-33-and-newer API is more capabable on paper, but this remains to be tested.

[mrmstn]

Hi @bk138,

First of all, sorry for the late reply, and thank you for your lightning-fast responses – although I wasn't expecting such speed! 😄

I tried debugging the issue myself and thought I would come up with a fancy solution, but the truth is, I dislike the pre-API-level-33 a11y API more than I ever imagined I could. The documentation is lacking, and some events just don't fire without any clear reason.

I'll continue working on this because I love solving puzzles, but I genuinely understand your points now and share your frustration. So everyone reading this comment and hoping for a quick fix, don't expect too much 😅

That's very probably nothing we can do anything about with the a11y current API.

Thanks for all your work! I love the project!

---

EDIT:
An oh, I forgot to answer your questions:

Does the server crash?

No, it doesn't, the input just doens't get applied to the focus element. So it's just like the feature “keyboard input” just doens't exist, so nothing to bother on that side, there is no step backwards 😄

Are you able to input after clicking somewhere else and then the text input again? This worked for me.

Not really, or at least I wasn't able to pint point it down to that action, sometime it works, sometimes it doesn't 😄 I wish I could provide you a more detailed error description

[bigrck64]

https://paypal.me/dankechristianbeier

Hi bk138 !
Sorry I did not had the time to test the keyboard implementation yet, but I just send 150€ on your paypal to thank you.
I will send 150€ more in two months, when I will be able to test it 👍

Looks like I'll have to file one when I find some time. Android 9 Galaxy S8 (older I know) with TigerVNC doesn't seem to support for the additional keys atm and just confirmed 2.4.0 and droidVNC-NG is already on in Installed services.

I also plan to test it over Samsung Galaxy S8 (SM-G950F) - Android 9 / ARM64 - so please crush all the bug you can first ❤

[bk138]

@bigrck64 Thank you very much for the financial support!