Keyboard mapping issues in HTML5 client

[totaam]

Issue migrated from trac ticket # 2154

component: html5 | priority: blocker | resolution: needinfo | keywords: keyboard

2019-02-18 12:55:58: berserker created the issue

---

I know that there is a long guide how to report keyboard layout mapping problems, but I think here the issue is clearly in the HTML5 JS code.

Most of the users to whom I tried to provide remote access through a browser report that input is not working even with the en_US layout. It is only possible to input numbers into xterm. Other keys cause beeps and either spaces or numbers are typed instead of, e.g., letters.

For me, en_US layout is working at the beginning. Switching to Russian layout breaks everything and switching back to en_US does not help. Here are logs from the JS console (Browser version: Firefox 65.0 (64-bit). OS: Ubuntu 18.04, LANG=en_US.UTF-8. Xpra 2.4.2 is running inside a CentOS 7 Docker container).

debug enabled for: 
Array []
1:438:5
connection_progress( Initializing ,   ,  20 ) 1:399:6
connection_progress( Connecting to server ,  localhost:8888/display/1/ ,  40 ) 1:399:6
we have webworker support Client.js:320:3
we can use websocket in webworker Client.js:329:5
connection_progress( Opening WebSocket connection ,  ws://localhost:8888/display/1/ ,  60 ) Client.js:228:2
connection_progress( WebSocket connection established ,   ,  80 ) Client.js:228:2
sending hello Client.js:942:3
return all encodings:  
Array(6) [ "jpeg", "png", "rgb", "rgb32", "h264", "mpeg1" ]
Client.js:865:3
return all encodings:  
Array(6) [ "jpeg", "png", "rgb", "rgb32", "h264", "mpeg1" ]
Client.js:865:3
getFirstBrowserLanguage()= en-US Utilities.js:135:3
getKeyboardLayout()= us Utilities.js:150:3
hello capabilities: [object Object] Client.js:956:2
got hello: server version 2.4.2 accepted our connection Client.js:228:2
server screen sizes: 
Array(1571) [ (2) […], (2) […], (2) […], (2) […], (2) […], (2) […], (2) […], (2) […], (2) […], (2) […], … ]
Client.js:1648:5
connection_progress( Session started ,   ,  100 ) Client.js:228:2
startup complete Client.js:228:2
connection-established Client.js:228:2
server connection is OK Client.js:890:4 

Now, if I type something into xterm having en_US layout active, everything works fine. If I try to switch the layout to Russian, I get the following in the JS console

input language changed from null to ru Client.js:595:3 

I don't see symbols in xterm, only spaces, but I think this is fonts-related issue.

Then, if I switch back to en_US layout and try to type something, I see nothing in the JS console about input language change. And typing causes beeps.

[totaam]

Most of the users to whom I tried to provide remote access through a browser report that input is not working even with the en_US layout

How do they select en_US?

Switching to Russian layout breaks everything and switching back to en_US does not help

How do you switch?

Then, if I switch back to en_US layout and try to type something, I see nothing in the JS console about input language change. And typing causes beeps.

How to you switch back?
It would help to have the server log, with or without -d keyboard.

[totaam]

2019-02-18 14:09:46: berserker uploaded file start-postprocessed.log (77.7 KiB)

[totaam]

2019-02-18 14:29:06: berserker uploaded file xpra-noasciivt.log (195.6 KiB)

[totaam]

2019-02-18 14:34:26: berserker commented

---

Replying to [comment:1 Antoine Martin]:

Most of the users to whom I tried to provide remote access through a browser report that input is not working even with the en_US layout

How do they select en_US?

Lets forget about other users for now.

Switching to Russian layout breaks everything and switching back to en_US does not help

How do you switch?

Using CapsLock while having browser tab with HTML5 client active.

Then, if I switch back to en_US layout and try to type something, I see nothing in the JS console about input language change. And typing causes beeps.

How to you switch back?

The same way.

It would help to have the server log, with or without -d keyboard.

I've attached the log xpra-noasciivt.log with keyboard debugging ON. Plz ignore and delete the other attachment.
At first, I typed ls into xterm and hit Enter.
Then I switched the layout using CapsLock, typed фыва and pushed Enter again.
Then I switched back to en_US and tried to type ls. Even though there are l and s keys logged, what I got in xterm was ~4

[totaam]

2019-03-08 23:29:50: berserker uploaded file xpra+xev.log (202.2 KiB)

[totaam]

2019-03-08 23:39:25: berserker commented

---

Hi. I've attached a log with xev running instead of xterm to see what X11 applications receive when I type.
As can be seen, after switching layout to ru and back to en, xev starting to get F10 when I press l and 4 when s.

Do you have any ideas? Need other logs?

[totaam]

Do you have any ideas? Need other logs?
I don't really know how to re-create this setup, having more information as per Keyboard would help.

Too late for 2.5, re-scheduling.

[totaam]

2019-06-08 17:07:24: berserker commented

---

Replying to [comment:4 Antoine Martin]:

Do you have any ideas? Need other logs?
I don't really know how to re-create this setup, having more information as per Keyboard would help.

It could be easily seen that the JS logics is flawed, without me going through that giant list of tasks you require to report bugs.

Let's take a look at Client.js:XpraClient.prototype._keyb_process

	var key_language = null;
// ...
	//next try mapping the actual character
	else if (str in CHAR_TO_NAME) {
		keyname = CHAR_TO_NAME[str];
		if (keyname.includes("_")) {
			//ie: Thai_dochada
			var lang = keyname.split("_")[0];
			key_language = KEYSYM_TO_LAYOUT[lang];
		}
	}
// ...
	this._check_browser_language(key_language);

where Client.js:XpraClient.prototype._check_browser_language(key_layout) contains

	var new_layout = null;
	if (key_layout && this.key_layout!=key_layout) {
		this.clog("input language changed from", this.key_layout, "to", key_layout);
		new_layout = key_layout;
		this.key_layout = key_layout;
	}
// ...
	if (new_layout!=null) {
		this.send(["layout-changed", new_layout, ""]);

When I press д, keyname = CHAR_TO_NAME[str] (## Cyrillic_de) includes "_", key_language (through KEYSYM_TO_LAYOUT) is mapped to ru, _check_browser_language, being invoked with key_layout "ru", sends a notification about layout change to the server, the server changes layout from us (initial layout, because browser language is en_US) to ru.
When I press l, keyname = CHAR_TO_NAME[str] (== l) does not include "_" and key_language remains null, which is passed to _check_browser_language which does almost nothing in this case, i.e. it does not notify the server about layout change!!! And if it was previously switched to ru and now receives en keys, one gets wrong key mapping.

[totaam]

2019-06-08 17:35:07: antoine uploaded file html-layout-change-back.patch (1.4 KiB)

change back to the default layout

[totaam]

How about this patch? Does it help?
/attachment/ticket/2154/html-layout-change-back.patch

Without steps, I don't know how to configure my keyboard, so I can't reproduce or test.

[totaam]

2019-06-08 19:37:32: berserker commented

---

Replying to [comment:6 Antoine Martin]:

How about this patch? Does it help?
[/attachment/ticket/2154/html-layout-change-back.patch]

No. When I repeatedly type д (Cyrillic_de), the HTML5 client switches layout every 2 seconds to ru and then back to en then back to ru... (starting from ru):

2019-06-08 18:16:13,243 setting keyboard layout to 'ru'
2019-06-08 18:16:15,293 setting keyboard layout to 'us'
2019-06-08 18:16:17,295 setting keyboard layout to 'ru'
2019-06-08 18:16:19,316 setting keyboard layout to 'us'
2019-06-08 18:16:21,367 setting keyboard layout to 'ru'
2019-06-08 18:16:30,962 setting keyboard layout to 'us'

I think it is clear why.
Here key_layout is always "ru".
At the first time, the if branch is taken, because key_layout && this.key_layout!=key_layout is true.
On the next invocation of _check_browser_language with key_layout ## "ru" the else if branch is taken, because this.key_layout!=null holds. And in that block of code, default_layout # this._get_keyboard_layout() is "us" and it is not equal to this.key_layout "ru". In this case you set new_layout to default_layout= "us", even though the symbol being typed is not from "us" layout.

[totaam]

2019-06-09 04:15:08: antoine uploaded file html-layout-change-back-v2.patch (1.5 KiB)

updated patch

[totaam]

2019-06-09 10:02:39: berserker commented

---

Replying to [comment:8 Antoine Martin]:

Please try the updated patch.

It seem to solve the issue for me if I don't switch the layout too often. I think embargo time has to be set to 100ms in all cases.
Also, I didn't test how this behaves in non-en browser locales.

[totaam]

2019-06-09 10:42:06: berserker commented

---

Replying to [comment:8 Antoine Martin]:

Please try the updated patch.
_check_browser_language is called from Client.js:do_window_mouse_move with no argument (key_layout is undefined).
In this case, the else if branch is selected, new_layout is set to "en" (or "us"?) and if this.key_layout == "ru", "layout-changed" message is sent to the server.
Not an optimal behavior, IMO.

[totaam]

2019-06-09 12:19:02: antoine uploaded file html-layout-change-back-v3.patch (2.4 KiB)

updated patch - don't change from motion events

[totaam]

This call to _check_browser_language was added in r15510. (for Xpra-org/xpra#1484)

How about the updated patch?

[totaam]

2019-06-09 19:56:49: berserker commented

---

Replying to [comment:11 Antoine Martin]:

This call to _check_browser_language was added in r15510. (for Xpra-org/xpra#1484)
I'm surprised that it worked for someone with Ru layout...

How about the updated patch?
Latin symbols are incorrectly handled in Chromium in Russian locale (started as LANGUAGE=ru chromium-browser ...). The server (looking at the user-agent?) sets the layout to ru on connection:

2019-06-09 18:33:04,726 Handshake complete; enabling connection
2019-06-09 18:33:04,777  automatic picture encoding enabled, also available:
2019-06-09 18:33:04,777   jpeg, png, rgb32, webp, h264, mpeg1
2019-06-09 18:33:04,779 HTML5 Linux Chrome client version 3.0
2019-06-09 18:33:04,907 setting keyboard layout to 'ru'

and when I type a latin character, _check_browser_language is called with the null argument when this.key_layout is also null:

2019-06-09 18:33:08,801 client keyboard processKeyEvent( true ,  [object KeyboardEvent] ) key= KeyL keycode= 76
2019-06-09 18:33:08,802 client keyboard Client.js:628: _check_browser_language( null )
2019-06-09 18:33:08,803 client keyboard Client.js:656 new_layout # null  this.key_layoutnull
2019-06-09 18:33:08,807 client keyboard key-action,1,l,true,mod2,76,l,76,0

(I've added some "keyboard" debug info to _check_browser_language).
So, neither if (key_layout) nor else if (this.key_layout!=null) branches are taken. new_layout!=null && this.key_layout!=new_layout is also false.
This l keypress is mapped to F10 keypress on the server side because of the incorrect "ru" locale.

[totaam]

2019-06-10 19:40:31: berserker uploaded file worksforme.patch (2.2 KiB)

[totaam]

2019-06-10 19:47:31: berserker commented

---

See the attached patch. It works for me.

Basic idea is that if _check_browser_language is called with null (~Latin characters) and the current layout is the same as the browser language, switch to the "us" layout. It also works in case the browser language is "us" and the locales being used are "us" and a non-Latin one.

It probably won't work for non-English Latin language/locale, so this is not a complete solution.

[totaam]

2019-06-10 19:59:29: berserker commented

---

Replying to [comment:6 Antoine Martin]:

Without steps, I don't know how to configure my keyboard, so I can't reproduce or test.

I think you don't even need to start Xpra or a browser to work on this.
There are: the current layout, the browser language and the argument to _check_browser_language.
The problem is: how to implement the _check_browser_language function which guarantees proper change of the current layout for the most common combinations:

• US + Lating layout, the browser layout is the same as the non-US Latin input layout

• US + non-Latin layout, the browser layout is the same as the non-Latin input layout
• US + Lating layout, US browser layout
• US + non-Latin layout, US browser layout

[totaam]

2019-06-11 05:52:50: antoine uploaded file html-layout-change-back-v4.patch (3.0 KiB)

updated patch

[totaam]

I think you don't even need to start Xpra or a browser to work on this.

I do, on macos, on windows, on Linux, with Firefox, with Safari, with chrome, etc.
Removing layout detection is not a solution.

AFAICT, the 'ru' keyboard layout does have support for all the latin characters, so there should be no need to switch to another layout.

From my limited testing using setxkbmap ru on Linux, the latest patch works fine.

Unless you can provide the information required, I will have to close this as 'invalid' or 'wontfix'.

[totaam]

2019-06-11 16:41:28: berserker commented

---

Replying to [comment:15 Antoine Martin]:

I think you don't even need to start Xpra or a browser to work on this.
I do, on macos, on windows, on Linux, with Firefox, with Safari, with chrome, etc.

As I see now, the problem is in the _check_browser_language function, which does not depend on the keyboard layout.

AFAICT, the 'ru' keyboard layout does have support for all the latin characters

Experience shows that it does not. Maybe I do not clearly understand what "layout" is. Or the system on the server side is misconfigured.

From my limited testing using setxkbmap ru on Linux, the latest patch works fine.

There is no immediate problem with the ru layout.
Try to test with this._get_keyboard_layout() returning "ru" (mocking a browser with the first language == Russian) and "us" layout. There is no difference between your third and fourth patches in this context.

Unless you can provide the information required, I will have to close this as 'invalid' or 'wontfix'.

I've tested my patch in Linux with Firefox (us) ("us" is the browser language, not layout) and chromium (us + ru), Windows with Firefox (ru), MacOS with chromium (ru). Both "us" and "ru" layouts work as expected when I switch between them in any order.
Your last two patches do not work for latin characters input when the browser language is "ru".
As I wrote above, I don't see how _check_browser_language depends on the information you would like me to provide, so I'm closing this. Maybe I'll reopen it later if there is any problem.
Thank you for your time and effort.