keyboard backspace bug for x230-hotp-maximized (x230t)

[Kawaiisu]

So I was excited to get heads installed and my nitrokey setup. But then realised I have some sort of bug on all three laptops. No matter the OS, if its linux it locks the screen or just doesn't do anything. Same issue on 1 x230 tablet and 2 x230 notebooks.

How can I go about fixing this? I'm not a seasoned developer so please have mercy on me.

Thank you

[tlaurion]

@Kawaiisu what is your backspace bug?

I had the following bug on x230t but in OS (QubesOS) not under Heads, if your bug is that the backspace is not linked to the right keymap for some reason: QubesOS/qubes-issues#3306

Please describe actual behavior and expected behavior, which OS, and if behavior is under Heads or not.
The operating system being kexec'ed, Heads should not have any influence on the OS behavior. We are interested in the behaviors happening inside of Heads.

[Kawaiisu]

Hi,

I went to powershell from Heads and they backspace and backspace keys seem to be fine.

Then I tested debian 10 live and found that the backslash and pipe key "\ |" seems not to function. As well as Backspace.

Again in Qubes OS, same as debian 10, the backspace or backslash doesn't seem to function.

On Tails backslash does nothing but backspace in this instance locks the screen.

I visited this: QubesOS/qubes-issues#3306

Am I supposed to overwrite xmodmap_x230t.txt from Dom 0?

But what about every other operating system? Doesn't it mean there's something I need to do in heads to make sure its permanent across all operating systems? We use many live operating systems.

Thank you.

[tlaurion]

@Kawaiisu The problem was never resolved. Last time I checked with QubesOS, fix was temporary, and was not able to map it correctly over reboot and was randomly showing back. Looked for help, found none, gave up.

AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all).
The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.

If you find the solution elsewhere, please link to the fix. It seems that no one cared, including myself, since I do not use my x230t but to test Heads builds. AFAIK that QubesOS issue included all the pertinent information that I had found from that time. If you find something new, please ressucitate that issue, since as you discovered, is still present. Also, as you saw, the issue is not having any relevance to Heads, since the behavior is not present.

So the issue here is irrelevant.

[fhvyhjriur]

AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all).
The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.

If you find the solution elsewhere, please link to the fix.

@tlaurion Could you use the x230t coreboot config for x230t? Skulls is building separate images for x230 and x230t because the devices are not exactly same. https://github.com/merge/skulls
x230t support was merged in coreboot on 03.2020 here: https://review.coreboot.org/c/coreboot/+/38482

It would also make more sense for people who switch from skulls to heads to flash a x230t image when they had before also a x230t skulls-coreboot image.

[tlaurion]

@ fhvyhjriur Heads is a community project.
There is a porting guide on heads-wiki. As you saw for #925 its not really useful neither for myself nor others to do ports for hardware I do not use. The people that have needs should be the one pushing changes. This is mentality of open source and is called Selfish Contributionism. Maintainership is the main problem of open source projects. It would be counter productive that I accept to bring a board to a project I do not use, for whichI will then be asked to maintain. It won't happen.

Consequently, I will not do the port for x230t but will invite x230t, t400 and other not supported board owners to propose pull requests.

And will hold my stance on maintaining and trying to continue to do quality work on my community time, for which i'm not paid for. Hope the message reached its audience.

Free software doesn't mean free as in free beer. Everyone has 24h in a day. If it's important for you, you will take the required time to make it happen, asking questions, proposing code; not requiring others to do the work at your place. This is how it works.

I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.

Letting people take their fish lines instead of providing unlimited fish supply:
cp config/coreboot-x230.config build/coreboot-ver/.config
cd build/coreboot-ver/
make menuconfig
(do config changes needed)
make savedefconfig
cp .config ../../config/coreboot-x230t.config
cd -
cp -r boards/x230/ boards/x230t/
mv boards/x230t/x230.config boards/x230t/x230t.config
vi boards/x230t/x230t.config
(change path for coreboot-x230.config to coreboot-x230t.config)
make BOARD=x230t
(adjust until it compiles)
git checkout -b x230t
git add board/* config/*
git commit
git push
create PR.

[fhvyhjriur]

@tlaurion
I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.

You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.

EDIT: When i wrote the text, your last edit of your text did not had this line: 'I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.'
Yes, this was the reason why i wrote what i had wrote before.

[tlaurion]

@tlaurion
I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.

You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.

I did'nt know. That would then be linked to x230-maximized images, being pushed to coreboot newer version, which are still not merged upstream under Heads. My comment is still relevant. Code should be borrowed, modified, tested functional, and then proposed for merged, tested and then upstreamed by the people who need the changes.

[fhvyhjriur]

@tlaurion I watched https://fosdem.org/2020/schedule/event/selfish_contributor/
My "selfish motivation" bringing up a idea how to maybe help @Kawaiisu and all other heads users is because heads is a way to do important work as a investigative journalist that travel around the world and have to leave the device for a moment at some place that is not secure.
@Kawaiisu is a new user that have register fresh to github to report this problem. So i expect that this user needs heads to do something important to the society and needs help with that.
I want to live a life in a knowledgeable society with free access to free(copyleft) knowledge. I am also "selfish motivated" to support old hardware because of freedom(low purchase price, low to none dependencies to startup based on closed source software) and environmental benefits. My "selfish motivation" in for example getting broken printers from electrical waste and soldering out their 8MiB and 16MiB spi chips and putting those SPI chips in thinkpads that have 4MiB or less. To get heads support for such devices is something i am interested in after learning facts about this world.
I am not a developer and i am not good at writing code. I depend, like mentioned in the https://fosdem.org/2020/schedule/event/selfish_contributor/ video, mainly on commit messages that explain what the commit have done.
And because T400 was mentioned - i have spend happily some hours into taking a T400 apart and writing with a external programmer some heads testing-image and reporting if its working or not. I would do this work again to help adding support for T400 and other devices that are not supported in heads recently. Adding heads-support for all x86_64 devices that coreboot support and that can run in perfect case with 100% free software except cpu-microcode is part of my "selfish motivation". It would be my "selfish motivation" until RISC-V based portable devices are available at the same price and amount like ~10 year old Thinkpads that heads support recently. At least until devices like the https://en.wikipedia.org/wiki/Novena_(computing_platform) and now https://mntre.com/reform get the same produced amount like the thinkpad **00, **10, **20 and **30 series combined together.

At some parts of this world the 11.01.2021 have begun. This is the 8th year since the death of https://en.wikipedia.org/wiki/Aaron_Swartz . My "selfish motivation" is to help the world to be a better place for people with such a free mind like Aaron had.

[Thrilleratplay]

@fhvyhjriur

Could you use the x230t coreboot config for x230t?

Yes. My Heads test device is a 230T. As far as I can tell is the only difference between the x230 and x230T coreboot configurations is gpio33 changed from output to input. This probably has to do with using the touchscreen as an input during boot up.

[tlaurion]

@Thrilleratplay did you figured out why backspace was not functionning properly on Qubesos and other OSes?

[tlaurion]

@Kawaiisu first question first. Is the bad behavior present with stock Lenovo BIOS on live cds?

@fhvyhjriur lookin at coreboot changes, I doubt sincerely that the changes would resolve the keyboard issue.

With that first question here answered, avenues of solutions can then be nrrowed down.

Vivid memories for Swartz. Sympathetic to the cause. Thanks for reminding me why we do this. Just remember time is a limited resource, I don't intend to be rude.

[Thrilleratplay]

@tlaurion I haven't had an issue with the backspace. I'll try the latest build tomorrow

[Thrilleratplay]

@tlaurion Open source projects are rife with burnout and you put in a considerable amount of work to Heads. It is important to be reminded why you it. The personal reasons that drive you to contribute so much to Heads are your own. I understand the importance and do what I can with my time, resources and energy but pales in comparison to you do to keep this project going. Thank you.

[Thrilleratplay]

@Kawaiisu I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries.

What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards.

I still have to test this myself.

[Thrilleratplay]

@tlaurion @Kawaiisu I may have reproduced this. I am confused on whether it is just the backspace key or all keys that do not work? I flashed the latest x230 maximized build artifact from CircleCI and everything seems to work fine but occasionally after a reboot, the keyboard does not seem to be initialized. It also only seems to happen when rebooting from the emergency shell either with the command reboot or ctrl-alt-del.

EDIT: Oops, I used a test build. Let me try one from master.....and osresearch does not allow/have any projects in circleCI because there is only a 404 when I click on projects.

EDIT 2: Updated master on my fork and used the CircleCI build heads-x230-maximized-v0.2.0-1006-g6bc40d7.rom. If I boot into a USB image of Debian 9, I can go into a prompt and still use the backspace and pipe characters on the keyboard.

[ghost]

@tlaurion @Thrilleratplay

i remember this bug with backspace key
sorry but its not only Qubes
same issue with x220/x220t (flashed year ago)
it works in heads shell
it lock screen on gnome desktop (lastest ubuntu)
it doesn't work everywhere in terminal/chat/text editor
never seen this bug on clean coreboot or stock bios.
replaced keyboard - nothing changed
maybe really xev or xmonad can help
I use classical six row kbd

I would never give more than $ 50 for these laptops,
ebay disappointed me a lot.
out of five, I have only one alive (librebooted x200 from Vikings)

[Kawaiisu]

RIP Aaron :(

Thank you all so much for your generosity.

Sorry for my delay. I hope you're all finding ways to prepare and maintain your safety during these trying times.

@tlaurion : you asked "Is the bad behavior present with stock Lenovo BIOS on live cds?"

I will double check this on a stock device and post my results today.

Also I think you're correct that there isn't this problem on the notebook non-tablet version of the x230. I'll double check this as well.

@Thrilleratplay you said "I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries."

No mods. I did do a backlit keyboard on one, but went back to stock keyboard.

@Thrilleratplay: "What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards."

Laptops were purchased from a recycler. But there was spanish post-it notes from the previous owners. Its possible they came from South America, would this matter? What could I do to figure this out? I think the recycler would not tell me for competitive reasons.

I will go through in detail everything else mentioned above in case I missed anything.

[Thrilleratplay]

@Kawaiisu I ask about where they are from because there are regions variants of keyboards (Japanese, Russian etc). Normally these are obvious, other times it could be subtle like the currency character not being $ if you are in the US. A few keys could be mapped differently on these variants for backspace and pipe. Although, I think the backspace would be the same on all of these.

[Kawaiisu]

@Thrilleratplay @tlaurion

Okay I've tried both tails and debian 10 live on both stock x230 notebook non-tablet version and x230 tablet versions. Backspace and and backslash work normally. I also swapped out the tablet with a backlit keyboard and backspace and backslash still work as expected.

Also not sure if it will become relevant but want to note it so I don't forget: Another X230 notebook I flashed using make BOARD=x230 and x230-flash rom creation process. On that system backspace and backslash work just fine.

The system that's giving me the problem is an x230 tablet and I used make BOARD=x230-hotp-mazimized. Not sure if the rom difference is relevant.

I am going to now flash the stock x230 tablet and notepad with the x230-hotp-mazimized rom and will report if the backspace-backslash problem comes back.

@tlaurion about whether I have country specific keyboards, they seem to be all US, except 1 backlit keyboard on the problem computer does have a random extra euro symbol on the 5 and % key. So three symbols on this one key. Which is strange. The other backlits don't have this and seem to be US-only. Could be something here.

Okay I'll get to flashing and report back my findings.

But first I have a question about preparing the x230-hotp-maximized roms.

In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:

"Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop."

I know I'm doing something really noob here. A hyphen or space or something is missing or needs to deleted, please have mercy on me ^_^.

To produce the roms while getting around the error we skipped this part and manually ran wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin
then made the roms after.

But if I know why the error was happening this will speed up our process.

Thank you. I'll get to flashing now.

[Kawaiisu]

@0rb677 @tlaurion @Thrilleratplay @fhvyhjriur

Greetings everyone. I'm reporting back my findings.

I newly flashed two laptops, a stock x230 Tablet and a stock x230 Notebook non-tablet.

The notebook's backspace and backslash work fine. However on the Tablet backspace and backslash don't work as intended. Locking the screen in Tails for instance.

I did test both stock laptops prior, they were both working as intended on multiple OS before the flash.

So at this point I'm stumped, is this a heads problem or something else?

Not really sure how to proceed here. Any advice would be greatly appreciated.

Thank you all for your time and insight. Be safe.

[Thrilleratplay]

@Kawaiisu Just to clarify, all of your x230 work as expected but the x230T still has an issue with the backspace key? What OSes have you tried after flashing Heads?

Lets try to figure out what the OS sees (link to helpful answer). At the moment I am on a x230 that is running coreboot and Arch Linux.

If I execute sudo showkey and press backspace. The key code is:

keycode  14 press
keycode  14 release

sudo showkey -s and press backspace. The scancodes is:

0x0e 
0x8e 

If the same OS and version of Heads is installed on both your x230 and x230T, it would be more beneficial if you compare the the results between those machines. Hopefully this will narrow down the issue.

[Kawaiisu]

Hi @Thrilleratplay

Yes you're correct on the x230 backspace and backslash works fine with Qubes.

But x230T with Qubes backspace and backslash doesn't work.

On x230, running sudo showkey and pressing backspace shows:

keycode 14 press
keycode 14 release

Then running sudo showkey -s and pressing backspace shows:

0x0e 0x8

Then running sudo showkey and pressing backslash shows:

keycode 43 press
keycode 43 release

Then running sudo showkey -s and pressing backslash shows:

0x2b 0xab

On x230T, running sudo showkey and pressing backspace shows:

keycode 152 press
keycode 152 release

Then running sudo showkey -s and pressing backspace shows:

0xe0 0x12 0xe0 0x92

Then running sudo showkey and pressing backslash shows:

keycode 139 press
keycode 139 release

Then running sudo showkey -s and pressing backslash shows:

0xe0 0x1e 0xe0 0x9e

Does that help?

[Thrilleratplay]

@Kawaiisu It sounds like a bug in Qubes and now that you know what the keys are, remapping seems like the easiest fix. I am not familiar enough with it to suggest an appropriate work around. @tlaurion Can you suggest a way to globally remap keys in Qubes?

[Kawaiisu]

@Thrilleratplay @tlaurion Want to give update. The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.

Also on the 230T, in heads powershell backspace works correctly. So I'm confused, why do all OSs work on a stock X230T, but when I flash heads, suddenly no OSs work?

Are we sure this is a Qubes specific problem? Isn't Qubes natively Xen? Should I test more OSs?

[tlaurion]

@Kawaiisu i'm confused on your conclusions.
You say that QubesOS, Tails and Debian 10 live OSes: the behavior is the same. I guess the behavior is bad? Or you mean by flashing Heads, the behavior changes to bad?

@Kawaiisu : can you send me a rom backup of lenovo x230t? I cannot guarantee when I will invest time in this, but my first debfugging steps, as said prior, would be to test behavior on stock BIOS. My hypothesis is that OSes are detecting keyboard and set mappings based on detected x230. Which doesn't match x230t. Another note: QubesOS is based on Xen, yes, but the underlying dom0 OS for QubesOS 4.0 is fedora-25. So issues disovered here are fedora-25 related for QubesOS. Or related to Xorg keyboard, or underlying mapping to keyboard config.

@Thrilleratplay I left the ticket as unfixed under QubesOS with all steps I took in the past trying to fix the issue. As of now, i'm still into uncertainty on what is the problem to replicate. On my past experience on X230t with QubesOS, the behavior was random. It was functional at first install (I could change LUKS key passphrase doing backstpace) and then after OS install, I couldn't. The issue I opened (QubesOS 3.2) was closed because noone else was confirming the issue. So the user base of X230t seems quite small. Noone else replicated the issue; noone +1. I investigated as far as I could. And left the x230t alone.

The heads related ticket is : #827 (comment)
The QubesOS related issue is: QubesOS/qubes-issues#3306

I think the problem is generalizing to all OSes.
And the fix to test is this: QubesOS/qubes-issues#3306 (comment)
Unfortunately, it seems that it was not persistent(the problem came back on next boot? I can't recall. 2018, after all!), since I after that posted QubesOS/qubes-issues#3306 (comment)
....And then, lost interest in the X230t, which I use to test flashes, and where Heads backspace works, where there is no added layers of overcomplications or guessing and overides of keymaps.

So I will ask again:

1. Is the behavior present on stock Lenovo bios for X230t?
2. Is the behavior limited to Heads?

If the behavior is limited to Heads, which is a duplicate of the X230 coreboot config, then maybe coreboot exposing the board as a x230 in dmi tables might be the culprit. Maybe the remapping of keys in OSes are basing themselves on exposed X230 mappings instead of X230t.

Please give feedback on 1 or 2 above. Instead of trying to resolve the issue and waste time. Troubleshooting steps from simple to complex is key, else you will loose yourselves. If we know the behavior is different between installed OSes from stock BIOS vs Heads, then we might be able to get the differences in assumptions and blame the x230 config inside of the x230t board. And start from there.

If I missed your conclusions, please re clarify and sorry to have missed them; I have not see clear statement of differentiation between 1 or 2 being the cultprit.

[Thrilleratplay]

@Kawaiisu I asked what OSes you had tried, as the respond was only pertained to Qubes, I assumed the issue was isolated to Qubes.

I booted Debian 9 on my x230T with maximized Heads and do not have this issue. The shell I entered from the Debian Graphical install does not have showkey. The best thing I can find is to run more /proc/bus/input/devices. On my x230T, the keyboard name is "AT Raw Set 2 keyboard". Interestingly, my x230 is "AT Translated Set 2 keyboard". Looking at them, they look like they are interchangeable.

Steps to take:

• Boot into Debian using the USB
• Go to a shell
• run more /proc/bus/input/devices, scroll until you find your keyboard.
• Google search the name and maybe the version ID to see if there are any issues with that specific keyboard.
• Afterwords, for shits and giggles, swap the keyboards on a working x230 and the x230T and see if that fixes it. If this works and no one knows what to correctly map the bad keyboard, it may be easier to buy a replacement keyboard at this time.

@tlaurion I saw the solution you posted in Qubes issues #3306 using xmodmap but, unless I am mistaken, this would only work in X11, not in a terminal if x11 failed to start or the user enters another TTY session by pressing Ctrl-alt-F2 (this may not be an issue in Qubes).

[Kawaiisu]

@tlaurion apologies, let me correct my poor grammar:

The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.

You asked:

1. Is the behavior present on stock Lenovo bios for X230t?

The bad behaviour isn't present on a stock lenovo bios for the x230t. I test qubes, debian live and tails.

2. Is the behavior limited to Heads?

Yes the bad behaviour of incorrectly mapped backspace and backslash only appears only after flashing.

I tried to upload .zip directly to github, either its not supported or my browser doesn't allow. I was able to upload here:

https://ufile.io/3vrd09o2

Inside is both heads that I flashed to the x230t and the factory bios before should you need.

[tlaurion]

So works on stock bios but not heads. Now we talk.

[fhvyhjriur]

@Kawaiisu
Could you give the skulls x230t image a try and report if you have the same issues? https://github.com/merge/skulls/releases/tag/x230t-0.0.2
Probably the best way to do the test would be to first install the x230t oem backup you have made at some point, test if the issue is gone with the OEM again (never know if EC does the problems) and then follow the skulls installation howto. Would be interesting to know if the issue is gone with that skulls-x230t-image that is made for x230t.
If the issue is gone with the x230t skulls image, you can flash the skulls-x230 https://github.com/merge/skulls/releases/tag/x230-0.1.11 release to the x230t to find out if it have to do something with that and the issue appear again.

[Kawaiisu]

@Thrilleratplay

Upon your suggestions I ran on debian 10:

more /proc/bus/input/devices

It showed on the x230t:

AT RAW set 2 keyboard" and ID: "ab84

I googled around and nothing came up relevant to this issue.

I swapped out the keyboard from a x230 notebook where the backspace is working on to the x230t, and the backspace still doesn't work. I ran more /proc/bus/input/devices and the readout is identical to before AT RAW set 2 keyboard" and ID: "ab84

You said heads maximised is working correctly with your x230t, do you think it's possible I built heads incorrectly?

Beforehand when I built heads, I was using make BOARD=x230 and x230-flash, I had to copy the original OEM roms to the flashing device and unlock them or remake them into new roms.

But there was a key difference when doing BOARD=x230-hotp-maximized, where I never unlocked the OEM roms from the original x230t. Instead the roms were downloaded. Is it possible I downloaded the wrong roms?

In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:

Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop.

I couldn't figure out what the custom invoke or call command was here so I tried to get around this step by manually running

wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin

then made the roms after.

Is it possible I missed a key step involving a difference in the way to make the roms for the x230 versus the x230t?

[fhvyhjriur]

Short report:
Thinkpad x220t
image: heads-x220-maximized-v0.2.0-1380-gc4b964c.rom
result: backspace not working

[bwachter]

I've just seen it on my x230 (without the t) for the first time, same behaviour as described above for my x230t. Heads is 6300dd1 (from the 4.19 pull request)

[tlaurion]

I've just seen it on my x230 (without the t) for the first time, same behaviour as described above for my x230t. Heads is 6300dd1 (from the 4.19 pull request)

Using 3a38ac0 on daily driver (x230-hotp-maximized). Cannot replicate under QubesOS that was already installed.

@bwachter what is your OS, was it already installed, anything special (different locales, different language) and can you replicate this across reboots?

[pcm720]

I can reliably reproduce this bug on ab1faf5 x230-maximized and x230-maximized-fhd_edp by coldbooting the X230 into the latest Arch Linux with GNOME DE. Just a single reboot is enough to fix the issue until the next coldboot.

Backspace and backslash/vertical slash keys seem to work fine during initramfs stage when the system's encrypt hook asks user to enter a disk unlock key, but stop working once the laptop boots into the root OS.

[pcm720]

While searching for information on i8042 and input lines in dmesg, I came across this post.
I tried downgrading systemd to 253.1-3-arch, and can confirm that it does indeed help.

The keyboard is still detected as AT Raw Set 2 keyboard after coldboot and AT Translated Set 2 keyboard after the reboot, but that doesn't prevent backspace and backslash keys from working properly. Now to figure out what's changed between 253.1 and 253.2/3...

[tlaurion]

Considering its a detection bug at runtime to init i8042, I think a good testing g avenue would be to look into proper parameters to pass to the module from coreboot kernel boot arguments.

That's one path I never checked before.

[pcm720]

I tried recompiling Heads with i8042.reset and i8042.kbdreset flags set, but had no luck getting the keyboard identify as AT Translated Set 2. In fact, with those parameters set, the keyboard is being consistently identified as AT Raw Set 2 even across reboots.

P.S. I'm doing all of my testing in Heads instead of Arch Linux since it's way faster and seems to produce the same results. If the keyboard was identified as Raw in Heads, it'll still be Raw in Arch, and the same goes for Translated.

[tlaurion]

Both coreboot and Linux can interact with i8042 and serio which setups the keyboard.

Coreboot coreboot/coreboot@a69d682

Will check history of linux since we are close to be able to propose 5.10.178 where 4.x might still have bogus code and once set per heads kernel, it is reused under final kexec'ed OS unless KERNEL_ADD board parameter is forcing behavior. Edit nothing changed there for a while.

I'm not aware of a way to force i8042 into translated, where coreboot last changes chanfed the behavior to init in translated 2 first maybe this would be the way: renable that coreboot config option for all and force Linux kernel boot option to leave it that way?

Otherwise someone will have to dig https://wiki.osdev.org/%228042%22_PS/2_Controller

[tlaurion]

From my memory recollection, this behavior was touching tablet variants only before and now with coreboot 4.19 x230 owners start reporting issues where I can't replicate. Fixating the behavior for all boards seem to be the way

[tlaurion]

Revisiting this issue from #958 (comment)

• x230 ps2 coreboot support was added in 2021 at 816259c
• Attempts to remove it didn't change anything
• x220 board was added with coreboot ps2 init in at d7ccd87 and never was removed since then.

I will do a quick branch with my codechanges laying around for 5.10.178, testing only for x230. Will test with coreboot ps2 init off and see how latest kernel behaves

[tlaurion]

Ok.

• https://app.circleci.com/pipelines/github/tlaurion/heads/1647/workflows/81a7116e-a9cb-4250-ad46-49d6fad9a278 is producing rom for x230 on 4.19 with 5.10.178 kernel bump config as baseline, still with coreboot 4.19 ps2 init.

---

I understand that people here are testing against x230-maximized board configurations. So I will modify the coreboot configurations in next commit to deactivate ps2 init from coreboot. It is said under ps2 Kconfig option:

  │ CONFIG_DRIVERS_PS2_KEYBOARD:                                                                                                                  │  
  │                                                                                                                                               │  
  │ Enable this option to initialize PS/2 keyboards found connected                                                                               │  
  │ to the PS/2 port.                                                                                                                             │  
  │                                                                                                                                               │  
  │ Some payloads (eg, filo) require this option.  Other payloads                                                                                 │  
  │ (eg, GRUB 2, SeaBIOS, Linux) do not require it.                                                                                               │  
  │ Initializing a PS/2 keyboard can take several hundred milliseconds.                                                                           │  
  │                                                                                                                                               │  
  │ If you know you will only use a payload which does not require                                                                                │  
  │ this option, then you can say N here to speed up boot time.                                                                                   │  
  │ Otherwise say Y.                                                                                                                              │  
  │                                                                                                                                               │  
  │ Symbol: DRIVERS_PS2_KEYBOARD [=y]                                                                                                             │  
  │ Type  : bool                                                                                                                                  │  
  │ Defined at src/drivers/pc80/pc/Kconfig:6                                                                                                      │  
  │   Prompt: PS/2 keyboard init                                                                                                                  │  
  │   Depends on: PC80_SYSTEM [=y]                                                                                                                │  
  │   Location:                                                                                                                                   │  
  │     Main menu                                                                                                                                 │  
  │       -> Generic Drivers                           

x230 roms without Ps2 init from coreboot are happening here:
https://app.circleci.com/pipelines/github/tlaurion/heads/1648/workflows/5caf96ef-ed36-4392-a72e-983e3467eaf7

@pcm720 @bwachter can you report cbmem and dmesg logs?

[tlaurion]

Testing x230-maximized on x230 (not x230t) (fastest to test just with tpm without hotp. Neglect build errors since related to librems which I forgot to remove from CircleCI in above post). Will edit post here with results

Flat 5.10.178 kernel config without coreboot ps2 init removal https://output.circle-artifacts.com/output/job/f5aa0a4c-dc69-4e3e-a9e0-71f746df14c7/artifacts/0/build/x86/x230-maximized/heads-x230-maximized-v0.2.0-1545-g41ca685.rom

Same kernel config without ps2 init from coreboot https://output.circle-artifacts.com/output/job/26ccdfb5-c49f-4c1c-955e-88a584a4ba51/artifacts/0/build/x86/x230-maximized/heads-x230-maximized-v0.2.0-1546-g08140b0.rom

Both has proper translated type 2 under tinycore x86_64 dd'ed iso boot

No regression can be confirmed on either master or those commits on x230. Will now test on my x230t

[tlaurion]

Testing x230-maximized on x230t (fastest to test just with tpm without hotp. Neglect build errors since related to librems which I forgot to remove from CircleCI in above post).

No synaptics/trackpad detected under Tinycore 14. Switched to console (alt-f1 to get screenshots)

Flat 5.10.178 kernel config without coreboot ps2 init removal https://output.circle-artifacts.com/output/job/f5aa0a4c-dc69-4e3e-a9e0-71f746df14c7/artifacts/0/build/x86/x230-maximized/heads-x230-maximized-v0.2.0-1545-g41ca685.rom

Same kernel config without ps2 init from coreboot https://output.circle-artifacts.com/output/job/26ccdfb5-c49f-4c1c-955e-88a584a4ba51/artifacts/0/build/x86/x230-maximized/heads-x230-maximized-v0.2.0-1546-g08140b0.rom

Sorry guys, but I cannot replicate with 5.10.178 kernel config (master is 4.14 which might be faulty).
@bwachter @i-c-o-n @pcm720 : If this resolves your issue in one flavor or the other, I just confirmed no regression over x230/x230t so your feeback will be helpful to me version bumping everything.

Note that next kernel/coreboot config won't have libgfxinit either, based on #1378 work conclusion and the need of going away of gnat6 on the host (to transition to Nix based buildsystem to ehlp in reproducible builds (with reproducible toolchain which is needed).

[pcm720]

Sadly, 5.10.178 did not fix the issue in my case.
I can even report a regression: I can't boot the system from internal SATA SSD anymore.
cryptsetup luksDump /dev/sda2 outputs the following:

Cannot initialize crypto backend.
Device /dev/sda2 is not a valid LUKS device.

It works fine after downgrading to 4.14-based kernel, but that's besides the point.

Anyway, here's the short summary:

With PS2 init enabled, it behaves just as 4.14 kernel. The only thing that seems different is the fact that input shows two Raw Set 2 keyboards.

• Cold boot:

cbmem:
[DEBUG]  PNP: 00ff.2 init
[DEBUG]  Keyboard init...
[INFO ]  Keyboard controller output buffer result timeout
[INFO ]  Keyboard controller output buffer result timeout
[INFO ]  Keyboard controller output buffer result timeout
[INFO ]  Keyboard controller output buffer result timeout
[INFO ]  Keyboard controller output buffer result timeout
[ERROR]  Keyboard reset failed ACK: 0xaa
[DEBUG]  PNP: 00ff.2 init finished in 2287 msecs
------------------
dmesg:
[    4.281377] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    4.282996] serio: i8042 KBD port at 0x60,0x64 irq 1
[    4.282998] serio: i8042 AUX port at 0x60,0x64 irq 12
[    4.284483] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
[    5.065425] psmouse serio1: synaptics: queried max coordinates: x [..5768], y [..5062]
[    5.096459] psmouse serio1: synaptics: queried min coordinates: x [1174..], y [790..]
[    5.157722] psmouse serio1: synaptics: Touchpad model: 1, fw: 8.1, id: 0x1e2b1, caps: 0xd002a3/0x940300/0x123800/0x0, board id: 1611, fw id: 1099905
[    5.157750] psmouse serio1: synaptics: serio: Synaptics pass-through port at isa0060/serio1/input0
[    5.196815] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input6
[    5.201443] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input7
[    6.551178] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/serio2/input/input8

• Warm boot:

cbmem:
[DEBUG]  PNP: 00ff.2 init
[DEBUG]  Keyboard init...
[INFO ]  Keyboard controller output buffer result timeout
[DEBUG]  PS/2 keyboard initialized on primary channel
[DEBUG]  PNP: 00ff.2 init finished in 509 msecs
------------------
dmesg:
[    4.281709] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    4.283211] serio: i8042 KBD port at 0x60,0x64 irq 1
[    4.283215] serio: i8042 AUX port at 0x60,0x64 irq 12
[    4.284661] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
[    5.118906] psmouse serio1: synaptics: queried max coordinates: x [..5768], y [..5062]
[    5.150391] psmouse serio1: synaptics: queried min coordinates: x [1174..], y [790..]
[    5.211104] psmouse serio1: synaptics: Touchpad model: 1, fw: 8.1, id: 0x1e2b1, caps: 0xd002a3/0x940300/0x123800/0x0, board id: 1611, fw id: 1099905
[    5.211114] psmouse serio1: synaptics: serio: Synaptics pass-through port at isa0060/serio1/input0
[    5.250102] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input6

With PS2 init disabled, it always detects the keyboard as AT Raw Set 2. Obviously, dmesg logs only since there is nothing of note in cbmem output. For some reason, input shows no touchpad after a reboot:

• Cold boot:

[    4.285593] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    4.287257] serio: i8042 KBD port at 0x60,0x64 irq 1
[    4.287260] serio: i8042 AUX port at 0x60,0x64 irq 12
[    4.288822] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
[    5.066957] psmouse serio1: synaptics: queried max coordinates: x [..5768], y [..5062]
[    5.097967] psmouse serio1: synaptics: queried min coordinates: x [1174..], y [790..]
[    5.158933] psmouse serio1: synaptics: Touchpad model: 1, fw: 8.1, id: 0x1e2b1, caps: 0xd002a3/0x940300/0x123800/0x0, board id: 1611, fw id: 1099905
[    5.158962] psmouse serio1: synaptics: serio: Synaptics pass-through port at isa0060/serio1/input0
[    5.197011] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input6
[    5.201814] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input7
[    6.630489] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/serio2/input/input8

• Warm boot:

[    4.281583] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    4.282491] serio: i8042 KBD port at 0x60,0x64 irq 1
[    5.218674] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input6

Full logs are here, file names should be pretty self-explanatory.

By the way, I'm using stock X230 keyboard, the only EC patch I applied was battery check removal.

[tlaurion]

Damnit. Missed important point between cold/warm boot. Will have to redo tests later on since x230t cold booted shows no cbmem log and raw (x230t). Will redo and repost but the point here with this old ticket is : we do not know how to fix this on x230t.

Will redo on x230 first because if I can replicate, its a new bug. Will also review crypto requirements on kernel from cryptsetup and reply back here when done.

[pcm720]

I recompiled the kernel with the following options set, cryptsetup is working fine now:

CONFIG_CRYPTO_XTS=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=y
CONFIG_CRYPTO_AES_NI_INTEL=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y

Can confirm that other than that, there has been no regressions for me on 5.10.178.

There's one thing about this that's been bugging me for a while: are we sure that Translated Set is the correct one?
I don't feel that's the case because disabling coreboot's PS2 init or adding reset+kbdreset to coreboot kernel args causes i8042 to consistently initialize the keyboard in AT Raw Set 2 mode.
Not to mention that both keys work fine in Heads and kexec'd kernel during initramfs stage and stop working properly only after systemd-udevd initializes the keyboard all over again. And, like I mentioned previously, keyboard works fine even in AT Raw Set 2 after downgrading systemd to 253.1.

[pcm720]

Nevermind all of that. I dug a little deeper into systemd.
Apparently keyboard issues with X230 were caused by systemd-udevd and this commit. Basically, it caused systemd-udevd to apply input remapping rules defined for X2?? Tablet.
It was fixed in this pull request and was backported into the stable tree only yesterday.
Manually adding rvn to the evdev line in 60-keyboard.hwdb fixes all my keyboard issues, both keys work fine with AT Raw Set 2 mode and disabled keyboard init in coreboot config.

I am very sorry for wasting your time on this.

By the way, doesn't this mean that key mapping issue with X230T in Raw Mode is well-known and should be handled by udev?

[tlaurion]

By the way, doesn't this mean that key mapping issue with X230T in Raw Mode is well-known and should be handled by udev?

For OSes that are udev based, yes. :/
The weird stuff for Heads, here, is that Heads behavior is Ok per plain kernel behavior and/or without coreboot interfering, which now by default tries to put in translated AT 2 by default, and SHOULD make sure no raw 2 is inititialized and still fails. for some reason. As said previously, board users should make sure tht the kernel is doing the right thing by pressuring upstream (x220t/x230t) otherwise I remember vaguely having asked coreboot devels in the past, and there doeesn't seem to be much more possible there. So the idea is to apply quirks upon detection, just like udev is doing here, but ideally, without depending on udev to do this.

Up to the task? For the x230, this seem sa local regression on arch that was unfortunate. But this bug still affects x220t and x230t afaik for OSes that do not apply quirks, outside of the kernel module itself. Ideally, this should be detected correctly. Second best would be to permit to fixate this directly through i8042 driver, specifying mode needed (per platform, which we could easily put on coreboot passed to kernel boot options and on KERNEL_ADD board config option, applied on kexec call to boot into final OS. Without that, we need OSes to do the right thing, which obviously is not universal and as we just saw, can regress in time)

[pcm720]

So the idea is to apply quirks upon detection, just like udev is doing here, but ideally, without depending on udev to do this.

Sounds like a good idea if there's nothing to be done on the Coreboot side.
So, in your opinion, what is the best way to do this detection in Heads?
Kernel patch for i8042 module that adds parameter for forcing translated mode, init script/binary that runs only when X220T/230T is detected or bare implementation of udev-like service that applies quirks on boot using pre-defined rules?

I'm not really familiar with raw device access in Linux, but it seems that the easiest (and probably the hackiest) way to do this is a script and SERIO_RAW kernel option that exposes raw access to all serio ports, including i8042.

[tlaurion]

@pcm720 please try to follow the rabbit https://review.coreboot.org/c/coreboot/+/52737

Which then leads to https://review.coreboot.org/c/coreboot/+/47594

Otherwise seems related to reset.
So reading and testing modules options at https://github.com/torvalds/linux/blob/master/drivers/input/serio/i8042.c seems the way.

Will check the full oldconfig of 4.19 x230 to see what is there tomorrow.

[pcm720]

Nah, I figured that I'll need to look at Coreboot code.
What I'm interested in is how this should be done on Heads side.

Right now I see two main options:

1. Kernel patch for i8042/atkbd that adds an option to switch to translated mode on module init
2. Script or binary that runs on Heads boot, detects X2??T and switches the keyboard into translated mode

As for testing existing i8042 options, I already did that. The only thing that changes keyboard behaviour is a combination of reset and kbdreset parameters, and it does the opposite. After resets, keyboard always initializes into Raw mode.

There's also atkbd module which I didn't play with all that much. I think it's worth to take a look at it before doing anything.
But I feel like if we need the module to force Translated mode, we'll have to add that option ourselves.

[tlaurion]

Next steps QubesOS/qubes-issues#3306 (comment)

[tlaurion]

Resolved with latest kernel bum + coreboot bump #1744