Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate DKMS autoinstall failure behavior #184

Closed
eloquence opened this issue Jul 23, 2020 · 5 comments · Fixed by #189
Closed

Investigate DKMS autoinstall failure behavior #184

eloquence opened this issue Jul 23, 2020 · 5 comments · Fixed by #189

Comments

@eloquence
Copy link
Member

Via #179 we resolved issues with the grsec kernel metapackage for some Qubes users, where the Qubes-specific u2mfn module was not successfully built in postinst, causing VM windows to no longer display correctly (freedomofpress/securedrop-workstation#590).

However, there are still open questions:

  • Why was the graphics issue not reproducible? Several testers didn’t hit it, then post-release several folks did. Was it a race condition, an issue with specific hardware, a combination, something else?
  • Why was the DKMS autoinstall failure not sufficient to cause apt/dpkg to error out? Can we make such a failure louder?

We should continue to investigate this, so we can better reason about similar failures, and ideally tweak the behavior so that the package install will fail if the required kernel module is not built successfully.

@eloquence
Copy link
Member Author

Post sprint-planning, @conorsch expressed that he wants to set aside at least 4 hours in the next couple of weeks for continued investigation here.

@eloquence
Copy link
Member Author

Unfortunately this issue may still be occurring even with the fixes in #179; @ninavizz reported it after a delayed update of her workstation yesterday. @conorsch has offered to investigate directly with her.

@conorsch
Copy link
Contributor

Given the importance of the u2mfn kernel module for graphical environments, at least until Qubes 4.1, the most straightforward process is to confirm the existence the .ko file in e.g. /usr/lib/modules/4.14.186-grsec-workstation/updates/dkms/u2mfn.ko. The postinst for the linux-image deb package should handle the dkms autoinstall just fine. By the time the metapackage's postinst runs, the dkms process should have exited. If for some reason the ko file remains missing, we can exit non-zero from the metapackage postinst, causing apt/dpkg and therefore the overall updater run to report failure.

@eloquence
Copy link
Member Author

@conorsch will attempt resolution in 8/20-9/2 sprint, and @eloquence has committed to helping with QA. (We may not be able to cut a new metapackage release just yet.)

@conorsch
Copy link
Contributor

This bug report was helpful in diagnosing why subsequent dkms autoinstall actions did not reliably correct the problem when observed in the wild: https://bugs.launchpad.net/dkms/+bug/1265136 If a race between dkms processes resulted in the module being compiled, but not successfully installed, then resolution requires an explicit removal of the module even though it was never fully present—as evidenced by the garbled output described in freedomofpress/securedrop-workstation#590. So our postinst logic can optimistically try to recover from that state, but still exit non-zero if unsuccessful.

There's a small trade-off, which is that the dkms removal option requires an explicit reference to the module's version string. For u2mfn, that's 4.0.30 presently. We'd have to update that version string if the upstream changes, but only when making changes to our own kernel versions, which is a reasonable trade-off for the gains in stability, particularly given that Qubes 4.1 intends to remove the dependency on u2mfn altogether.

conorsch pushed a commit to freedomofpress/securedrop-apt-test that referenced this issue Aug 26, 2020
Further refinements to the postinst logic, see for details
freedomofpress/securedrop-builder#184
@rmol rmol closed this as completed in #189 Aug 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants