-
Notifications
You must be signed in to change notification settings - Fork 687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace cron-apt with unattended-upgrades on Focal #5684
Conversation
a8240b6
to
0a8334c
Compare
Codecov Report
@@ Coverage Diff @@
## develop #5684 +/- ##
===========================================
+ Coverage 81.41% 81.48% +0.07%
===========================================
Files 53 53
Lines 3965 3965
Branches 496 496
===========================================
+ Hits 3228 3231 +3
+ Misses 632 630 -2
+ Partials 105 104 -1
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Took a first pass through, visual review only, did not run through the test plan. At a high level, I like the changes, although I'm now increasingly convinced that forking the securedrop-config
metapackage is going to free us up. During prior discussions, @emkll, I'd feared the complexity of maintaining multiple packages there, but given that a similar implementation was required for #5691, I'm pleased to say it wasn't all that daunting. If you agree, we can wait until that PR is merged and then easily add a Focal-only metapackage for "config".
While it would be more straightforward to also update the Suite field, this will have an impact on running instances
Only for Focal, though! And there aren't any Focal instances in prod yet, so it sounds like we should make this change now, no? We can make it for Focal only but not for Xenial, by modifying only the Focal component in https://github.com/freedomofpress/securedrop-dev-packages-lfs/blob/75073cfd59dd98c76757a27ec1f3bd516ec70217/repo/conf/distributions#L9-L10
perhaps forking the securedrop-config package for focal and populate 50unattended-upgrades and 20auto-upgrades via deb package [...] we will need to configure the reboot time somehow instead of relying on a jinja template via Ansible
As mentioned above, yes, I do think that forking the package is a good idea. As for the template-vs-deb-pkg issue, my understanding of the conf.d logic is that we could set the Unattended-Upgrade::Automatic-Reboot-Time
option in a separate via, e.g. /etc/apt/apt.conf.d/80securedrop
, and manage the 50unattendedupgrades file in the deb package. More testing required, of course, to confirm that behavior.
Given the complexity of these changes, plus the fact I'm requesting further changes, I suggest we defer merge until after feature freeze for 1.7.0, currently scheduled for 2021-01-12.
@@ -1,3 +1,4 @@ | |||
Origin: {{ rep_origin }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an important change! It mirrors what's also provided by our reprepro setup, e.g.
$ curl -s https://apt.freedom.press/dists/xenial/Release | grep ^Origin
Origin: SecureDrop
Although the repo config is managed outside of this repository, we should add tests (for apt-test & apt) that verify that Origin line: if the apt server config ever changes, then unattended-upgrade will stop working.
securedrop_common_packages: | ||
- apt-transport-https | ||
- aptitude | ||
- unattended-upgrades |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
cron: | ||
name: Indicate that a reboot is required at the scheduled time. | ||
job: "touch /var/run/reboot-required" | ||
hour: "*/12" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creative! By ensuring that /var/run/reboot-required
always exists, unattended-upgrades will always reboot after it runs. That's similar to what we're doing manually with cron-apt by appending &&
to the job.
In this case, it'd give us more control to handle the cron job in /etc/
and configure it via the securedrop-config
package. We'd have to fork the package for Focal specifically, but I'm leaning toward that as a simpler option for maintenance anyway.
|
||
// Automatically reboot *WITHOUT CONFIRMATION* | ||
// if the file /var/run/reboot-required is found after the upgrade | ||
Unattended-Upgrade::Automatic-Reboot "true"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be worth appending
// Automatically reboot even if there are users currently logged in
// when Unattended-Upgrade::Automatic-Reboot is set to true
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
since we use tmux, and dangling session isn't impossible.
hour: "*/12" | ||
tags: | ||
- cron | ||
- unatted-upgrades |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: typo in tag name.
|
||
### Security fixes for distribution packages | ||
deb http://security.ubuntu.com/ubuntu {{ ansible_distribution_release }}-security main | ||
deb http://security.ubuntu.com/ubuntu {{ ansible_distribution_release }}-security universe |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good, clean restrictions here. Essentially, we're removing use of "multiverse" and "restricted", and allowing everything else. Confirmed that e.g. intel-microcode
will be caught by these rules:
sdadmin@sd-staging-app:~$ apt-cache policy intel-microcode
intel-microcode:
Installed: 3.20201110.0ubuntu0.20.04.2
Candidate: 3.20201110.0ubuntu0.20.04.2
Version table:
*** 3.20201110.0ubuntu0.20.04.2 500
500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
500 http://us.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages
100 /var/lib/dpkg/status
3.20191115.1ubuntu3 500
500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
8628255
to
6d1d194
Compare
Thanks @conorsch for the review. I have implemented all your suggestions, and this is now ready for re-review. In my local testing I have confirmed this will also resolve the longstanding #3376 as well, updated the test plan and PR to that effect. I think this is ready for another review, and @conorsch you can return for a final pass after a functional review. |
This will need to be updated once #5684 is merged. |
Took a quick look visually again, and looks great. As has been said above, let's revisit this post-release of 1.7.0, since some of the string changes will need to be updated as part of recent close-out. Still haven't tested functionally, but pleased with the approach presented. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Testing
Code wise this looks good.
- Changes to the apt config in distribution.j2 matches the configuration of both apt.freedom.press and apt-test.freedom.press
- Run
molecule converge libvirt-staging-focal
, the unattended-upgrades configuration is as expected - [ ] Optionally, add a package in
Required
of a control file, to observe dependencies also also downloaded by unattended-upgrades - Bump SecureDrop version locally:
securedrop/bin/dev-shell ../update_version.sh 1.7.0
and build focal packages:make build-debs-focal
- Deploy a local apt server
vagrant up apt-local
- Update or add an apt source to point to local server (dev[arch=amd64] http://10.0.1.7 focal main
If using staging, you will need to un-hold packages:- app :
sudo aptitude unhold securedrop-app-code securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-agent
and mon, un - mon :
sudo aptitude unhold securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-server
- app :
-
sudo unattended-upgrades --dry-run
makes sense,sudo unattended-upgrade d
will apply the upgrades - Bump SecureDrop version once again:
securedrop/bin/dev-shell ../update_version.sh 1.7.1
and build focal packages:make build-debs-focal
- Redeploy a local apt server
vagrant destroy apt-local && vagrant up apt-local
(maybevagrant provision apt-local
will work here as well, untested. - Observe the package being updated automatically
- Observe the machine is rebooted at the daily reboot time
I could managed to get the last parts working and that made me to look at a few more points.
- I can not see securedrop-config package is building for me with the latest
securedrop_app_code_version
I tried, 1.8.0 first and 1.8.1 and 1.8.2, the build directory looks like:
❯ ls build/focal/
ossec-agent-3.6.0-amd64.deb securedrop-keyring-0.1.4+1.6.0~rc1-amd64.deb
ossec-server-3.6.0-amd64.deb securedrop-keyring-0.1.4+1.7.0~rc1-amd64.deb
securedrop-app-code_1.6.0~rc1+focal.tar.gz securedrop-keyring-0.1.4+1.8.0-amd64.deb
securedrop-app-code_1.6.0~rc1+focal_amd64.deb securedrop-keyring-0.1.4+1.8.0~rc1-amd64.deb
securedrop-app-code_1.7.0~rc1+focal.tar.gz securedrop-keyring-0.1.4+1.8.1-amd64.deb
securedrop-app-code_1.7.0~rc1+focal_amd64.deb securedrop-keyring-0.1.4+1.8.2-amd64.deb
securedrop-app-code_1.8.0+focal.tar.gz securedrop-ossec-agent-3.6.0+1.6.0~rc1-amd64.deb
securedrop-app-code_1.8.0+focal_amd64.deb securedrop-ossec-agent-3.6.0+1.7.0~rc1-amd64.deb
securedrop-app-code_1.8.0~rc1+focal.tar.gz securedrop-ossec-agent-3.6.0+1.8.0-amd64.deb
securedrop-app-code_1.8.0~rc1+focal_amd64.deb securedrop-ossec-agent-3.6.0+1.8.0~rc1-amd64.deb
securedrop-app-code_1.8.1+focal.tar.gz securedrop-ossec-agent-3.6.0+1.8.1-amd64.deb
securedrop-app-code_1.8.1+focal_amd64.deb securedrop-ossec-agent-3.6.0+1.8.2-amd64.deb
securedrop-app-code_1.8.2+focal.tar.gz securedrop-ossec-server-3.6.0+1.6.0~rc1-amd64.deb
securedrop-app-code_1.8.2+focal_amd64.deb securedrop-ossec-server-3.6.0+1.7.0~rc1-amd64.deb
securedrop-config-0.1.3+1.6.0~rc1-amd64.deb securedrop-ossec-server-3.6.0+1.8.0-amd64.deb
securedrop-config-0.1.3+1.7.0~rc1+focal-amd64.deb securedrop-ossec-server-3.6.0+1.8.0~rc1-amd64.deb
securedrop-config-0.1.3+1.7.0~rc1-amd64.deb securedrop-ossec-server-3.6.0+1.8.1-amd64.deb
securedrop-config-0.1.3+1.8.0~rc1-amd64.deb securedrop-ossec-server-3.6.0+1.8.2-amd64.deb
securedrop-grsec-4.14.188+focal-amd64.deb tor-geoipdb_0.4.4.5-1~focal+1_all.deb
securedrop-grsec-4.14.188-amd64.deb tor_0.4.4.5-1~focal+1_amd64.deb
Also, all the reference i found online about unattended-upgrades, they suggest to enable apt-daily
and apt-daily-upgrade
timers to be used. But, I could not figure how that part is happening in the current implementation in Focal in this PR. I changed the time a few times, and that did not reboot the box or updated any packages.
c = host.run('sudo unattended-upgrades -d') | ||
assert c.rc == 0 | ||
expected_origins = ( | ||
"Allowed origins are: origin=Ubuntu,archive=focal, origin=Ubuntu,archive=focal-security" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is nice.
@@ -29,6 +29,7 @@ | |||
rep_dist: "focal" | |||
molecule_dir: "../molecule/upgrade" | |||
dpkg_dir: /var/repos/debs | |||
rep_origin: SecureDrop |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://wiki.debian.org/DebianRepository/Format#Origin has the definition (for the person who is looking at this in future).
@kushaldas you can find the times for the two daily tasks in |
dc72c7d
to
4b62bc2
Compare
@kushaldas thanks for the feedback you've provided out of band. I've resolved the two issues you've identified:
This should unblock you for further testing, thanks for your continued attention on this PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This still did not start the timers.
vagrant@app-staging:~$ sudo systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2021-02-02 14:48:41 UTC 4h 13min left Tue 2021-02-02 09:18:57 UTC 1h 16min ago motd-news.timer motd-news.service
Tue 2021-02-02 19:13:07 UTC 8h left Sun 2020-08-16 03:36:36 UTC 5 months 18 days ago fwupd-refresh.timer fwupd-refresh.service
Wed 2021-02-03 00:00:00 UTC 13h left Tue 2021-02-02 09:12:11 UTC 1h 23min ago logrotate.timer logrotate.service
Wed 2021-02-03 00:00:00 UTC 13h left Tue 2021-02-02 09:12:11 UTC 1h 23min ago man-db.timer man-db.service
Wed 2021-02-03 09:40:11 UTC 23h left Tue 2021-02-02 09:40:11 UTC 55min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2021-02-07 03:10:29 UTC 4 days left Tue 2021-02-02 09:12:53 UTC 1h 22min ago e2scrub_all.timer e2scrub_all.service
Mon 2021-02-08 00:00:00 UTC 5 days left Tue 2021-02-02 09:12:11 UTC 1h 23min ago fstrim.timer fstrim.service
7 timers listed.
Pass --all to see loaded but inactive timers, too.
vagrant@app-staging:~$ sudo systemctl status apt-daily-upgrade.timer
● apt-daily-upgrade.timer - Daily apt upgrade and clean activities
Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.timer; disabled; vendor preset: enabled)
Active: inactive (dead)
Trigger: n/a
Triggers: ● apt-daily-upgrade.service
When we have the timers running, we will see something similar with the details below (after starting both the services).
vagrant@app-staging:~$ sudo systemctl start apt-daily-upgrade.timer
vagrant@app-staging:~$ sudo systemctl status apt-daily-upgrade.timer
● apt-daily-upgrade.timer - Daily apt upgrade and clean activities
Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.timer; disabled; vendor preset: enabled)
Active: active (waiting) since Tue 2021-02-02 10:38:17 UTC; 1s ago
Trigger: Wed 2021-02-03 06:59:27 UTC; 20h left
Triggers: ● apt-daily-upgrade.service
Feb 02 10:38:17 app-staging systemd[1]: Started Daily apt upgrade and clean activities.
vagrant@app-staging:~$ sudo systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2021-02-02 14:02:29 UTC 2h 53min left Sun 2020-08-16 03:36:36 UTC 5 months 18 days ago apt-daily.timer apt-daily.service
Tue 2021-02-02 14:48:41 UTC 3h 39min left Tue 2021-02-02 09:18:57 UTC 1h 49min ago motd-news.timer motd-news.service
Tue 2021-02-02 19:13:07 UTC 8h left Sun 2020-08-16 03:36:36 UTC 5 months 18 days ago fwupd-refresh.timer fwupd-refresh.service
Wed 2021-02-03 00:00:00 UTC 12h left Tue 2021-02-02 09:12:11 UTC 1h 56min ago logrotate.timer logrotate.service
Wed 2021-02-03 00:00:00 UTC 12h left Tue 2021-02-02 09:12:11 UTC 1h 56min ago man-db.timer man-db.service
Wed 2021-02-03 06:59:27 UTC 19h left Tue 2021-02-02 10:38:17 UTC 30min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2021-02-03 09:40:11 UTC 22h left Tue 2021-02-02 09:40:11 UTC 1h 28min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2021-02-07 03:10:29 UTC 4 days left Tue 2021-02-02 09:12:53 UTC 1h 56min ago e2scrub_all.timer e2scrub_all.service
Mon 2021-02-08 00:00:00 UTC 5 days left Tue 2021-02-02 09:12:11 UTC 1h 56min ago fstrim.timer fstrim.service
Now, the following patch adds this to our ansible role.
diff --git a/install_files/ansible-base/roles/common/tasks/unattended_upgrades.yml b/install_files/ansible-base/roles/common/tasks/unattended_upgrades.yml
index 29e1b876e..eee592cb2 100644
--- a/install_files/ansible-base/roles/common/tasks/unattended_upgrades.yml
+++ b/install_files/ansible-base/roles/common/tasks/unattended_upgrades.yml
@@ -25,3 +25,15 @@
tags:
- apt
- unattended-upgrades
+
+- name: Ensure apt-daily and apt-daily-upgrade timers are started, and enabled.
+ systemd:
+ name: "{{ item }}"
+ state: started
+ enabled: yes
+ with_items:
+ - 'apt-daily.timer'
+ - 'apt-daily-upgrade.timer'
+ tags:
+ - apt
+ - unattended-upgrades
But, still does not solve the timer issues, as you can notice, the apt-daily
and apt-daily-upgrade
are running on strange time automatically. @emkll please let me know what do you think.
Also, the config is missing the -focal
from ./update_version.sh.
securedrop-config-0.1.3+1.8.0~rc1+focal-amd64.deb
securedrop-config-0.1.3+1.9.0-amd64.deb
Universe channel is required for `ntp` and `aptitude`
The current configuration of SecureDrop apt servers does not contain the `Suite` field which, with `Origin`, is the default combination used by unattended-upgrades to allow certain package lists to be updated. In order to avoid using a wildcard (e.g. SecureDrop:*, which would allow any (configured in apt sources) Suite for the SecureDrop Origin to be updated through unatteded upgrades, we use the newer Allowed-Origins syntax to specify the `Codename` field in lieu of the `Suite` field for SecureDrop.
This is inline with the configuration provided by apt(-test).freedom.press.
Unattended upgrades will reboot the system if and only if the dependency that is being updated requires a reboot. However, SecureDrop relies on cron-apt's reboot to ensure the memory is regularly cleared from the system. In order to reboot on a daily basis, we drop the updates-required flag in /var/run to tell unattended-upgrades that the system should be rebooted at the scheduled time. The absence of update-notifier-common package will make daily reboots silently fail, so adding explicitly to the apt package install step to ensure it's installed as it's not pulled in the Depends field of unattended-upgrades.
4b62bc2
to
cd445c1
Compare
securedrop-config package will provide the configuration for focal only
44b1ae6
to
19ea23f
Compare
Based on feedback from @kushaldas, in some conditions the apt-daily and apt-daily upgrade would be masked and stopped after install. This should ensure the services and timers are properly enabled after install. They are automatically started by cron when the upgrade processes are invoked.
19ea23f
to
733bd11
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did one more round of testing today: now everything looks good to me. I will wait till tomorrow to check for reboot, but everything else is working as planned.
- Changes to the apt config in distribution.j2 matches the configuration of both apt.freedom.press and apt-test.freedom.press
- Run
molecule converge libvirt-staging-focal
, the unattended-upgrades configuration is as expected - [ ] Optionally, add a package in
Required
of a control file, to observe dependencies also also downloaded by unattended-upgrades - Bump SecureDrop version locally:
securedrop/bin/dev-shell ../update_version.sh 1.7.0
and build focal packages:make build-debs-focal
- Deploy a local apt server
vagrant up apt-local
- Update or add an apt source to point to local server (dev[arch=amd64] http://10.0.1.7 focal main
If using staging, you will need to un-hold packages:- app :
sudo aptitude unhold securedrop-app-code securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-agent
and mon, un - mon :
sudo aptitude unhold securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-server
- app :
-
sudo unattended-upgrades --dry-run
makes sense,sudo unattended-upgrade d
will apply the upgrades - Bump SecureDrop version once again:
securedrop/bin/dev-shell ../update_version.sh 1.7.1
and build focal packages:make build-debs-focal
- Redeploy a local apt server
vagrant destroy apt-local && vagrant up apt-local
(maybevagrant provision apt-local
will work here as well, untested. - Observe the package being updated automatically
Looks great! Thanks for changes, @emkll. Test plan checks out, made sure to observe upgrades to unheld staging packages via |
Status
Ready for Review
See freedomofpress/securedrop-docs#146 for documentation changes
Description of Changes
Fixes #5295 , fixes #3376
A couple of comments for the reviewer:
''
tofocal
Testing
molecule converge libvirt-staging-focal
, the unattended-upgrades configuration is as expectedRequired
of a control file, to observe dependencies also also downloaded by unattended-upgradessecuredrop/bin/dev-shell ../update_version.sh 1.7.0
and build focal packages:make build-debs-focal
vagrant up apt-local
If using staging, you will need to un-hold packages:
sudo aptitude unhold securedrop-app-code securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-agent
and mon, unsudo aptitude unhold securedrop-config securedrop-grsec securedrop-keyring securedrop-ossec-server
sudo unattended-upgrades --dry-run
makes sense,sudo unattended-upgrade d
will apply the upgradessecuredrop/bin/dev-shell ../update_version.sh 1.7.1
and build focal packages:make build-debs-focal
vagrant destroy apt-local && vagrant up apt-local
(maybevagrant provision apt-local
will work here as well, untested.Deployment
Changes will be applied to new Focal installs via Ansible, and behavior for Xenial should remain unchanged
Checklist
If you made changes to the server application code:
make lint
) and tests (make test
) pass in the development containerIf you made changes to the system configuration:
If you made non-trivial code changes:
Choose one of the following: