Makes unattended-upgrades sequencing more similar to cron-apt #5855
Conversation
Builds on the work by @rmol in #5853. Slots in overrides to the apt-daily{,-upgrade} timers, shipped with the 'apt' package, to provide fine-grained control over the update and reboot times. Ensures that the apt lists are updated approximately 1h prior to the package upgrade. Lowered the time-fuzzing to 20m on each action, so that even at the extremes, there's still a 20m window for an apt update to complete. Uses a modulus to determine the sooner update time.
"Now" was the default value, we previously set it to the 'daily_reboot_time' to provide some predictability around reboots, but that came at the cost of separating the package updates from the reboot logic. Ideally, we'll have as narrow gap as possible between: * apt update * apt upgrade * reboot and these changes implement that.
Follow-up to #5852. The options parsing for apt configs requires that list options be carefully specified, otherwise the last declared value wins, clobbering all preceding values.
|
I'm currently waiting on the apt timers to fire in the local staging instance I've got configured on this branch. (Fun fact, it's 3AM UTC right now, so I've got about another hour to wait, give or take, with the random delays. Will leave these running and check back for results. |
| Unattended-Upgrade::Automatic-Reboot-Time "{{ daily_reboot_time }}:00"; | ||
| // Reboot should happen after nightly upgrades. Timing of upgrade | ||
| // is configured via apt.daily.timer | ||
| Unattended-Upgrade::Automatic-Reboot-Time "now"; |
There was a problem hiding this comment.
Warning: my understanding of this option is that running sudo unattended-upgrades -d interactively will reboot the system out from under you once updates are complete. This is because we set Unattended-Upgrade::Automatic-Reboot-WithUsers=true. I haven't observed that behavior myself yet, but warning others because it could be surprising.
There was a problem hiding this comment.
Thanks for calling this out. If we can confirm this behavior & the PR lands, I'll add a small note the docs to this effect.
There was a problem hiding this comment.
Even though I can see the timers getting fired, the systems did not reboot.
There was a problem hiding this comment.
To guarantee a reboot, we'd have to update the reboot-flag to fire perhaps hourly. Right now, it fires once every 12 hours. So it's likely that when the update ran, /var/run/reboot-required did not (yet) exist.
| @@ -0,0 +1,5 @@ | |||
| [Timer] | |||
| OnCalendar= | |||
| OnCalendar=*-*-* {{ (daily_reboot_time|int - 1) % 24 }}:00 | |||
There was a problem hiding this comment.
The goal of this modulus is to ensure that apt-daily.timer is configured to run 1h before apt-daily-upgrade, so the apt lists are quite new before installing upgrades.
There was a problem hiding this comment.
This works on the staging instance.
There was a problem hiding this comment.
Running the update more frequently would increase the chance of applying upgrades as soon as possible. You could update every hour except the one in which upgrade is performed with:
OnCalendar=*-*-* {{ range(0,24) | difference([ (daily_reboot_time|int) % 24 ]) | join(",") }}:00
| Dpkg::Options { | ||
| "--force-confdef"; | ||
| "--force-confold"; | ||
| } |
There was a problem hiding this comment.
Here's the change that will re-close #5849.
Uses apt-config directly, rather than naively checking the config files. That gives us a more accurate picture of what the system state is.
conorsch
force-pushed
the
adjust-apt-timers-and-more
branch
from
5829d5e to
1bc7452
Compare
Codecov Report
@@ Coverage Diff @@
## develop #5855 +/- ##
========================================
Coverage 85.66% 85.66%
========================================
Files 53 53
Lines 3885 3885
Branches 484 484
========================================
Hits 3328 3328
Misses 449 449
Partials 108 108 Continue to review full report at Codecov.
|
|
Confirmed working upgrade to 1.8.0~rc3 via cron, at 04:18 UTC. Ready for review! |
| // Default: "now" | ||
| Unattended-Upgrade::Automatic-Reboot-Time "{{ daily_reboot_time }}:00"; | ||
| // Reboot should happen after nightly upgrades. Timing of upgrade | ||
| // is configured via apt.daily.timer |
There was a problem hiding this comment.
typo in comment: apt.daily.timer -> apt-daily.timer?
|
In both staging and prod instances the the timers are getting fired properly. But, the actual
Test steps used
I am digging into this trying to figure out what is wrong here. |
|
I can not find any proper logs for this in the staging server other than the following entry in |
|
Note: the manual update just works, but the |
|
A few more tries shows that the timer worked, but the |
|
Adding |
There was a problem hiding this comment.
It appears the timer/overrides are working as expected, but the apt-daily timer/service is not updating the apt cache:
vagrant@app-staging:~$ stat /var/cache/apt/srcpkgcache.bin
File: /var/cache/apt/srcpkgcache.bin
Size: 40471018 Blocks: 79048 IO Block: 4096 regular file
Device: fc00h/64512d Inode: 526795 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-03-09 15:48:45.692051161 +0000
Modify: 2021-03-09 15:48:45.728051630 +0000
Change: 2021-03-09 15:48:45.728051630 +0000
Birth: -
after the timer is run:
vagrant@app-staging:~$ systemctl list-timers
[...]
Wed 2021-03-10 16:04:48 UTC 23h left Tue 2021-03-09 16:09:42 UTC 3min 56s ago apt-daily.timer apt-daily.service
The apt cache is still not updated on disk:
vagrant@app-staging:~$ stat /var/cache/apt/pkgcache.bin
File: /var/cache/apt/pkgcache.bin
Size: 40491682 Blocks: 79088 IO Block: 4096 regular file
Device: fc00h/64512d Inode: 526916 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-03-09 16:00:59.487188672 +0000
Modify: 2021-03-09 15:48:45.780052307 +0000
Change: 2021-03-09 15:48:45.780052307 +0000
Birth: -
setting APT::Periodic::Update-Package-Lists "always"; in 20auto-upgrades as suggested in another issue did not resolve, nor did removing the stamps in /var/lib/apt/periodic
simply put, the apt-daily service is never run more than once daily, despite the timer or configuration.
setting
to always in `20 does not resolve, the apt-daily timer is simply not updateing the apt cache,
unless i am missing something
vagrant@app-staging:~$ stat /var/cache/apt/srcpkgcache.bin
File: /var/cache/apt/srcpkgcache.bin
Size: 40471018 Blocks: 79048 IO Block: 4096 regular file
Device: fc00h/64512d Inode: 526795 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-03-09 15:48:45.692051161 +0000
Modify: 2021-03-09 15:48:45.728051630 +0000
Change: 2021-03-09 15:48:45.728051630 +0000
Birth: -
vagrant@app-staging:~$ stat /var/cache/apt/
archives/ pkgcache.bin srcpkgcache.bin
vagrant@app-staging:~$ stat /var/cache/apt/^C
vagrant@app-staging:~$ sudo apt update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Hit:2 https://apt-test.freedom.press focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Fetched 223 kB in 1s (348 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
vagrant@app-staging:~$ stat /var/cache/apt/srcpkgcache.bin
File: /var/cache/apt/srcpkgcache.bin
Size: 40471018 Blocks: 79048 IO Block: 4096 regular file
Device: fc00h/64512d Inode: 526795 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-03-09 16:41:37.192161214 +0000
Modify: 2021-03-09 16:41:37.228161626 +0000
Change: 2021-03-09 16:41:37.228161626 +0000
Birth: -
vagrant@app-staging:~$ ls -lah /var/lib/apt/periodic/update-success-stamp
-rw-r--r-- 1 root root 0 Mar 9 16:41 /var/lib/apt/periodic/update-success-stamp
| // time instead of immediately | ||
| // Default: "now" | ||
| Unattended-Upgrade::Automatic-Reboot-Time "{{ daily_reboot_time }}:00"; | ||
| // Reboot should happen after nightly upgrades. Timing of upgrade |
There was a problem hiding this comment.
since we are no longer templating this file (using the reboot time) we could move this configuration to 50unattended-upgrades for simplicity
There was a problem hiding this comment.
Good point, I can move that logic back to the Focal-specific config package now.
| @@ -0,0 +1,5 @@ | |||
| [Timer] | |||
| OnCalendar= | |||
| OnCalendar=*-*-* {{ (daily_reboot_time|int - 1) % 24 }}:00 | |||
There was a problem hiding this comment.
Running the update more frequently would increase the chance of applying upgrades as soon as possible. You could update every hour except the one in which upgrade is performed with:
OnCalendar=*-*-* {{ range(0,24) | difference([ (daily_reboot_time|int) % 24 ]) | join(",") }}:00
|
@emkll That's odd. My staging server updated from rc1 to rc3 without me doing any more than the test plan suggested. I'll watch the cache more closely my next time through. |
@rmol I don't follow. My understanding is that I'll work on a parallel spike to fall back to cron-apt to give us some options. All, please push to this branch at will. |
|
Confirmed the same behaviour as @kushaldas - apt-daily-uprade timer is triggered but nothing is actually upgraded. Tried clearing the timestamp files, and resetting the timer with: Ran again (by editing timer override), nothing happened, and nothing of note recorded in logs bar the timer itself being triggered. |
Right. I was thinking we should use intervals of But it turns out that I can even reboot the system by creating that file before updating the timer overrides with the We could reschedule the cron job that creates Or, we could just go back to |
|
@zenmonkeykstop helpfully pointed out that the bento box configs are different from Qubes staging HVMs, in that the apt-periodic settings are explicitly disabled: https://github.com/chef/bento/blob/b5c6cb3c75bb111b9d55bb26bdb2740c162f1b89/packer_templates/ubuntu/scripts/update.sh#L7-L23 That explains the variation in testing we were seeing. @zenmonkeykstop has pushed a change to be more explicit about that config option, and I've just added a commit to re-implement reboots on the time of |
|
With the two commits above the behaviour should be (where T is
Notes for folks testing this:
The timers only trigger actions if the interval (1 day) since the last action has elapsed. To reliably trigger an action at a specific time:
|
Revision, based on discussion. It turns out that setting reboot time to "now" causes reboots even during download-only stages, which isn't the behavior we want. Let's stagger the actions: * apt-get update * apt-get upgrade * reboot all approximately 1h apart (with 20m random delay on the apt-get actions). This reverts commit bddeb5e.
conorsch
force-pushed
the
adjust-apt-timers-and-more
branch
from
944389a to
74fadb8
Compare
|
Pushed against to tweak a typo in the failing tests. I ever-so-slightly adjusted the |
|
I was wrong about seeing a reboot during the "update" task. I think I must have been running the Having The system was rebooted the next time So, chalk it up to too many iterations of manual timer adjustments. 🙁 I think it might be safe to |
should be |
There was a problem hiding this comment.
Thanks @conorsch @rmol and @zenmonkeykstop for working on these changes, I think we are now observing the expected behavior, Tested the following in staging VMs and the behavior is as we expect:
apt-mark showhold | xargs -r sudo apt-mark unholdrm -f /var/lib/apt/periodic/{upgrade,update}-stamp- set
APT::Periodic::Verbose "3" - imported the apt test key
set curl https://raw.githubusercontent.com/freedomofpress/securedrop/develop/install_files/ansible-base/roles/install-fpf-repo/files/apt-test-signing-key.pub | sudo apt-key add - - edited the timers
- waited
- observed journalctl output:
Wed 2021-03-10 23:26:59 UTC 23h left Tue 2021-03-09 23:26:37 UTC 5min ago apt-daily.timer apt-daily.service
Wed 2021-03-10 23:27:20 UTC 23h left Tue 2021-03-09 23:27:20 UTC 4min 44s ago apt-daily-upgrade.timer apt-daily-upgrade.service
[...]
Mar 9 23:26:42 app-staging systemd[1]: apt-daily.service: Succeeded.
[...]
Mar 09 23:27:55 app-staging systemd[1]: apt-daily-upgrade.service: Succeeded.
Mar 09 23:27:55 app-staging systemd[1]: Finished Daily apt upgrade and clean activities.
Broadcast message from root@app-staging (Tue 2021-03-09 23:28:00 UTC):
The system is going down for reboot at Tue 2021-03-09 23:30:00 UTC!
The reboot occurred, the apt-cache was updated right after apt-daily.service was run
vagrant@app-staging:~$ stat /var/cache/apt/pkgcache.bin
File: /var/cache/apt/pkgcache.bin
Size: 40491536 Blocks: 79088 IO Block: 4096 regular file
Device: fc00h/64512d Inode: 526262 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-03-09 23:27:52.959581115 +0000
Modify: 2021-03-09 23:27:52.735582601 +0000
Change: 2021-03-09 23:27:52.735582601 +0000
Birth: -
It was updated to rc3 packages:
securedrop-app-code/unknown,now 1.8.0~rc3+focal amd64 [installed]
securedrop-config/unknown,now 0.1.4+1.8.0~rc3+focal all [installed]
securedrop-grsec/unknown,now 5.4.97+focal amd64 [installed]
securedrop-keyring/unknown,now 0.1.4+1.8.0~rc3+focal amd64 [installed]
securedrop-ossec-agent/unknown,now 3.6.0+1.8.0~rc3+focal amd64 [installed]
Let's merge and include this in 1.8.0-rc4
|
|
||
| # Template declaration for setting the upgrade time to a predictable time, | ||
| # matching the 'daily_reboot_time' time via sdconfig. | ||
| unattended_upgrades_timer_overrides: |
There was a problem hiding this comment.
Initially i thought this was a template block but it's just declaring the variables 👍
| with_items: "{{ unattended_upgrades_timer_overrides }}" | ||
|
|
||
| # Ensure daemon-reload has happened before starting/enabling | ||
| - meta: flush_handlers |
There was a problem hiding this comment.
|
Pleased to report that I've observed working upgrades to During that testing, I had neglected to |
emkll
dismissed
rmol’s stale review
https://github.com/freedomofpress/securedrop/pull/5855/files#diff-12366c76335b8b6ecc14b0a02f3cd0cce6c6d06d17e2ecd4b282b029926396dfR65-R67 will ensure freshness of apt cache, we can address this in later iterations if there's a need to update apt cache more frequently
Status
Ready for review
Description of Changes
Fixes #5851. Fixes #5849.
Builds on the great work in #5852 & #5853.
apt-config, to avoid problems like Focal: force confold and force confdef in apt options for unattended-upgrades #5852 in the futureTesting
daily_reboot_time1-2h in the future. Re-run the config of just those files to apply, like so:molecule converge -s qubes-staging-focal -- --tags ua -e daily_reboot_time=15apt-mark unhold securedrop-app-code, or evenapt-mark showhold | xargs -r sudo apt-mark unholdso that ua can make changes. Also runrm -f /var/lib/apt/periodic/{upgrade,update}-stampso that ua doesn't defer updates based on recent changes.apt-cache policy securedrop-app-codeand confirm its outdated.securedrop-app-codeis at least1.8.0~rc3.apt-daily.timerandapt-daily-upgrade.timer, different by one hour. For example, formolecule converge -s qubes-staging-focal -- --tags ua -e daily_reboot_time=0you should see a value of midnight for the upgrade, but 11PM (23:00) for the apt update.Deployment
Critical for Focal behavior starting in in 1.8.0.