Switch cronjobs to systemd timers

[legoktm]

Status

Ready for review

Description of Changes

systemd timers are easier to define and offer more flexibility in running commands. For example, 9169606 would've been a 2 line change instead of a complex and fragile bash migration.

We clear the old crontabs in the postinst and let debhelper enable the timers.

Turns out our usage of dh_systemd_enable and dh_systemd_start was wrong, because we were calling it multiple times without specifying explicit unit files, it was adding the same stanza to the postinst/postrm multiple times. We now invoke it explicitly for each unit we want enabled/started. Hopefully when we switch to compat 12/dh_installsystemd we don't need to manually list each one.

Fixes #6705.
Fixes #6748.

Testing

• Run make build-debs. Copy the securedrop-app-code deb into a temporary directory (so it doesn't get messy) and extract the maintainer scripts with ar x securedrop-app-code_2.6.0\~rc1+focal_amd64.deb && tar xvf control.tar.xz. Verify that the 2 timers and 4 pre-existing service units all have identical stanzas in the postinst, postrm, and prerm.
  • You can also run make build-debs without this PR and diffoscope against the new build with this PR, but I found enough stuff got re-arranged that the diff wasn't very useful to look at.
• Set up a staging/prod instance with either 2.5.1 or develop before this PR. Then with this PR, run make build-debs and install the securedrop-app-code deb with sudo apt install ./securedrop-app-code_2.6.0~rc1+focal_amd64.deb
• Run systemctl list-timers, see the 2 new timers appear
  • securedrop-clean-tmp should be scheduled for 00:00 UTC.
  • securedrop-submissions-today should be scheduled for 1 hour before the reboot. Easiest way to verify this is add the amount of hours left (e.g. mine says 9h left) with the uptime (mine is 14h), it should add up to 23h.
• Manually create a backdated temporary file to be deleted: sudo -u www-data touch /var/lib/securedrop/tmp/delete-me -mt 2303300000
• Run sudo crontab -l and sudo crontab -u www-data -l, both should say "no crontab for root/www-data"
• Run systemctl status securedrop-clean-tmp.service, verify it is not enabled and not active. Then run systemctl status securedrop-clean-tmp.timer, verify it is active and enabled.
  • Repeat for securedrop-submissions-today.{service,timer}
• Wait a day
• verify you receive the next "Submissions in the past 24h" email at the same time as before.
• Run sudo -u www-data ls -l /var/lib/securedrop/tmp/ and verify the delete-me file is gone.
• Run sudo cat /var/log/auth.log | grep cron | grep www-data, verify the lines like app CRON[268825]: pam_unix(cron:session): session opened for user www-data by (uid=0) have stopped.

Deployment

Any special considerations for deployment? Consider both:

1. Upgrading existing production instances.
   • The postinst will clear any cron entries and the packaging sets up the systemd services and timers
2. New installs.
   • The postinst will gracefully skip over the fact no cron entries exist, and the packaging still sets up the systemd services + timers

Checklist

• Configuration tests pass
• I have written a test plan and validated it for this PR
• These changes do not require documentation

[legoktm]

Still in draft because I need to wait a day to ensure the timers + units are executed properly and on time. I'm also not sure whether the OnBootSec=23h hack is acceptable. E.g. if the instance reboots in the middle of the day for whatever reason (power loss, maintenance, etc.), then that day's notification will get skipped.

A more robust approach might be to run the timer every hour and have the script check the time and do nothing if it's not correct.

[cfm]

I've started with a code review (comments inline) and an inspection of the securedrop-app-code package according to the test plan:

• Run make build-debs. Copy the securedrop-app-code deb into a temporary directory (so it doesn't get messy) and extract the maintainer scripts with ar x securedrop-app-code_2.6.0\~rc1+focal_amd64.deb && tar xvf control.tar.xz. Verify that the 2 timers and 4 pre-existing service units all have identical stanzas in the postinst, postrm, and prerm.

Identical within each script, not across scripts, per https://gist.github.com/cfm/389f7ea72a526e558f7b6b2bb9a3224e. :-)

Tomorrow I'll spin up a staging instance for testing.

[cfm]

I've kicked off the test plan in both staging VMs (for cleanliness) and QA hardware (for e-mail notifications) in https://pad.riseup.net/p/securedrop-6780-cfm. I'll have results tomorrow.

[cfm]

Trivial changes since yesterday are trivial, and the test plan checked out nicely on both staging VMs and QA hardware:

• Set up a staging/prod instance with either 2.5.1 or develop before this PR. Then with this PR, run make build-debs and install the securedrop-app-code deb with sudo apt install ./securedrop-app-code_2.6.0~rc1+focal_amd64.deb

+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop-clean-tmp.timer
+ deb-systemd-helper --quiet was-enabled securedrop-clean-tmp.timer
+ deb-systemd-helper enable securedrop-clean-tmp.timer
Created symlink /etc/systemd/system/timers.target.wants/securedrop-clean-tmp.timer → /lib/systemd/system/securedrop-clean-tmp.timer.
+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop-submissions-today.timer
+ deb-systemd-helper --quiet was-enabled securedrop-submissions-today.timer
+ deb-systemd-helper enable securedrop-submissions-today.timer
Created symlink /etc/systemd/system/timers.target.wants/securedrop-submissions-today.timer → /lib/systemd/system/securedrop-submissions-today.timer.
+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop_rqrequeue.service
+ deb-systemd-helper --quiet was-enabled securedrop_rqrequeue.service
+ deb-systemd-helper enable securedrop_rqrequeue.service
+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop_rqworker.service
+ deb-systemd-helper --quiet was-enabled securedrop_rqworker.service
+ deb-systemd-helper enable securedrop_rqworker.service
+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop_shredder.service
+ deb-systemd-helper --quiet was-enabled securedrop_shredder.service
+ deb-systemd-helper enable securedrop_shredder.service
+ '[' configure = configure ']'
+ deb-systemd-helper unmask securedrop_source_deleter.service
+ deb-systemd-helper --quiet was-enabled securedrop_source_deleter.service
+ deb-systemd-helper enable securedrop_source_deleter.service
+ '[' configure = configure ']'
+ '[' -d /run/systemd/system ']'
+ systemctl --system daemon-reload
+ deb-systemd-invoke start securedrop-clean-tmp.service securedrop-clean-tmp.timer securedrop-submissions-today.service securedrop-submissions-today.timer securedrop_rqrequeue.service securedrop_rqworker.service securedrop_shredder.service securedrop_source_deleter.service
securedrop-clean-tmp.service is a disabled or a static unit, not starting it.
securedrop-submissions-today.service is a disabled or a static unit, not starting it.

• Run systemctl list-timers, see the 2 new timers appear
  • securedrop-clean-tmp should be scheduled for 00:00 UTC.
  • securedrop-submissions-today should be scheduled for 1 hour before the reboot. Easiest way to verify this is add the amount of hours left (e.g. mine says 9h left) with the uptime (mine is 14h), it should add up to 23h.

vagrant@app-staging:~$ systemctl list-timers | grep "securedrop"
Thu 2023-05-25 00:00:00 UTC 4h 53min left n/a                         n/a       securedrop-clean-tmp.timer         securedrop-clean-tmp.service        
Thu 2023-05-25 17:51:57 UTC 22h left      n/a                         n/a       securedrop-submissions-today.timer securedrop-submissions-today.service
vagrant@app-staging:~$ uptime
 19:07:19 up 15 min,  2 users,  load average: 0.13, 0.10, 0.04

• Manually create a backdated temporary file to be deleted: sudo -u www-data touch /var/lib/securedrop/tmp/delete-me -mt 2303300000
• Run sudo crontab -l and sudo crontab -u www-data -l, both should say "no crontab for root/www-data"

vagrant@app-staging:~$ sudo crontab -l
no crontab for root
vagrant@app-staging:~$ sudo crontab -u www-data -l
no crontab for www-data

• Run systemctl status securedrop-clean-tmp.service, verify it is not enabled and not active. Then run systemctl status securedrop-clean-tmp.timer, verify it is active and enabled.
  • Repeat for securedrop-submissions-today.{service,timer}

vagrant@app-staging:~$ systemctl status securedrop-clean-tmp.service
● securedrop-clean-tmp.service - job to clean SecureDrop tmp dir daily
     Loaded: loaded (/lib/systemd/system/securedrop-clean-tmp.service; static; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● securedrop-clean-tmp.timer
vagrant@app-staging:~$ systemctl status securedrop-clean-tmp.timer
● securedrop-clean-tmp.timer - Run clean-tmp daily
     Loaded: loaded (/lib/systemd/system/securedrop-clean-tmp.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Wed 2023-05-24 19:05:49 UTC; 3min 57s ago
    Trigger: Thu 2023-05-25 00:00:00 UTC; 4h 50min left
   Triggers: ● securedrop-clean-tmp.service
vagrant@app-staging:~$ systemctl status securedrop-submissions-today.service
● securedrop-submissions-today.service - job to clean SecureDrop tmp dir daily
     Loaded: loaded (/lib/systemd/system/securedrop-submissions-today.service; static; vendor preset: e>
     Active: inactive (dead)
TriggeredBy: ● securedrop-submissions-today.timer
vagrant@app-staging:~$ systemctl status securedrop-submissions-today.timer
● securedrop-submissions-today.timer - Run submissions-today 1h before daily reboot
     Loaded: loaded (/lib/systemd/system/securedrop-submissions-today.timer; enabled; vendor preset: en>
     Active: active (waiting) since Wed 2023-05-24 19:05:49 UTC; 4min 10s ago
    Trigger: Thu 2023-05-25 17:51:57 UTC; 22h left
   Triggers: ● securedrop-submissions-today.service

• Wait a day
• verify you receive the next "Submissions in the past 24h" email at the same time as before.

(Not configured on this staging server, but I've also installed this package on my QA instance, where it is.)

• Run sudo -u www-data ls -l /var/lib/securedrop/tmp/ and verify the delete-me file is gone.
• Run sudo cat /var/log/auth.log | grep cron | grep www-data, verify the lines like app CRON[268825]: pam_unix(cron:session): session opened for user www-data by (uid=0) have stopped.