Disable do-release-upgrade OSSEC notification #3546

redshiftzero · 2018-06-20T22:53:44Z

Status

Ready for review

Description of Changes

Fixes #3205

Changes proposed in this pull request:

Currently admins are getting weekly alerts encouraging them to upgrade to Xenial, but upgrading to Xenial actually breaks SecureDrop. This addition to the postinst scripts of the securedrop OSSEC packages will suppress that notification.

Testing

These instructions assume that you are testing this in staging VMs.

Setup

Make sure you're getting OSSEC emails

First, since the alert does not appear in vagrant VMs, and it actually only appears once per week, we should address both of those things as setup.

git checkout develop
make build-debs

Now make sure you have prod like creds and modify your staging.yml group vars to enable postfix (i.e. apply this diff):

vagrant up /staging/

Now ensure OSSEC alerts are flowing.

Make the `do-release-upgrade` alert happen, and frequently

On each machine, SSH in and in /etc/update-manager/release-upgrades replace Prompt=never with Prompt=lts (this is what is set on hardware).

Also delete the flag file at /var/lib/ubuntu-release-upgrader/release-upgrade-available if it exists.

Finally for easy testing, in /etc/crontab run the weekly job every 5 mins:

*/5 * * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly)

At this point you should be getting the alert about do-release-upgrade to your OSSEC email address every 5 minutes. Wait and verify this is the case.

Note: I only ever get the alert from mon on both hardware and prod VMs. However I think it is best to apply this change to both hosts to ensure this alert doesn't pop up in the future for anyone.

Test the fix

git checkout  resolve-do-release-notification
make build-debs
vagrant provision /staging/

Verify on each server that /etc/update-manager/release-upgrades now has Prompt=never

At this point, the alert will continue to fire, because the weekly job is running /usr/lib/ubuntu-release-upgrader/release-upgrade-motd, and the logic there is to only update the "do we need to do an upgrade" status every 24 hours. This is tracked via the flag file that we deleted earlier, so you need to do one of the following:

Wait 24+ hours to wait for the alert to go away
Edit /usr/lib/ubuntu-release-upgrader/release-upgrade-motd to expire the flag file more rapidly OR
Delete the stamp file at /var/lib/ubuntu-release-upgrader/release-upgrade-available so the check (the actual checking for a new release is done in /usr/lib/ubuntu-release-upgrader/check-new-release) is performed again (this is what I did)

Once one of these three things is done:

Verify the alert no longer appears

Deployment

Will be automatically deployed when SD OSSEC packages update

Checklist

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

At least one instance accidentally upgraded to Xenial due to the now weekly alerts being sent to administrators. This addition to postinst of the securedrop OSSEC packages will suppress that notification.

redshiftzero · 2018-06-20T22:54:36Z

Apologies for the novella in the testing plan here, but it's a little annoying to test this so I erred on the side of more detailed. Let me know if anything is confusing

codecov-io · 2018-06-20T23:32:03Z

Codecov Report

Merging #3546 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3546   +/-   ##
========================================
  Coverage    85.12%   85.12%           
========================================
  Files           37       37           
  Lines         2367     2367           
  Branches       260      260           
========================================
  Hits          2015     2015           
  Misses         289      289           
  Partials        63       63

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdb54d3...d23134b. Read the comment docs.

msheiny · 2018-06-21T17:39:40Z

❤️ the detailed test plan .. stepping thru now

msheiny · 2018-06-21T17:46:50Z

install_files/securedrop-ossec-agent/DEBIAN/postinst

@@ -18,6 +18,8 @@ set -e

 case "$1" in
    configure)
+        # Disable do-release-upgrade notification
+        sed -i 's/Prompt=.*/Prompt=never/' /etc/update-manager/release-upgrades


Just a quick observation.. i think it would make more sense to drop this in securedrop-config since that package already goes on both servers. thoughts @redshiftzero ?

Ahhh, yep - that is probably better

👍 k i can make that change locally and test against it and commit when all is well

@redshiftzero

@redshiftzero did some great work at debugging and preventing ubuntu from emailing out update alerts. This is a tiny tweak to that logic to stuff it in the securedrop-config package which already executes on both servers.

msheiny

This was a pain in the butt to test... thanks for the detailed guidance @redshiftzero . I did tweak my testing to utilize the upgrade testing scenario.. i really want to get that on virtualbox for ya. I think it'll help out your local debugging a ton

msheiny · 2018-06-21T19:42:00Z

on a side note.. you dont have to wait for cron to fire... if you execute the same cron command as root with the same arguments.. just verify no output is shown.. like this:

$ cd /
$ SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin  run-parts --report /etc/cron.weekly

redshiftzero · 2018-06-21T19:42:33Z

Thanks for the review! and definitely, looking forward to getting that upgrade testing on VirtualBox 😍

msheiny · 2018-06-21T19:43:08Z

Apologies for the novella in the testing plan here, but it's a little annoying to test this so I erred on the side of more detailed. Let me know if anything is confusing

Just to be clear.. i super appreciated the full testing plan :)

redshiftzero · 2018-06-21T20:31:05Z

mergin' and backportin'...

[0.8.0] Backports #3546 disabling the do-release-upgrade notification

Disable do-release-upgrade notification

19846ba

At least one instance accidentally upgraded to Xenial due to the now weekly alerts being sent to administrators. This addition to postinst of the securedrop OSSEC packages will suppress that notification.

redshiftzero added the ops/deployment label Jun 20, 2018

redshiftzero requested review from conorsch and msheiny as code owners June 20, 2018 22:53

msheiny reviewed Jun 21, 2018

View reviewed changes

Move upgrade notifier logic to shared config pkg

d23134b

@redshiftzero did some great work at debugging and preventing ubuntu from emailing out update alerts. This is a tiny tweak to that logic to stuff it in the securedrop-config package which already executes on both servers.

msheiny approved these changes Jun 21, 2018

View reviewed changes

redshiftzero merged commit eacb423 into develop Jun 21, 2018

redshiftzero deleted the resolve-do-release-notification branch June 21, 2018 20:31

redshiftzero mentioned this pull request Jun 21, 2018

[0.8.0] Backports #3546 disabling the do-release-upgrade notification #3548

Merged

redshiftzero added a commit that referenced this pull request Jun 21, 2018

Merge pull request #3548 from freedomofpress/backports-3546-0.8.0

d979b24

[0.8.0] Backports #3546 disabling the do-release-upgrade notification

eloquence mentioned this pull request Feb 6, 2019

[xenial] Reinstate LTS upgrade notification #4104

Closed

redshiftzero mentioned this pull request Feb 15, 2019

[xenial] ensure OSSEC alert is sent pointing to xenial upgrade docs #4147

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable do-release-upgrade OSSEC notification #3546

Disable do-release-upgrade OSSEC notification #3546

redshiftzero commented Jun 20, 2018 •

edited by msheiny

Loading

redshiftzero commented Jun 20, 2018

codecov-io commented Jun 20, 2018 •

edited

Loading

msheiny commented Jun 21, 2018

msheiny Jun 21, 2018

redshiftzero Jun 21, 2018

msheiny Jun 21, 2018

redshiftzero Jun 21, 2018

msheiny left a comment

msheiny commented Jun 21, 2018

redshiftzero commented Jun 21, 2018

msheiny commented Jun 21, 2018

redshiftzero commented Jun 21, 2018

Disable do-release-upgrade OSSEC notification #3546

Disable do-release-upgrade OSSEC notification #3546

Conversation

redshiftzero commented Jun 20, 2018 • edited by msheiny Loading

Status

Description of Changes

Testing

Setup

Make sure you're getting OSSEC emails

Make the do-release-upgrade alert happen, and frequently

Test the fix

Deployment

Checklist

If you made changes to the system configuration:

If you made non-trivial code changes:

redshiftzero commented Jun 20, 2018

codecov-io commented Jun 20, 2018 • edited Loading

Codecov Report

msheiny commented Jun 21, 2018

msheiny Jun 21, 2018

Choose a reason for hiding this comment

redshiftzero Jun 21, 2018

Choose a reason for hiding this comment

msheiny Jun 21, 2018

Choose a reason for hiding this comment

redshiftzero Jun 21, 2018

Choose a reason for hiding this comment

msheiny left a comment

Choose a reason for hiding this comment

msheiny commented Jun 21, 2018

redshiftzero commented Jun 21, 2018

msheiny commented Jun 21, 2018

redshiftzero commented Jun 21, 2018

redshiftzero commented Jun 20, 2018 •

edited by msheiny

Loading

Make the `do-release-upgrade` alert happen, and frequently

codecov-io commented Jun 20, 2018 •

edited

Loading