Updated restore playbook to only compare tor configs when configs are being updated. #5834

zenmonkeykstop · 2021-03-01T19:30:17Z

Status

Ready for review

Description of Changes

Fixes #5829 .

Updates the restore playbook tor config comparison logic such that:

if --preserve-tor-config is set, the results of the comparison are ignored and a data-only restore is performed
if the backup and server have the same tor service types defined, a full restore is performed
if the backup is 'v2+v3' and the server is v3, a restore is performed of v3 services and data only.
the restore is not performed in any other cases.

Testing

On a production v2+v3 install (vms or h/w):

take a backup with ./securedrop-admin backup
restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
- Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
- Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
- Verify that the restore finishes immediately without applying data or config changes
flip the install to v3-only using the sdconfig and install admin subcommands, then:
- restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
  - Verify that the Extract backup using v3-only task runs, but the other Extract tasks do not
  - verify that the restore completes and that only the v3 JI/SI and SSH configs are present.
- restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
  - Verify that the Extract backup, skipping tor service configurations task runs, but the other Extract task do not
  - verify that the restore completes and that the v3 JI/SI and SSH configs are unchanged and the v2 ssh configs have not been restored
- if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
  - Verify that the restore finishes immediately without applying data or config changes

Checklist

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

Choose one of the following:

I have opened a PR in the docs repo for these changes, or will do so later
I would appreciate help with the documentation
These changes do not require documentation

kushaldas

take a backup with ./securedrop-admin backup
restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
- Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
- Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
- Verify that the restore finishes immediately without applying data or config changes
flip the install to v3-only using the sdconfig and install admin subcommands, then:
- restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
  - Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not. I got the exact Extract backup, using v3 services only.
  - verify that the restore completes and that only the v3 JI/SI and SSH configs are present.
- restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
  - Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not
  - verify that the restore completes and that the v3 JI/SI and SSH configs are unchanged and the v2 ssh configs have not been restored
- if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
  - Verify that the restore finishes immediately without applying data or config changes

Worked well, now in a separate PR I will work on the new safety failures.

conorsch

These changes resolved the error reported in #5829 for me. However, according to

if --preserve-tor-config is set, the results of the comparison are ignored and a data-only restore is performed

That's not actually what happened for me. Using a v2-only Xenial backup tarball, the data was restored (full db and prior submissions), but the v2 services were destroyed. After running the restore, I logged into the Application Server, and saw only v3 services: sshv3, sourcev3, journalistv3.

That's not really a problem, since it matches our Focal-is-v3-only expectations. Still, calling it out for attention in case since it'll likely be relevant in support contexts.

conorsch · 2021-03-02T15:19:43Z

install_files/ansible-base/roles/restore/tasks/main.yml

+      - "This backup's tor configuration cannot be applied on this server."
+      - "A data-only restore can be applied using the --preserve-tor-config argument"
+      - "More info: {{ compare_result.stdout }}"
+  when: restore_skip_tor is not defined


Nit: consider setting restore_skip_tor to False in defaults/main.yml, then update this line to be when: not restore_skip-tor. Checking for definedness works in this case, but only because it's non-existent, or false.

eloquence · 2021-03-02T16:59:16Z

Using a v2-only Xenial backup tarball, the data was restored (full db and prior submissions), but the v2 services were destroyed.

Yes, that's also my expectation of what should happen with a restore to Focal, and we'll make sure to make that clear in the docs.

zenmonkeykstop · 2021-03-02T19:06:37Z

If @conorsch 's instance wasn't actually Focal tho (I'm guessing Xenial v2+v3, trying to apply a data-only v2 backup?) then this isn't the desired behaviour. There are a bunch of tasks to clear out V2 configs that should only run if it's a Focal target or if we're restoring just the v3 component of a v2+v3 backup. They shouldn't run in the case where we're doing a data-only backup.

Given that this is a config-removing bug, I'll update the PR real quick to address.

conorsch · 2021-03-02T19:19:58Z

instance wasn't actually Focal tho

Process was:

Install clean v2-only on Xenial prod VMs.
Create v2-only backup tarball.
Destroy Xenial prod VMs.
Create Focal prod VMs.
Install clean v3-only on Focal prod VMs, after running sdconfig to shift v2->v3.
Attempt to restore v2-only backup tarball on Focal VMs.

Step 6 is where it failed for me in #5829, but when testing with these changes, the restore process completed. This works as expected, and I don't recommend that we change the behavior further.

…ing the server's config: - if --preserve-tor-config is set, tor configs are ignored completely and only data is restored - if configs match, complete restore performed - if v2+v3 restore to v3 server, only v3 is restored - playbook exits early without restoring anything in all other cases

rocodes · 2021-03-03T22:29:57Z

this will be slighty out of date, I tested before 43c0e44 was a thing. About to retest, same scenarios (testing on a v3-only install)

Testing In progress

[...]

flip the install to v3-only using the sdconfig and install admin subcommands, then:
- restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
  - [:x: ] Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not
    (from me: Extract backup, using v3 services only runs, followed by Extract backup, skipping tor service configuration)
  - (Also from me: The playbook times out waiting for Tor to reload.** (App ssh creds still being changed?))
  - verify that the restore completes and that only the v3 JI/SI and SSH configs are present.
- restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
  - [:x:] Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not **neither task showed in the output. **
  - verify that the restore completes and that the v3 JI/SI and SSH configs are unchanged and the v2 ssh configs have not been restored
- if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
  - Verify that the restore finishes immediately without applying data or config changes

rmol · 2021-03-03T23:33:15Z

Installed 1.7.1 on Xenial with v2 and v3 enabled.
Took a backup with ./securedrop-admin backup.
Ran securedrop-qa playbook and performed cron-apt upgrade to get to
1.8.0.
Checked out this branch.
restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
- Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
  ⚠️ the v2 services were removed, but the v3 configs were unchanged.
restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
- Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
  ⚠️ again, the v2 services were removed.

At this point, I can't restore the v2 services, even if I check out 1.7.1 again. The Tor configuration comparison fails and the restore is aborted. I'm told to contact myself via the support portal. 🙂

rocodes · 2021-03-03T23:39:21Z

Testing

[...]

flip the install to v3-only using the sdconfig and install admin subcommands, then:
- restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
  - Verify that the Extract backup using v3-only task runs, but the other Extract tasks do not
  - verify that the restore completes and that only the v3 JI/SI and SSH configs are present.
  - Following the procedure specified in the release test plan (Wait for Tor to reload times out, ssh access is repaired by copying in app-ssh.auth_private, etc), this works as expected
- restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
  - Verify that the Extract backup, skipping tor service configurations task runs, but the other Extract task do not
  - verify that the restore completes and that the v3 JI/SI and SSH configs are unchanged and the v2 ssh configs have not been restored
- if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
  - Verify that the restore finishes immediately without applying data or config changes

kushaldas · 2021-03-04T09:38:01Z

⚠️ the v2 services were removed, but the v3 configs were unchanged.

I think as we were already accessing with v3 ssh service, many of us did not notice this.

./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here

But, for a Xenial system, this should not happen. Once again, as a tester we could still ssh into the box with the new v3 ssh access. After spending sometime thinking about this I think this is an okay behavior. Because once one box got v3 ssh access, why do we keep (or restore) v2 ssh access?

I would love to hear what the rest of the team think about this.

rmol · 2021-03-04T13:12:52Z

I think my second scenario, with --preserve-tor-config, was actually OK. The Tor configuration left by the previous step -- v3-only -- was preserved. It just might be frustrating that v2 can't easily be gotten back onto a Xenial system once removed.

I reinstalled and ran that test successfully, with v2 services being left enabled after the restore.

rmol · 2021-03-04T15:17:31Z

rocodes · 2021-03-04T15:32:39Z

Ok so to recap (and check my assumptions), the case @rmol is describing is, an instance is (still) on Xenial v2+v3, restores a Xenial v2+v3 backup without using preserve-tor-config, ends up on v3-only.

That is kind of unfriendly/surprising, but it will only apply to v2v3 instances trying to restore onto Xenial before April 30, which... is probably not going to come up (and doubtful it would come up without someone filing a support ticket, anyway). My opinion is we are pushing people towards Focal and v3 within a couple weeks anyway, so given the time constraints I think it kind of falls under the "if you have a specific restore scenario you need help with, contact Support," and is okay.

If others feel similarly and want to go ahead with this, I can make more explicit note of this behaviour in the migration docs PR.

zenmonkeykstop · 2021-03-04T15:58:14Z

This wasn't the intent of the PR tho - so it's a bug. My take is that this behaviour is going to be unexpected (and will break v2->v2 backups altogether) so it should be fixed rather than documented. Gonna take one final stab at correcting the logic, if I can prevail upon ye for a final round of testing in the next little while!

zenmonkeykstop · 2021-03-04T16:32:03Z

OK @rmol or @rocodes if you're set up for v2 or v2+v3 testing (Xenial obvs) please take a spin through. The behaviour should be correct now - the v2 service removal should only be triggered when a v2+v3 backup is applied to a v3 instance and --preserve-tor-config is not set.

Apologies for the back and forth on this one!

rmol · 2021-03-04T22:17:17Z

After last fix:

installed 1.7.1 v2-only, backed it up, restored using this
branch. no errors, and v2 services were not disabled.

On a production v2+v3 install (vms or h/w):

take a backup with ./securedrop-admin backup
restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
- Verify that the Extract backup, skipping tor service configurations task runs, but the Extract backup task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
- Verify that the Extract backup task runs, but the Extract backup, skipping tor service configurations task does not
- verify that the restore completes and that the JI/SI and SSH configs are unchanged.
if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
- Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
- Verify that the restore finishes immediately without applying data or config changes
flip the install to v3-only using the sdconfig and install admin subcommands, then:
- restore the v2+v3 backup with ./securedrop-admin --force restore your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and passes.
  - Verify that the Extract backup using v3-only task runs, but the other Extract tasks do not
  - verify that the restore completes and that only the v3 JI/SI and SSH configs are present.
- restore the backup again with ./securedrop-admin --force restore --preserve-tor-config your/backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task is not executed.
  - Verify that the Extract backup, skipping tor service configurations task runs, but the other Extract task do not
  - verify that the restore completes and that the v3 JI/SI and SSH configs are unchanged and the v2 ssh configs have not been restored
- if a v2 backup is available from any test server, attempt to apply it with ./securedrop-admin --force restore your/v2backupfile/path/here
  - Verify that the "Verify that the backup Tor config is compatible with the server Tor config" task runs and fails.
  - Verify that the restore finishes immediately without applying data or config changes

zenmonkeykstop requested review from conorsch, emkll and kushaldas as code owners March 1, 2021 19:30

zenmonkeykstop requested a review from rocodes March 1, 2021 19:30

kushaldas previously approved these changes Mar 2, 2021

View reviewed changes

conorsch previously approved these changes Mar 2, 2021

View reviewed changes

zenmonkeykstop dismissed stale reviews from conorsch and kushaldas via 4cd77c7 March 2, 2021 20:04

zenmonkeykstop force-pushed the 5829-fix-restore-without-tor branch from 2af982a to 5bd7313 Compare March 2, 2021 21:40

zenmonkeykstop mentioned this pull request Mar 2, 2021

Release SecureDrop 1.8.0 #5794

Closed

27 tasks

rocodes assigned rocodes and unassigned rocodes Mar 3, 2021

zenmonkeykstop added this to the 1.8.0 milestone Mar 3, 2021

eloquence added the release blocker label Mar 3, 2021

fixed incorrect conditional on data-only extract task

43c0e44

rmol previously approved these changes Mar 4, 2021

View reviewed changes

updated v2 services removal condition

f2b9f41

zenmonkeykstop dismissed rmol’s stale review via f2b9f41 March 4, 2021 16:16

rmol approved these changes Mar 4, 2021

View reviewed changes

rmol merged commit 60f8d54 into develop Mar 4, 2021

rmol deleted the 5829-fix-restore-without-tor branch March 4, 2021 22:18

emkll mentioned this pull request Mar 4, 2021

[1.8.0] Bump to 1.8.0 and bump to RC2 #5846

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated restore playbook to only compare tor configs when configs are being updated. #5834

Updated restore playbook to only compare tor configs when configs are being updated. #5834

zenmonkeykstop commented Mar 1, 2021 •

edited

Loading

kushaldas left a comment •

edited

Loading

conorsch left a comment

conorsch Mar 2, 2021

eloquence commented Mar 2, 2021

zenmonkeykstop commented Mar 2, 2021

conorsch commented Mar 2, 2021

rocodes commented Mar 3, 2021 •

edited

Loading

rmol commented Mar 3, 2021

rocodes commented Mar 3, 2021 •

edited

Loading

kushaldas commented Mar 4, 2021

rmol commented Mar 4, 2021

rmol commented Mar 4, 2021

rocodes commented Mar 4, 2021 •

edited

Loading

zenmonkeykstop commented Mar 4, 2021

zenmonkeykstop commented Mar 4, 2021

rmol commented Mar 4, 2021

Updated restore playbook to only compare tor configs when configs are being updated. #5834

Updated restore playbook to only compare tor configs when configs are being updated. #5834

Conversation

zenmonkeykstop commented Mar 1, 2021 • edited Loading

Status

Description of Changes

Testing

Checklist

If you made non-trivial code changes:

kushaldas left a comment • edited Loading

Choose a reason for hiding this comment

conorsch left a comment

Choose a reason for hiding this comment

conorsch Mar 2, 2021

Choose a reason for hiding this comment

eloquence commented Mar 2, 2021

zenmonkeykstop commented Mar 2, 2021

conorsch commented Mar 2, 2021

rocodes commented Mar 3, 2021 • edited Loading

Testing In progress

rmol commented Mar 3, 2021

rocodes commented Mar 3, 2021 • edited Loading

Testing

kushaldas commented Mar 4, 2021

rmol commented Mar 4, 2021

rmol commented Mar 4, 2021

rocodes commented Mar 4, 2021 • edited Loading

zenmonkeykstop commented Mar 4, 2021

zenmonkeykstop commented Mar 4, 2021

rmol commented Mar 4, 2021

zenmonkeykstop commented Mar 1, 2021 •

edited

Loading

kushaldas left a comment •

edited

Loading

rocodes commented Mar 3, 2021 •

edited

Loading

rocodes commented Mar 3, 2021 •

edited

Loading

rocodes commented Mar 4, 2021 •

edited

Loading