Update Admin USB rebuild instructions for v3 + 20.04 instances

[rocodes]

Status

Ready for review

Description of Changes

• Description: Update Admin rebuild instructions for use with Focal / v3 onions

• Fixes Instructions for rebuilding Admin Workstation are not applicable for v3 #138

Testing

• visual review
• step through instructions with fresh Admin USB on hardware

Release

• n/a

Checklist (Optional)

• Doc linting (make docs-lint) passed locally
• [:x: ] Doc link linting (make docs-linkcheck) passed - error on unrelated link (https://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html), appears to be ssl-related
• You have previewed (make docs) docs at http://localhost:8000

[rocodes]

Comments for reviewer:

• Currently, the backup happens before SSH-over-Tor has been set up again. So the steps are: enable ssh-over-LAN, grab required files, sdconfig, backup, enable ssh-over-Tor, install, tailsconfig. This seems less than ideal, but I guess it's safer than proceeding with the install step without having an existing backup. I'm kind of hoping someone will vote to move the backup til later in the process.
• I've included grabbing the journalist alerts ossec configuration info in this rebuild; it was not previously included.

[eloquence]

One early observation: The instructions for single-user mode may need to be updated. At least on my NUC10s with Ubuntu 20.04:

• The GRUB menu does not appear unless I hold "Escape" during boot
• The Linux boot options look different:
  

@rocodes would you like to consider that in the scope of this PR, or would you prefer if I file a separate issue for it?

[rocodes]

Thanks for your review! I've addressed your comments in a5a6392.

I didn't provide updated instructions for getting into the GRUB menu because it does vary based on hardware model and I think it's fair to expect admins to figure that part out themselves if need be, but I did update the line about the kernelopts in single user mode.

[eloquence]

I stepped through a full rebuild using these docs and while I encountered a couple of stumbles (I specified the wrong IP to iptables; later Ansible errored out once during "Timed out waiting for last boot time check (timeout=600)"; I had to restart Tor manually to regain SSH access), none of this was due to the instructions here, which worked beautifully. 🎉

I may push some minor tweaks but otherwise I think this looks pretty good to go.

[eloquence]

This looks good from my end! @zenmonkeykstop, it would be great if you could have a look. I think a visual review & input on overall structure and open comments is probably sufficient; I tested the overall process (except for the HTTPS configuration -- I only tested that the tar syntax works), and it worked well for me.

[zenmonkeykstop]

I haven't attempted a run-thru yet, but I think this needs some changes to address the fact that authentication keys will change. At minimum all existing keys need to be reprovisioned (this wasn't the case for v2 because the hidden service auth token didn't change).

[rocodes]

(update: more work tk on this on Monday)

[eloquence]

(Starting another round of review on this today, may not get all the way through until tomorrow.)

[eloquence]

I stepped through the whole process one more time and I think this is good to go. Thanks for all your hard work on this, @rocodes.

I hit one snag: When running ./securedrop-admin install, it kept rebooting the servers, which kept destroying the iptables rules, causing subsequent commands to hang indefinitely. I'm not quite sure why it kept rebooting, after the task "Add overrides for apt-daily timers", with Ansible then including the reboot.yml file. Commenting that file out let me proceed, but I'd like to understand if this is an error real users may encounter, and if so, how we may be able to mitigate it.

[eloquence]

Thanks @rocodes! As per our discussion last week, I've not re-tested whether the SSH-over-LAN setting resolves the reboot issue during the playbook run, but logically it makes sense to me, and I trust in your own test results. @zenmonkeykstop, I believe your previous review comments were all addressed - I think this is good to merge.

Note I've appended one small commit to clarify the implications of the SSH-over-Tor vs. SSH-over-LAN setting.

Thanks a lot for the hard work on this, glad to be getting these updated docs in.

pr updated