Adds support for v3 Onion urls to ./securedrop-admin tailsconfig` #4675

zenmonkeykstop · 2019-08-16T16:41:18Z

Status

Work in progress

Description of Changes

This PR adds support for setting up v3 onion urls in 2 cases:

v2 and v3 support are enabled
v3 support only is enabled.

For an Admin Workstation, when v3 support is enabled, ./securedrop-admin tailsconfig expects to find v3 service info in 4 files:

~/Persistent/securedrop/install_files/ansible-base/app-sourcev3-ths, with format:

<56CharAddress>.onion

~/Persistent/securedrop/install_files/ansible-base/app-journalist.auth_private, with format:

<56CharAddress>:<descriptor>:x25519:<clientAuthPrivateKey>

~/Persistent/securedrop/install_files/ansible-base/app-ssh.auth_private, with format:

<56CharAddress>:<descriptor>:x25519:<clientAuthPrivateKey>

~/Persistent/securedrop/install_files/ansible-base/mon-ssh.auth_private, with format:

<56CharAddress>:<descriptor>:x25519:<clientAuthPrivateKey>

An Admin Workstation will also require that the instance's site-specific file be present, with the appropriate values for the required configuration.

For v3 support on a Journalist Workstation, the app-sourcev-3ths and app-journalist.auth_private files must be present, and other files mentioned above must not.

Testing

Admin Workstation testing

Case 1: v2 only (previous state)

Setup:

instance installed with v2_onionservice=true, v3_onion_service=false.
v2 -ths and -aths files must be present in install_files/ansible-base.
v3 files must not be present.

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfuly
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v2 addresses
verify that ~/.ssh/config contains entries for app and mon with their v2 URLs
verify that servers are accessible via ssh app and ssh mon
verify that a subsequent run of ./securedrop-admin install completes successfully

Case 2: v2 and v3 enabled

Setup:

instance installed with v2_onionservice=true, v3_onion_service=true
v2 -ths and -aths files must be present in install_files/ansible-base.
v3 files must be present in install_files/ansible-base.

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses
verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that ~/.ssh/config contains entries for app-legacy and mon-legacy with their v2 URLs
verify that servers are accessible via ssh app and ssh mon
verify that servers are accessible via ssh app-legacy and ssh mon-legacy
verify that a subsequent run of ./securedrop-admin install completes successfully

Case 3: v3 only

Setup:

instance installed with v2_onionservice=false, v3_onion_service=true
v2 -ths and -aths files must not be present
v3 files must be present in install_files/ansible-base.

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses
verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that servers are accessible via ssh app and ssh mon
verify that a subsequent run of ./securedrop-admin install completes successfully

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Setup:

instance installed with v2_onionservice=true, v3_onion_service=false
v2 app-source-ths and app-journalist-aths files must be present in install_files/ansible-base.
v3 files must not be present.
install_files/ansible-base/group_vars/all/site-specific must not be present

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v2 addresses

Case 2: v3 only (previous state)

Setup:

instance installed with v2_onionservice=true, v3_onion_service=true or v2_onion_service=false, v3_onion-service=true
v3 app-sourcev3-ths and app-journalist.auth_private files must be present in install_files/ansible-base.
v2 files must not be present.
install_files/ansible-base/group_vars/all/site-specific must not be present

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains a single .auth_private file with the contents of the app-journalist.auth_private file
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses

Deployment

Any special considerations for deployment? Consider both:

Checklist

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

codecov-io · 2019-08-17T01:18:32Z

Codecov Report

Merging #4675 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #4675   +/-   ##
========================================
  Coverage    82.67%   82.67%           
========================================
  Files           45       45           
  Lines         3122     3122           
  Branches       338      338           
========================================
  Hits          2581     2581           
  Misses         454      454           
  Partials        87       87

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b91c437...bbad1cb. Read the comment docs.

kushaldas

For case 1

TASK [tails-config : Set normal user ownership on subset of directories.] **************************************
ok: [localhost] => (item=/home/amnesia/Persistent/.securedrop)
ok: [localhost] => (item=/home/amnesia/Desktop)
ok: [localhost] => (item=/home/amnesia/.local/share/applications)

TASK [tails-config : Assemble desktop icon info.] **************************************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout'\n\nThe error appears to have been in '/home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/tails-config/tasks/create_desktop_shortcuts.yml': line 62, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# Storing as host fact so we can loop over the data in one task.\n- name: Assemble desktop icon info.\n  ^ here\n"}

NO MORE HOSTS LEFT *********************************************************************************************

NO MORE HOSTS LEFT *********************************************************************************************
	to retry, use: --limit @/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-tails.retry

PLAY RECAP *****************************************************************************************************
localhost                  : ok=18   changed=2    unreachable=0    failed=1

kushaldas

Case 2: v2 and v3 enabled

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses

Did not test ssh values as my v3 is not automatically setup yet

verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that ~/.ssh/config contains entries for app-legacy and mon-legacy with their v2 URLs
verify that servers are accessible via ssh app and ssh mon
verify that servers are accessible via ssh app-legacy and ssh mon-legacy

Case 3: v3 only

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses

Did not test ssh values as my v3 is not automatically setup yet

verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that servers are accessible via ssh app and ssh mon

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Setup:

instance installed with v2_onionservice=true, v3_onion_service=false, using the Add v3 onion support to tor-hidden-services ansible role #4652 candidate branch if the changes in Add v3 onion support to tor-hidden-services ansible role #4652 are not yet merged and rebased into this branch
v2 app-source-ths and app-journalist-aths files must be present in install_files/ansible-base.
v3 files must not be present.

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v2 addresses

^^^ this failed as mentioned in the comment above.

Case 2: v3 only (previous state)

Setup:

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains a single .auth_private file with the contents of the app-journalist.auth_private file
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses

Ansible overrides facts if they are reused even with when statements. We are creating two varaiables from either v2 or from v3 addresses based on what is available and then using those.

kushaldas

This is now approved from my side. 🦄

Preserves the v2 Onion lookup logic, but prefers v3 Onions if those are found on the Admin Workstation. Ensures that Admins connecting to the servers after migrating to v3 Onion URLs are using them for SSH.

kushaldas · 2019-08-22T16:59:11Z

@rmol Do you want to review it once? I can then merge it after your review.

Needs work on dynamic-inventory side.

conorsch · 2019-08-22T17:43:24Z

Based on visual review, looks like there's only one piece missing: updates to the inventory-dynamic script, so that post-v3 migration, Admins are using the v3 Onions to connect to the servers.

Example patch for proposed changes

commit bbad1cbae4bef0da73fc648e75d83852acdfcbf6
Author: Conor Schaefer <[email protected]>
Date:   Thu Aug 22 09:46:47 2019 -0700

    Adds support for v3 Onion to SSH inventory script
    
    Preserves the v2 Onion lookup logic, but prefers v3 Onions if those are
    found on the Admin Workstation. Ensures that Admins connecting to the
    servers after migrating to v3 Onion URLs are using them for SSH.

diff --git a/install_files/ansible-base/inventory-dynamic b/install_files/ansible-base/inventory-dynamic
index 0f2adbefe..6520be98d 100755
--- a/install_files/ansible-base/inventory-dynamic
+++ b/install_files/ansible-base/inventory-dynamic
@@ -92,9 +92,9 @@ def lookup_admin_username():
     return admin_username
 
 
-def lookup_tor_hostname(hostname):
+def lookup_tor_v2_hostname(hostname):
     """
-    Extract Onion URL from HidServAuth file that was fetched back locally.
+    Extract Onion v2 URL from HidServAuth file that was fetched back locally.
     Returns Onion URL for given inventory hostname.
     """
     aths_path = os.path.join(SECUREDROP_ANSIBLE_DIRECTORY,
@@ -106,13 +106,32 @@ def lookup_tor_hostname(hostname):
             # assuming the file is a raw `hostname` file generated by tor,
             # but the SD playbooks format the line with `HidServAuth` prefix,
             # so it can be concatenated into the torrc file on Tails.
-            tor_hostname = tor_config[1]
+            tor_v2_hostname = tor_config[1]
         except IndexError:
-            msg = ("Tor config file for '{}' ",
+            msg = ("Tor v2 config file for '{}' ",
                    "appears to be empty").format(hostname)
             raise Exception(msg=msg)
 
-    return tor_hostname
+    return tor_v2_hostname
+
+
+def lookup_tor_v3_hostname(hostname):
+    """
+    Extract Onion v3 URL from .auth_private file that was fetched back locally.
+    Returns Onion URL for given inventory hostname.
+    """
+    aths_path = os.path.join(SECUREDROP_ANSIBLE_DIRECTORY,
+                             "{}-ssh.auth_private".format(hostname))
+    with io.open(aths_path, 'r') as f:
+        tor_config = f.readline().rstrip().split(":")
+        try:
+            tor_v3_hostname = "{}.onion".format(tor_config[0])
+        except IndexError:
+            msg = ("Tor v3 config file for '{}' ",
+                   "appears to be empty").format(hostname)
+            raise Exception(msg=msg)
+
+    return tor_v3_hostname
 
 
 def lookup_ssh_address(hostname):
@@ -122,10 +141,13 @@ def lookup_ssh_address(hostname):
     """
     ssh_address = lookup_local_ipv4_address(hostname)
     try:
-        ssh_address = lookup_tor_hostname(hostname)
+        ssh_address = lookup_tor_v3_hostname(hostname)
     # Don't assume ATHS files are present; they won't be on first run.
     except (IndexError, EnvironmentError):
-        pass
+        try:
+            ssh_address = lookup_tor_v2_hostname(hostname)
+        except (IndexError, EnvironmentError):
+            pass
 
     return ssh_address

Not pushing that commit because I haven't tested it in Tails. Take a look and holler if you have different thoughts on implementation, @zenmonkeykstop.

conorsch

In the v3-only scenario for the Admin Workstation, this step will pass:

verify that servers are accessible via ssh app and ssh mon

However I expect that a subsequent ./securedrop-admin install will fail, because of missing updates to install_files/ansible-base/inventory-dynamic. See patch in comment suggesting an approach; if you agree the changes are warranted, @zenmonkeykstop, please add a commit and holler back for follow-up review!

rmol · 2019-08-22T22:03:10Z

I was trying admin case 2 (v2 and v3 enabled), and got an error when installing the servers, at Generate Onion v3 keys if required the Tails admin system: '_X25519PrivateKey' object has no attribute 'private_bytes'.

This was at bbad1cb. I'm out of time today, but will dig into it in the morning.

conorsch · 2019-08-22T22:49:50Z

@rmol Ack, we bumped the cryptography version in #4652, so make sure to run ./securedrop-admin setup to get in sync! That should resolve the problem for you.

zenmonkeykstop · 2019-08-22T22:59:29Z

Thanks for the dynamic inventory fix @conorsch ! Running through the v3-only scenario now and adding a second ./securedrop-admin install command, which is working well. I'll update the test plan accordingly

rmol · 2019-08-23T22:03:28Z

I got through the admin test plans today, despite several apparent flakes that were resolved by rebooting the admin workstation and trying again. 🙄 The only joy I had with the journalist workstation was that the source interface was visible in the v3-only scenario (haven't reinstalled to v2+v3 yet), but /var/lib/tor/onion_auth is not getting populated, so the journalist interface is not available.

kushaldas · 2019-08-26T12:17:45Z

I got through the admin test plans today, despite several apparent flakes that were resolved by rebooting the admin workstation and trying again. roll_eyes The only joy I had with the journalist workstation was that the source interface was visible in the v3-only scenario (haven't reinstalled to v2+v3 yet), but /var/lib/tor/onion_auth is not getting populated, so the journalist interface is not available.

@rmol Can you please show us all the files available under install_files/ansible-base?

In my Tails vm:

$ sudo ls -l /var/lib/tor/onion_auth
[sudo] password for amnesia: 
total 12
-r-------- 1 debian-tor debian-tor 128 Aug 26 12:31 app-journalist.auth_private
-r-------- 1 debian-tor debian-tor 128 Aug 26 12:31 app-ssh.auth_private
-r-------- 1 debian-tor debian-tor 128 Aug 26 12:31 mon-ssh.auth_private

kushaldas

Approving once again. Works for me as it should be.

rmol · 2019-08-27T21:43:40Z

Sorry to hold this up, everyone. I ran through the journalist workstation tests fresh today and everything worked.

Testing

Admin Workstation testing

Case 1: v2 only (previous state)

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfuly
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v2 addresses
verify that ~/.ssh/config contains entries for app and mon with their v2 URLs
verify that servers are accessible via ssh app and ssh mon

Case 2: v2 and v3 enabled

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses
verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that ~/.ssh/config contains entries for app-legacy and mon-legacy with their v2 URLs
verify that servers are accessible via ssh app and ssh mon
verify that servers are accessible via ssh app-legacy and ssh mon-legacy

Case 3: v3 only

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains 3 distinct .auth_private files with the contents of the v3 files
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses
verify that ~/.ssh/config contains entries for app and mon with their v3 URLs
verify that servers are accessible via ssh app and ssh mon

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v2 addresses

Case 2: v3 only (previous state)

Test plan:

run ./securedrop-admin tailsconfig and verify that it completes successfully
Verify that /etc/tor/torrc contains a line ClientOnionAuthDir /var/lib/tor/onion_auth
verify that the /var/lib/tor/onion_auth directory was created and contains a single .auth_private file with the contents of the app-journalist.auth_private file
verify that the Journalist Interface and Source Interface are available via their desktop shortcuts at their v3 addresses

redshiftzero · 2019-08-27T21:51:38Z

thanks for testing @rmol! @conorsch are you happy with the changes here?

conorsch · 2019-08-27T22:15:39Z

Looks good to me! Approving based on visual review. We'll be banging on this logic quite a bit in the QA period...

zenmonkeykstop added this to the 1.0.0 milestone Aug 16, 2019

zenmonkeykstop force-pushed the 4629-v3-tailsconfig branch 2 times, most recently from eb554af to 23decc7 Compare August 16, 2019 21:19

zenmonkeykstop force-pushed the 4629-v3-tailsconfig branch from e47f7a1 to b0709cf Compare August 17, 2019 02:34

zenmonkeykstop marked this pull request as ready for review August 17, 2019 02:46

zenmonkeykstop requested review from conorsch, emkll, heartsucker, kushaldas, msheiny and redshiftzero as code owners August 17, 2019 02:46

kushaldas suggested changes Aug 21, 2019

View reviewed changes

zenmonkeykstop and others added 6 commits August 22, 2019 08:11

Removing SecureDrop 0.3.x cleanup and migration logic

c9ea74b

Adding v3 ClientAuth files

fd21c5f

Adding v3 SSH support

1cfe967

Adding v3 desktop shortcut support

d19bab6

Updated v3 service filenames

473b7cd

Uses the correct v2 or v3 URLS

7b302c0

Ansible overrides facts if they are reused even with when statements. We are creating two varaiables from either v2 or from v3 addresses based on what is available and then using those.

zenmonkeykstop force-pushed the 4629-v3-tailsconfig branch from 9c42b9a to 7b302c0 Compare August 22, 2019 15:12

kushaldas previously approved these changes Aug 22, 2019

View reviewed changes

Adds support for v3 Onion to SSH inventory script

bbad1cb

Preserves the v2 Onion lookup logic, but prefers v3 Onions if those are found on the Admin Workstation. Ensures that Admins connecting to the servers after migrating to v3 Onion URLs are using them for SSH.

conorsch suggested changes Aug 22, 2019

View reviewed changes

kushaldas approved these changes Aug 26, 2019

View reviewed changes

conorsch approved these changes Aug 27, 2019

View reviewed changes

conorsch merged commit 1e00822 into freedomofpress:develop Aug 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for v3 Onion urls to ./securedrop-admin tailsconfig` #4675

Adds support for v3 Onion urls to ./securedrop-admin tailsconfig` #4675

zenmonkeykstop commented Aug 16, 2019 •

edited

Loading

codecov-io commented Aug 17, 2019 •

edited

Loading

kushaldas left a comment

kushaldas left a comment

kushaldas left a comment

kushaldas commented Aug 22, 2019

conorsch commented Aug 22, 2019

conorsch left a comment

rmol commented Aug 22, 2019

conorsch commented Aug 22, 2019

zenmonkeykstop commented Aug 22, 2019 •

edited

Loading

rmol commented Aug 23, 2019

kushaldas commented Aug 26, 2019 •

edited

Loading

kushaldas left a comment

rmol commented Aug 27, 2019

redshiftzero commented Aug 27, 2019

conorsch commented Aug 27, 2019

Adds support for v3 Onion urls to ./securedrop-admin tailsconfig` #4675

Adds support for v3 Onion urls to ./securedrop-admin tailsconfig` #4675

Conversation

zenmonkeykstop commented Aug 16, 2019 • edited Loading

Status

Description of Changes

Testing

Admin Workstation testing

Case 1: v2 only (previous state)

Setup:

Test plan:

Case 2: v2 and v3 enabled

Setup:

Test plan:

Case 3: v3 only

Setup:

Test plan:

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Setup:

Test plan:

Case 2: v3 only (previous state)

Setup:

Test plan:

Deployment

Checklist

If you made changes to securedrop-admin:

If you made non-trivial code changes:

codecov-io commented Aug 17, 2019 • edited Loading

Codecov Report

kushaldas left a comment

Choose a reason for hiding this comment

For case 1

kushaldas left a comment

Choose a reason for hiding this comment

Case 2: v2 and v3 enabled

Test plan:

Case 3: v3 only

Test plan:

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Setup:

Test plan:

Case 2: v3 only (previous state)

Setup:

Test plan:

kushaldas left a comment

Choose a reason for hiding this comment

kushaldas commented Aug 22, 2019

conorsch commented Aug 22, 2019

conorsch left a comment

Choose a reason for hiding this comment

rmol commented Aug 22, 2019

conorsch commented Aug 22, 2019

zenmonkeykstop commented Aug 22, 2019 • edited Loading

rmol commented Aug 23, 2019

kushaldas commented Aug 26, 2019 • edited Loading

kushaldas left a comment

Choose a reason for hiding this comment

rmol commented Aug 27, 2019

Testing

Admin Workstation testing

Case 1: v2 only (previous state)

Test plan:

Case 2: v2 and v3 enabled

Test plan:

Case 3: v3 only

Test plan:

Journalist Workstation testing

Case 1: v2 enabled (previous state)

Test plan:

Case 2: v3 only (previous state)

Test plan:

redshiftzero commented Aug 27, 2019

conorsch commented Aug 27, 2019

zenmonkeykstop commented Aug 16, 2019 •

edited

Loading

If you made changes to `securedrop-admin`:

codecov-io commented Aug 17, 2019 •

edited

Loading

zenmonkeykstop commented Aug 22, 2019 •

edited

Loading

kushaldas commented Aug 26, 2019 •

edited

Loading