Tor Apt Repo mirror #2113

msheiny · 2017-08-15T20:45:33Z

Status

Ready for review 👍

Description of Changes

Addresses #2106 , in response to the big ol' #2105 issue of 2017.

Changes proposed in this pull request:

Switches tor apt repo to point from http://deb.torproject.org/torproject.org to http://tor-apt.ops.freedom.press. Which is a snapshot in time of the upstream tor repo.

Testing

This switches the network location of the tor packages pulled in by the securedrop servers. So the big part in testing is to ensure that:

tor doesnt break
securedrop-app deb packages correctly switch the repo to point to our mirror

Deployment

Any special considerations for deployment? Consider both:

Upgrading existing production instances. --- the securedrop-app deb package contains basic bash logic run on updates that attempts to modify the tor apt repo location. This needs to be tested more before this PR is pulled in.
New installs. --- Do the tor packages get installed and does securedrop onion services come up 👍

Checklist

If you made changes to the app code:

Unit and functional tests pass on the development VM

If you made changes to the system configuration:

Configuration tests pass

redshiftzero

A test needs updating (comment inline), otherwise 👍 from me

redshiftzero · 2017-08-16T15:48:00Z

testinfra/common/test_tor_config.py

@@ -9,7 +9,7 @@ def test_tor_apt_repo(File):
    The version of Tor in the Trusty repos is not up to date.
    """
    f = File('/etc/apt/sources.list.d/deb_torproject_org_torproject_org.list')
-    repo_regex = re.escape('deb http://deb.torproject.org/torproject.org trusty main')
+    repo_regex = re.escape('deb http://tor-apt.ops.freedom.press trusty main')
    assert f.contains(repo_regex)



test_cron_apt_repo_list in testinfra/common/test_cron_apt.py also needs updating

msheiny · 2017-08-16T20:02:06Z

rebased on develop and attempting to address broken testinfra test -- waiting for CI

msheiny · 2017-08-16T20:55:51Z

Sweet CI is passing -- the only situation I havent really tested is the upgrade path (taking the debs)... where is #1689 when we need it!!?

conorsch · 2017-08-18T04:17:12Z

install_files/securedrop-app-code/DEBIAN/postinst

+        if [ -f "/etc/apt/$file" ]; then
+            sed -i 's/deb.torproject.org\/torproject.org/tor-apt.ops.freedom.press/g' "/etc/apt/$file"
+        fi
+    done


This strategy will only update the apt config on the App Server, not on the Mon Server. Definitely it's critical that we minimize breakage for the App Server interfaces, but we need a way to land config changes across the board. Debian configuration packages come to mind: https://debathena.mit.edu/config-packages/

conorsch · 2017-08-18T04:17:36Z

testinfra/common/test_tor_config.py

@@ -10,8 +10,7 @@ def test_tor_apt_repo(File):
    The version of Tor in the Trusty repos is not up to date.
    """
    f = File('/etc/apt/sources.list.d/deb_torproject_org_torproject_org.list')
-    repo_regex = re.escape('deb http://deb.torproject.org/torproject.org '
-                           'trusty main')
+    repo_regex = re.escape('deb http://tor-apt.ops.freedom.press trusty main')


We also need to test for the absence of the old repo URL, otherwise the change is moot.

conorsch

Good first pass, but we're only touching the Application Server via these automatic updates. That doesn't seem good enough to me.

Also requesting changes as an intentional blocker until we have more robust testing on the new backend service, otherwise we're trading one potential point of failure for another.

Edited PR to implement requested changes

conorsch · 2017-09-28T00:42:29Z

Created new deb package securedrop-config to handle the repo swap. Deliberately dismissed old reviews since the diff has changed.

We'll need vigorous QA regarding upgrades to make sure the path is smooth. Reminder to @redshiftzero and @msheiny that #1689 remains unresolved.

ghost

A minor leftover and two nits, otherwise looks great :-)

ghost · 2017-10-03T14:32:05Z

install_files/securedrop-config/DEBIAN/control

+Homepage: https://securedrop.org
+Package: securedrop-config
+Version: 0.1.0+0.4.3
+Architecture: amd64


It should be all instead of amd64 because the package is not architecture dependent.

https://www.debian.org/doc/debian-policy/index.html#architecture

ghost · 2017-10-03T14:44:32Z

testinfra/common/test_tor_mirror.py

+    assert tor_gpg_pub_key_info in c.stdout
+
+
+@pytest.mark.parametrize('filename', [


this is probably a leftover of a previous implementation ?

@dachary Can you clarify what "this" refers to? The test beginning on L36 of testinfra/common/test_tor_mirror.py checks for absence of the official Tor apt repository, thereby complementing the tests that check for presence of the FPF Tor mirror.

the filename parameter is not used by the test below and it will fail testinfra

ghost · 2017-10-03T14:56:08Z

install_files/securedrop-keyring/DEBIAN/control

@@ -6,5 +6,5 @@ Homepage: https://securedrop.org
 Package: securedrop-keyring
 Version: 0.1.0+0.4.3
 Architecture: amd64
-Depends: gnupg
+Depends: gnupg,securedrop-config


securedrop-config should be a direct dependency of securedrop-ossec-* and securedrop-app-code

ghost · 2017-10-03T16:36:39Z

./testinfra/test.py app-staging locally fails with

INFO:testinfra:RUN CommandResult(command="grep -riP 'deb\\.torproject\\.org' /etc/apt*", exit_status=0, stdout=u'Binary file /etc/apt/trusted.gpg~ matches\nBinary file /etc/apt/trusted.gpg.d/deb.torproject.org-keyring.gpg~ matches\nBinary file /etc/apt/trusted.gpg.d/deb.torproject.org-keyring.gpg matches', stderr=u'')

conorsch · 2017-10-03T16:37:19Z

@dachary Addressed most of the changes you requested, but I could use some clarification on one of the comments: #2113 (comment)

redshiftzero · 2017-10-03T16:51:31Z

testinfra/common/test_tor_mirror.py

+    """
+    f = File('/etc/apt/sources.list.d/tor_apt_ops_apt_freedom_press.list')
+
+    regex = ('^deb \[arch=amd64\] http:\/\/tor-apt\.ops\.freedom\.press '


Needs updating since c4159dd

conorsch · 2017-10-03T17:13:51Z

Great review, @dachary and @redshiftzero. I'm going to rebase this on top of latest develop and clean up a bit. Will ping you both when it's ready for re-review.

This addresses issue #2106 - mitigate side-effects from a tor package breaking securedrop instances in the field. The actual logic that creates our mirror is in source control in another repo.

We don't have the mirror reflected in the config yet, but these tests describe the system state we want. Time for some TDD!

New Debian package designed for shipping unattended upgrades that affect system state (rather than simply installing new software). We're breaking some of the Debian packaging rules, specifically by touching contents inside `/etc/apt/`, but SecureDrop is not an official Debian package, and we're highly opinionated about what gets to run on SecureDrop servers, so I'm comfortable with the lintian errors for now. Creates postinst for securedrop-config package Running the sed replace actions via postinst, so that current official Tor apt repo settings are replaced with the FPF mirror. Removes postinst tor repo logic from the `securedrop-app-code` package, since that would only have affected the Application Server.

Reusing the generic-pkg role to build the new `securedrop-config` package along with the others. Sets securedrop-config as dependency of securedrop-keyring Adds local pkg vars for securedrop-keyring. Since `securedrop-keyring` depends on `securedrop-config`, the latter must be installed first, so the order of items the local packages list var is important. Runs the usual collection of tests over the new deb package, as well. Includes entry in the `update_version.sh` script so that the config package will have updates made to it, as well.

@dachary

The package isn't architecture-dependent, so don't lie about it. The SecureDrop install scripts only setup "amd64" package access via the FPF tor repository. Will shake out any inconsistencies in pre-release QA with release candidate debs. Pointed out by @dachary during review. Further reading at: https://www.debian.org/doc/debian-policy/index.html#architecture Updates associated config tests, as well.

Previously tried to piggyback on securedrop-keyring, setting the securedrop-config package as a dependency thereof. It's more explicit to list securedrop-config as a dependency of packages required for the Application and Monitor Server roles: * securedrop-app-code * securedrop-ossec-agent * securedrop-ossec-server Therefore, removed the dependency in securedrop-keyring.

The parametrization was improperly implemented, causing the test suite to fail. Rather than recursive grep across all of `/etc/apt`—which will cause false positives in the tor keyring under /etc/apt/trusted.gpg.d—let's look only at the two rationale locations, and recursively search through /etc/apt/sources.list.d (since multiple Ansible versions could have written to different paths).

conorsch · 2017-10-06T16:28:48Z

During yesterday's SecureDrop Engineering meeting, the team decided the S3 bucket mirror strategy is insufficient, given that it requires substantial manual maintenance, and creates scheduling problems for QA, via Valid-Until. @msheiny is working on more comprehension solution that involves using the SecureDrop Release Signing Key to re-sign Tor packages, for distribution to SecureDrop servers.

msheiny requested review from conorsch and redshiftzero August 15, 2017 20:45

redshiftzero previously requested changes Aug 16, 2017

View reviewed changes

msheiny force-pushed the TorAptRepoMirror branch 2 times, most recently from c81f904 to 99162fb Compare August 16, 2017 20:01

conorsch reviewed Aug 18, 2017

View reviewed changes

conorsch previously requested changes Aug 18, 2017

View reviewed changes

msheiny added this to the 0.4.4 milestone Sep 14, 2017

conorsch added the blocked label Sep 22, 2017

conorsch mentioned this pull request Sep 22, 2017

Create configuration packages for managing server settings #2362

Open

conorsch force-pushed the TorAptRepoMirror branch from 99162fb to 6007640 Compare September 28, 2017 00:08

conorsch removed the blocked label Sep 28, 2017

conorsch force-pushed the TorAptRepoMirror branch 2 times, most recently from a6883e3 to 02f83f9 Compare September 28, 2017 00:33

conorsch force-pushed the TorAptRepoMirror branch from 02f83f9 to ee664c6 Compare September 28, 2017 00:40

conorsch force-pushed the TorAptRepoMirror branch 2 times, most recently from 12f2d31 to 2d3ba28 Compare September 28, 2017 00:52

conorsch requested a review from a user October 2, 2017 17:01

ghost suggested changes Oct 3, 2017

View reviewed changes

redshiftzero reviewed Oct 3, 2017

View reviewed changes

Swap tor apt repo to point to our mirror

1d089eb

This addresses issue #2106 - mitigate side-effects from a tor package breaking securedrop instances in the field. The actual logic that creates our mirror is in source control in another repo.

msheiny and others added 7 commits October 3, 2017 10:56

Update tests to accompany tor repo mirror

200a850

Creates config tests for FPF Tor apt mirror

705879a

We don't have the mirror reflected in the config yet, but these tests describe the system state we want. Time for some TDD!

conorsch force-pushed the TorAptRepoMirror branch from 8e8fd56 to d91ff6f Compare October 3, 2017 17:57

conorsch closed this Oct 6, 2017

msheiny mentioned this pull request Oct 13, 2017

Avoid SecureDrop server identification due to apt traffic analysis #1204

Open

3 tasks

conorsch mentioned this pull request Oct 13, 2017

Uses apt mirror for Tor via securedrop-config package #2441

Merged

3 tasks

legoktm deleted the TorAptRepoMirror branch January 9, 2025 19:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tor Apt Repo mirror #2113

Tor Apt Repo mirror #2113

msheiny commented Aug 15, 2017 •

edited

Loading

redshiftzero left a comment

redshiftzero Aug 16, 2017

msheiny commented Aug 16, 2017

msheiny commented Aug 16, 2017

conorsch Aug 18, 2017

conorsch Aug 18, 2017

conorsch left a comment

conorsch commented Sep 28, 2017

ghost left a comment

ghost Oct 3, 2017

ghost Oct 3, 2017

conorsch Oct 3, 2017

ghost Oct 3, 2017

ghost Oct 3, 2017

ghost commented Oct 3, 2017

conorsch commented Oct 3, 2017

redshiftzero Oct 3, 2017

conorsch commented Oct 3, 2017

conorsch commented Oct 6, 2017

		assert tor_gpg_pub_key_info in c.stdout


		@pytest.mark.parametrize('filename', [

Tor Apt Repo mirror #2113

Tor Apt Repo mirror #2113

Conversation

msheiny commented Aug 15, 2017 • edited Loading

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the app code:

If you made changes to the system configuration:

redshiftzero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msheiny commented Aug 16, 2017

msheiny commented Aug 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conorsch left a comment

Choose a reason for hiding this comment

conorsch commented Sep 28, 2017

ghost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Oct 3, 2017

conorsch commented Oct 3, 2017

Choose a reason for hiding this comment

conorsch commented Oct 3, 2017

conorsch commented Oct 6, 2017

msheiny commented Aug 15, 2017 •

edited

Loading