-
Notifications
You must be signed in to change notification settings - Fork 687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"securedrop-admin tailsconfig" fails after updating admin workstation checkout from 0.14.0 to 1.0.0~rc2 #4762
Labels
Comments
I can reproduce this bug.
|
kushaldas
added a commit
that referenced
this issue
Sep 7, 2019
5 tasks
redshiftzero
added a commit
that referenced
this issue
Sep 9, 2019
Fixes #4762 adds missing variables for tailsconfig
redshiftzero
pushed a commit
that referenced
this issue
Sep 9, 2019
(cherry picked from commit dd174f8)
kushaldas
added a commit
that referenced
this issue
Sep 25, 2019
kaiiyer
added a commit
to kaiiyer/securedrop
that referenced
this issue
Oct 20, 2019
* Adds support for v3 Onion to SSH inventory script Preserves the v2 Onion lookup logic, but prefers v3 Onions if those are found on the Admin Workstation. Ensures that Admins connecting to the servers after migrating to v3 Onion URLs are using them for SSH. * Tweak icon alignment, spacing; fix mobile regressions * Add footer.html to Apache whitelist * update mkvirtualenv command to reflect update to python3 * Invalidate Session When Admin Resets Journalist Password * Improve deletion of submissions Replace srm with shred, which is faster, reducing the chance that deletion of a submission will be interrupted. Update rq and redis requirements, to eliminate long-standing bugs. Add manage.py tasks for detecting and correcting submissions that have been disconnected from their files on disk, and vice versa. Update manage.py to explicitly run with the production virtualenv. Also specify the virtualenv in WSGI scripts and the run-test script. In the dev/test Docker container, install requirements in a virtualenv at the same path as production. Add a supervisor script for requeuing interrupted rq jobs. If the app server is rebooted while an rq job is running, that job has already been deleted from the queue and rq will not automatically resume it on reboot, but it does have a record of it in the queue's started job registry. This script checks that registry for jobs that aren't already queued or being run, and requeues them. * Fix CI lint requirements installation * Fix mobile regression in footer layout, rename class Resolves freedomofpress#4701 * Use final footer wording Resolves freedomofpress#4700 * Adjust tooltip sizing/weight to reduce likelihood of abbreviation Resolves freedomofpress#4716 * Use config.SECUREDROP_ROOT for worker path in test_worker.py * Delete Submissions with Sources that were deleted This is accomplished by an alembic migration that removes any orphaned submissions from the database. * store: add method to find a file without its filesystem_id if a source has been deleted, we no longer have its filesystem_id in the database. as such, we will need to find the file, being careful to guard against potential duplicates due to the journalist_designations not necessarily being unique (this is a very rare case) * alembic: add replies to the migration and enqueue jobs to cleanup the files * create new exception class and handle in migration for resiliency two situations _could_ potentially arise trying to think of the edge cases here: - Admin has been deleting files manually by digging around for files that consume a lot of disk in the securedrop data directory - A very rare situation can occur where there is a colliding journalist_designation * alembic: we should also delete objects that have invalid source_ids invalid objects being those which do not have a matching source_id in the sources table * alembic: ensure row is deleted if the file is gone easiest to do with a custom exception, but for files that were manually deleted by a curious admin who e.g. removed large files that were not properly deleted due to this bug, we should also remove the corresponding row in the database. * alembic: migration contains raw SQL, add `# nosec` bandit flags this because the table name is passed in as a variable. It's not user controlled so this does not introduce a security problem (function is in the migration only). I could suppress the alert by removing the function but in the spirit of not making this alembic migration even more repetitive I'm adding `# nosec`. * alembic: guard against config.py not existing yet one of the quirks of SecureDrop is that config.py might not yet exist until the app Ansible role has ran this means for fresh installs, the database must be created via alembic upgrade head without any errors raising, else the securedrop-app-code package install will fail * deletion cleanup: srm -> secure_delete, new session_nonce * Update documentation for translators * deletion cleanup: worker.py now imports config.py so it won't import on existing installs * deletion cleanup: mv two imports that import worker.py into try/except * Fixes freedomofpress#4708, adds sdconfig prompt for v2/v3 services Now it asks if the admin wants to enable v2 or v3 services. It will not let you to disable v3 if you already disabled v2. * Adds test cases and related change for v2/v3 user input We now have two more integration tests to verify that we can have "enabled v2 and disabled v3" and both versions enabled at the same time. * Updates based on feedback of strings * Exposes Onion URLs (v2 & v3) to webapp We write plaintext files to /var/lib/securedrop so that the application can provide user-facing messaging about upcoming v2 -> v3 Onion URL transitions. For example, if a v3 Source Onion URL exists, but a source is browsing via the v2 URL, we can recommend the v3 URL instead. * Documents running subset of testinfra tests Since testinfra is based on pytest, the pytest env var for passing additional options works just fine. We can use the `-k` option to perform a substring search on test names, and run only those tests matching the selector. * Renames admin _config var for clarity There's already a `.config` attribute in the sdconfig flow, so let's rename `._config` -> `._config_in_progress` to make the intended use case explicit to future maintainers. The contents of the _config_in_progress dict are consulted during validation steps as part of `sdconfig`. * Fixes freedomofpress#4677 warns admin if v3 and https both are enabled * Use dev-shell for make translate * Add Weblate translation status widgets Weblate provides widgets showing translation progress. Add a badge to our README, and larger widgets to the Translators section of our Contributing doc. * admin: add warning message for v3 onions and HTTPS * Update Tor Debian package version to 0.4.1.5-1 * Bump Tor Browser in Dockerfile to 8.5.4 Fixes 404 on Tor Browser Download for dev env * Incorporate source string feedback from KwadroNaut * Update the path to rqworker in dev-deps supervisor config The supervisor configuration in the dev container was still pointing to /usr/local/bin/rqworker, which has been moved to /opt/venvs/securedrop-app-code/bin. Correcting the path fixes submission hashing and deletion. * Correct cleanup of disconnected submission files In securedrop.management.submissions.find_disconnected_fs_submissions, check Replies as well as Submissions for a valid association to any file found in the store, before reporting it as disconnected. * Add install-time check for JSON keyfile if client_auth files present on Admin Workstation * Updated to allow for SSH-over-LAN scenarios * Updating tailsconfig to always run network hook * Update Python script interpreter directives For Python scripts run where the /opt/venvs/securedrop-app-code virtualenv is used, specify that in the interpreter directive line. Remove the shebang line from crypto_util.py, since it's never run. Leave unmodified the scripts used on workstations or in packaging VMs. * Teach update_version.sh to bump the expected sdist tarball version * ci: bump staging timeout to 35m * Remove custom_logo.png from app code build logic rsync filter rules are processed on a first-match basis, which means the exclusion rule for custom_logo.png must appear before the wildcard inclusion of all png files. * Update release management docs to ensure repository is clean prior to building packages. * build: fetch sources only during securedrop-app-code build * Make build-debs more verbose For securedrop-app-code, give pip the --verbose option. In build-debs.sh, give molecule the --debug option if the environment variable SD_DEBUG_BUILD_DEBS is "yes". * Fixes freedomofpress#4762 adds missing variables for tailsconfig * docs: fix typos severs -> servers that so that -> so that * Add instructions on updating Tails USBS. Refer to updates instead of upgrades. Clarify generic Tails manual update process. Add warning to Tails Setup docs on verifying iso image. Additional changes in wording per pr review. * Default v3_onion_services to off in validate_tails_environment.yml The lack of this variable has tripped us up in several roles, and depending on how the validate role is used, adding it to defaults/main.yml within that role might not fix it, so let's add a fact in the validate_tails_environment playbook defaulting it to off. * Internal links on Tails backup process * Fix typo naming Tails instead of Ubuntu * update torbrowser version * Update screenshots in docs for 1.0.0 Also update source guide to reflect button label changes. Cropped the Tor download dialogs to not show the background, so they have a longer shelf life. Resolves freedomofpress#4721 * Update Tor restart dialog screenshot * Remove v3 onion secrets when transitioning to ssh over local - v2 aths secret file pattern does not match the one for v3 - Use async task for reboot for more immediate feedback - Instruct admin to run tailsconfig to update the ssh config on tails workstation * updated documentation to include references to v3 services * Expand .gitgnore for venv The wilcard did not match `.venv/` folder used for python2 virtualenv in the securedrop-admin context * Rework Ansible restore role to detect Onion service version mismatch Make the restore playbook extract the Tor configuration from the backup, retrieve the app server's Tor configuration, and compare them. If the service versions differ, abort the restore with an explanation. * Instruct localization manager to squash translation commits * Overhaul Transfer Device and export recommendations Resolves freedomofpress#4620 Resolves freedomofpress#4646 Resolves freedomofpress#4434 Resolves freedomofpress#4670 In addition to introducing the Transfer and Export Device, this commit clearly breaks out optional hardware into its own checklist, more strongly recommends purchase of a printer, and strengthens recommendations for malware mitigation. It removes some outdated recommendations and a reference to storing the journalist's GPG passphrase in KeePassX (not mentioned anywhere else). It removes instructions for journalists to set up individual GPG keys, as they are unlikely to be followed, and the more critical recommendation is to ensure tha the Export Device is encrypted. It updates the overview diagram and data flow diagram consistent with the above changes. This update also makes the diagrams more consistent with terminology and current practices. It removes OnionShare from the data flow diagram, as it is not currently mentioned elsewhere in the docs. * Rename files for consistency with new organization/content * Improve loading behavior of language selector in Tor Specifying a minimum width ensures the widget is always rendered as it should be. Fixes freedomofpress#4810 * added setup command to pull in dependencies for v3 key generation * Update Tails desktop icons, updater banner, updater icon These were still using the old logo. Note that resources_rc.py needed to be regenerated for these updates to be reflected in the updater UI. * updates based on PR review * Fix journalist test_delete_one JournalistNavigationStepsMixin's _journalist_delete_one is more reliable with an ActionChains move/click incantation, but switching to that in the functional test safe_click* methods breaks a bunch of other tests, so I'm just rolling with the Selenium voodoo. * Update logo in README `docs/images/logo.png` is used for no other purpose, so deleting. * Add documentation of submission cleanup * Update builder hash Builder image needed security updates * l10n: updated Arabic (ar) contributors: Thalia Rahme updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Catalan (ca) contributors: Benet (BennyBeat) R. i Camps John Smith updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated German (de_DE) contributors: Ettore Atalan John Hensley Robin Schubert updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Greek (el) contributors: Dimitris Maroulidis updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Spanish (es_ES) contributors: Adolfo Jayme-Barrientos carlos erinm updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated French (fr_FR) contributors: AO updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Hindi (hi) contributors: AbhayKaushik Chandan Kumar (raukadah) Drashti updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Icelandic (is) contributors: Oktavia Sveinn í Felli updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Italian (it_IT) contributors: Beatrice Martini Claudio Arseni Giandomenico Lombardi updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Norwegian (nb_NO) contributors: John Hensley Øyvind Bye Skille updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Dutch (nl) contributors: Pander Thom kwadronaut updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Portuguese, Brasil (pt_BR) contributors: communiaa updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Romanian (ro) contributors: mickael e updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Russian (ru) contributors: Adham Kurbanov erinm updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Swedish (sv) contributors: Jonas Franzén updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Turkish (tr) contributors: Kaya Zeren tekrei updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * l10n: updated Chinese, Traditional (zh_Hant) contributors: Chi-Hsun Tsai updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 75d090c * Adds ./ to manage.py invocations in admin docs Same as we do elsewhere, in codeblocks, let's explicitly prefix the "manage.py" executable with "./", since Admins will need to type that invocation exactly in order for the subcommand to work. * Add 0.14.0 to 1.0.0 upgrade guide, remove oldest one, DRY up docs a bit * Update builder hash Builder needed security updates * Clearer hardware prioritization, updated threat model The hardware recommendations more clearly prioritize choices we can support and are familiar with (VeraCrypt, USB w/ write switch) over ones we cannot currently support well (hardware-encrypted drives, write blockers). The threat model has been updated consistent with the data flow updates; note that the clarified data flow means that the Transfer Device never stores _decrypted_ submissions. The use of write blockers for the Transfer Device has been clarified, as well. * added StartupNotify=true to desktop shortcuts * Add flaky to translation-test * SecureDrop 1.0.0~rc1 (cherry picked from commit 4b4e557) * SecureDrop 1.0.0-rc2 (cherry picked from commit 09bfa45) * SecureDrop 1.0.0-rc3 (cherry picked from commit 874d48c) * Updates to 1.0.0~rc4 (cherry picked from commit 392a6e7) * Fix PR # for orphaned submission entry in changelog (cherry picked from commit 7291100) * Release 1.0.0 (cherry picked from commit 162288f) * Updates to 1.1.0~rc1 * added HTTPS EV cert note * Stronger warning against using `manage.py reset` * docs: improvements to RM guide after 1.0.0 release * Move grsec role before common role The in securedrop-staging.yml, the grsec role is bring run before the common role. [1] : https://github.com/freedomofpress/securedrop/blob/9f32d46136861546af556d3f54d92e31cebe2eb0/install_files/ansible-base/securedrop-staging.yml * Fixes v3 keypair generation in upgrade box prep During 1.0, we did not test the v3 keypair generation in the upgrade scenario. Makes sense: we only run that scenario *after* a release is final. The git-repo-root logic assumed we were in Tails or running against normal staging VMs. In the vagrant-package logic (which builds "upgrade" boxes for testing), the proper path structure requires that filepaths be relative to the secondary git repo, cloned in order to prepare an old version from the known-good tag. This var override ensures that the v3 keys are stored inside the secondary git repo. We could instead update the rev-parse logic to be relative to the role, but that'd require additional testing in Tails, and this works just as well. * Add upgrade boxes for SecureDrop 1.0.0 * updated Tor version in python2 dockerfile * workaround for Tails detection in securedrop-admin setup * added check for Tails in /etc/os-release in validate role * added Tails 4.0 check in inventory-dynamic * Update upgrade playbook for v3 onion services Use v2 and v3 onion services * reordered manage.py args in parser to put common operations first and dev-only operations last * ci: remove python 2 CI steps/jobs * dev: remove python 2 support from dev env recall that we have a separate dev env for the securedrop-admin CLI tool in admin/ * app: remove python 2 support * bandit: add exception for input() since it's safe in Python 3 * code owners: @msheiny has been offboarded from team SD 😢 * docs: update mac dev and type annotation instructions for python 3 * securedrop-app-code: remove python 2 reference in comment (we're using python 3.5) * Update the app test instructions It's now just "make test", instead of "make -C securedrop test". * Bump wekzeug to 0.16.0 Due to CVE-2019-14806: Pallets Werkzeug before 0.15.3, when used with Docker, has insufficient debugger PIN randomness because Docker containers share the same machine id. * Bump ansible to 2.6.18 Addresses CVE-2019-10156. Given that SecureDrop would not process untrusted variables in templates, this vulnerability did not impact the system. Also updates SecureDrop installer's ansible version check * Bump molecule to 2.22 This is required to upgrade pyyaml to 5.1 (resolving CVE-2017-18342) and jinja2 (resolving CVE-2019-10906) * Don't sudo pip install in container for linting sudo pip installing (even in the virtualenv) would cause permission issues for the molecule tmp files at runtime * Update Pyyaml in admin requirements Resolves CVE-2017-18342 * Add dependency diff review to pull request template * Fix tests for new version of testinfra * Pin hashes for dev requirements Also updates docs with correct python3 requirements path * Bump ansible to 2.6.19 Resolves CVE-2019-10206 * Apply mix fixes in install_files/securedrop-app-code/debian/copyright * Also updates Upstream-Name These "securedrop-proxy" strings were leftover from packaging logic updates in freedomofpress#4870. Hat tip to @rmol for catching this one during review. * Reload extension to refresh desktop icons in Tails 4 * Fix scaling of diagrams in docs Resolves freedomofpress#4873 * Improve specification of securedrop-app-code dependencies The securedrop-app-code Debian package didn't fully specify its dependencies, crucially libpython3.5, which is almost always installed on SecureDrop servers built according to our instructions, but doesn't have to be. This fixes that, and makes the securedrop-app-code package the sole source of truth about its dependencies, eliminating the out-of-band installation of some of them via the "app" Ansible role. This also fixes a few dependency specifications in the instructions for setting up a development environment. * Clarifications on diff review in PR template During review of freedomofpress#4865, I misinterpreted the new checkboxes as a sequential list, rather than an either/or choice. Adding a small clarification to prevent future misunderstandings. =) * Moves securedrop-admin to Python3 This will inform the admins to rerun ./securedrop-admin setup as this will create .venv3 dirctory for Python3 dependencies. * Works for both Tor4 and Tor3 As Tor4 has Python 3.7.3, we should just mention without any particular version. * Python3 compat for Admin tooling A smattering of the familiar Python 2 -> 3 changes, mostly around string handling, filepaths to interpreters and virtualenvs, and syntax updates. Specifically: * coerces major release number to int * converts iteritems() -> items() in dynamic inventory * updates string handling in dynamic inventory * ensures securedrop_init runs under python3 Tested under Tails 3.16 (based on Stretch), running Python 3.5. Worked well, after these fixes landed, although I've observed a handful of the "weakref" stderr output. The weakref output doesn't break anything, but it may be alarming to Admins who haven't updated to Tails 4 & Python 3.7 yet. * added check for admin password in GUI updater * Update Ansible to 2.7.13 * Resolve Ansible 2.7 deprecation warnings for apt Several instances of the warning below: [DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ unused_packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. * l10n: updated Czech (cs) contributors: 1000101 Michal Stanke darmozrac michaela-bot updated from: repo: https://github.com/freedomofpress/securedrop-i18n revision: 570f297 * Add Czech to list of supported languages Also update the translation-test script so it gets the list of supported locales from i18n_tool.py and will no longer have to be updated when we change that list. * Edit README section about documentation Edits for readability and concision. Also remove last of language related to RTD migration. * Add pip, setuptools and supervisor to develop requirements These requirements were installed via pip in the Dev/Admin Docker containers. We should instead add them to the develop-requirements file and pin hashes. The `--allow-unsafe` flags should not impact security: it is to allow pip to pin setuptools and pip-tools (see issue freedomofpress#4686). * Pin hashes for test-requirements.txt Also adds --allow-unsafe to allow pinning hashes of setuptools in test-requirements.txt The `--allow-unsafe` flags should not impact security: it is to allow pip to pin setuptools and pip-tools (see issue freedomofpress#4686). * Bump tor to 0.4.1.6 * docs: update the list of supported languages * Fix column/button alignment problems on source start page Certain combinations of language and viewport size could result in the two columns on the source start page not lining up with each other. This changes the columns' justify-content property to push the top and bottom containers apart, and controls the borders and content of the buttons to align them better. * Fix link to Tails update guide The page moved to a new URL without a redirect * Bump app tests timeout to 15 minutes (See freedomofpress#4691) * Reduce the space between index row content and buttons * Fix securedrop-admin crash when no operation given under Python 3 Under Python 3, running securedrop-admin with no positional argument results in an ugly error, due to https://bugs.python.org/issue16308. Under Python 3.7, we could simply add dest and required arguments to the add_subparsers call, but required isn't available in Python 3.5, so would break under Tails 3. * Remove Tails 3.x admin virtualenv when upgrading to Tails 4.x Fixes freedomofpress#4910 * Add tests for tails 3 admin venv removal in tails 4 * Check that securedrop-admin has a good virtualenv Adds a "checkenv" argument to admin/bootstrap.py, used in securedrop-admin to ensure that the admin virtualenv is functional. Also make sure bootstrap.py runs under Python 3. * Fix pipenv docs URLs (HTTPS on redirect is broken) * Restore dropped virtualenv activation in securedrop-admin * Fix Tails detection in Python 3 + Tails 3.x The final comparison failed to account for the fact that the subprocess module returns a byte literal. Incorrect Tails detection caused torify not to be used, which caused a network error during the creation of the virtualenv on Tails 3.x. Decoding the string first reduces the likelihood of such coding errors. Fixes freedomofpress#4925 * Add 1.0.0->1.1.0 upgrade guide Includes instructions to rebuild the venv after the Tails 4 upgrade, so securedrop-admin and the graphical updater continue to work with Python 3. Removed older upgrade guides along the way per standard procedure. * Delete newly created virtualenv if creation fails Fixes freedomofpress#4929: If a virtualenv is created, but a transient network error prevents core dependencies to be installed in this virtualenv (for example, pip3), we should delete the virtualenv to prevent subsequent `./securedrop-admin setup` runs from failing. * Add "Troubleshooting securedrop-admin" section * Add test to ensure envsetup deletes venv if exception is raise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Running
securedrop-admin tailsconfig
after updating the admin workstation repo from 0.14.0 to 1.0.0~rc2 will fail with an error frominstall_files/ansible-base/roles/validate/tasks/validate_tails_environment.yml
, when it checks v3 SSH files inConfirm that a valid set of SSH auth files is present
andv3_onion_services
is undefined.Steps to Reproduce
On an admin workstation, with a
site-specific
file that does not contain the new onion service variablesv2_onion_services
andv3_onion_services
, run:cd ~/Persistent/securedrop
git checkout -b release/1.0.0 origin/release/1.0.0
./securedrop-admin tailsconfig
Expected Behavior
That
securedrop-admin tailsconfig
should run normally.Actual Behavior
An error is thrown at the
Confirm that a valid set of SSH auth files is present
task.Comments
Copying the
v2_onion_services
andv3_onion_services
variable initialization from e.g.roles/app/defaults/main.yml
intoroles/tails-config/defaults/main.yml
will solve this.Running
securedrop-admin sdconfig
beforetailsconfig
, to add the new variables togroup_vars/all/site-specific
, would fix it as well, but that hasn't been part of our previous upgrade instructions, and isn't otherwise necessary, unless the site is planning on adding v3 Onion service support.The text was updated successfully, but these errors were encountered: