Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Ubuntu 20.04 (Focal) #4768

Closed
47 of 53 tasks
redshiftzero opened this issue Sep 9, 2019 · 25 comments
Closed
47 of 53 tasks

Support for Ubuntu 20.04 (Focal) #4768

redshiftzero opened this issue Sep 9, 2019 · 25 comments
Labels
epic Meta issue tracking child issues

Comments

@redshiftzero
Copy link
Contributor

redshiftzero commented Sep 9, 2019

Description

Ubuntu 16.04 (Xenial) is the Ubuntu LTS release that is currently supported for SecureDrop and it will be supported until April 2021.

This issue tracks support for Ubuntu 20.04 (supported until April 2025).

Subtickets

Completed

CI work

Post migration

Deferring for next OS upgrade or other major release

Pending discussion

@eloquence 2020-08-06: Issue description refactored.
@eloquence 2020-09-03: Refactored per sprint planning
@kushaldas / @emkll 2020-09-17: Refactored issues based on Sprint 59 work
@eloquence 2020-10-08: Edited per tech meeting discussions
@eloquence 2020-11-23: Updated per discussion w/ @emkll and @conorsch
@eloquence 2020-12-02: Updated per discussion w/ @emkll and @conorsch
@eloquence 2021-02-25: Updated per 1.8.0 progress

@redshiftzero redshiftzero added the epic Meta issue tracking child issues label Sep 9, 2019
@redshiftzero redshiftzero added this to the Long Term Product Backlog milestone Sep 9, 2019
This was referenced Sep 9, 2019
@zenmonkeykstop
Copy link
Contributor

Moving to Bionic for internal tooling also unlocks goodness like an up-to-date version of dh_virtualenv, with support for explicitly-set build package versions.

@eloquence
Copy link
Member

Perhaps we should consider enforcing v3 as part of the Bionic switch. We'll have to do a lot of outreach related to this upgrade, and having admins change the .onion address as part of an upgrade or reinstall seems like a good way to get everyone over the hump. This would also give us a clear date we can communicate to the Tor team by which we know that v2 is no longer supported for SecureDrop, which they could use for end-of-life planning (i.e. April 30, 2021, the EOL date for Ubuntu Xenial).

@eloquence
Copy link
Member

eloquence commented Jun 4, 2020

Recap of discussions today:

  • Potential upgrade strategies, roughly in decreasing order of desirability:
    • unattended upgrade to 18.04 or 20.04
    • attended upgrade to 18.04, then 20.04
    • reinstall on 20.04 (high effort for admins, though could still be least effort and least error-prone if other upgrade paths are very dicey)
    • attended upgrade to 18.04 (only buys us a couple of years)
  • To assess feasibility of these strategies, we'll do exploratory spikes in future sprints, both for fresh installs and upgrades
  • In terms of aligning this with v2 deprecation, there's broad team interest in deprecating/removing v2 support soon, with some team members advocating removal even before the Xenial EOL. [*]
  • There's interest in addressing the longstanding cron-apt update logic only allows addition of apt dependencies from /etc/apt/security.list  #3376 as part of this upgrade.
  • There's interest in transitioning from cron-apt to unattended-upgrades as part of this upgrade. This is tracked as Use unattended-upgrades instead of cron-apt #5295.

We're not planning to transition to a different server OS this time around, but if we can get all the way to 20.04, this will buy us some headroom to have that conversation if/when we focus on making major server-side/admin improvements.

I will update this issue's description and start organizing into an epic before the next sprint.

[*] My view: We should remove v2 support by April 30, 2021, and issue prominent deprecation warnings well before then. This will IMO simplify messaging to admins of v2 instances. That said, we all agreed that the final timetable should be informed by the rate of v3 adoption in the real world.

@eloquence
Copy link
Member

We have a longstanding PR to reduce the retention of Apache logs to 30 days (#3714). This is a candidate for rolling into the OS upgrade release, since running the upgrade will require a playbook run anyway, so we can avoid using postinst to update existing configurations.

@eloquence eloquence changed the title Support for Ubuntu 18.04 Support for Ubuntu 18.04 (Bionic) or 20.04 (Focal) Jun 23, 2020
@eloquence eloquence removed this from the Long Term Product Backlog milestone Jun 24, 2020
@eloquence
Copy link
Member

RESEARCH SPIKE: upgrade a prod install via do-release-upgrade to Ubuntu 18.04, try to unbreak, see where you get stuck, document

@zenmonkeykstop will take a stab at this one during the 6/24-7/8 sprint, time-boxed to 8 hours.

RESEARCH SPIKE: attempt to package the SecureDrop app on Ubuntu 20.04

@kushaldas will take this one, also time-boxed to 8 hours.

@kushaldas
Copy link
Contributor

NOTE: On going work.

The goal

We should be able to build SecureDrop packages using make build-debs command for Ubuntu Focal, that is 20.04.

Update Makefile/devops script

The make build-debs uses /devops/scripts/build-debs.sh script, I modified the line 13 to be using -focal from xenial.

TARGET_PLATFORM="${2:-focal}"

This means, we should have a molecule scenario, I copied builder-xenial to builder-focal first. And then replaced
xenial to focal like s/xenial/focal in the following files.

✦ ❯ rg focal
Makefile
2:BUILDER_IMAGE ?= "quay.io/freedomofpress/sd-docker-builder-focal:$(DATE_STR)"

molecule.yml
7:  - name: focal-sd-app
10:  - name: focal-sd-generic-ossec-agent
13:  - name: focal-sd-generic-ossec-server
16:  - name: focal-sd-generic-ossec-agent2
19:  - name: focal-sd-generic-ossec-server2
22:  - name: focal-sd-grsec
25:  - name: focal-sd-config
28:  - name: focal-sd-keyring
31:  - name: focal-sd-sec-update
34:  - name: focal-sd-dpkg-verification
58:  name: builder-focal
76:    SECUREDROP_TARGET_PLATFORM: focal

create.yml
12:    default_image: "quay.io/freedomofpress/sd-docker-builder-focal@sha256:{{image_hash}}"

Dockerfile
3:FROM ubuntu:focal
8:LABEL image_name="focal-sd-builder-app"

Major point to remember, I need sd-docker-builder-focal container image to build the actual packages.

Building the container

docker build -t sd-docker-build-focal:latest . is the command to build the
container. But, it showed the dh-virtualenv package is missing. It seems the
package was last built for Ubuntu
19.10
as part of the
Universe packages. So, the next target is to build that package first for
focal.

In the same focal container image I installed all the other dependencies, and
the downloaded the sources using dget http://archive.ubuntu.com/ubuntu/pool/universe/d/dh-virtualenv/dh-virtualenv_1.1-1.dsc, and then dpkg -x filename.dsc. Then I changed all dependencies to their equivalent python3
packages. Then a rebuild using dpkg-buildpackage -uc -us command. I moved the
whole packaging directory out of the container so that we can reuse.

Next, I modified the Dockerfile to add the locally built dh-virtualenv
package, and then finally the docker build command worked.

Updating the SecureDrop package related files

diff --git a/install_files/securedrop-app-code/debian/securedrop-app-code.triggers b/install_files/securedrop-app-code/debian/securedrop-app-code.triggers
index d12cc4e54..8962f21aa 100644
--- a/install_files/securedrop-app-code/debian/securedrop-app-code.triggers
+++ b/install_files/securedrop-app-code/debian/securedrop-app-code.triggers
@@ -1,7 +1,7 @@
 # Register interest in Python interpreter changes; and
 # don't make the Python package dependent on the virtualenv package
 # processing (noawait)
-interest-noawait /usr/bin/python3.5
+interest-noawait /usr/bin/python3.8
 
 # Also provide a symbolic trigger for all dh-virtualenv packages
 interest dh-virtualenv-interpreter-update

Now, CFFI python source fails to compile with errors like following.

 c/_cffi_backend.c: In function ‘b_callback’:", "    c/_cffi_backend.c:5911:5: warning: ‘ffi_prep_closure’ is deprecated: use ffi_prep_closure_loc instead [-Wdeprecated-declarations]", "     5911 |     if (ffi_prep_closure(closure, &cif_descr->cif,", "          |     ^~", "    In file included from c/_cffi_backend.c:15:", "    /usr/include/x86_64-linux-gnu/ffi.h:334:1: note: declared here", "      334 | ffi_prep_closure (ffi_closure*,", "          | ^~~~~~~~~~~~~~~~", "    In file included from c/cffi1_module.c:20,", "                     from c/_cffi_backend.c:7370:", "    c/call_python.c: In function ‘_get_interpstate_dict’:", "    c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState’ {aka ‘struct _is’}", "       20 |     builtins = tstate->interp->builtins;", "          |                              ^~", "    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1",

Updates cffi to 1.14.0 and argon2-cffi to 20.1.0

Next error: dh_virtualenv: not found\nmake[1]:.

~# dpkg -L dh-virtualenv
/.
/usr
/usr/share
/usr/share/debhelper
/usr/share/debhelper/autoscripts
/usr/share/debhelper/autoscripts/postinst-dh-virtualenv
/usr/share/doc
/usr/share/doc/dh-virtualenv
/usr/share/doc/dh-virtualenv/changelog.Debian.gz
/usr/share/doc/dh-virtualenv/copyright
/usr/share/doc/dh-virtualenv/html
/usr/share/doc/dh-virtualenv/html/_modules
/usr/share/doc/dh-virtualenv/html/_modules/dh_virtualenv
/usr/share/doc/dh-virtualenv/html/_modules/dh_virtualenv/cmdline.html
/usr/share/doc/dh-virtualenv/html/_modules/dh_virtualenv/deployment.html
/usr/share/doc/dh-virtualenv/html/_modules/index.html
/usr/share/doc/dh-virtualenv/html/_sources
/usr/share/doc/dh-virtualenv/html/_sources/api
/usr/share/doc/dh-virtualenv/html/_sources/api/dh_virtualenv.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/changes.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/examples.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/howtos.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/index.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/source.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/trouble-shooting.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/tutorial.rst.txt
/usr/share/doc/dh-virtualenv/html/_sources/usage.rst.txt
/usr/share/doc/dh-virtualenv/html/_static
/usr/share/doc/dh-virtualenv/html/_static/basic.css
/usr/share/doc/dh-virtualenv/html/_static/css
/usr/share/doc/dh-virtualenv/html/_static/css/custom.css
/usr/share/doc/dh-virtualenv/html/_static/documentation_options.js
/usr/share/doc/dh-virtualenv/html/_static/file.png
/usr/share/doc/dh-virtualenv/html/_static/fonts
/usr/share/doc/dh-virtualenv/html/_static/img
/usr/share/doc/dh-virtualenv/html/_static/img/logo-180px.png
/usr/share/doc/dh-virtualenv/html/_static/img/logo.svg
/usr/share/doc/dh-virtualenv/html/_static/js
/usr/share/doc/dh-virtualenv/html/_static/language_data.js
/usr/share/doc/dh-virtualenv/html/_static/minus.png
/usr/share/doc/dh-virtualenv/html/_static/plus.png
/usr/share/doc/dh-virtualenv/html/_static/pygments.css
/usr/share/doc/dh-virtualenv/html/api
/usr/share/doc/dh-virtualenv/html/api/dh_virtualenv.html
/usr/share/doc/dh-virtualenv/html/changes.html
/usr/share/doc/dh-virtualenv/html/examples.html
/usr/share/doc/dh-virtualenv/html/genindex.html
/usr/share/doc/dh-virtualenv/html/howtos.html
/usr/share/doc/dh-virtualenv/html/index.html
/usr/share/doc/dh-virtualenv/html/objects.inv
/usr/share/doc/dh-virtualenv/html/py-modindex.html
/usr/share/doc/dh-virtualenv/html/search.html
/usr/share/doc/dh-virtualenv/html/searchindex.js
/usr/share/doc/dh-virtualenv/html/source.html
/usr/share/doc/dh-virtualenv/html/trouble-shooting.html
/usr/share/doc/dh-virtualenv/html/tutorial.html
/usr/share/doc/dh-virtualenv/html/usage.html
/usr/share/doc-base
/usr/share/doc-base/dh-virtualenv
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/dh_virtualenv.1.gz
/usr/share/perl5
/usr/share/perl5/Debian
/usr/share/perl5/Debian/Debhelper
/usr/share/perl5/Debian/Debhelper/Buildsystem
/usr/share/perl5/Debian/Debhelper/Buildsystem/dh_virtualenv.pm
/usr/share/perl5/Debian/Debhelper/Sequence
/usr/share/perl5/Debian/Debhelper/Sequence/python_virtualenv.pm
/usr/share/doc/dh-virtualenv/html/_static/css/badge_only.css
/usr/share/doc/dh-virtualenv/html/_static/css/theme.css
/usr/share/doc/dh-virtualenv/html/_static/doctools.js
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Bold.ttf
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Bold.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-BoldItalic.ttf
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-BoldItalic.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Italic.ttf
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Italic.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Regular.ttf
/usr/share/doc/dh-virtualenv/html/_static/fonts/Lato-Regular.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/RobotoSlab-Bold.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/RobotoSlab-Regular.woff2
/usr/share/doc/dh-virtualenv/html/_static/fonts/fontawesome-webfont.eot
/usr/share/doc/dh-virtualenv/html/_static/fonts/fontawesome-webfont.svg
/usr/share/doc/dh-virtualenv/html/_static/fonts/fontawesome-webfont.ttf
/usr/share/doc/dh-virtualenv/html/_static/fonts/fontawesome-webfont.woff
/usr/share/doc/dh-virtualenv/html/_static/fonts/fontawesome-webfont.woff2
/usr/share/doc/dh-virtualenv/html/_static/jquery.js
/usr/share/doc/dh-virtualenv/html/_static/js/modernizr.min.js
/usr/share/doc/dh-virtualenv/html/_static/js/theme.js
/usr/share/doc/dh-virtualenv/html/_static/searchtools.js
/usr/share/doc/dh-virtualenv/html/_static/underscore.js

So, let me try to repackage once again.

dpkg-source -x dh-virtualenv_1.1-1.dsc 
dpkg-source: warning: extracting unsigned source package (dh-virtualenv_1.1-1.dsc)
dpkg-source: info: extracting dh-virtualenv in dh-virtualenv-1.1
dpkg-source: info: unpacking dh-virtualenv_1.1.orig.tar.gz
dpkg-source: info: unpacking dh-virtualenv_1.1-1.debian.tar.xz

root@2f7a74b349e9:~/packaging/dh-virtualenv-1.1# sudo apt install python2-minimal
dh_auto_build: warning: Please use the third-party "pybuild" build system instead of python-distutils
dh_auto_build: warning: This feature will be removed in compat 12.
pyversions: missing X(S)-Python-Version in control file, fall back to debian/pyversions
pyversions: missing debian/pyversions file, fall back to supported versions

dpkg-source -x dh-virtualenv_1.1-1.dsc
dpkg-source: warning: extracting unsigned source package (dh-virtualenv_1.1-1.dsc)
dpkg-source: info: extracting dh-virtualenv in dh-virtualenv-1.1
dpkg-source: info: unpacking dh-virtualenv_1.1.orig.tar.gz
dpkg-source: info: unpacking dh-virtualenv_1.1-1.debian.tar.xz fatal: [focal-sd-app]: FAILED! => {"changed": true, "cmd": ["dpkg-buildpackage", "-us", "-uc"], "delta": "0:00:02.805439", "end": "2020-06-26 10:44:03.571255", "msg": "non-zero return code", "rc": 2, "start": "2020-06-26 10:44:00.765816", "stderr": " dpkg-source --before-build .\n debian/rules clean\n dpkg-source -b .\ndpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)\ndpkg-source: warning: source directory 'securedrop-app-code_1.5.0rc1+xenial_amd64' is not - 'securedrop-app-code-1.5.0rc1+xenial'\n debian/rules build\ndh_auto_configure: warning: Please use the third-party "pybuild" build system instead of python-distutils\ndh_auto_configure: warning: This feature will be removed in compat 12.\n debian/rules binary\nTraceback (most recent call last):\n File "/usr/bin/dh_virtualenv", line 29, in \n from debpython.debhelper import DebHelper\nModuleNotFoundError: No module named 'debpython'\nmake[1]: *** [debian/rules:28: override_dh_virtualenv] Error 1\nmake: *** [debian/rules:8: binary] Error 2\ndpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2", "stderr_lines": [" dpkg-source --before-build .", " debian/rules clean", " dpkg-source -b .", "dpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)", "dpkg-source: warning: source directory 'securedrop-app-code_1.5.0rc1+xenial_amd64' is not - 'securedrop-app-code-1.5.0rc1+xenial'", " debian/rules build", "dh_auto_configure: warning: Please use the third-party "pybuild" build system instead of python-distutils", "dh_auto_configure: warning: This feature will be removed in compat 12.", " debian/rules binary", "Traceback (most recent call last):", " File "/usr/bin/dh_virtualenv", line 29, in ", " from debpython.debhelper import DebHelper", "ModuleNotFoundError: No module named 'debpython'", "make[1]: *** [debian/rules:28: override_dh_virtualenv] Error 1", "make: *** [debian/rules:8: binary] Error 2", "dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2"], "stdout": "dpkg-buildpackage: info: source package securedrop-app-code\ndpkg-buildpackage: info: source version 1.5.0rc1+xenial\ndpkg-buildpackage: info: source distribution xenial\ndpkg-buildpackage: info: source changed by SecureDrop Team [email protected]\ndpkg-buildpackage: info: host architecture amd64\ndh clean --with python-virtualenv --with systemd\n dh_clean\ndpkg-source: info: using source format '1.0'\ndpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0rc1+xenial.tar.gz\ndpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0rc1+xenial.dsc\ndh build --with python-virtualenv --with systemd\n dh_update_autotools_config\n dh_auto_configure\ndh binary --with python-virtualenv --with systemd\n dh_testroot\n dh_prep\n dh_install\n dh_installdocs\n dh_installchangelogs\n dh_installdebconf\n debian/rules override_dh_virtualenv\nmake[1]: Entering directory '/tmp/securedrop-app-code_1.5.0rc1+xenial_amd64'\ndh_virtualenv \\n\t--python=/usr/bin/python3.5 \\n\t--setuptools \\n\t--extra-pip-arg "--verbose" \\n\t--extra-pip-arg "--ignore-installed" \\n\t--extra-pip-arg "--no-deps" \\n\t--extra-pip-arg "--no-binary=:all:" \\n\t--extra-pip-arg "--no-cache-dir"\nmake[1]: Leaving directory '/tmp/securedrop-app-code_1.5.0rc1+xenial_amd64'", "stdout_lines": ["dpkg-buildpackage: info: source package securedrop-app-code", "dpkg-buildpackage: info: source version 1.5.0rc1+xenial", "dpkg-buildpackage: info: source distribution xenial", "dpkg-buildpackage: info: source changed by SecureDrop Team [email protected]", "dpkg-buildpackage: info: host architecture amd64", "dh clean --with python-virtualenv --with systemd", " dh_clean", "dpkg-source: info: using source format '1.0'", "dpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0rc1+xenial.tar.gz", "dpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0rc1+xenial.dsc", "dh build --with python-virtualenv --with systemd", " dh_update_autotools_config", " dh_auto_configure", "dh binary --with python-virtualenv --with systemd", " dh_testroot", " dh_prep", " dh_install", " dh_installdocs", " dh_installchangelogs", " dh_installdebconf", " debian/rules override_dh_virtualenv", "make[1]: Entering directory '/tmp/securedrop-app-code_1.5.0rc1+xenial_amd64'", "dh_virtualenv \", "\t--python=/usr/bin/python3.5 \", "\t--setuptools \", "\t--extra-pip-arg "--verbose" \", "\t--extra-pip-arg "--ignore-installed" \", "\t--extra-pip-arg "--no-deps" \", "\t--extra-pip-arg "--no-binary=:all:" \", "\t--extra-pip-arg "--no-cache-dir"", "make[1]: Leaving directory '/tmp/securedrop-app-code_1.5.0rc1+xenial_amd64'"]}


 fatal: [focal-sd-app]: FAILED! => {"changed": true, "cmd": ["dpkg-buildpackage", "-us", "-uc"], "delta": "0:00:02.805439", "end": "2020-06-26 10:44:03.571255", "msg": "non-zero return code", "rc": 2, "start": "2020-06-26 10:44:00.765816", "stderr": " dpkg-source --before-build .\n debian/rules clean\n dpkg-source -b .\ndpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)\ndpkg-source: warning: source directory 'securedrop-app-code_1.5.0~rc1+xenial_amd64' is not <sourcepackage>-<upstreamversion> 'securedrop-app-code-1.5.0~rc1+xenial'\n debian/rules build\ndh_auto_configure: warning: Please use the third-party \"pybuild\" build system instead of python-distutils\ndh_auto_configure: warning: This feature will be removed in compat 12.\n debian/rules binary\nTraceback (most recent call last):\n  File \"/usr/bin/dh_virtualenv\", line 29, in <module>\n    from debpython.debhelper import DebHelper\nModuleNotFoundError: No module named 'debpython'\nmake[1]: *** [debian/rules:28: override_dh_virtualenv] Error 1\nmake: *** [debian/rules:8: binary] Error 2\ndpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2", "stderr_lines": [" dpkg-source --before-build .", " debian/rules clean", " dpkg-source -b .", "dpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)", "dpkg-source: warning: source directory 'securedrop-app-code_1.5.0~rc1+xenial_amd64' is not <sourcepackage>-<upstreamversion> 'securedrop-app-code-1.5.0~rc1+xenial'", " debian/rules build", "dh_auto_configure: warning: Please use the third-party \"pybuild\" build system instead of python-distutils", "dh_auto_configure: warning: This feature will be removed in compat 12.", " debian/rules binary", "Traceback (most recent call last):", "  File \"/usr/bin/dh_virtualenv\", line 29, in <module>", "    from debpython.debhelper import DebHelper", "ModuleNotFoundError: No module named 'debpython'", "make[1]: *** [debian/rules:28: override_dh_virtualenv] Error 1", "make: *** [debian/rules:8: binary] Error 2", "dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2"], "stdout": "dpkg-buildpackage: info: source package securedrop-app-code\ndpkg-buildpackage: info: source version 1.5.0~rc1+xenial\ndpkg-buildpackage: info: source distribution xenial\ndpkg-buildpackage: info: source changed by SecureDrop Team <[email protected]>\ndpkg-buildpackage: info: host architecture amd64\ndh clean --with python-virtualenv --with systemd\n   dh_clean\ndpkg-source: info: using source format '1.0'\ndpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0~rc1+xenial.tar.gz\ndpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0~rc1+xenial.dsc\ndh build --with python-virtualenv --with systemd\n   dh_update_autotools_config\n   dh_auto_configure\ndh binary --with python-virtualenv --with systemd\n   dh_testroot\n   dh_prep\n   dh_install\n   dh_installdocs\n   dh_installchangelogs\n   dh_installdebconf\n   debian/rules override_dh_virtualenv\nmake[1]: Entering directory '/tmp/securedrop-app-code_1.5.0~rc1+xenial_amd64'\ndh_virtualenv \\\n\t--python=/usr/bin/python3.5 \\\n\t--setuptools \\\n\t--extra-pip-arg \"--verbose\" \\\n\t--extra-pip-arg \"--ignore-installed\" \\\n\t--extra-pip-arg \"--no-deps\" \\\n\t--extra-pip-arg \"--no-binary=:all:\" \\\n\t--extra-pip-arg \"--no-cache-dir\"\nmake[1]: Leaving directory '/tmp/securedrop-app-code_1.5.0~rc1+xenial_amd64'", "stdout_lines": ["dpkg-buildpackage: info: source package securedrop-app-code", "dpkg-buildpackage: info: source version 1.5.0~rc1+xenial", "dpkg-buildpackage: info: source distribution xenial", "dpkg-buildpackage: info: source changed by SecureDrop Team <[email protected]>", "dpkg-buildpackage: info: host architecture amd64", "dh clean --with python-virtualenv --with systemd", "   dh_clean", "dpkg-source: info: using source format '1.0'", "dpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0~rc1+xenial.tar.gz", "dpkg-source: info: building securedrop-app-code in securedrop-app-code_1.5.0~rc1+xenial.dsc", "dh build --with python-virtualenv --with systemd", "   dh_update_autotools_config", "   dh_auto_configure", "dh binary --with python-virtualenv --with systemd", "   dh_testroot", "   dh_prep", "   dh_install", "   dh_installdocs", "   dh_installchangelogs", "   dh_installdebconf", "   debian/rules override_dh_virtualenv", "make[1]: Entering directory '/tmp/securedrop-app-code_1.5.0~rc1+xenial_amd64'", "dh_virtualenv \\", "\t--python=/usr/bin/python3.5 \\", "\t--setuptools \\", "\t--extra-pip-arg \"--verbose\" \\", "\t--extra-pip-arg \"--ignore-installed\" \\", "\t--extra-pip-arg \"--no-deps\" \\", "\t--extra-pip-arg \"--no-binary=:all:\" \\", "\t--extra-pip-arg \"--no-cache-dir\"", "make[1]: Leaving directory '/tmp/securedrop-app-code_1.5.0~rc1+xenial_amd64'"]}

Not sure why it still says --python=/usr/bin/python3.5 as I updated it to python3.8

@zenmonkeykstop
Copy link
Contributor

naive do-release-upgrade from xenial to bionic, prod NUC5s

summary

  • did a rough run-thru with VMs to start, hit some issues with multiple NICs that aren't relevant to prod installs, so switched to prod hw.
  • do-release-upgrade is more straightforward than with trusty->xenial, and does not have the intermediate step where sshd and tor are restarted. Time for upgrade process is 35-40 min per server. (playbook not run afterwards due to errors below.
  • current securedrop-app-code package fails to install, as no bionic version available. Install from xenial repo also fails as python3.5 dependency isn't satisfied. (bionic default is 3.6)
  • I put a little time into trying to build a python3.6 version of the package, but without luck. Will come back to that later.

notes

fresh prod install ,ssh over tor, v3 only, also nucs, following the procedure for 14.04->16.04 as closely
as possible.
On mon (ssh mon):

sudo apt update
sudo apt upgrade
sudo apt dist-upgrade
sudo apt autoremove
  • edited /etc/update-manager/release-upgrades, flipped Prompt value to lts
  • sudo do-release-upgrade
  • press ENTER
  • ncurses (busted) Postfix configuration: No configuration
  • cli: /etc/sysctl.conf: N (default keep local)
  • ncurses: PAM config: No (default, keep local)
  • ncurses: /etc/default/grub: Keep local
  • cli ssh: N (default)
  • ncurses sshd: Keep local
  • cli: paxctld: N (keep local)
  • cli remove obsolete packages: Y
  • restart? *y
    (no restart due to tor or ssh dropping was needed, improvement over Trusty-> Xenial)

On app (ssh app):

sudo apt update
sudo apt upgrade
sudo apt dist-upgrade
sudo apt autoremove
  • edited /etc/update-manager/release-upgrades, flipped Prompt value to lts

  • sudo do-release-upgrade

  • press ENTER

  • ncurses (busted) Configuring dma: app (default)

  • ncurses: dma ip address: [blank]

  • cli: /etc/sysctl.conf: N (default keep local)

  • ncurses: PAM config: No (default, keep local)

  • ncurses: /etc/default/grub: Keep local

  • cli: apache config: N (keep local)

  • cli ssh: N (default)

  • ncurses sshd: Keep local

  • cli: paxctld: N keep local

  • cli remove obsolete packages: Y

  • restart? y

  • Apache not running after restart, as python3.5-specific libraries not present, mod_wsgi not set up

  • ./securedrop-admin install fails early, at FPF apt server setup phase: apt cache update failed

  • uncommented fpf repos in /etc/apt/sources.list.d/*- got big giant failure errors coz fpf repos doesn't have a Bionic release file. (so edited the line to point back at xenial again)

  • ran apt -f install to fix broken packages where possible

  • ran it again, removed securedrop-app-code

  • ran apt-get install paxctld redis-server sqlite

  • dpkg -i securedrop-app-code now complains about libpython3.5 not installed, python 3.6.7 being too new

  • ran dpkg -i --ignore-depends=libpython3.5,python3, fails to complete with "ImportError: no module named 'encodings'"

@conorsch
Copy link
Contributor

conorsch commented Jul 8, 2020

@kushaldas Regarding the dh-virtualenv problems, see spotify/dh-virtualenv#298 which discusses building for 20.04. There's some work-in-progress that would be useful, although for a spike, consider

You can just use sid's version, since it is not platform-dependent.

@eloquence
Copy link
Member

eloquence commented Jul 8, 2020

It looks to me like version 1.2 which just came out four days ago may be Python3-ready:
spotify/dh-virtualenv@c73d7da

So perhaps just a matter of using that version? In any event, for the 7/8-7/22 sprint, @kushaldas will continue the packaging journey to see if we can unblock that part of the process, one way or another.

@kushaldas
Copy link
Contributor

As a quick note: I finally have Focal packages, I will push the WIP branch in sometime.

@conorsch
Copy link
Contributor

@kushaldas Great news! I'd love to take a look at your WIP—mind pushing a branch and documenting here?

@kushaldas
Copy link
Contributor

kushaldas commented Jul 13, 2020

Steps:

Go inside of molecule/builder-focal.

First download dh-virtualenv from Sid
.
https://packages.debian.org/sid/all/dh-virtualenv/download

docker build -t sd-docker-builder-focal:latest .

Then, from the main directory,

BUILDER_IMAGE=sd-docker-builder-focal:latest make build-debs

https://github.com/freedomofpress/securedrop/tree/on_focal this is the branch.

@conorsch
Copy link
Contributor

@kushaldas Following the steps you provide, I'm able to build packages. I've tacked on a commit that performs the dh-virtualenv-from-sid step inside the dockerfile. The purpose of the deployment.py dh-virtualenv helper script is not immediately obvious to me, so would appreciate some additional detail there. Happy to talk it through with you 1:1, then we can provide a report-back as a summary of changes on this ticket when you're satisfied with the findings.

@kushaldas
Copy link
Contributor

kushaldas commented Jul 15, 2020

I pushed a few more updates related to Python3.5 to Python3.8. The package installation fails with the following details

**
+ chmod 0700 /var/www/securedrop
+ '[' -e /var/lib/securedrop/keys/gpg-agent.conf ']'
+ echo allow-loopback-pinentry
+ '[' '!' -d /var/lib/securedrop/keys/private-keys-v1.d ']'
+ chown -R www-data:www-data /var/lib/securedrop /var/www/securedrop
+ chown -R www-data:www-data /var/www/securedrop
+ chown www-data:www-data /var/www/journalist.wsgi
+ chown www-data:www-data /var/www/source.wsgi
+ a2dissite 000-default
Site 000-default disabled.
To activate the new configuration, you need to run:
  systemctl reload apache2
+ a2dissite default-ssl
Site default-ssl already disabled
+ service apache2 stop
+ /opt/venvs/securedrop-app-code/bin/mod_wsgi-express module-config
+ a2enmod wsgi
Enabling module wsgi.
To activate the new configuration, you need to run:
  systemctl restart apache2
+ '[' -e /etc/apparmor.d/disable/usr.sbin.apache2 ']'
+ aa-enforce /etc/apparmor.d/usr.sbin.tor
Setting /etc/apparmor.d/usr.sbin.tor to enforce mode.
+ aa-enforce /etc/apparmor.d/usr.sbin.apache2
Setting /etc/apparmor.d/usr.sbin.apache2 to enforce mode.
+ adjust_wsgi_configuration
+ journalist_conf=/etc/apache2/sites-available/journalist.conf
+ test -f /etc/apache2/sites-available/journalist.conf
+ service apache2 restart
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
dpkg: error processing package securedrop-app-code (--configure):
 installed securedrop-app-code package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (245.4-4ubuntu3) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9) ...
Processing triggers for ufw (0.36-6) ...
Errors were encountered while processing:
 securedrop-app-code
E: Sub-process /usr/bin/dpkg returned an error code (1)

In our post-installation of the debian package, we execute this following command.

${SDBIN}/mod_wsgi-express module-config > /etc/apache2/mods-available/wsgi.load

And this is creating wrong configuration as shown below.

LoadModule wsgi_module "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.so"

But, the real path for the mod_wsgi module is /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so.

Maybe updating mod_wsgi package is the solution, I am yet to reach that level. Our current version is 4.6.7 and the upstream version is 4.7.1.

I also noticed the following error:

[Wed Jul 15 10:43:12.889104 2020] [core:warn] [pid 17375] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot

@eloquence
Copy link
Member

eloquence commented Jul 23, 2020

We won't be directly working on this during the 7/23-8/5 sprint, but we'll aim to:

  • review findings from Kev's and Kushal's spikes as a team;
  • agree on the build strategy we want to pursue for maintaining 18.04 and/or 20.04 builds alongside 16.04 (e.g., do we want to achieve greater parity between the build strategy used in SD Core with the build strategy used for the SecureDrop Workstation).

Keeping this ticket on the sprint to track those commitments.

@kushaldas
Copy link
Contributor

I updated my on_focal branch with more findings. That includes mod-wsgi upgrade, and more apparmor rules for usr.sbin.apache2.

mod-wsgi is still a problem. I had to do the following manually (we can add the things into postinstallation of the debian package I think) to have the right file path via a symlink.

root@app-prod:~# cat /etc/apache2/mods-enabled/wsgi.load
LoadModule wsgi_module "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.so"
WSGIPythonHome "/opt/venvs/securedrop-app-code"
root@app-prod:~# ls /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server/
apxs_config.py  environ.py  __init__.py  management  mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so  __pycache__
root@app-prod:~# cd /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server/
root@app-prod:/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/mod_wsgi/server# ln -s mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so mod_wsgi-py38.so

@nailor
Copy link

nailor commented Nov 2, 2020

FYI: I've made dh-virtualenv available on Focal, Bionic and Xenial here: https://launchpad.net/~jyrki-pulliainen/+archive/ubuntu/dh-virtualenv/

Edit: Added Xenial and Bionic to the list

@eloquence eloquence changed the title Support for Ubuntu 18.04 (Bionic) or 20.04 (Focal) Support for Ubuntu 20.04 (Focal) Dec 2, 2020
@dkorzhevin
Copy link

dkorzhevin commented Dec 24, 2020

Hello guys,

Anything I can help you with to speedup official support of 20.04 LTS for securedrop? Looking forward to help

@eloquence
Copy link
Member

@dkorzhevin Sorry, I missed your note on this issue. We would absolutely appreciate your contributions; given the scope and complexity of this issue, it might be easiest to chat in real-time about how to best get involved, e.g. on https://gitter.im/freedomofpress/securedrop or in our virtual standups Mo-Thu at 9AM PST / 5 PM UTC (announced on Gitter).

@kushaldas
Copy link
Contributor

It is a good thing that we are not doing in place upgrade of the servers, https://bugs.launchpad.net/ubuntu/+source/libxcrypt/+bug/1903838

@eloquence
Copy link
Member

Closing this epic:

@eloquence eloquence unpinned this issue Mar 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic Meta issue tracking child issues
Projects
None yet
Development

No branches or pull requests

7 participants