Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds package builds for Focal #5465

Merged
merged 7 commits into from
Sep 3, 2020
+101 −662
Merged

Adds package builds for Focal #5465

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
77be0b1
Fixes #5449 changes to build securedrop-app-code in focal
kushaldas Aug 24, 2020
ae1a78c
Fixes #5449 builds ossec packages on Focal
kushaldas Aug 24, 2020
7826458
Updates based on review feedback
kushaldas Aug 26, 2020
bd58aff
Updates focal debs target to run tests
Aug 27, 2020
fe90068
Adds libevent-dev package dep for Focal
kushaldas Aug 31, 2020
dd872db
Uses new experimental build for Focal
kushaldas Sep 3, 2020
c4d69c9
Consolidates packaging tests
Sep 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,23 +318,28 @@ update-user-guides: ## Run the page layout tests to regenerate screenshots.
###########

.PHONY: build-debs
build-debs: ## Build and test SecureDrop Debian packages.
@echo "Building SecureDrop Debian packages..."
build-debs: ## Build and test SecureDrop Debian packages (for Xenial)
@echo "Building SecureDrop Debian packages for Xenial..."
@$(SDROOT)/devops/scripts/build-debs.sh
@echo

.PHONY: build-debs-notest
build-debs-notest: ## Build SecureDrop Debian packages without running tests.
@echo "Building SecureDrop Debian packages; skipping tests..."
build-debs-notest: ## Build SecureDrop Debian packages (for Xenial) without running tests.
@echo "Building SecureDrop Debian packages for Xenial; skipping tests..."
@$(SDROOT)/devops/scripts/build-debs.sh notest
@echo

.PHONY: build-debs-focal
build-debs-focal: ## Build and test SecureDrop Debian packages.
@echo "Building SecureDrop Debian packages..."
@$(SDROOT)/devops/scripts/build-debs.sh notest focal
build-debs-focal: ## Build and test SecureDrop Debian packages (for Focal)
@echo "Building SecureDrop Debian packages for Focal..."
@$(SDROOT)/devops/scripts/build-debs.sh test focal
@echo

.PHONY: build-debs-notest-focal
build-debs-notest-focal: ## Build SecureDrop Debian packages (for Focal) without running tests.
@echo "Building SecureDrop Debian packages for Focal; skipping tests..."
@$(SDROOT)/devops/scripts/build-debs.sh notest focal
@echo


########################
Expand Down
9 changes: 8 additions & 1 deletion install_files/ansible-base/roles/build-ossec-deb-pkg/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
---
build_ossec_deb_pkg_dependencies: ['libevent1-dev','libpcre2-dev']
build_ossec_deb_pkg_dependencies:
- "libpcre2-dev"
- "{{ 'libevent-dev' if securedrop_build_focal_support else 'libevent1-dev' }}"

ossec_server_hostname: ossec-server
ossec_version: 3.6.0
Expand All @@ -23,3 +25,8 @@ ossec_build_rsync_ansible_hack_opt:
- "--rsync-path='sudo rsync'"

ossec_source_checksum: sha256:653828a19137b8a7e98af65e873318f7bb48137fe1e61b80577e13c316e04708

# Default is xenial.
securedrop_build_focal_support: False
securedrop_package_dist: "{{ 'focal' if securedrop_build_focal_support else 'xenial' }}"

0 ...les/securedrop-ossec-agent/DEBIAN/control → ...s/build-ossec-deb-pkg/files/control-focal
View file
Open in desktop
File renamed without changes.
4 changes: 4 additions & 0 deletions install_files/ansible-base/roles/build-ossec-deb-pkg/templates/agent_control.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Homepage: http://ossec.net
Package: ossec-agent
Version: {{ ossec_version }}
Architecture: amd64
{% if securedrop_build_focal_support %}
Depends: libc6,libssl1.1,expect,inotify-tools,adduser
{% else %}
Depends: libc6,libssl1.0.0,expect,inotify-tools,adduser
{% endif %}
Conflicts: ossec-server
Description: Installs the generic ossec agent
4 changes: 4 additions & 0 deletions install_files/ansible-base/roles/build-ossec-deb-pkg/templates/server_control.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Homepage: http://ossec.net
Package: ossec-server
Version: {{ ossec_version }}
Architecture: amd64
{% if securedrop_build_focal_support %}
Depends: libc6 (>=2.7),libssl1.1,adduser,expect,inotify-tools
{% else %}
Depends: libc6,libssl1.0.0,adduser,expect,libc6 (>= 2.7),inotify-tools
{% endif %}
Conflicts: ossec-agent
Description: Installs generic OSSEC server
19 changes: 13 additions & 6 deletions install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,6 @@ securedrop_code_filtered: "{{ securedrop_app_code_prep_dir }}/var/www/securedrop

securedrop_pip_requirements: "{{ securedrop_code_filtered }}/requirements/python3/securedrop-app-code-requirements.txt"

# SecureDrop virtualenv location
securedrop_venv: "/opt/venvs/securedrop-app-code"
securedrop_venv_bin: "{{ securedrop_venv }}/bin"
securedrop_venv_site_packages: "{{ securedrop_venv }}/lib/python3.5/site-packages"

# SecureDrop code installation directory
securedrop_code: /var/www/securedrop

Expand Down Expand Up @@ -45,4 +40,16 @@ securedrop_app_rsync_opts:

securedrop_local_build: "../../build"

securedrop_package_dist: xenial
# Default is xenial.
securedrop_build_focal_support: False
securedrop_package_dist: "{{ 'focal' if securedrop_build_focal_support else 'xenial' }}"

# SecureDrop virtualenv location
securedrop_venv: "/opt/venvs/securedrop-app-code"
securedrop_venv_bin: "{{ securedrop_venv }}/bin"
securedrop_python_version: "{{ '3.8' if securedrop_build_focal_support else '3.5' }}"
securedrop_venv_site_packages: "{{ securedrop_venv }}/lib/python{{ securedrop_python_version }}/site-packages"

securedrop_app_focal_files:
- src: securedrop-app-code.triggers-focal
dest: "{{ securedrop_app_code_prep_dir }}/debian/securedrop-app-code.triggers"
5 changes: 5 additions & 0 deletions install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/files/changelog-focal
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
securedrop-app-code (1.6.0~rc1+focal) focal; urgency=medium

*

-- SecureDrop Team <[email protected]> Thu, 18 Jun 2020 21:58:23 +0000
7 changes: 7 additions & 0 deletions ...ble-base/roles/build-securedrop-app-code-deb-pkg/files/securedrop-app-code.triggers-focal
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Register interest in Python interpreter changes; and
# don't make the Python package dependent on the virtualenv package
# processing (noawait)
interest-noawait /usr/bin/python3.8

# Also provide a symbolic trigger for all dh-virtualenv packages
interest dh-virtualenv-interpreter-update
5 changes: 5 additions & 0 deletions install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/files/usr.sbin.apache2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
/etc/magic r,
/etc/mime.types r,
/etc/python3.5/sitecustomize.py r,
/etc/python3.8/sitecustomize.py r,
/etc/services r,
/etc/timezone r,
/lib/x86_64-linux-gnu/libbz2.so.* mr,
Expand All @@ -87,6 +88,7 @@
/run/apache2/wsgi.*.sock rw,
/run/lock/apache2/rewrite-map.* rw,
/run/lock/apache2/ssl-cache.* rwk,
/run/systemd/userdb/io.systemd.DynamicUser r,
/run/shm rw,
/sbin/ldconfig rix,
/sbin/ldconfig.real rix,
Expand All @@ -107,6 +109,9 @@
/opt/venvs/securedrop-app-code/bin/python3 r,
/opt/venvs/securedrop-app-code/lib/python3.5/ r,
/opt/venvs/securedrop-app-code/lib/python3.5/** rm,
/opt/venvs/securedrop-app-code/lib/python3.8/ r,
/opt/venvs/securedrop-app-code/lib/python3.8/** rm,
/opt/venvs/securedrop-app-code/pyvenv.cfg r,
/var/lib/securedrop/ r,
/var/lib/securedrop/db.sqlite kw,
/var/lib/securedrop/db.sqlite rwk,
Expand Down
14 changes: 13 additions & 1 deletion install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,18 @@
src: "changelog-{{ securedrop_package_dist }}"
dest: "{{ securedrop_app_code_prep_dir }}/debian/changelog"

- name: Replace the files required for focal package
copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
with_items: "{{ securedrop_app_focal_files }}"
when: securedrop_build_focal_support

- name: Create the control file based on distribution
template:
src: "control.j2"
dest: "{{ securedrop_app_code_prep_dir }}/debian/control"

- name: Create lib/systemd/services directory in prep directory
file:
state: directory
Expand Down Expand Up @@ -107,7 +119,7 @@
environment:
DH_PIP_EXTRA_ARGS: "--verbose --ignore-installed --no-deps --no-binary=:all: --no-cache-dir"
DH_UPGRADE_SETUPTOOLS: "46.0.0"
DH_VIRTUALENV_ARGUMENTS: "--python=/usr/bin/python3.5 --setuptools"
DH_VIRTUALENV_ARGUMENTS: "{{ '--python=/usr/bin/python3.8 --setuptools 46.0.0' if securedrop_build_focal_support else '--python=/usr/bin/python3.5 --setuptools' }}"
DH_VIRTUALENV_INSTALL_ROOT: "/opt/venvs"

- name: Find newly built Debian package
Expand Down
5 changes: 4 additions & 1 deletion ..._files/securedrop-app-code/debian/control → ...rop-app-code-deb-pkg/templates/control.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,14 @@ Maintainer: SecureDrop Team <[email protected]>
Homepage: https://securedrop.org
Build-Depends: debhelper (>= 9), dh-python, python3-all, python3-setuptools, dh-systemd, dh-virtualenv
Standards-Version: 3.9.8
X-Python3-Version: >= 3.5

Package: securedrop-app-code
Architecture: amd64
Conflicts: libapache2-mod-wsgi,supervisor
Replaces: libapache2-mod-wsgi,supervisor
{% if securedrop_build_focal_support %}
Depends: ${dist:Depends}, ${misc:Depends}, ${python3:Depends}, apache2, apparmor-utils, coreutils, gnupg2, haveged, libapache2-mod-xsendfile, libpython3.8, paxctld, python3, redis-server, securedrop-config, securedrop-keyring, sqlite3
{% else %}
Depends: ${dist:Depends}, ${misc:Depends}, ${python3:Depends}, apache2, apparmor-utils, coreutils, gnupg2, haveged, libapache2-mod-xsendfile, libpython3.5, paxctld, python3 (>= 3.5), python3 (<< 3.6), redis-server, securedrop-config, securedrop-keyring, sqlite3
{% endif %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much improved! As discussed today in tech meeting, can we further consolidate the dependencies for libpython3.8 -> libpython3 for both platforms? Maybe that's not possible, but if it is, it'd greatly simplify the logic in play.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can not be done.

# apt install libpython3  
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libpython3

Description: SecureDrop application code, dependencies, Apache configuration, systemd services, and AppArmor profiles. This package will put the AppArmor profiles in enforce mode.
18 changes: 18 additions & 0 deletions install_files/securedrop-ossec-agent/DEBIAN/control.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Source: ossec.net
Section: web
Priority: optional
Maintainer: SecureDrop Team <[email protected]>
Homepage: https://securedrop.org
Package: securedrop-ossec-agent
Version: 3.6.0+1.6.0~rc1
Architecture: amd64
{% if securedrop_build_focal_support %}
Depends: libevent-2.1.7,libpcre2-8-0,ossec-agent,securedrop-keyring,securedrop-config
{% else %}
Depends: libevent-1.4-2,libpcre2-8-0,ossec-agent,securedrop-keyring,securedrop-config
{% endif %}
Replaces: ossec-agent
Conflicts: securedrop-ossec-server
Description: Installs the securedrop pre-configured OSSEC agent
This package installs an OSSEC agent pre-configured for the
SecureDrop app server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid approach to multi-distro support. 👍 However, elsewhere in the diff I see that install_files/securedrop-ossec-agent/DEBIAN/control → ...s/build-ossec-deb-pkg/files/control-focal. Given thecontrol.j2introduction here, it apepars thatbuild-ossec-deb-pkg/files/control-focal` is no longer necessary, and should be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@conorsch These are two different packages :) ossec-agent and securedrop-ossec-agent.

4 changes: 4 additions & 0 deletions ...es/securedrop-ossec-server/DEBIAN/control → ...securedrop-ossec-server/DEBIAN/control.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,11 @@ Homepage: https://securedrop.org
Package: securedrop-ossec-server
Version: 3.6.0+1.6.0~rc1
Architecture: amd64
{% if securedrop_build_focal_support %}
Depends: libevent-2.1.7,libpcre2-8-0,ossec-server,securedrop-keyring,securedrop-config
{% else %}
Depends: libevent-1.4-2,libpcre2-8-0,ossec-server,securedrop-keyring,securedrop-config
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, perhaps we can standardize on a single libevent metapackage that would work well. Can you check whether that's possible and add if so?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

# apt install libevent    
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libevent

This is also not possible due to package naming.

{% endif %}
Replaces: ossec-server
Conflicts: securedrop-ossec-agent
Description: Installs the pre-packaged OSSEC server
Expand Down
4 changes: 3 additions & 1 deletion molecule/builder-focal/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ verifier:
n: auto
env:
SECUREDROP_TARGET_PLATFORM: focal
directory: tests/
SECUREDROP_PYTHON_VERSION: "3.8"
# Reuse the same test suite for all packages
directory: ../builder-xenial/tests/
lint:
name: flake8
22 changes: 0 additions & 22 deletions molecule/builder-focal/tests/conftest.py
View file
Open in desktop

This file was deleted.

40 changes: 0 additions & 40 deletions molecule/builder-focal/tests/test_build_dependencies.py
View file
Open in desktop

This file was deleted.

20 changes: 0 additions & 20 deletions molecule/builder-focal/tests/test_legacy_paths.py
View file
Open in desktop

This file was deleted.

Loading