Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add note about how we update the builder image #137

Merged
merged 2 commits into from
Mar 8, 2021
Merged

add note about how we update the builder image #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
511db58
more information about build container
Jan 21, 2021
ecf3f42
Fix typo
eloquence Feb 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 26 additions & 13 deletions docs/development/dockerbuildmaint.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@
Docker Build Maintenance
========================
.. _build_container:

Get your Quay account squared away
-----------------------------------
The container that performs builds of Debian packages is version controlled in
a docker repository at **quay.io/freedomofpress/sd-docker-builder**.
There are tight restrictions over who can make edits here. If you have permissions
to do so, you'll need to make sure your local docker client has credentials to push.
Build container
===============
We use a Docker build container to build our debian packages for SecureDrop (via ``make build-debs``
in the ``securedrop`` Github repository root directory). We keep images of this our container in a
Docker repository at **quay.io/freedomofpress**. The images are organized by Ubuntu release
version. For instance, you can find the images for Xenial at
**quay.io/freedomofpress/sd-docker-builder-xenial** and, for Focal, at
**quay.io/freedomofpress/sd-docker-builder-focal**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make these bolded URLs actual URLs to be a bit easier on folks, but that's not a change that came in via this PR.


Maintaining images of our build container for each release is our way of recording the exact version
of each dependency used to build our production debian packages for SecureDrop.

Who can update the build container?
===================================
There are tight restrictions over who can make edits to our Docker repository. If you have
permissions to do so, you'll need to make sure your local Docker client has credentials to push.

* First login into your quay.io account via the web-portal at https://quay.io/
* Drill into your **Account settings** via the upper right drop-down (where your username is)
Expand All @@ -15,12 +24,16 @@ to do so, you'll need to make sure your local docker client has credentials to p
obtained from the previous step.
* Proceed with update instructions

Updating the build container
============================
We know the build container needs to be updated when **test_ensure_no_updates_avail** fails during
``make build-debs`` in the ``securedrop`` Github reprository root directory. This test fails if any
of the dependencies required to build the debian packages have security updates. If you have access
rights to push to quay.io, then you can build and push a new container to the Quay repository by
following the steps below.

Performing container updates
----------------------------
If one of the dependencies requires security updates, the build may fail at
**test_ensure_no_updates_avail** . If you have access rights to push to quay.io,
here is the process to build and push a new container:
.. note:: The reason we don't update the container at runtime is that we use the container image as
a way of recording our build environment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could equivalent security properties be achieved by including the exact version information in the build logs automatically, as suggested in freedomofpress/securedrop#4533 (comment) ? If so, it does not seem a compelling argument to include in favor of the current process.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs presented here are definitely accurate in terms of the process we actually adhere to. I remain interested in getting the SD core debs fully reproducible, so we can stop fussing with the build container so much. After Focal, that seems a reasonable goal again, consistent with discussion in freedomofpress/securedrop#4533.

All that to say, the "note" as written here LGTM!


.. code:: sh

Expand Down
38 changes: 23 additions & 15 deletions docs/development/release_management.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,12 @@ Pre-Release
goal is to make sure we test against the lastest Tails release, including release candidates,
so that we can report bugs early to Tails.

#. Check if there is a new stable release of Tor that can be QAed and released as part of the
SecureDrop release. Also check for any new release candidates so that we're aware of any
upcoming major bug fixes and communicate them to the team. You can find releases by checking the
`Tor blog <https://blog.torproject.org/category/tags/stable-release>`_. If there is a new
stable release, file an issue and upgrade Tor following these steps:
#. Check the Tor blog for new
`release candidates <https://blog.torproject.org/category/tags/release-candidate>`_ and new
`stable releases <https://blog.torproject.org/category/tags/stable-release>`_. Let the team
know about any new release candidates during the SecureDrop release process in case there are
critical bug fixes. For a new stable release, file an issue and upgrade Tor following these
steps:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much clearer, thank you for including the explicit URLs!


a. Bump the version in `fetch-tor-packages
<https://github.com/freedomofpress/securedrop/blob/develop/molecule/fetch-tor-packages/
Expand Down Expand Up @@ -127,23 +128,30 @@ Pre-Release
#. Build Debian packages:

a. Check out the tag for the release candidate.
#. Build the packages with ``make build-debs``.
#. Build logs should be saved and published according to the `build
log guidelines
b. Build the packages with ``make build-debs``

.. note:: If the :ref:`build container <build_container>` used by ``make build-debs`` has
security updates, then you will see ``test_ensure_no_updates_avail`` fail in the
build output. To get around the bottleneck of tight restrictions around who can
update the build container, you can ignore this test failure until you are building a
production release.

c. Build logs should be saved and published according to the `build log guidelines
<https://github.com/freedomofpress/securedrop/wiki/Build-logs>`_.
#. Open a PR on `securedrop-dev-packages-lfs
d. Open a PR on `securedrop-dev-packages-lfs
<https://github.com/freedomofpress/securedrop-dev-packages-lfs>`_ that targets the ``main``
branch with the new debs. Do not include tarballs or any debs that would overwrite
existing debs. Changes merged to this branch will be published to ``apt-test.freedom.press``
within 15 minutes.

.. warning:: Only commit deb packages with an incremented version number: do not clobber existing
packages. That is, if there is already a deb called e.g.
``ossec-agent-3.6.0-amd64.deb`` in ``main``, do not commit a new version of this
deb.
.. warning:: Only commit deb packages with an incremented version number: do not clobber
existing packages. That is, if there is already a deb called e.g.
``ossec-agent-3.6.0-amd64.deb`` in ``main``, do not commit a new version of this
deb.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Locally, the way I determine this is by running git status and checking if a file was modified by my changes. If so, I run git reset <file> on each of them, and commit only the new packages. That's something CI could check for us, even in the dev-lfs repo, that might be a little friendlier to developers. Weighed against continued progress toward reproducibility, when we can trust a bot to file the PR, I very much prefer these docs you've got here.


.. note:: If the release contains other packages not created by ``make build-debs``, such as Tor
or kernel updates, make sure that they also get pushed to ``apt-test.freedom.press``.
.. note:: If the release contains other packages not created by ``make build-debs``, such as
Tor or kernel updates, make sure that they also get pushed to
``apt-test.freedom.press``.

#. Write a test plan that focuses on the new functionality introduced in the release. Post for
feedback and make changes based on suggestions from the community. Once it's ready, publish the
Expand Down