Use latest Buster CI image

[maeve-fpf]

Status

Ready for review

Description of Changes

Works towards #5928.

Changes proposed in this pull request: rebuild the Buster-based CI image.

Testing

I think this should just require make ci-go.

Deployment

Any special considerations for deployment? No.

Checklist

• These changes do not require documentation

[conorsch]

The staging-test-with-rebase job took >1h on this PR:

That's less of an improvement than I'd expect. @maeve-fpf I believe there's a bit more we can do than just rebuilding the CI image, as I initially suggested: it looks like the 202008.16.0 versions of both the Xenial & Focal boxes are included here. The Xenial one isn't necessary at all, so we can remove, but for Focal, we should bump that to some newer Vagrant box that will have correspondingly more recent apt lists bundled inside of it. That's where I expect to save some time—although admittedly only 5-15m per run.

[conorsch]

Let's also bump the vagrant box versions inside the ci image, and re-evaluate the CI run time.

[maeve-fpf]

Looks like this failed with:

    TASK [app : Install apache packages.] ******************************************
    fatal: [app-staging]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: "}

Naming change in the packages?

[conorsch]

I'm guessing that's a flake... see apt cache flakes reported elsewhere in #5960 (comment) . I've kicked CI to rerun, let's see if it reoccurs.

[conorsch]

Same failure. =(

TASK [app : Install apache packages.] ******************************************
    fatal: [app-staging]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: "}

Clearly the apt cache updated fine in the other tasks, will debug interactively if it occurs again. I think also not using state=latest on the package tasks would help here.

[conorsch]

Reliably failing on the same task. @rmol could you take a look at the new Vagrant box included here, version 202105.25.0 for Focal, and see if you can reproduce this error? I'd say forget about CI for the moment, and debug locally, or you can use an interactive session on the next failure run.

[zenmonkeykstop]

Log here of a run with debug enabled - it looks like it isn't apt that's failing, but rather there's a connectivity issue: https://circleci.com/api/v1.1/project/github/freedomofpress/securedrop/55291/output/104/0?file=true&allocation-id=60d9fcc8e56ba4530bc45a7a-0-build%2F7F2C4FF5

excerpt of interest:

    TASK [Expose source v3 onion service info to app] ******************************
    task path: /home/sdci/securedrop-source/install_files/ansible-base/roles/app/tasks/copy_tor_url_info_to_app_dir.yml:9
    --- before
    +++ after: /home/sdci/.ansible/tmp/ansible-local-9952clfpcjsg/tmpbcyrhkc1
    @@ -0,0 +1 @@
    +psqvgnvs4mqfohvqihk5tofmlothzomfijkq3wa6sthvrfvkbkiqp6id.onion
    
    changed: [app-staging] => {
        "changed": true,
        "checksum": "76c776b822d5adb4fac4d189cb82edf15c94b6f2",
        "dest": "/var/lib/securedrop/source_v3_url",
        "diff": [
            {
                "after": "psqvgnvs4mqfohvqihk5tofmlothzomfijkq3wa6sthvrfvkbkiqp6id.onion\n",
                "after_header": "/home/sdci/.ansible/tmp/ansible-local-9952clfpcjsg/tmpbcyrhkc1",
                "before": ""
            }
        ],
        "gid": 33,
        "group": "www-data",
        "invocation": {
            "module_args": {
                "_original_basename": "tmpbcyrhkc1",
                "attributes": null,
                "backup": false,
                "checksum": "76c776b822d5adb4fac4d189cb82edf15c94b6f2",
                "content": null,
                "delimiter": null,
                "dest": "/var/lib/securedrop/source_v3_url",
                "directory_mode": null,
                "follow": false,
                "force": true,
                "group": "www-data",
                "local_follow": null,
                "mode": "0644",
                "owner": "www-data",
                "regexp": null,
                "remote_src": null,
                "selevel": null,
                "serole": null,
                "setype": null,
                "seuser": null,
                "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1624901174.2667842-11616-202946281823193/source",
                "unsafe_writes": null,
                "validate": null
            }
        },
        "md5sum": "b8ceaa51e9f696ad7819fd6e0ac3315d",
        "mode": "0644",
        "owner": "www-data",
        "size": 63,
        "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1624901174.2667842-11616-202946281823193/source",
        "state": "file",
        "uid": 33
    }
    Using module file /home/sdci/securedrop-source/.venv/lib/python3.7/site-packages/ansible/modules/packaging/os/apt.py
    Pipelining is enabled.
    <192.168.121.101> ESTABLISH SSH CONNECTION FOR USER: vagrant
    <192.168.121.101> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o Port=22 -o 'IdentityFile="/home/sdci/.vagrant.d/insecure_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o ControlPath=/home/sdci/.ansible/cp/05cef265d7 192.168.121.101 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-vmunnjxbouhdpbknwggmmihscxtcstag ; LC_ALL=C /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
    Escalation succeeded
    <192.168.121.101> (1, b'\n{"msg": "Failed to update apt cache: ", "failed": true, "invocation": {"module_args": {"pkg": ["apache2", "libapache2-mod-xsendfile"], "state": "present", "update_cache": true, "cache_valid_time": 3600, "package": ["apache2", "libapache2-mod-xsendfile"], "purge": false, "force": false, "dpkg_options": "force-confdef,force-confold", "autoremove": false, "autoclean": false, "only_upgrade": false, "force_apt_get": false, "allow_unauthenticated": false, "deb": null, "default_release": null, "install_recommends": null, "upgrade": null, "policy_rc_d": null}}}\n', b'')
    <192.168.121.101> Failed to connect to the host via ssh:
    
    TASK [app : Install apache packages.] ******************************************
    task path: /home/sdci/securedrop-source/install_files/ansible-base/roles/app/tasks/install_and_harden_apache.yml:2
    fatal: [app-staging]: FAILED! => {
        "changed": false,
        "invocation": {
            "module_args": {
                "allow_unauthenticated": false,
                "autoclean": false,
                "autoremove": false,
                "cache_valid_time": 3600,
                "deb": null,
                "default_release": null,
                "dpkg_options": "force-confdef,force-confold",
                "force": false,
                "force_apt_get": false,
                "install_recommends": null,
                "only_upgrade": false,
                "package": [
                    "apache2",
                    "libapache2-mod-xsendfile"
                ],
                "pkg": [
                    "apache2",
                    "libapache2-mod-xsendfile"
                ],
                "policy_rc_d": null,
                "purge": false,
                "state": "present",
                "update_cache": true,
                "upgrade": null
            }
        },
        "msg": "Failed to update apt cache: "
    }

[eloquence]

@maeve-fpf Do you have time to continue work on this, or would you prefer the SD team investigate further?

[zenmonkeykstop]

(rebased on current develop to clear CI errors)

[maeve-fpf]

I've amended the use-shell-for-apt-get update commit, but it doesn't seem like an ideal solution, because it is not clear why it works (to me, at least!). However it is still passing the test-with-rebase CI job now (there is what looks like an unrelated failure in the dependency-CVE-check job).

[conorsch]

Rebased again on current develop to address the safety CI check.

[conorsch]

Works as advertised, with the separate apt-cache update. Wish we understood at root what the problem was there, but judging by some quick searches, we are not alone in finding Ansible's apt cache update mechanism to be finicky. Merging. Thanks, @maeve-fpf!