CI staging runs are timing out while waiting for VM reboot

[conorsch]

Description

The staging-test-with-rebase CI job has been timing out frequently, blocking review and merge.

Steps to Reproduce

Observe example CI failures:

• https://circleci.com/gh/freedomofpress/securedrop/27844
• https://circleci.com/gh/freedomofpress/securedrop/27847

Expected Behavior

CI should pass if no bugs are found. CI should not halt early because VMs did not successfully reboot.

Actual Behavior

CI frequently fails on while waiting for the staging VMs to report. Example text:

    RUNNING HANDLER [common : Wait for server to come back.] ***********************
    fatal: [app-staging -> localhost]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string OpenSSH in 192.168.121.45:22"}
    fatal: [mon-staging -> localhost]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string OpenSSH in 192.168.121.81:22"}
    

Comments

The GCP CI image hasn't changed since 2018-10. On every start, however, cloud-init settings will upgrade all apt packages. Perhaps an upstream version mismatch in the Xenial repos (for the GCP CI host, not for the staging VMs themselves) is causing problems with the libvirt config.

[conorsch]

Temporary fix in #4435 yielded at least one passing build: https://circleci.com/gh/freedomofpress/securedrop/27875 Will re-run it in an attempt to shake out problems.

[conorsch]

Another successful build: https://circleci.com/gh/freedomofpress/securedrop/27883

[conorsch]

More successful builds:

• https://circleci.com/gh/freedomofpress/securedrop/27892
• https://circleci.com/gh/freedomofpress/securedrop/27937

The fix has been merged into develop via #4432, and will shortly be backported to release/0.13, so I'm marking this resolved. Please open a new issue if CI problems rear their heads again.