Running ./securedrop-admin install on 0.14.0 against server updated to 1.0.0 will fail and not immediately recoverable

[emkll]

Description

Steps to Reproduce

• install on 0.14.0
• cron-apt upgrade to 1.0.0
• from the admin workstation, ./securedrop-admin install from a previous version (0.14.0)
• observe source interface returning 404
• attempt to fix by running playbooks on 1.0.0
• Observe failure during installing of securedrop-app-code, due to error in apache config

Expected Behavior

The install should ideally not break the server. The probability of users having the wrong branch checked out may be low, but it would be preferable for the error to be recoverable.

Actual Behavior

The apache module step fails with the following error, when installing using the 0.14.0 branch on a 1.0.0-rc2 updated server:

TASK [app : Enable required Apache modules.] *******************************************************************
failed: [app] (item=access_compat) => {"changed": false, "item": "access_compat", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=authn_core) => {"changed": false, "item": "authn_core", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=alias) => {"changed": false, "item": "alias", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=authz_core) => {"changed": false, "item": "authz_core", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=authz_host) => {"changed": false, "item": "authz_host", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=authz_user) => {"changed": false, "item": "authz_user", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=deflate) => {"changed": false, "item": "deflate", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=filter) => {"changed": false, "item": "filter", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=dir) => {"changed": false, "item": "dir", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=headers) => {"changed": false, "item": "headers", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=mime) => {"changed": false, "item": "mime", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=mpm_event) => {"changed": false, "item": "mpm_event", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=negotiation) => {"changed": false, "item": "negotiation", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=reqtimeout) => {"changed": false, "item": "reqtimeout", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=rewrite) => {"changed": false, "item": "rewrite", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=ssl) => {"changed": false, "item": "ssl", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=wsgi) => {"changed": false, "item": "wsgi", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}
failed: [app] (item=xsendfile) => {"changed": false, "item": "xsendfile", "msg": "Error executing /usr/sbin/apache2ctl: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory\n"}

NO MORE HOSTS LEFT *********************************************************************************************

NO MORE HOSTS LEFT *********************************************************************************************
	to retry, use: --limit @/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.retry

PLAY RECAP *****************************************************************************************************
app                        : ok=91   changed=6    unreachable=0    failed=1   
localhost                  : ok=4    changed=0    unreachable=0    failed=0   
mon                        : ok=101  changed=5    unreachable=0    failed=0   

It also breaks the install, the source interface returns 404:

re-running the installer on 1.0.0 does NOT resolve, because the postinstall task for for the securedrop-app-code package fails, due to a syntax error in the apache configuration:

Sep 11 18:34:48 app-prod apache2[10723]: apache2: Syntax error on line 15 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/wsgi.load: Cannot load /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so into server: /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

Comments

[eloquence]

This is applicable to an outdated release no longer used in production, so closing. I would recommend prioritizing #5317 soon so we can minimize the risk of similar admin errors in future.