Updated testinfra tests to optionally run against a prod instance

[zenmonkeykstop]

Status

Ready for Review

Description of Changes

Fixes #4216

Updates testinfra tests to allow them to be run against a production SecureDrop hardware instance from an Admin Workstation.

Testing

Verify staging unchanged:

• check out this branch and run:

  make build-debs && make staging && make testinfra
  

  • testinfra tests complete without error

test against prod hardware (assuming default IPs, DNS settings, and both v2 and v3 services enabled):

• on the admin workstation for a previously configured instance, cd ~/Persistent/securedrop and check out this branch

• run ./securedrop-admin setup -t to install testinfra dependencies

• run ./securedrop-admin verify

  • testinfra tests complete with no errors

• remove the test dependencies with rm -rf ~/Persistent/securedrop/admin/.venv3

• run ./securedrop-admin setup to install vanilla dependencies

• verify testinfra module not installed with:

  cd ~/Persistent/securedrop
  source admin/.venv3/bin/activate
  pip list | grep testinfra  # there should be no output
  

Deployment

No special requirements for deployment

Checklist

because securedrop-admin changes:

• make -C admin test passing locally.

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

[kushaldas]

Note: TEST_ENV='prodVM' ./devops/scripts/run_prod_testinfra this step requires a lot of system package dependencies. I am slowly and painfully installing them. maybe we can add them or via the setup step.

[kushaldas]

• testinfra tests complete without error

This works, I could not finish up the prod testing thanks to tails.

[zenmonkeykstop]

I'm not 100% into adding the test dependencies in the regular setup step, as they're not required for securedrop-admin, so they're going to make that step longer for users for a start. But yeah it's a bit of a pain. At least it's faster on repeated runs (assuming the virtualenv wasn't nuked in between).

What problem did you hit with Tails and prod?

[zenmonkeykstop]

flipped back to draft to allow for some updates

[zenmonkeykstop]

Have addressed @kushaldas's point above by adding a new requirements file including testinfra deps that can optionally be installed at setup (using the -t flag), and making the call out to the testinfra script a securedrop-admin command for good measure. Also working around some remaining issues with iptables tests by suppressing the test in prod - not ideal but prod instances do have the daily check recently added by @rmol. This should be ready for review.

[conorsch]

Changes look great! Confirmed staging is unaffected (CI appears happy, as well). For testing the Tails behavior, I didn't use prod HW, rather prod VMs. The IP addresses for the vagrant environment were different from the HW defaults, and those failures were reported in the testinfra run, as expected. Everything else passed.

Also confirmed that the virtualenv only installs testinfra if the -t flag is passed to the setup subcommand. Only question I have is given that

Also working around some remaining issues with iptables tests by suppressing the test in prod

there might be a bit of leftover prod iptables info in the diff, that's now unused. Even if that's the case, I wouldn't be opposed to merging, if it's partial progress toward full coverage.

[conorsch]

@kushaldas It'd be great to have your approval on here too, since you performed active review previously!

[kushaldas]

Went through all the test steps (used prod vms), this looks good to me. 💯

[emkll]

Thanks @zenmonkeykstop for your hard work on these changes, this is very exciting and these changes will be of great help with the upcoming release testing.

Took one last pass through the test plan, all tests are passing with the exception of (3) tests for IP addresses related to OSSEC (using non-standard IP addresses, so this is expected, we can always follow-up later).

[emkll]

Noting for posterity that the versions of cffi, cryptography, jinja and others are different from the ones locked in develop-requirements.txt and admin/requirements.txt. Given that the changes here work, and that this is exclusively used for testing, we can always update the other requirements files at a later date.

[zenmonkeykstop]

A good future improvement against this would be to have testinfra read from site-specific when run in prod for the IP addresses etc mentioned above (also reinstating the iptables checks). When I can stand to look at this code again I'll see about adding that :)