[functional testing] Determine how to handle users when testing external server

[redshiftzero]

Description

In a release QA situation, the users credentials must be provided to the functional tests, such that logins can occur. This is done via a config file. We should document this for pre-release QA. We should fall back to the existing pattern if we are not using an external server.

Epic: #3659

[kushaldas]

I have a branch under my github account named tbb_in_dev which has all of my latest ongoing work.

A few notes:

• ./bin/dev-shell ./bin/run-test --capture=no -v tests/functional/test_source.py tests are working against an external server. You will have to fill in the functional/instance_information.json with correct details.
• If you see Address not available error while creating the Tor browser driver, it means the Tor could not connect on time.
• If you see capabilities not found or similar error, it means we have different version of geckodriver and firefox-esr.

Current my workstation laptop is not allow me to import gnupg shows an error like following::

>>> import gnupg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/gnupg/__init__.py", line 23, in <module>
    from . import gnupg
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/gnupg/gnupg.py", line 42, in <module>
    from ._meta    import GPGBase
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/gnupg/_meta.py", line 134, in <module>
    class GPGBase(object):
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/gnupg/_meta.py", line 72, in __new__
    if cls._find_agent():
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/gnupg/_meta.py", line 115, in _find_agent
    if (proc.name() == "gpg-agent") and proc.is_running():
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/psutil/__init__.py", line 641, in name
    name = self._proc.name()
  File "/home/kdas/code/sd/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 1340, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: psutil.AccessDenied (pid=2)

[kushaldas]

Today I found out the cause behind our python script failing to download any message/file from the server. The torsocks available in the Ubuntu Trusty is too old, and can not have a custom port as proxy port.

To solve that I have a modified copy of the torsocks shell script added the Dockerfile which works. I am still testing the branch as I managed to destroy my prod instances.

[kushaldas]

It seems we still need the latest torsocks package in the Trusty container image. Any tips anyone how to get that?

[conorsch]

@kushaldas Can you clarify a bit the steps necessary to use the TBB-based tests? Some step-by-step instructions would be quite helpful. I've consulted the README at securedrop/tests/functional/README.md, but still struggling to get the tests running appropriately.

./bin/dev-shell ./bin/run-test --capture=no -v tests/functional/test_source.py tests are working against an external server. You will have to fill in the functional/instance_information.json with correct details.

Here are some questions to kickstart discussion:

1. Is filling out functional/instance_information.json mandatory? From a readthrough of securedrop/tests/functional/functional_test.py, it is not mandatory—defaults will be set to use the dev containers.

2. Are the pip install actions in securedrop/tests/functional/README.md intended to be run on the host machine, i.e. inside the virtualenv used to develop SecureDrop?

3. Are the tbb install instructions up to date? Most of that logic is already in securedrop/Dockerfile, but the comments in the README say "This is yet to be in the Ansible." (I assume Update ansible and Dockerfile for tbb #3592 will resolve.)

It seems we still need the latest torsocks package in the Trusty container image. Any tips anyone how to get that?

You commented separately that using the Bionic torsocks package resolved the issue. That's one way to go, but I would prefer using nc instead of torsocks—the nc version in Trusty should be more than sufficient to set a custom port on the socks5 proxy, and we wouldn't have to fetch any non-dist packages to accommodate.

[kushaldas]

Is filling out functional/instance_information.json mandatory? From a readthrough of securedrop/tests/functional/functional_test.py, it is not mandatory—defaults will be set to use the dev containers.

Yes, the dev container currently do not start a server locally. I was working on that when I got into this mess.

Are the pip install actions in securedrop/tests/functional/README.md intended to be run on the host machine, i.e. inside the virtualenv used to develop SecureDrop?

No, those came in from the old branch of mine, right now everything is done inside of the Docker container.

Are the tbb install instructions up to date? Most of that logic is already in securedrop/Dockerfile, but the comments in the README say "This is yet to be in the Ansible." (I assume #3592 will resolve.)

Yes (inside the container), as I said above that README is from the old branch.

[kushaldas]

To test the current branch, create an instance_information.json pointing to a production instance, and have the journalist user there, then:

./bin/dev-shell ./bin/run-test --capture=no -v tests/functional/test_submit_and_retrieve_message.py

[conorsch]

Running against a separate prod instance helped somewhat, so thanks for clarifying, @kushaldas. For others testing, when I tried to run the tests against the local dev containers, I got a segfault each and every time. Tested on two different kernels, 4.14.51-grsec and 4.14.41-1.pvops.qubes.x86_64, with same results.

These are the steps I took to test locally:

1. git remote add kushaldas https://github.com/kushaldas/securedrop
2. git fetch kushaldas
3. git checkout tbb_in_dev
4. cd securedrop/
5. cp tests/functional/README.md tests/functional/instance_information.json
6. vim tests/functional/instance_information.json
7. Delete everything except JSON example block in instance_information.json.
8. Fill out all fields within JSON block with valid data for a running prod-like instance.
9. ./bin/dev-shell ./bin/run-test --capture=no -v tests/functional/test_submit_and_retrieve_message.py

On step 9, no segfaults, but the test still fails. I do see output including several successful interactions with the remote server, so at a networking level the calls appear to be successful. If I edit the instance_information.json to include bad data on the Onion URLs, I instead see an error about connections failing. That's certainly progress!

As a next step, it would be helpful for collaboration, @kushaldas, if you edited the file at tests/functional/README.md to include accurate information on the testing steps required to get the new TBB Selenium work up and running.

I'll continue to tinker with the setup locally, and see if I can get any of the functional tests to pass on my machine. If you have any tips, @kushaldas, happy to hear them! Also, if you're still blocked on any aspect of this work, please be explicit, so we can team up to finish it up.

[kushaldas]

I have updated the README with the latest Dockerfile based tests.

I tried in two remote vms, one with Fedora 28, and the other (on DigitalOcean) Debian stretch. On both machines I checked out the branch, and filled up the instance_information.json with the prod vm details (prod vms are running on my laptop). The test_submit_and_retrieve_message.py worked successfully.

I am currently working to get a local test instance running for the functional tests (inside of the container).

[kushaldas]

There is also now a local_tests branch in my github. If you run any functional test with instance_information.json file in the source tree, it will run the tests against a local server inside the container.

[conorsch]

During review of #3628, we've focused on debugging the test logic against local containers. No progress has been made recently regarding managing users in the app-staging database. @kushaldas yesterday suggested we can perhaps use create-dev-data.py. That's certainly an option.

Given that we already have the instance_configuration.json method @kushaldas explained above, I propose that's sufficient for for resolving this ticket. We'll still need documentation, as @redshiftzero stated when opening this ticket. The instance_configuration.json approach would require developers to fill out that file manually, rather than automatically creating the accounts. That's reasonable, as part of the goal of targeting external servers with the functional tests is to enable testing of hardware instances, which use the prod playbooks.

[conorsch]

@kushaldas just opened #3671 to handle the creation of users in the staging db. Beyond that, we'll also need to run the command that @kushaldas proposes. For the CI use case, it should be sufficient to modify molecule/aws/side_effect.yml to run the --staging command prior to running the app tests.

[eloquence]

This was an investigation task initially scoped to the pre-release QA process. The remaining work is tracked in #3671 and #3661. Closing this issue.