Implement pytest-randomly in CI and Development workflows

[shreyb]

While looking to improve unit tests, I checked out a plugin (thanks to the @talkpython podcast for recommending it) called pytest-randomly. This plugin to pytest randomizes the tests in a test suite, so we can be sure that the unit tests are not order-dependent. Once installed, pytest can automatically pick it up and will randomize the tests that pytest selects.

It turns out that we DO have about 15% of our unit tests that fail if called out of order:

Without pytest-randomly:

$  python3 -m pytest -m unit
========================================== test session starts ===========================================
platform linux -- Python 3.6.8, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/sbhat/jobsub_lite
collected 163 items / 44 deselected / 119 selected
....test output...
====================== 119 passed, 44 deselected, 118 warnings in 119.56s (0:01:59) ======================

After pytest-randomly installed:

$  python3 -m pytest -m unit
========================================== test session starts ===========================================
platform linux -- Python 3.6.8, pytest-7.0.1, pluggy-1.0.0
Using --randomly-seed=2174338830
rootdir: /home/sbhat/jobsub_lite
plugins: randomly-3.10.3
collected 163 items / 44 deselected / 119 selected
....test output...
======================================== short test summary info =========================================
FAILED tests/test_creds_unit.py::TestCredUnit::test_get_creds_proxy_only - KeyError: 'HOME'
FAILED tests/test_creds_unit.py::TestCredUnit::test_get_creds_token_only - KeyError: 'HOME'
FAILED tests/test_creds_unit.py::TestCredUnit::test_get_creds_default_role - KeyError: 'HOME'
FAILED tests/test_creds_unit.py::TestCredUnit::test_get_creds_invalid_auth - KeyError: 'HOME'
FAILED tests/test_creds_unit.py::TestCredUnit::test_get_creds_1 - KeyError: 'HOME'
FAILED tests/test_tarfiles_unit.py::TestTarfilesUnit::test_dcache_persistent_path_1 - KeyError: 'USER'
FAILED tests/test_jobsub_submit_unit.py::TestJobsubSubmitUnit::test_do_dataset_defaults_1 - KeyError: '...
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_get_principal_1 - KeyError: 'USER'
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--site FERMIGRID --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC,OFFSITE (wrong case for fermigrid)]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--onsite --resource_provides=usage_model=DEDICATED,OFFSITE --resource-provides=IWANT=this_resource]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--site=Fermigrid,Random_Site --resource-provides=usage_model=DEDICATED,OFFSITE]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--site=Fermigrid,Random_Site --resource_provides=usage_model=DEDICATED,OFFSITE --resource-provides=IWANT=this_resource]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--site=Fermigrid,Random_Site]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[onsite] - AssertionEr...
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--onsite --resource_provides=usage_model=DEDICATED,OFFSITE]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--resource-provides=usage_model=OFFSITE --site Fermigrid]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--resource-provides=usage_model=DEDICATED,OFFSITE --site Fermigrid]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--onsite --resource-provides=usage_model=DEDICATED,OFFSITE]
FAILED tests/test_utils_unit.py::TestUtilsUnit::test_resolve_site_and_usage_model[--site Fermigrid] - A...
ERROR tests/test_packages_unit.py::TestPackagesUnit::test_pkg_orig_env_1 - KeyError: 'SETUPS_CVMFS_SAVE'
ERROR tests/test_tarfiles_unit.py::TestTarfilesUnit::test_tarfile_publisher_glob_path - Exception: No k...
ERROR tests/test_tarfiles_unit.py::TestTarfilesUnit::test_do_tarballs_1 - Exception: No kerberos creden...
ERROR tests/test_tarfiles_unit.py::TestTarfilesUnit::test_do_tarballs_3 - Exception: No kerberos creden...
ERROR tests/test_tarfiles_unit.py::TestTarfilesUnit::test_tarfile_publisher_1 - Exception: No kerberos ...
ERROR tests/test_tarfiles_unit.py::TestTarfilesUnit::test_do_tarballs_2 - Exception: No kerberos creden...
ERROR tests/test_utils_unit.py::TestUtilsUnit::test_get_client_dn_no_proxy_provided_no_env - Exception:...
ERROR tests/test_utils_unit.py::TestUtilsUnit::test_get_client_dn_env_plus_proxy_provided - Exception: ...
ERROR tests/test_utils_unit.py::TestUtilsUnit::test_set_extras_1 - Exception: No kerberos credentials f...
ERROR tests/test_utils_unit.py::TestUtilsUnit::test_get_client_dn_no_proxy_provided - Exception: No ker...
ERROR tests/test_utils_unit.py::TestUtilsUnit::test_get_client_dn_valid_proxy_provided - Exception: No ...
================= 19 failed, 90 passed, 44 deselected, 118 warnings, 11 errors in 27.92s =================

This proposal is three parts:

1. We should go through and fix these tests so they aren't order-dependent. The metric for success will be that with pytest-randomly installed on a dev machine, unit tests run with python3 -m pytest -m unit.
2. We currently run our integration and unit tests by hand before PRs are open. The reason we haven't added these to CI is mainly due to credential management. We could for the time-being mark unit tests as needing credentials or not, and then run just those that don't need credentials as either another pre-commit hook (so it gets run with each commit by a dev), or as a part of a github action.
3. Our "Development Workflow" currently has nothing. We should at least sketch out a skeleton there where we encourage devs to use pre-commit and pytest-randomly.