diff --git a/docs/marks.rst b/docs/marks.rst index 436fb480..a7d848b8 100644 --- a/docs/marks.rst +++ b/docs/marks.rst @@ -113,42 +113,99 @@ You can then run tests that are relevant only to the selected ticket: cd src/tests/system pytest --mh-config=mhc.yaml --mh-lazy-ssh -v --ticket=gh#3433 -@pytest.mark.tier -================= +@pytest.mark.importance +======================= -The `tier mark `__ can -associate a test with a specific tier. +The `importance mark `__ can +associate a test with a level of importance. This is used by quality engineers to +prioritize and group the test determined by the level of impact to the customer. -The ``@pytest.mark.tier`` takes single number as an argument. +The ``@pytest.mark.importance`` takes a string as an argument. The values used +are "critical", "high", "medium" and "low". .. code-block:: python :caption: Examples - @pytest.mark.tier(0) - def test_tier0() + @pytest.mark.importance("critical") + def test_importance_critical() pass - @pytest.mark.tier(1) - def test_tier1() + @pytest.mark.importance("high") + def test_importance_high() pass -You can then run tests that are relevant only to the selected ticket: +You can then run the tests by importance: .. code-block:: text cd src/tests/system - pytest --mh-config=mhc.yaml --mh-lazy-ssh -v --tier=1 + pytest --mh-config=mhc.yaml --mh-lazy-ssh -v --importance="critical" -Tier definition -=============== +Importance definition +===================== -.. list-table:: Tier definition - :align: center +.. list-table:: :widths: 10 90 :header-rows: 1 :stub-columns: 1 - * - Tier + * - Importance - Description - * - @TODO - - + * - critical + - Core subset of tests that covers all operational features. This is used for gating where it maybe ran several + times a day. To manage resources, the execution all these tests should be kept under an hour. + * - high + - The most comprehensive set of tests, that covers all operational features. + * - medium + - Tests that do not impact the operational function, like the CLI commands included in sss-tools package. + * - low + - Tests that may have a long execution time or are heavy on resources so will be executed during our comprehensive + test cycle. + +@pytest.mark.custom +=================== + +The use of `custom markers `__ +help group tests into categories; identity, authentication, authorization are +some examples. + +.. code-block:: python + :caption: Examples + + @pytest.mark.authentication + def test_authenticate_user() + pass + +You can run tests by custom markers: + +.. code-block:: text + + cd src/tests/system + pytest --mh-config-mhc=mhc.yaml --mh-lazy-ssh -v -m authentication + +Custom Marker definitions +========================= + +.. list-table:: + :widths: 20 80 + :header-rows: 1 + :stub-columns: 1 + + * - Marker + - Description + * - authentication + - Tests checking user and password policies and anything to do with the login prompt. + * - authorization + - Tests checking user access after login like sudo and credential delegation. + * - cache + - Test checking the local cache like timeout, negative cache and refresh. + * - contains_workaround + - Test requires workaround for an existing bug. (gh=...,bz=...) + * - identity + - Tests checking user identity lookups, group memberships, domain priority and id mapping. + * - schema + - Tests checking ldap schemas, rfc2307, rfc2307bis, AD and directory attributes. + * - slow + - Tests that are slow. (deselect with '-m "not slow"') + * - tools + - Tests for all SSSD CLI commands; sssctl, sss_cache, sss_override, etc.