doc: added importance and custom markers section

[danlavu]

No description provided.

[jakub-vavra-cz]

Hello, looks mostly good.
We might want to add also "security" marker from the start and label tests that cover logins with wrong password, expired/disabled accounts eg. . WDYIT @pbrezina?

[jakub-vavra-cz]

LGTM

[danlavu]

@jakub-vavra-cz , some of it should be under authentication, and we don't have any access control tests ported in yet, so that group will be added eventually.

[pbrezina]

Hello, looks mostly good. We might want to add also "security" marker from the start and label tests that cover logins with wrong password, expired/disabled accounts eg. . WDYIT @pbrezina?

AFAIK this is the "authentication" or "authorization" marker. So my opinion is to either use these, or scratch them in favor of "security".

---

From my upstream point of view, both importance and custom markers do not really sense. Before release and before pull request push, all tests must be green. So IIUIC it is more related to how you want to run the tests in downstream and this should be explained here so however reads it knows the meaning of these markers

[danlavu]

@pbrezina, it's simply a way to organize the tests when we execute regression runs, so we can run a subset of tests that a patch, build, and errata effects without executing all tests. Gating, for example, gating is supposed to be a quick, core tests that touch all features, aka "critical". While "high" will be a comprehensive run of the features. Medium will be features that are not used all the time or will not impact customer's operations if they fail, like the CLI tools, sssctl, sss_override.

I'll expand the documentation and add this kind of thing to the doc.

[pbrezina]

@pbrezina, it's simply a way to organize the tests when we execute regression runs, so we can run a subset of tests that a patch, build, and errata effects without executing all tests. Gating, for example, gating is supposed to be a quick, core tests that touch all features, aka "critical". While "high" will be a comprehensive run of the features. Medium will be features that are not used all the time or will not impact customer's operations if they fail, like the CLI tools, sssctl, sss_override.

Yes, exactly this kind of thing should go to the docs.

I'll expand the documentation and add this kind of thing to the doc.

[jakub-vavra-cz]

@pbrezina, @sidecontrol as for difference between critical, high, medium, low.
The categorization of tests is important internally for dividing test to tiers (that will have a different frequence of execution), it is useful to evaluate seriousness of possible failure. (Failure of low importance test will have a different impact that critical test.

"Critical" tests should contain only small sample of tests for the most important functionality.
These should have very short runtime so they can be run often (like on PRs).
They should be covering only the "sunny day" (positive test) scenarios.

"High" tests that cover all functionality properly still focusing on positive scenarios.
Consider tests with properly configured sssd and kerberos, automount, ... making sure that everything works properly.

"Medium" tests comprehensively including some less used functionality and more complex setups.
Tests with slightly misconfigured sssd, less common setups, basic failover.

"Low" tests for a rarely used functionality, edge cases and very specific complex setups, performance/stress tests.
Tests where the sssd / environment is misconfigured.

[danlavu]

@jakub-vavra-cz , how does this sound?

* - critical
  - Core subset of tests that covers all operational features. This is used for gating where it maybe ran several
    times a day. To manage resources, the execution all these tests should be kept under an hour.
* - high
  - The most comprehensive set of tests, that covers all operational features.
* - medium
  - Tess that do not impact operational functionality, like the CLI commands included in sss-tools package.
* - low
  - Tests that may have a long execution time, edge cases or complex scenarios that demand a lot of resources.

[jakub-vavra-cz]

Hi @sidecontrol ,
I would shift some stuff from critical to high. Gating is Crit+High, I would keep Critical for smoke tests that we will execute
on commits/PRs.

• • critical
  • Core subset of tests that covers most important operational features. This is used in pipelines where it maybe ran multiple times a day. The execution should be kept as short as possible.
• • high
  • The comprehensive set of tests, that covers all operational features. This is used for gating where it maybe ran several times a day. To manage resources, the execution all these tests should be kept under an hour.
• • medium
  • Extended set that covers tests that do not impact operational functionality, like the CLI commands included in sss-tools package. Tests that cover negative scenarios and misconfigured environment fit here as well.
• • low
  • Tests that may have a long execution time, edge cases or complex scenarios that demand a lot of resources. Consider performance and stress tests as part of this set.

[danlavu]

Updated.

[jakub-vavra-cz]

LGTM

[pbrezina]

Thanks, see minor comments inside.

[patriki01]

Hi @sidecontrol,
It seems that you forgot to put config marker to your original PR to Testing Framework. Jakub has already added the marker. Could you please add it to the docs as well?
Thanks

[danlavu]

@patriki01 Done.

[jakub-vavra-cz]

LGTM