twister: ztest: short test case names on --no-detailed-test-id

Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.

For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.

This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.

There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.

Signed-off-by: Dmitrii Golovanov <[email protected]>

doc/develop/test/twister.rst

@@ -261,8 +261,7 @@ A Test Suite is a collection of Test Cases which are intended to be used to test
 a software program to ensure it meets certain requirements. The Test Cases in a
 Test Suite are either related or meant to be executed together.
 
-The name of each Test Scenario needs to be unique in the context of the overall
-test application and has to follow basic rules:
+The name of each Test Scenario must follow to these basic rules:
 
 #. The format of the Test Scenario identifier shall be a string without any spaces or
    special characters (allowed characters: alphanumeric and [\_=]) consisting
@@ -272,7 +271,8 @@ test application and has to follow basic rules:
    subsection names delimited with a dot (``.``). For example, a test scenario
    that covers semaphores in the kernel shall start with ``kernel.semaphore``.
 
-#. All Test Scenario identifiers within a ``testcase.yaml`` file need to be unique.
+#. All Test Scenario identifiers within a Test Configuration (``testcase.yaml`` file)
+   need to be unique.
    For example a ``testcase.yaml`` file covering semaphores in the kernel can have:
 
    * ``kernel.semaphore``: For general semaphore tests
@@ -295,6 +295,18 @@ test application and has to follow basic rules:
      Test Case name, for example: ``debug.coredump.logging_backend``.
 
 
+The ``--no-detailed-test-id`` command line option modifies the above mentioned rules:
+
+#. Test Suite name has only ``<Test Scenario identifier>`` component.
+   The Application Project path can be found in ``twister.json`` report (``path:`` property).
+
+#. In this mode, with short Test Suite naming, all Test Scenario names in Twister execution
+   scope (current test plan) must be unique.
+
+#. **Ztest** Test Case name has only Ztest components ``<Ztest suite name>.<Ztest test name>``.
+   The Test Scenario identifier is the parent Test Suite object name.
+
+
 The following is an example test configuration with a few options that are
 explained in this document.
 

scripts/pylib/twister/twisterlib/environment.py

@@ -568,15 +568,21 @@ def add_parse_arguments(parser = None) -> argparse.ArgumentParser:
 
     parser.add_argument(
         '--detailed-test-id', action='store_true',
-        help="Include paths to tests' locations in tests' names. Names will follow "
-             "PATH_TO_TEST/SCENARIO_NAME schema "
-             "e.g. samples/hello_world/sample.basic.helloworld")
+        help="Compose each test Suite name from its configuration path (relative to root) and "
+             "the appropriate Scenario name using PATH_TO_TEST_CONFIG/SCENARIO_NAME schema. "
+             "Also (for Ztest only), prefix each test Case name with its Scenario name. "
+             "For example: 'kernel.common.timing' Scenario with test Suite name "
+             "'tests/kernel/sleep/kernel.common.timing' and 'kernel.common.timing.sleep.usleep' "
+             "test Case (where 'sleep' is its Ztest suite name and 'usleep' is Ztest test name.")
 
     parser.add_argument(
         "--no-detailed-test-id", dest='detailed_test_id', action="store_false",
-        help="Don't put paths into tests' names. "
-             "With this arg a test name will be a scenario name "
-             "e.g. sample.basic.helloworld.")
+        help="Don't prefix each test Suite name with its configuration path, "
+             "so it is the same as the appropriate Scenario name. "
+             "Also (for Ztest only), don't prefix each Ztest Case name with its Scenario name. "
+             "For example: 'kernel.common.timing' Scenario name, the same Suite name, "
+             "and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name "
+             "and 'usleep' is Ztest test name.")
 
     # Include paths in names by default.
     parser.set_defaults(detailed_test_id=True)

scripts/pylib/twister/twisterlib/harness.py

@@ -745,13 +745,13 @@ def get_testcase(self, tc_name, phase, ts_name=None):
         for ts_name_ in ts_names:
             if self.started_suites[ts_name_]['count'] < (0 if phase == 'TS_SUM' else 1):
                 continue
-            tc_fq_id = "{}.{}.{}".format(self.id, ts_name_, tc_name)
+            tc_fq_id = self.instance.compose_case_name(f"{ts_name_}.{tc_name}")
             if tc := self.instance.get_case_by_name(tc_fq_id):
                 if self.trace:
                     logger.debug(f"On {phase}: Ztest case '{tc_name}' matched to '{tc_fq_id}")
                 return tc
         logger.debug(f"On {phase}: Ztest case '{tc_name}' is not known in {self.started_suites} running suite(s).")
-        tc_id = "{}.{}".format(self.id, tc_name)
+        tc_id = self.instance.compose_case_name(tc_name)
         return self.instance.get_case_or_create(tc_id)
 
     def start_suite(self, suite_name):

scripts/pylib/twister/twisterlib/runner.py

@@ -1104,8 +1104,7 @@ def demangle(self, symbol_name):
         return symbol_name
 
     def determine_testcases(self, results):
-        yaml_testsuite_name = self.instance.testsuite.id
-        logger.debug(f"Determine test cases for test suite: {yaml_testsuite_name}")
+        logger.debug(f"Determine test cases for test suite: {self.instance.testsuite.id}")
 
         elf_file = self.instance.get_elf_file()
         elf = ELFFile(open(elf_file, "rb"))
@@ -1132,7 +1131,7 @@ def determine_testcases(self, results):
                         logger.warning(f"Unexpected Ztest suite '{new_ztest_suite}' "
                                        f"not present in: {self.instance.testsuite.ztest_suite_names}")
                     test_func_name = m_[2].replace("test_", "", 1)
-                    testcase_id = f"{yaml_testsuite_name}.{new_ztest_suite}.{test_func_name}"
+                    testcase_id = self.instance.compose_case_name(f"{new_ztest_suite}.{test_func_name}")
                     detected_cases.append(testcase_id)
 
         if detected_cases:
@@ -1141,16 +1140,15 @@ def determine_testcases(self, results):
             self.instance.testcases.clear()
             self.instance.testsuite.testcases.clear()
 
-            # When the old regex-based test case collection is fully deprecated,
-            # this will be the sole place where test cases get added to the test instance.
-            # Then we can further include the new_ztest_suite info in the testcase_id.
-
             for testcase_id in detected_cases:
                 testcase = self.instance.add_testcase(name=testcase_id)
                 self.instance.testsuite.add_testcase(name=testcase_id)
 
                 # Keep previous statuses and reasons
                 tc_info = tc_keeper.get(testcase_id, {})
+                if not tc_info:
+                    logger.warning(f"Ztest case '{testcase_id}' is not expected - "
+                                   f"'{self.instance.testsuite.source_dir_rel}' has {list(tc_keeper)}")
                 testcase.status = tc_info.get('status', TwisterStatus.NONE)
                 testcase.reason = tc_info.get('reason')
 

scripts/pylib/twister/twisterlib/testinstance.py

@@ -163,6 +163,9 @@ def __setstate__(self, d):
     def __lt__(self, other):
         return self.name < other.name
 
+    def compose_case_name(self, tc_name) -> str:
+        return self.testsuite.compose_case_name(tc_name)
+
     def set_case_status_by_name(self, name, status, reason=None):
         tc = self.get_case_or_create(name)
         tc.status = status

scripts/pylib/twister/twisterlib/testsuite.py

@@ -455,20 +455,25 @@ def load(self, data):
         if self.harness == 'console' and not self.harness_config:
             raise Exception('Harness config error: console harness defined without a configuration.')
 
+    @staticmethod
+    def compose_case_name_(test_suite, tc_name) -> str:
+        return f"{test_suite.id}.{tc_name}" if test_suite.detailed_test_id else f"{tc_name}"
+
+    def compose_case_name(self, tc_name) -> str:
+        return self.compose_case_name_(self, tc_name)
+
     def add_subcases(self, data, parsed_subcases=None, suite_names=None):
         testcases = data.get("testcases", [])
         if testcases:
             for tc in testcases:
-                self.add_testcase(name=f"{self.id}.{tc}")
+                self.add_testcase(name=self.compose_case_name(tc))
         else:
             if not parsed_subcases:
                 self.add_testcase(self.id, freeform=True)
             else:
                 # only add each testcase once
-                for sub in set(parsed_subcases):
-                    name = "{}.{}".format(self.id, sub)
-                    self.add_testcase(name)
-
+                for tc in set(parsed_subcases):
+                    self.add_testcase(name=self.compose_case_name(tc))
         if suite_names:
             self.ztest_suite_names = suite_names
 

scripts/tests/twister/test_harness.py

@@ -30,6 +30,7 @@
     Test,
 )
 from twisterlib.statuses import TwisterStatus
+from twisterlib.testsuite import TestSuite
 from twisterlib.testinstance import TestInstance
 
 GTEST_START_STATE = " RUN      "
@@ -594,6 +595,7 @@ def test_get_harness(name):
 
 TEST_DATA_7 = [
     (
+        True,
         "",
         "Running TESTSUITE suite_name",
         ["suite_name"],
@@ -604,16 +606,18 @@ def test_get_harness(name):
         TwisterStatus.NONE,
     ),
     (
+        True,
         "On TC_START: Ztest case 'testcase' is not known in {} running suite(s)",
         "START - test_testcase",
         [],
         {},
-        { 'test_id.testcase': { 'count': 1 } },
+        { 'dummy.test_id.testcase': { 'count': 1 } },
         TwisterStatus.STARTED,
         True,
         TwisterStatus.NONE
     ),
     (
+        True,
         "On TC_END: Ztest case 'example' is not known in {} running suite(s)",
         "PASS - test_example in 0 seconds",
         [],
@@ -624,6 +628,7 @@ def test_get_harness(name):
         TwisterStatus.NONE,
     ),
     (
+        True,
         "On TC_END: Ztest case 'example' is not known in {} running suite(s)",
         "SKIP - test_example in 0 seconds",
         [],
@@ -634,6 +639,7 @@ def test_get_harness(name):
         TwisterStatus.NONE,
     ),
     (
+        True,
         "On TC_END: Ztest case 'example' is not known in {} running suite(s)",
         "FAIL - test_example in 0 seconds",
         [],
@@ -644,21 +650,34 @@ def test_get_harness(name):
         TwisterStatus.NONE,
     ),
     (
-        "not a ztest and no state for test_id",
+        True,
+        "not a ztest and no state for dummy.test_id",
         "START - test_testcase",
         [],
         {},
-        { 'test_id.testcase': { 'count': 1 } },
+        { 'dummy.test_id.testcase': { 'count': 1 } },
         TwisterStatus.PASS,
         False,
         TwisterStatus.PASS,
     ),
     (
-        "not a ztest and no state for test_id",
+        False,
+        "not a ztest and no state for dummy.test_id",
         "START - test_testcase",
         [],
         {},
-        { 'test_id.testcase': { 'count': 1 } },
+        { 'testcase': { 'count': 1 } },
+        TwisterStatus.PASS,
+        False,
+        TwisterStatus.PASS,
+    ),
+    (
+        True,
+        "not a ztest and no state for dummy.test_id",
+        "START - test_testcase",
+        [],
+        {},
+        { 'dummy.test_id.testcase': { 'count': 1 } },
         TwisterStatus.FAIL,
         False,
         TwisterStatus.FAIL,
@@ -667,12 +686,12 @@ def test_get_harness(name):
 
 
 @pytest.mark.parametrize(
-    "exp_out, line, exp_suite_name, exp_started_suites, exp_started_cases, exp_status, ztest, state",
+    "detailed_id, exp_out, line, exp_suite_name, exp_started_suites, exp_started_cases, exp_status, ztest, state",
     TEST_DATA_7,
-    ids=["testsuite", "testcase", "pass", "skip", "failed", "ztest pass", "ztest fail"],
+    ids=["testsuite", "testcase", "pass", "skip", "failed", "ztest pass", "ztest pass short id", "ztest fail"],
 )
 def test_test_handle(
-    tmp_path, caplog, exp_out, line,
+    tmp_path, caplog, detailed_id, exp_out, line,
     exp_suite_name, exp_started_suites, exp_started_cases,
     exp_status, ztest, state
 ):
@@ -682,24 +701,27 @@ def test_test_handle(
     mock_platform.name = "mock_platform"
     mock_platform.normalized_name = "mock_platform"
 
-    mock_testsuite = mock.Mock(id="id", testcases=[])
-    mock_testsuite.name = "mock_testsuite"
+    mock_testsuite = mock.Mock(id="dummy.test_id", testcases=[])
+    mock_testsuite.name = "dummy_suite/dummy.test_id"
     mock_testsuite.harness_config = {}
     mock_testsuite.ztest_suite_names = []
-
-    outdir = tmp_path / "gtest_out"
-    outdir.mkdir()
-
-    instance = TestInstance(
-        testsuite=mock_testsuite, platform=mock_platform, outdir=outdir
-    )
+    mock_testsuite.detailed_test_id = detailed_id
+    mock_testsuite.source_dir_rel = "dummy_suite"
+    mock_testsuite.compose_case_name.return_value = TestSuite.compose_case_name_(mock_testsuite, "testcase")
+
+    outdir = tmp_path / "ztest_out"
+    with mock.patch('twisterlib.testsuite.TestSuite.get_unique', return_value="dummy_suite"):
+        instance = TestInstance(
+            testsuite=mock_testsuite, platform=mock_platform, outdir=outdir
+        )
 
     test_obj = Test()
     test_obj.configure(instance)
-    test_obj.id = "test_id"
+    test_obj.id = "dummy.test_id"
     test_obj.ztest = ztest
     test_obj.status = state
-    test_obj.id = "test_id"
+    test_obj.started_cases = {}
+
     # Act
     test_obj.handle(line)