Added scripts to get metadata (runner arguments) associated with the …

…python_testing scripts for automating the process (project-chip#32752)

* Create unit test to test metadata parser

* Create test script metadata parser

* Create test environment file

* Added Copyright Code

* Added Copyright code

* Added copyright code

* Added run arguments comment header

* Created BUILD.gn for metadata parser

* Added reference to the metadata parser BUILD.gn

* Added TC_RR_1_1_Test.py for testing purpose

* Reverted back the testing changes done to the original file

* Updated TC_RR_1_1_Test.py with comments as it was

* Update test_metadata.py to correct the location for TC_RR* test script

* Updated BUILD.gn with the correct naming tests

* Rename env.yaml to test_env.yaml as it is for testing only

* Updated test_metadata.py with the corrected test env file

* Updated and rename TC_RR_1_1_Test.py to simple_run_args.py

Just keeping the runner arguments required for testing purpose and removing all the unwanted test script code for this current testing use case

* Updated test_metadata.py to the new name reference of the testing file

* Renamed test method to test_run_metadata_parser

* Removed the unwanted print method

* Removed handling for int/bool/hex args and trace args

* Remove try-catch block

* Replaced == to is for null comparsion

* Removed argument validation method

* Added the dataclass to be optional

* Updated the test script to be a .txt file and changed the designed to be all run arguments in one line

* Rename simple_run_args.py to simple_run_args.txt

* Updated BUILD.gn

* Rename scripts/tests/py/test_env.yaml to env_example.yaml

* Rename env_example.yaml to scripts/tests/py/env_example.yaml

* Updated path of the environment file

* Change the test script file type

* Updated metadata_parser reference

* Rename env_example.yaml to env_test.yaml

* Update environment file name

* Created __init__.py file

* Added reference for __init__.py in the source

* Created pyproject.toml file

* Created setup.py file

* Created setup.cfg file

* Changed the path of the folder where the scripts are located

* Added print statement

* Updated print statement

* Added inputs to the BUILD.gn file

* Updated paths for file references

* Added test script path to the inputs

* Added sources for the file not find error fix

* Updated metadata.py changes as suggested in the PR comments review

* Updated test script

* Updated license comment

* Removed from pathlib import Path

* Added Print Statement

* Removed the print debug line

* Updated it to use relative path instead of absolute path

* Changed copyright year and removed unwanted sources

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Update comment

Co-authored-by: Andrei Litvin <[email protected]>

* Updated the class name to MetadataReader

* Updated class name to MetadataReader

* Removed optional on factoryreset

* Renamed env_yaml_file_path and solved inline inside constructor

* Update metadata.py

* Update metadata.py

* Delete scripts/tests/py/simple_run_args.txt

* Update BUILD.gn

* Updated test_metadata.py

* Updated metadata.py

* Create simple_run_args.txt

* Update BUILD.gn

* Delete scripts/tests/py/simple_run_args.txt

* Update BUILD.gn

* Updated test_metadata.py

* Restyled by whitespace

* Restyled by gn

* Restyled by prettier-yaml

* Restyled by autopep8

* Restyled by isort

* Fixed the spell error

* Fixed comments from the PR

* Removed extra print line

* Restyled by autopep8

* Test via comment space

* Test via removed comment space

* Test via comment space

* Restyled by autopep8

* Fixed mypy erros and test script line readability fixes

* Restyled by autopep8

* Removed unawanted variables

* Removing unused Union

* Restyled by autopep8

---------

Co-authored-by: Andrei Litvin <[email protected]>
Co-authored-by: Restyled.io <[email protected]>

BUILD.gn

@@ -233,6 +233,7 @@ if (current_toolchain != "${dir_pw_toolchain}/default:default") {
         "//scripts/build:build_examples.tests",
         "//scripts/py_matter_idl:matter_idl.tests",
         "//scripts/py_matter_yamltests:matter_yamltests.tests",
+        "//scripts/tests/py:metadata_parser.tests",
         "//src:tests_run",
       ]
 

scripts/tests/py/BUILD.gn

@@ -0,0 +1,36 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import("//build_overrides/build.gni")
+import("//build_overrides/chip.gni")
+
+import("//build_overrides/pigweed.gni")
+import("$dir_pw_build/python.gni")
+
+pw_python_package("metadata_parser") {
+  setup = [
+    "setup.py",
+    "setup.cfg",
+    "pyproject.toml",
+  ]
+
+  inputs = [ "env_test.yaml" ]
+
+  sources = [
+    "__init__.py",
+    "metadata.py",
+  ]
+
+  tests = [ "test_metadata.py" ]
+}

scripts/tests/py/env_test.yaml

@@ -0,0 +1,32 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# test environment argument definitions for metadata parser
+discriminator: 1234
+KVS: kvs1
+trace-to-appjson: json:out/trace_data/app-{SCRIPT_BASE_NAME}.json
+enable-key: 000102030405060708090a0b0c0d0e0f
+
+storage-path: admin_storage.json
+commissioning-method: on-network
+passcode: 20202021
+endpoint: 1
+trace-to-testjson: json:out/trace_data/test-{SCRIPT_BASE_NAME}.json
+trace-to-testperfetto: perfetto:out/trace_data/test-{SCRIPT_BASE_NAME}.perfetto
+manual-code: 10054912339
+PICS: src/app/tests/suites/certification/ci-pics-values
+
+app:
+    all-clusters: out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app
+    lock: out/linux-x64-lock-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-lock-app

scripts/tests/py/metadata.py

@@ -0,0 +1,217 @@
+#!/usr/bin/python3
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+from dataclasses import dataclass
+from typing import Dict, List, Optional
+
+import yaml
+
+
+@dataclass
+class Metadata:
+    py_script_path: Optional[str] = None
+    run: Optional[str] = None
+    app: Optional[str] = None
+    discriminator: Optional[str] = None
+    passcode: Optional[str] = None
+
+    def copy_from_dict(self, attr_dict: Dict[str, str]) -> None:
+        """
+        Sets the value of the attributes from a dictionary.
+
+        Attributes:
+
+        attr_dict:
+          Dictionary that stores attributes value that should
+          be transferred to this class.
+        """
+
+        if "app" in attr_dict:
+            self.app = attr_dict["app"]
+
+        if "run" in attr_dict:
+            self.run = attr_dict["run"]
+
+        if "discriminator" in attr_dict:
+            self.discriminator = attr_dict["discriminator"]
+
+        if "passcode" in attr_dict:
+            self.passcode = attr_dict["passcode"]
+
+        if "py_script_path" in attr_dict:
+            self.py_script_path = attr_dict["py_script_path"]
+
+        # TODO - set other attributes as well
+
+
+class MetadataReader:
+    """
+    A class to parse run arguments from the test scripts and 
+    resolve them to environment specific values.
+    """
+
+    def __init__(self, env_yaml_file_path: str):
+        """
+        Reads the YAML file and Constructs the environment object
+
+        Parameters:
+
+        env_yaml_file_path:
+          Path to the environment file that contains the YAML configuration.
+        """
+        with open(env_yaml_file_path) as stream:
+            self.env = yaml.safe_load(stream)
+
+    def __resolve_env_vals__(self, metadata_dict: Dict[str, str]) -> None:
+        """
+        Resolves the argument defined in the test script to environment values.
+        For example, if a test script defines "all_clusters" as the value for app
+        name, we will check the environment configuration to see what raw value is
+        assocaited with the "all_cluster" variable and set the value for "app" option
+        to this raw value.
+
+        Parameter:
+
+        metadata_dict:
+          Dictionary where each key represent a particular argument and its value represent
+          the value for that argument defined in the test script.
+        """
+
+        for run_arg, run_arg_val in metadata_dict.items():
+
+            if not type(run_arg_val) == str or run_arg == "run":
+                metadata_dict[run_arg] = run_arg_val
+                continue
+
+            if run_arg_val is None:
+                continue
+
+            sub_args = run_arg_val.split('/')
+
+            if len(sub_args) not in [1, 2]:
+                err = """The argument is not in the correct format. 
+                The argument must follow the format of arg1 or arg1/arg2. 
+                For example, arg1 represents the argument type and optionally arg2 
+                represents a specific variable defined in the environment file whose
+                value should be used as the argument value. If arg2 is not specified,
+                we will just use the first value associated with arg1 in the environment file."""
+                raise Exception(err)
+
+            if len(sub_args) == 1:
+                run_arg_val = self.env.get(sub_args[0])
+
+            elif len(sub_args) == 2:
+                run_arg_val = self.env.get(sub_args[0]).get(sub_args[1])
+
+            # if a argument has been specified in the comment header
+            # but can't be found in the env file, consider it to be
+            # boolean value.
+            if run_arg_val is None:
+                run_arg_val = True
+
+            metadata_dict[run_arg] = run_arg_val
+
+    def __read_args__(self, run_args_lines: List[str]) -> Dict[str, str]:
+        """
+        Parses a list of lines and extracts argument
+        values from it.
+
+        Parameters:
+
+        run_args_lines:
+          Line in test script header that contains run argument definition.
+          Each line will contain a list of run arguments separated by a space.
+          Line below is one example of what the run argument line will look like:
+          "app/all-clusters discriminator KVS storage-path"
+
+          In this case the line defines that app, discriminator, KVS, and storage-path
+          are the arguments that should be used with this run.
+
+          An argument can be defined multiple times in the same line or in different lines.
+          The last definition will override any previous definition. For example,
+          "KVS/kvs1 KVS/kvs2 KVS/kvs3" line will lead to KVS value of kvs3.
+        """
+        metadata_dict = {}
+
+        for run_line in run_args_lines:
+            for run_arg_word in run_line.strip().split():
+                '''
+                We expect the run arg to be defined in one of the 
+                following two formats:
+                1. run_arg
+                2. run_arg/run_arg_val
+
+                Examples: "discriminator" and "app/all_clusters"
+
+                '''
+                run_arg = run_arg_word.split('/', 1)[0]
+                metadata_dict[run_arg] = run_arg_word
+
+        return metadata_dict
+
+    def parse_script(self, py_script_path: str) -> List[Metadata]:
+        """
+        Parses a script and returns a list of metadata object where
+        each element of that list representing run arguments associated
+        with a particular run.
+
+        Parameter:
+
+        py_script_path:
+          path to the python test script
+
+        Return:
+
+        List[Metadata]
+          List of Metadata object where each Metadata element represents
+          the run arguments associated with a particular run defined in
+          the script file.
+        """
+
+        runs_def_ptrn = re.compile(r'^\s*#\s*test-runner-runs:\s*(.*)$')
+        args_def_ptrn = re.compile(r'^\s*#\s*test-runner-run/([a-zA-Z0-9_]+):\s*(.*)$')
+
+        runs_arg_lines: Dict[str, List[str]] = {}
+        runs_metadata = []
+
+        with open(py_script_path, 'r', encoding='utf8') as py_script:
+            for line in py_script.readlines():
+
+                runs_match = runs_def_ptrn.match(line.strip())
+                args_match = args_def_ptrn.match(line.strip())
+
+                if runs_match:
+                    for run in runs_match.group(1).strip().split():
+                        runs_arg_lines[run] = []
+
+                elif args_match:
+                    runs_arg_lines[args_match.group(1)].append(args_match.group(2))
+
+        for run, lines in runs_arg_lines.items():
+            metadata_dict = self.__read_args__(lines)
+            self.__resolve_env_vals__(metadata_dict)
+
+            # store the run value and script location in the
+            # metadata object
+            metadata_dict['py_script_path'] = py_script_path
+            metadata_dict['run'] = run
+
+            metadata = Metadata()
+
+            metadata.copy_from_dict(metadata_dict)
+            runs_metadata.append(metadata)
+
+        return runs_metadata

scripts/tests/py/pyproject.toml

@@ -0,0 +1,17 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[build-system]
+requires = ['setuptools', 'wheel']
+build-backend = 'setuptools.build_meta'

scripts/tests/py/setup.cfg

@@ -0,0 +1,19 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[metadata]
+name = metadata_parser
+version = 0.0.1
+author = Project CHIP Authors
+description = Scripts to get metadata (runner arguments) associated with the python_testing scripts

scripts/tests/py/setup.py

@@ -0,0 +1,20 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""The metadata_parser package."""
+
+import setuptools  # type: ignore
+
+setuptools.setup()  # Package definition in setup.cfg