Fixed python test runner code to error out if there are 0 test runs (#…

…34164) * Create python_test_arguments.txt * Fixed runner code that 0 runs are an error * Restyled by autopep8 * Update hello_test.py * Update python_test_arguments.txt * Update python.md * Update python.md * Delete docs/testing/python_test_arguments.txt * Restyled by prettier-markdown * Update python.md * Update python.md * Update run_python_test.py * Restyled by prettier-markdown * Update python.md * Restyled by prettier-markdown --------- Co-authored-by: Restyled.io <[email protected]>
project-chip · Jul 3, 2024 · f8a633e · f8a633e
1 parent 7557e54
commit f8a633e
Show file tree

Hide file tree

Showing 3 changed files with 76 additions and 1 deletion.
diff --git a/docs/testing/python.md b/docs/testing/python.md
@@ -19,6 +19,11 @@ Python tests located in src/python_testing
 
 ## Writing Python tests
 
+-   Defining arguments in the test script
+    -   In order to streamline the configuration and execution of tests, it is
+        essential to define arguments at the top of the test script. This
+        section should include various parameters and their respective values,
+        which will guide the test runner on how to execute the tests.
 -   All test classes inherit from MatterBaseTest in
     [matter_testing_support.py](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/matter_testing_support.py)
     -   support for commissioning using the python controller
@@ -36,11 +41,17 @@ Python tests located in src/python_testing
 -   Use Mobly assertions for failing tests
 -   self.step() along with a steps\_ function to mark test plan steps for cert
     tests
--
 
 ### A simple test
 
 ```
+# test-runner-runs: run1
+# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
+# test-runner-run/run1/factoryreset: True
+# test-runner-run/run1/quiet: True
+# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
+# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
 class TC_MYTEST_1_1(MatterBaseTest):
 
     @async_test_body
@@ -74,6 +85,59 @@ The default_matter_test_main() function is used to run the test on the command
 line. These two lines should appear verbatim at the bottom of every python test
 file.
 
+## Defining the test arguments
+
+Below is the format:
+
+```
+# test-runner-runs: <run_identifier>
+# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
+# test-runner-run/<run_identifier>/factoryreset: <True|False>
+# test-runner-run/<run_identifier>/quiet: <True|False>
+# test-runner-run/<run_identifier>/app-args: <app_arguments>
+# test-runner-run/<run_identifier>/script-args: <script_arguments>
+```
+
+### Description of Parameters
+
+-   test-runner-runs: Specifies the identifier for the run. This can be any
+    unique identifier.
+
+    -   Example: run1
+
+-   test-runner-run/<run_identifier>/app: Indicates the application to be used
+    in the test. Different app types as needed could be referenced from section
+    [name: Generate an argument environment file ] of the file
+    [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)
+
+        -   Example: \${TYPE_OF_APP}
+
+-   test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
+    reset should be performed before the test.
+
+    -   Example: True
+
+-   test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
+    run. When set to True, the test run will be quieter.
+
+    -   Example: True
+
+-   test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
+    passed to the application during the test.
+
+    -   Example: --discriminator 1234 --KVS kvs1 --trace-to
+        json:\${TRACE_APP}.json
+
+-   test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
+    passed to the test script.
+    -   Example: --storage-path admin_storage.json --commissioning-method
+        on-network --discriminator 1234 --passcode 20202021 --trace-to
+        json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
+This structured format ensures that all necessary configurations are clearly
+defined and easily understood, allowing for consistent and reliable test
+execution.
+
 ## Cluster Codegen
 
 -   [Objects.py](https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/clusters/Objects.py)

diff --git a/scripts/tests/run_python_test.py b/scripts/tests/run_python_test.py
@@ -110,6 +110,10 @@ def main(app: str, factoryreset: bool, factoryreset_app_only: bool, app_args: st
             )
         ]
 
+    if not runs:
+        raise Exception(
+            "No valid runs were found. Make sure you add runs to your file, see https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/python.md document for reference/example.")
+
     for run in runs:
         print(f"Executing {run.py_script_path.split('/')[-1]} {run.run}")
         main_impl(run.app, run.factoryreset, run.factoryreset_app_only, run.app_args,

diff --git a/src/python_testing/hello_test.py b/src/python_testing/hello_test.py
@@ -15,6 +15,13 @@
 #    limitations under the License.
 #
 
+# test-runner-runs: run1
+# test-runner-run/run1/app: ${TYPE_OF_APP}
+# test-runner-run/run1/factoryreset: True
+# test-runner-run/run1/quiet: True
+# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
+# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
 import logging
 
 import chip.clusters as Clusters