Merge branch 'master' into launch-contentapp-on-command

project-chip · Jul 10, 2024 · 14789b1 · 14789b1
2 parents 4d692bb + c6af674
commit 14789b1
Showing 1 changed file with 62 additions and 52 deletions.
diff --git a/docs/testing/python.md b/docs/testing/python.md
@@ -85,58 +85,8 @@ The default_matter_test_main() function is used to run the test on the command
 line. These two lines should appear verbatim at the bottom of every python test
 file.
 
-## Defining the test arguments
-
-Below is the format:
-
-```
-# test-runner-runs: <run_identifier>
-# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
-# test-runner-run/<run_identifier>/factoryreset: <True|False>
-# test-runner-run/<run_identifier>/quiet: <True|False>
-# test-runner-run/<run_identifier>/app-args: <app_arguments>
-# test-runner-run/<run_identifier>/script-args: <script_arguments>
-```
-
-### Description of Parameters
-
--   test-runner-runs: Specifies the identifier for the run. This can be any
-    unique identifier.
-
-    -   Example: run1
-
--   test-runner-run/<run_identifier>/app: Indicates the application to be used
-    in the test. Different app types as needed could be referenced from section
-    [name: Generate an argument environment file ] of the file
-    [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)
-
-        -   Example: \${TYPE_OF_APP}
-
--   test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
-    reset should be performed before the test.
-
-    -   Example: True
-
--   test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
-    run. When set to True, the test run will be quieter.
-
-    -   Example: True
-
--   test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
-    passed to the application during the test.
-
-    -   Example: --discriminator 1234 --KVS kvs1 --trace-to
-        json:\${TRACE_APP}.json
-
--   test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
-    passed to the test script.
-    -   Example: --storage-path admin_storage.json --commissioning-method
-        on-network --discriminator 1234 --passcode 20202021 --trace-to
-        json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
-
-This structured format ensures that all necessary configurations are clearly
-defined and easily understood, allowing for consistent and reliable test
-execution.
+The structured comments above the class definition are used to set up the CI for
+the tests. Please see [Running tests in CI](#running-tests-in-ci).
 
 ## Cluster Codegen
 
@@ -610,3 +560,63 @@ example DUT on the host and includes factory reset support
 -   if there are things in your test that will fail on CI (ex. test vendor
     checks), gate them on the PICS_SDK_CI_ONLY
     -   is_ci = self.check_pics('PICS_SDK_CI_ONLY')
+
+The CI test runner uses a structured environment setup that can be declared
+using structured comments at the top of the test file. To use this structured
+format, use the --load-from-env flag with the run_python_tests.py runner.
+
+Ex:
+`scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDM_2_1.py'`
+
+## Defining the CI test arguments
+
+Below is the format:
+
+```
+# test-runner-runs: <run_identifier>
+# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
+# test-runner-run/<run_identifier>/factoryreset: <True|False>
+# test-runner-run/<run_identifier>/quiet: <True|False>
+# test-runner-run/<run_identifier>/app-args: <app_arguments>
+# test-runner-run/<run_identifier>/script-args: <script_arguments>
+```
+
+### Description of Parameters
+
+-   test-runner-runs: Specifies the identifier for the run. This can be any
+    unique identifier.
+
+    -   Example: run1
+
+-   test-runner-run/<run_identifier>/app: Indicates the application to be used
+    in the test. Different app types as needed could be referenced from section
+    [name: Generate an argument environment file ] of the file
+    [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)
+
+        -   Example: \${TYPE_OF_APP}
+
+-   test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
+    reset should be performed before the test.
+
+    -   Example: True
+
+-   test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
+    run. When set to True, the test run will be quieter.
+
+    -   Example: True
+
+-   test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
+    passed to the application during the test.
+
+    -   Example: --discriminator 1234 --KVS kvs1 --trace-to
+        json:\${TRACE_APP}.json
+
+-   test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
+    passed to the test script.
+    -   Example: --storage-path admin_storage.json --commissioning-method
+        on-network --discriminator 1234 --passcode 20202021 --trace-to
+        json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
+This structured format ensures that all necessary configurations are clearly
+defined and easily understood, allowing for consistent and reliable test
+execution.