Documentation for YAML tests (#31575)

* YAML schema: add generator file

* Docs: Start of testing documentation

Pulls from the test harness presentations in Korea and Geneva

* It's good to save before committing

* docs: add generator for pseudocluster tables

* add pseudoclusters into yaml doc

* a bit more

* fix loader

* add titles

* fix linter

* Restyled by prettier-markdown

* Restyled by autopep8

* Restyled by isort

* fix spelling

* add pics and pixit placeholder file

* ignore generated files in spellcheck

* update word list

---------

Co-authored-by: Restyled.io <[email protected]>

.github/.wordlist.txt

@@ -95,6 +95,7 @@ ASR
 AssertionError
 AST
 ASYNC
+ATLs
 atomics
 att
 attId
@@ -160,6 +161,7 @@ blockquote
 bluetoothd
 bluez
 BOOL
+booleans
 BooleanState
 bootable
 Bootloader
@@ -380,6 +382,7 @@ DefaultOTARequestor
 DefaultOTARequestorDriver
 DefaultOTARequestorStorage
 DefaultSuccess
+defaultValue
 definedValue
 DehumidificationControl
 DelayedActionTime
@@ -503,6 +506,7 @@ emberAfExternalAttributeReadCallback
 emberAfExternalAttributeWriteCallback
 EmberAfInitializeAttributes
 emberAfSetDynamicEndpoint
+emsp
 EnableNetwork
 EnableWiFiNetwork
 endian
@@ -604,6 +608,7 @@ GenericWiFiConfigurationManagerImpl
 GetDeviceId
 GetDeviceInfo
 GetDns
+GetInDevelopmentTests
 GetIP
 getManualTests
 GetSafeAttributePersistenceProvider
@@ -673,6 +678,7 @@ IasWd
 iaszone
 ibb
 ICA
+ICAC
 ICD
 ICDs
 iCloud
@@ -724,6 +730,7 @@ IoT
 ipaddr
 iPadOS
 ipadr
+IPK
 ipp
 iptables
 iputils
@@ -935,6 +942,7 @@ namespacing
 nano
 natively
 navpad
+nbsp
 NCP
 ncs
 nding
@@ -1076,6 +1084,7 @@ Pigweed
 PinCode
 pinrequest
 PIXIT
+PIXITs
 pkgconfig
 PKI
 plaintext
@@ -1155,6 +1164,7 @@ RADVD
 raspberryPi
 RasPi
 rAv
+RCAC
 RCP
 ReadAttribute
 ReadConfigValue
@@ -1219,6 +1229,7 @@ RTOS
 RTT
 RTX
 runArgs
+runIf
 RUNAS
 RunMain
 runtime
@@ -1227,6 +1238,7 @@ rw
 RXD
 sandboxed
 saveAs
+saveDataVersschemaionAs
 sbin
 scalability
 scalable
@@ -1377,6 +1389,7 @@ TestArray
 TestCluster
 TestConstraints
 TestEmptyString
+TestEqualities
 TestGenExample
 TestGroupDemoConfig
 TestMultiRead

.spellcheck.yml

@@ -20,7 +20,7 @@
 #
 #   Actual run:
 #
-#      pyspelling pyspelling --config .spellcheck.yml
+#      pyspelling --config .spellcheck.yml
 
 matrix:
   - name: markdown
@@ -65,6 +65,6 @@ matrix:
       # converts markdown to HTML
       - pyspelling.filters.markdown:
     sources:
-      - '**/*.md|!third_party/**|!examples/common/**/repo/**|!docs/ERROR_CODES.md|!docs/clusters.md'
+      - '**/*.md|!third_party/**|!examples/common/**/repo/**|!docs/ERROR_CODES.md|!docs/clusters.md|!docs/testing/yaml_schema.md|!docs/testing/yaml_pseudocluster.md'
     aspell:
       ignore-case: true

docs/index.md

@@ -16,6 +16,7 @@ cluster_and_device_type_dev/index
 guides/index
 style/index
 examples/index
+testing/index
 tools/index
 BUG_REPORT
 code_generation

docs/testing/ci_testing.md

@@ -0,0 +1,6 @@
+# CI testing
+
+This file is a placeholder for information on how to run tests in the CI.
+
+NOTE: discuss in particular triggers direct to the device, test event triggers
+and the CI pics.

docs/testing/index.md

@@ -0,0 +1,31 @@
+# Testing Guides
+
+The following guide provide an introduction to the testing mechanisms available
+in the SDK.
+
+```{toctree}
+:glob:
+:maxdepth: 1
+:hidden:
+
+*
+```
+
+## Unit testing
+
+-   [Unit tests](./unit_testing.md)
+
+## Integration and Certification tests
+
+-   [Integration and Certification tests](./integration_tests.md)
+-   [YAML](./yaml.md)
+-   [Python testing framework](./python.md)
+-   [Enabling tests in the CI](./ci_testing.md)
+
+## PICS and PIXIT
+
+-   [PICS and PIXIT](./pics_and_pixit.md)
+
+## Testing in the CI
+
+-   [CI testing](./ci_testing.md)

docs/testing/integration_tests.md

@@ -0,0 +1,46 @@
+# Integration and Certification Tests
+
+Integration tests use a server and a controller or controllers to test the
+behavior of a device. Certification tests are all integration tests. For
+certified products, the device under test (DUT) is tested against one of the SDK
+controller implementations (either chip-tool or the python-based controller,
+depending on the test type). For software component certification, the software
+component is tested against a sample device built from the SDK.
+
+Certification tests require an accompanying certification test plan in order to
+be used in the certification testing process. More information about test plans
+can be found in the
+[test plans repository](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/docs).
+Integration testing can also be used outside of the certification testing
+program to test device behavior in the SDK. Certification tests are all run in
+the [CI](./ci_testing).
+
+There are two main integration test types:
+
+-   [YAML](./yaml.md)
+-   [Python framework](./python.md)
+
+YAML is a human-readable serialization language that uses structured tags to
+define test steps. Tests are defined in YAML, and parsed and run through a
+runner that is backed by the chip-tool controller.
+
+The Python framework tests are written in python and use the
+[Mobly](https://github.com/google/mobly) test framework to execute tests.
+
+## Which test framework to use
+
+Both types of tests can be run through the Test Harness for certification
+testing, locally for the purposes of development and in the CI for the SDK. The
+appropriate test framework to use is whatever lets you automate your tests in a
+way that is understandable, readable, and has the features you need
+
+-   YAML
+    -   pros: more readable, simpler to write, easy for ATLs to parse and
+        understand
+    -   cons: conditionals are harder (not all supported), no branch control,
+        schema not well documented
+-   python
+    -   pros: full programming language, full control API with support for core
+        (certs, commissioning, etc), less plumbing if you need to add features,
+        can use python libraries
+    -   cons: more complex, can be harder to read

docs/testing/pics_and_pixit.md

@@ -0,0 +1,10 @@
+# PICS and PIXITs
+
+Placeholder file for PICS and PIXIT info
+
+-   PICS formats - XML vs. test harness
+-   PICS in CI
+-   PICS tool and how we practically use it in Matter
+-   PICS checker test
+-   PIXITs in tests and how to set them
+-   Why you should avoid using both of these things.

docs/testing/python.md

@@ -0,0 +1,6 @@
+# Python framework tests
+
+This file is a placeholder for python framework test information.
+
+NOTE: be sure to include information about how you need to commission with the
+python controller, not chip-tool and how to do that in the scripts

docs/testing/unit_testing.md

@@ -0,0 +1,3 @@
+# Unit testing
+
+This doc is a placeholder for an guide around unit tests.