[Feature] Python yamltest parsing that runs using chip-repl

[tehampson]

Feature description

This is a list of work items that we need to close out to get chip-repl version of the yaml test up to the chip-tool code gen parity

• Integrate PICS support from [matter_yamltests] Add PICS checker #24525
  • Integrate PICS on a per test step level
  • Integrate PICS on a test top level. PR is on hold because the PICS list provided to CI will actually prevent a bunch of tests from running that we actually want run (Coming in Add ability to check for PICS top level support #24741)
• Pseudo clusters that need implementation DelayCommands, DiscoveryCommands, LogCommands, CommissionerCommands (specially Unpair), SystemCommands. Common defaults can be found in [matter_yamltests] Add an abstract interface for pseudo clusters and a base class to register/use them #24620
  • Use common defaults for DelayCommands, LogCommands, SystemCommands (PR inprogress Add support for default pseudo cluster for chip-repl based yamltests #24669)
  • Make sure SystemCommands hook into into scripts/tests/chiptest/accessories.py:AppsRegister for system commands when CI is running (PR inprogress h#24669)
  • Implement DiscoveryCommands using DiscoverCommissionableNodes inside chip-repl (PR Add "DiscoveryCommands" to yaml repl tests, make TestDiscovery.yaml pass #24763)
  • Support CommissionerCommands::GetCommissionerNodeId. This was added in very recently and is very easy to add. Added when Add GetCommissionerNodeId to chip-reply yamltest runner #24793 merges
• ReadEvent, subscribeEvent - common parser needs to be able to to handle multiple events. In Terence's local environment he runner setup to be able to parse read events. Problem is that parser is not able to handle multiple events at the time. We need to develop a way that works for both apple's chip-tool version and out chip-repl based runner/adapter. Below both issues need to get merged in the order mentioned below.
  • Update to common parser [matter_yamltests] Add events supports #24831
  • Update to chip-repl adapter Add read/subscribe event to chip-repl based yamltests #24904
• Chip-repl needs to support issuing group commands such as WriteGroupAttribute and SendGroupCommand in order to get tests like TestGroupMessaging.yaml to work. Add WriteGroupAttributes to chip-repl #25260
• TestMultiAdmin.yaml Fails when trying to call pair when it is expecting a time out. The CommissionWithCode call hangs. Real issue somewhere in chip-repl or c++ controller code that it calls. Fix TestMultiAdmin to run on chip-repl based yamltest #24919
• Fix test flake from Test_TC_OO_2_4.yaml. Remove flaky test from list now that delay is added #25256

Nice to get to:

• Look into error spew from IDL parser. This was fixed iteratively over time
• Integrate chip-repl runner into scripts/tests/yaml/runner.py that was added here.
  • This needs to get merged Adding chip-repl into runner script #25900
• Implement CommissionerCommands::Unpair pseudo cluster. This is a nice to have since currently I don't see any YAML test that currently has this, but it is needed for codegen feature parity.
• Why are global attributes missing from spec definition from IDL parser. Search for comment "When we cannot find the attribute it is because it is a global attribute like" in src/controller/python/chip/yaml/runner.py
• Audit all ParseErrors, most should be converted to UnexpectedParseError to make them fatal. Example of one that should remains as a ParseError is inside InvokeAction when command is not found. Increase strictness of error that are raised by chip-repl runner/adapter #25357

General Tracking for things discovered while working on this:

• Needs csg spec change. Currently spec change PR is held up as of Feb 27, 2023 because if it got in now they would need to revalidate the fix in a TE if there is a change to the cert test. https://github.com/CHIP-Specifications/chip-test-plans/pull/2464

Platform

python

Platform Version(s)

1.1

Anything else?

No response