Invoke Batch Commands Feature for 1.3

[tehampson]

Nice to complete TODO, but shouldn't affect feature launch

• When CommandSender times out figure out what we need to do.
  Remove comment about what should be done for batch commands on timeout #32261
• Batch commands TestEventTrigger to force sending one InvokeResponse per InvokeResponseMessage #31434
  • Teststep 9 in IDM_1_4
• Reorder CommandSender::Callback and CommandSender::ExtendableCallback.
  Reorder CommandSender::Callback and CommandSender::ExtendableCallback. #31370
• Can we reduce the size of CommandPathRegistry * mCommandPathRegistry for the default case
• Can we remove mRefForResponse in CommandHandler, by passing the value on stack instead of taking up memory here.
• Updated structure of things in commandDataIB and such in https://github.com/project-chip/connectedhomeip/blob/master/src/lib/format/protocol_messages.matter#L319 with ref to help with automated decoding and it understanding datastructures in the future.
  I believe a "ref" item is needed after the spec updates and that would make automated decoding work a bit better and it helps understanding data structures in the future.
  • adding to --trace-to json:log or --trace-to json:/tmp/messages.json will make for a good debug experience. It probably works without as well, but instead of saying nice text of CommandRef: ... it will be something like Context(2): ...
• Remove CommandResponseSender from friend in CommandHandler. Introduced in #31378. Needed right now for CommandResponseSenderto callClose()`
• src/app/tests/TestCommandInteraction.cpp hardcodes is creating multiple commands manually. We should use newly created APIs.
  Cleanup and fix batch command tests in TestCommandInteraction #32288
• It should be possible for SetExchangeContext to internally call WillSendMessage. But unit tests start failing
  No longer relevant after refactor Refactor CommandHandler to be more unittestable #32467
• TestCommandInteraction::FillCurrentInvokeResponseBuffer uses a magic number to fill up InvokeResponseMessage exactly. This should be derived from other sources instead of being a magic number
  Replace hardcoded assumption with one dynamically calculated #32291
• Add unit testing for TestOnlyInvokeCommandRequestWithFaultsInjected

TODO issues discovered while working on this feature that would be good to follow up on

• When CommandHandler fails to send the first message (when calling CommandHandler::StartSendingCommandResponses), we should be sending back a failure if possible. Existing unit tests start failing when this was added in
• Spec mentions:
  If this action is marked with TimedRequest as TRUE, but this action is not part of a Timed Invoke transaction (i.e. there was no immediately previous Timed Invoke action), then a Status Response action with the TIMED_REQUEST_MISMATCH Status Code SHALL be submitted to the message layer and this interaction SHALL terminate.
  But code send UnsupportedAccess. Note TestPlan expected behavior is looking for UnsupportedAccess
  https://github.com/CHIP-Specifications/chip-test-plans/pull/3841
  Fix IM error when TimeRequest flag doesn't state of isTimedInvoke #31485
• It is pretty dangerous for CommandHandler to provide access to GetExchangeContext(). Unfortunately there is already a precedent of using in, but it seems like everything wants info that could be alternatively provided instead of giving access to the ExchangeContext directly. More info here
• CommandHandler::GetCommandDataIBTLVWriter should not be public. This is a footgun in causing issue with CommandHandler state that is a larger issue for batch commands
• Rolling back InvokeResponseMessag and InvokeRequestMessage's Builders requires manually resetting internal errors directly instead of it being handled for us. This would be a good quality of life improvement.

Archived section (as work is completed)

TODO before FC (Dec 2023):

• Add new SessionParameters add to spec for 1.3 (Review in progress)
  Update session parameters to be inline with matter 1.3 spec #30405
• Add required attributes to BasicInformationCluster add to spec for 1.3 (PR ready and will be uploaded once above PR lands)
  Add Spec version and Max Paths Per Invoke to Basic Information cluster #30496
• Update CommandDataIB to allow Building and Parsing multiple commands
  Changes to CommandDataIB to reflect 1.3 spec #30522
• Update InvokeResponseMessage to allow Building and Parsing MoreChunkedMessages
  Changes to InvokeResponseMessage to reflect 1.3 spec #30639
• Additional changes to SessionParamters plumbing so that SDK APIs know MaxPathsPerInvoke how many batch commands remote peer can accept.
  Allow plumbing of SessionParameters from Session #30525
• Update CommandSender to support sending multiple commands.
  Update CommandSender to support sending batch commands #30675
• Update CommandHandler to support receiving and processing multiple commands.
  Update CommandHandler to support batch commands #30614
• CommandSender::Callback API contract issue when Command has path-specific error response #30991
  Introduce CommandSender::ExtendedCallback #31324

CSG Cert work (Before TE2):

• Enable chip-repl to send multiple commands
  Enable chip-repl to send batch commands #30851
• Enable chip-repl to get SessionParameters
• Create IDM python test script
  • Initial PR to get basic test steps passing
    Implement updates to IDM test spec for batch commands #31042
  • Create TestOnly_SendBatchCommands in chip-repl so we can explicitly do things we are not supposed to be able to do Currently blocked on For CFFI, remove variadic arguments #31159
    Add TestOnlySendBatchCommands to chip-repl #31455
  • Rest of test cases that involved path specific errors.
    Update skipped test step in IDM_1_4 #31458
• Update CSG test spec to reflect what IDM python test script and add client test
  https://github.com/CHIP-Specifications/chip-test-plans/pull/3734
• Enable batch commands on linux all cluster app for CSG folks to more easily test for TE2
  Enable batch commands for the linux example all-clusters-app #31531

Critical bug fixes required (Before TE2):

• Allow for sending multiple individual InvokeResponse
  • Enabling MoreChunkedMessages for Invoke Interaction plumbing
    Enabling MoreChunkedMessages for Invoke Interaction #31378
  • Have CommandHandler actually send multiple InvokeResponseMessage. We should also make sure that when we run out of space, send out all currently complete command responses
    Upon running out of space in InvokeResponseMessage for additional responses, allocate another #31516

Between TE2 and SVE

These items focus on improvements not with the feature itself, but with QoL improvements for those to more easily send or receive batch commands:

• Update CommandSender APIs to be more extendable in the future
  Refactor CommandSender newly added APIs to be more extendable #31837
• CommandSender should be responsible for managing/matching response to requests that have come back, instead of application code.
  • Remove restriction from forcing CommandRefs to be provided in order starting from 0
    CommandSender: For batch commands, track requests are responded to #32105
• Make csg testing more automatiable
  • Expose way for chip-repl to know how many InvokeResponseMessages came in. This allows the check to be automatable instead of manual
    Allow chip-repl to know how many InvokeResponseMessages were recieved #31781
  • Use PayloadTestRequest to finish off all automation for IDM_1_4
    Add test steps to IDM_1_4 to validate multiple InvokeResponseMessages #31979
  • Identify if there is a way for IDM_1_3 to be more automatable in the response all-cluster-app sends
    Use FaultInjection to help with IDM_1_3 automation testing #32045
  • Update IDM_1_3 yaml to use nlfault injections
    Update IDM_1_3 yaml to latest testplan spec which uses fault injection #32149

Important TODOs:

• In PairingSession::EncodeSessionParameters figure out what MRP values we should sent when providedMrpConfig.HasValue() == false #30456
  Have PairingSession::EncodeSessionParameters require local MRP values #32324
• Fix incorrectly reported NoReponse when single batch command sent
  Fix incorrectly reported NoReponse when single batch command sent to … #32362
• Add CI integration tests
  Add batch invoke integration tests to CI #31642
• Now that we have ExtendedCallbacks we should remove CHIP_CONFIG_SENDING_BATCH_COMMANDS_ENABLED. For context read Introduce CommandSender::ExtendedCallback #31324
  Remove CHIP_CONFIG_SENDING_BATCH_COMMANDS_ENABLED #31371
• Use Reserve to make sure there is enough space to end messages in CommandSender and CommandHandler
  CommandSender and CommandHandler reserves space for Finalize to succeed #31077
  • Prevent TLVWrite from possibly performing buffer overrun if trying to reserve
    Prevent TLVWrite from possibly performing buffer overrun if trying to reserve #30714
• Validate entire InvokeRequestMessage before processing any commands
  Validate InvokeRequestMessage TLV properly terminated before running commands #31218
• Add integration tests to CI
  Add batch invoke integration tests to CI #31642