Decode TLV payload data and present it in a human readable format (#2…

…7638) * Start with a flat tree library for a human TLV format * Temp change for test * Switch to a flat list and more flexible finding ... expect I want to find by id AND name eventually * Clang-format * Add tests for searching by name in a flat tree * Provide non-array find-entry * Have a good tree position that works for navigating and descend/ascend * Added more documentation * Add more unit tests * Fix naming * Support current path for flat tree positions * Restyle * Add IM message encoding, to have pretty-print of data once available * Added secure channel message formats * Add UDC defintions * Rename things * make the matter file parseable * Attempt to start a codegen for tlv meta mapping * Restyle * Add missing files * Restyle * Have some codegen working, start defining types and names * Start implementing a bit of a table generation. not done, but tables start to exist * Support events for tables * Add support for commands (untested though) * More work, all except lists and constants are code-generated * Restyle * List support and better tag support including anonymous support * Make tags specific: many tags are NOT context tags currently * Restyle * Add some test data for development tests * Start adding some test support ... to be removed later * Code compiles * Add a unit test that compiles and runs * Starting some decoding support. Still very much broken * A bit more decoding, this time we handle lists. TLV interface is VERY bad * Better decoding, we now show data * Add some item information, to prepare for enum and bitmap decoding * Restyle * Add error messages on usage of command line * remove a non yes/no argument * Update error syntax for 2 more arguments * update the help. using both true/false and yes/no is a mess * Update logic for decoding * Do not allow restyle * Better StringBuilder formatting * Test adjustment * Naming update * Restyle * Add Format option for buffer writers and string builders * Update comments * Updated to only have printf inside stringbuilder and NOT bufferwriter * Restyle * Add missing file * remove cpp file comments * Fix cast to make clang happy * Much better formatting and make the compile clang-friendly * minor const correctness change. TLVReader has non-const getters * Add special tags for payloads of things * Added logic for binary data and payloads, to process things * Start adding clusters metadata, make everything const-correct * Minor update * Start updating formats * Never pass null pointer in vsnprintf, since our size available is never 0 * Restyle * Restyle * Better decoding * Clean up some printfs * Iterator decoding seems to work, including getting sub-data types * Better organization of code ... protocols decoder is actually a class now * Allow passing in decode trees for protocol understanding * Better arg parsing - was able to test for invalid data * Restyle * Add back reset call to stringbuilder * Restyle and make protocol decoding actually work * Unformatted protocols/cluster meta * Add more trace data for testing, fix SEGFAULT in decoder * Support non-struct list entries * Switch list decoding logic to be inside generated metadata * Restyle * Fix compilation and generation * Start having codegen support for protocols metadata * Move clusters meta to compile time codegen as well * Restyle * Cleanup dependencies a bit * Start making TestDecoding be actual unit tests * more unit tests, without protocol decoding * Slightly better formatting * More unit tests ... although invalid data looks odd * Better formatting of unknown attributes * Updated tests * Restyle * Better messaging, test overflows * Removed unused file * Undo submodule update * Add some tests for invoke. Command list is NOT complete * Restyle * Yield commands that have no request structure * Fix comment * Start adding some unit tests for cpp-tlvmeta codegen * Add tests for real * Allow both hex and json at the same time for output * Rename log_json to just json * Add file output option for json tracing * make the output look like a json array when outputing to file * Restyle * Fix support of "json:log" * Fix support of "json:log" * make things compile * Rename open/close to openfile/closefile to avoid override errors * Restyle * StartsWith should be available now globally as it is always used * StartsWith should be available now globally as it is always used * Forward declare json to make arm cross compile pass * Forward declare json to make arm cross compile pass * Restyle * Add some support for formatting enums and bitmaps * Restyle * Add json_tracing exceptions for includes checks * Proper bitmap support with tests, status codes are bitmaps now * Restyle * Update test for overflow to have more unique values * Restyle * Fix decoding of command inputs and names * Add a fuzz test for payload decoder * Handle invalid TLV * Add more error handling. Fuzzing runs longer now * Restyle * Make clang happy * Fix efr32 unit test compilation * Fix python lint * Restyle * Fix typo and restyle * Restyle * Add dependencies to flat-tree for generated code: they are needed * make tests use uppercase for unknown tags as this is the code update I made recently * Undo submodule update * Make clang-tidy happy * Fix subscribe response message indexing to match spec --------- Co-authored-by: Andrei Litvin <[email protected]>
project-chip · Feb 23, 2024 · 1488040 · 1488040
1 parent a67ec66
commit 1488040
Show file tree

Hide file tree

Showing 38 changed files with 3,953 additions and 13 deletions.
diff --git a/BUILD.gn b/BUILD.gn
@@ -54,6 +54,7 @@ if (current_toolchain != "${dir_pw_toolchain}/default:default") {
         "${chip_root}/src/credentials/tests:fuzz-chip-cert",
         "${chip_root}/src/lib/core/tests:fuzz-tlv-reader",
         "${chip_root}/src/lib/dnssd/minimal_mdns/tests:fuzz-minmdns-packet-parsing",
+        "${chip_root}/src/lib/format/tests:fuzz-payload-decoder",
       ]
     }
   }

diff --git a/build/chip/chip_codegen.gni b/build/chip/chip_codegen.gni
@@ -53,9 +53,19 @@ template("_chip_build_time_codegen") {
       rebase_path(target_gen_dir, root_build_dir),
       "--expected-outputs",
       rebase_path(_expected_outputs, root_build_dir),
-      rebase_path(_idl_file, root_build_dir),
     ]
 
+    if (defined(invoker.options)) {
+      foreach(option, invoker.options) {
+        args += [
+          "--option",
+          option,
+        ]
+      }
+    }
+
+    args += [ rebase_path(_idl_file, root_build_dir) ]
+
     inputs = [
       _idl_file,
       _expected_outputs,
@@ -313,12 +323,15 @@ template("chip_codegen") {
                                "generator",
                                "input",
                                "outputs",
+                               "options",
                                "public_configs",
                              ])
     }
   } else {
     _name = target_name
 
+    not_needed(invoker, [ "options" ])
+
     # This constructs a path like:
     #  FROM all-clusters-app.matter (inside examples/all-clusters-app/all-clusters-common/)
     #  USING "cpp-app" for generator:

diff --git a/examples/chip-tool/args.gni b/examples/chip-tool/args.gni
@@ -27,3 +27,6 @@ matter_enable_tracing_support = true
 
 # Perfetto requires C++17
 cpp_standard = "gnu++17"
+
+matter_log_json_payload_hex = true
+matter_log_json_payload_decode_full = true
diff --git a/scripts/pregenerate/__init__.py b/scripts/pregenerate/__init__.py
@@ -20,7 +20,8 @@
 from typing import Iterator, List, Optional
 
 from .types import IdlFileType, InputIdlFile
-from .using_codegen import CodegenCppAppPregenerator, CodegenJavaClassPregenerator, CodegenJavaJNIPregenerator
+from .using_codegen import (CodegenCppAppPregenerator, CodegenCppClustersTLVMetaPregenerator,
+                            CodegenCppProtocolsTLVMetaPregenerator, CodegenJavaClassPregenerator, CodegenJavaJNIPregenerator)
 from .using_zap import ZapApplicationPregenerator
 
 
@@ -94,6 +95,8 @@ def FindPregenerationTargets(sdk_root: str, external_roots: Optional[List[str]],
         CodegenJavaJNIPregenerator(sdk_root),
         CodegenJavaClassPregenerator(sdk_root),
         CodegenCppAppPregenerator(sdk_root),
+        CodegenCppClustersTLVMetaPregenerator(sdk_root),
+        CodegenCppProtocolsTLVMetaPregenerator(sdk_root),
 
         # ZAP codegen
         ZapApplicationPregenerator(sdk_root),

diff --git a/scripts/pregenerate/using_codegen.py b/scripts/pregenerate/using_codegen.py
@@ -27,11 +27,12 @@
 class CodegenTarget:
     """A target that uses `scripts/codegen.py` to generate files."""
 
-    def __init__(self, idl: InputIdlFile, generator: str, sdk_root: str, runner):
+    def __init__(self, idl: InputIdlFile, generator: str, sdk_root: str, runner, options=[]):
         self.idl = idl
         self.generator = generator
         self.sdk_root = sdk_root
         self.runner = runner
+        self.options = options
 
         if idl.file_type != IdlFileType.MATTER:
             raise Exception(
@@ -51,8 +52,12 @@ def Generate(self, output_root: str):
             '--log-level', 'fatal',
             '--generator', self.generator,
             '--output-dir', output_dir,
-            self.idl.full_path
         ]
+        for option in self.options:
+            cmd.append("--option")
+            cmd.append(option)
+
+        cmd.append(self.idl.full_path)
 
         logging.debug(f"Executing {cmd}")
         self.runner.run(cmd)
@@ -97,6 +102,9 @@ def Accept(self, idl: InputIdlFile):
         if idl.file_type != IdlFileType.MATTER:
             return False
 
+        if '/lib/format/' in idl.relative_path:
+            return False
+
         # we should not be checked for these, but verify just in case
         if '/tests/' in idl.relative_path:
             return False
@@ -105,3 +113,29 @@ def Accept(self, idl: InputIdlFile):
 
     def CreateTarget(self, idl: InputIdlFile, runner):
         return CodegenTarget(sdk_root=self.sdk_root, idl=idl, generator="cpp-app", runner=runner)
+
+
+class CodegenCppProtocolsTLVMetaPregenerator:
+    """Pregeneration logic for "cpp-app" codegen.py outputs"""
+
+    def __init__(self, sdk_root):
+        self.sdk_root = sdk_root
+
+    def Accept(self, idl: InputIdlFile):
+        return (idl.file_type == IdlFileType.MATTER) and idl.relative_path.endswith('/protocol_messages.matter')
+
+    def CreateTarget(self, idl: InputIdlFile, runner):
+        return CodegenTarget(sdk_root=self.sdk_root, idl=idl, generator="cpp-tlvmeta", options=["table_name:protocols_meta"], runner=runner)
+
+
+class CodegenCppClustersTLVMetaPregenerator:
+    """Pregeneration logic for "cpp-app" codegen.py outputs"""
+
+    def __init__(self, sdk_root):
+        self.sdk_root = sdk_root
+
+    def Accept(self, idl: InputIdlFile):
+        return (idl.file_type == IdlFileType.MATTER) and idl.relative_path.endswith('/controller-clusters.matter')
+
+    def CreateTarget(self, idl: InputIdlFile, runner):
+        return CodegenTarget(sdk_root=self.sdk_root, idl=idl, generator="cpp-tlvmeta", options=["table_name:clusters_meta"], runner=runner)
diff --git a/scripts/py_matter_idl/files.gni b/scripts/py_matter_idl/files.gni
@@ -3,24 +3,38 @@ import("//build_overrides/chip.gni")
 
 # Templates used for generation
 matter_idl_generator_templates = [
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/application/CallbackStubSource.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/application/PluginApplicationCallbacksHeader.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_cpp.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_h.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/CHIPCallbackTypes.jinja",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ChipClustersCpp.jinja",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ChipClustersRead.jinja",
-  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ClusterWriteMapping.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/CHIPGlobalCallbacks_cpp.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/CHIPReadCallbacks_h.jinja",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ClusterIDMapping.jinja",
-  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/application/CallbackStubSource.jinja",
-  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/application/PluginApplicationCallbacksHeader.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ClusterReadMapping.jinja",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/ClusterWriteMapping.jinja",
 ]
 
 matter_idl_generator_sources = [
   "${chip_root}/scripts/py_matter_idl/matter_idl/__init__.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/__init__.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/__init__.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/application/__init__.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/__init__.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/filters.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/java/__init__.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/generators/registry.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/generators/types.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/lint/__init__.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/lint/lint_rules_parser.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/lint/types.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/matter_idl_parser.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/matter_idl_types.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/test_generators.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/test_matter_idl_parser.py",
+  "${chip_root}/scripts/py_matter_idl/matter_idl/test_xml_parser.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/xml_parser.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/zapxml/__init__.py",
   "${chip_root}/scripts/py_matter_idl/matter_idl/zapxml/handlers/__init__.py",

diff --git a/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_cpp.jinja b/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_cpp.jinja
@@ -0,0 +1,44 @@
+#include <tlv/meta/{{table_name}}.h>
+
+namespace chip {
+namespace TLVMeta {
+namespace {
+
+using namespace chip::FlatTree;
+using namespace chip::TLV;
+
+{%- for table in sub_tables %}
+
+const Entry<ItemInfo> _{{table.full_name}}[] = {
+  {%-  for entry in table.entries %}
+  { { {{entry.code}}, "{{entry.name}}", ItemType::{{entry.item_type}} }, {{entry.reference | indexInTable(sub_tables)}} }, // {{entry.real_type}}
+  {%- endfor %}
+};
+{%- endfor %}
+
+const Entry<ItemInfo> _all_clusters[] = {
+{%- for cluster in clusters | sort(attribute='code') %}
+  { { ClusterTag({{"0x%02X" | format(cluster.code)}}), "{{cluster.name}}", ItemType::kDefault }, {{cluster.name | indexInTable(sub_tables)}} },
+{%- endfor %}
+
+};
+
+// For any non-structure list like u64[] or similar.
+const Entry<ItemInfo> _primitive_type_list[] = {
+  { { AnonymousTag(), "[]", ItemType::kDefault }, kInvalidNodeIndex },
+};
+
+} // namespace
+
+#define _ENTRY(n) { sizeof(n) / sizeof(n[0]), n}
+
+const std::array<const Node<ItemInfo>, {{ sub_tables | length }} + 2> {{table_name}} = { {
+  _ENTRY(_all_clusters), // 0
+  _ENTRY(_primitive_type_list), // 1
+{%- for table in sub_tables %}
+  _ENTRY(_{{table.full_name}}), // {{loop.index + 1}}
+{%- endfor %}
+} };
+
+} // namespace TLVMeta
+} // namespace chip
diff --git a/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_h.jinja b/scripts/py_matter_idl/matter_idl/generators/cpp/tlvmeta/TLVMetaData_h.jinja
@@ -0,0 +1,12 @@
+#include <lib/format/tlv_meta.h>
+#include <lib/format/FlatTree.h>
+
+#include <array>
+
+namespace chip {
+namespace TLVMeta {
+
+extern const std::array<const FlatTree::Node<ItemInfo>, {{ sub_tables | length }} + 2> {{table_name}};
+
+} // namespace TLVMeta
+} // namespace chip