Support codegen by namespace (open-telemetry#243)

Co-authored-by: Armin Ruech <[email protected]>
Co-authored-by: Alexander Wert <[email protected]>

.github/workflows/semconvgen.yml

@@ -13,7 +13,7 @@ jobs:
   tests:
     runs-on: ubuntu-latest
     defaults:
-      run: 
+      run:
         working-directory: semantic-conventions/
     steps:
     - uses: actions/checkout@v4
@@ -39,7 +39,7 @@ jobs:
       run: pylint *.py src/
     - name: Type checking (mypy)
       run: mypy src/
-      
+
 
   build-and-publish-docker:
     runs-on: ubuntu-latest
@@ -68,3 +68,13 @@ jobs:
         else
           tag_and_push "${GITHUB_REF#"refs/tags/"}"
         fi
+    - name: Push the Dev Docker image
+      if: startsWith(github.ref, 'refs/heads/feature/')
+      run: |
+        echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
+        function tag_and_push {
+          docker tag semconvgen "otel/semconvgen:${1}" && docker push "otel/semconvgen:${1}"
+        }
+        TAG="${GITHUB_REF#"refs/heads/"}"
+        TAG="${TAG/"/"/"-"}"
+        tag_and_push "${TAG}"

semantic-conventions/CHANGELOG.md

@@ -12,6 +12,10 @@ Please update the changelog as part of any significant pull request.
   ([#271](https://github.com/open-telemetry/build-tools/pull/271))
 - Add link to requirement levels definition from Markdown table title.
   ([#222](https://github.com/open-telemetry/build-tools/pull/222))
+- Added code-generation mode that groups attributes by the root namespace and ability to write each group into individual file.
+  [BREAKING] The `--file-per-group <pattern>` that used to create multiple directories (like `output/<pattern>/file`) now generates
+  multiple files (`output/<pattern>file`) instead.
+  ([#243](https://github.com/open-telemetry/build-tools/pull/243))
 
 ## v0.23.0
 

semantic-conventions/README.md

@@ -107,13 +107,14 @@ This way, multiple files are generated. The value of `pattern` can be one of the
 - `semconv_id`: The id of the semantic convention.
 - `prefix`: The prefix with which all attributes starts with.
 - `extends`: The id of the parent semantic convention.
+- `root_namespace`: The root namespace of attribute to group by.
 
 Finally, additional value can be passed to the template in form of `key=value` pairs separated by
 comma using the `--parameters [{key=value},]+` or `-D` flag.
 
 ### Customizing Jinja's Whitespace Control
 
-The image also supports customising
+The image also supports customizing
 [Whitespace Control in Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/#whitespace-control)
 via the additional flag `--trim-whitespace`. Providing the flag will enable both `lstrip_blocks` and `trim_blocks`.
 
@@ -145,3 +146,59 @@ Following checks are performed
     so it's considered non-critical and it's possible to suppress it with `--ignore-warnings`
 
 
+### Accessing Semantic Conventions in the template
+
+When template is processed, it has access to a set of variables that depends on the `--file-per-group` value (or lack of it).
+You can access properties of these variables and call Jinja or Python functions defined on them.
+
+#### Single file (no `--file-per-group` pattern is provided)
+
+Processes all parsed semantic conventions
+
+- `semconvs` - the dictionary containing parsed `BaseSemanticConvention` instances with semconv `id` as a key
+- `attributes_and_templates` - the dictionary containing all attributes (including template ones) grouped by their root namespace.
+  Each element in the dictionary is a list of attributes that share the same root namespace. Attributes that don't have a namespace
+  appear under `""` key.
+- `attributes` - the list of all attributes from all parsed semantic conventions. Does not include template attributes.
+- `attribute_templates` - the list of all attribute templates from all parsed semantic conventions.
+
+#### The `root_namespace` pattern
+
+Processes a single namespace and is called for each namespace detected.
+
+- `attributes_and_templates` - the list containing all attributes (including template ones) in the given root namespace.
+- `root_namespace` - the root namespace being processed.
+
+#### Other patterns
+
+Processes a single pattern value and is called for each distinct value.
+
+- `semconv` - the instance of parsed `BaseSemanticConvention` being processed.
+
+### Filtering and mapping
+
+Jinja templates has a notion of [filters](https://jinja.palletsprojects.com/en/2.11.x/templates/#list-of-builtin-filters) allowing to transform objects or filter lists.
+
+Semconvgen supports following additional filters to simplify common operations in templates.
+
+#### `SemanticAttribute` operations
+
+1. `is_definition` - Checks if the attribute is the original definition of the attribute and not a reference.
+2. `is_deprecated` - Checks if the attribute is deprecated. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.DEPRECATED"`
+3. `is_experimental` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.EXPERIMENTAL"`
+4. `is_stable` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.STABLE"`
+5. `is_template` - Checks if the attribute is a template attribute.
+
+#### String operations
+
+1. `first_up` - Upper-cases the first character in the string. Does not modify anything else
+2. `regex_replace(text, pattern, replace)` - Makes regex-based replace in `text` string using `pattern``
+3. `to_camelcase` - Converts a string to camel case (using `.` and `_` as words delimiter in the original string).
+   The first character of every word is upper-cased, other characters are lower-cased. E.g. `foo.bAR_baz` becomes `fooBarBaz`
+4. `to_const_name` - Converts a string to Python or Java constant name (SNAKE_CASE) replacing `.` or `-` with `_`. E.g.
+   `foo.bAR-baz` becomes `FOO_BAR_BAZ`.
+5. `to_doc_brief` - Trims whitespace and removes dot at the end. E.g. ` Hello world.\t` becomes `Hello world`
+
+#### `BaseSemanticConvention` operations
+
+1. `is_metric` - Checks if semantic convention describes a metric.

semantic-conventions/src/opentelemetry/semconv/main.py

@@ -155,7 +155,8 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--output",
         "-o",
-        help="Specify the output file for the code generation.",
+        help="Specify the output file name for the code generation. "
+        "See also `--file-per-group` on how to generate multiple files.",
         type=str,
         required=True,
     )
@@ -169,8 +170,10 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--file-per-group",
         dest="pattern",
-        help="Each Semantic Convention is processed by the template and store in a different file. PATTERN is expected "
-        "to be the name of a SemanticConvention field and is prepended as a prefix to the output argument",
+        help="Semantic conventions are processed by the template and stored in a different file. "
+        "File names start with a 'pattern' and end with the name specified in the 'output' argument. "
+        "The 'pattern' can either match 'root_namespace' to group attributes by the root namespace or "
+        "match a name of Semantic Convention property which value will be used as a file name prefix.",
         type=str,
     )
     parser.add_argument(

semantic-conventions/src/opentelemetry/semconv/model/semantic_attribute.py

@@ -59,6 +59,7 @@ class SemanticAttribute:
     sampling_relevant: bool
     note: str
     position: List[int]
+    root_namespace: str
     inherited: bool = False
     imported: bool = False
 
@@ -200,6 +201,10 @@ def parse(
             fqn = fqn.strip()
             parsed_brief = TextWithLinks(brief.strip() if brief else "")
             parsed_note = TextWithLinks(note.strip())
+
+            namespaces = fqn.split(".")
+            root_namespace = namespaces[0] if len(namespaces) > 1 else ""
+
             attr = SemanticAttribute(
                 fqn=fqn,
                 attr_id=attr_id,
@@ -215,6 +220,7 @@ def parse(
                 sampling_relevant=sampling_relevant,
                 note=parsed_note,
                 position=position,
+                root_namespace=root_namespace,
             )
             if attr.fqn in attributes:
                 position = position_data[list(attribute)[0]]
@@ -323,6 +329,7 @@ def parse_deprecated(deprecated, position_data):
             return deprecated.strip()
         return None
 
+
     def equivalent_to(self, other: "SemanticAttribute"):
         if self.attr_id is not None:
             if self.fqn == other.fqn:

semantic-conventions/src/opentelemetry/semconv/model/semantic_convention.py

@@ -266,6 +266,9 @@ def __init__(self, group, strict_validation=True):
         self.metric_name = group.get("metric_name")
         self.unit = group.get("unit")
         self.instrument = group.get("instrument")
+
+        namespaces = self.metric_name.split(".")
+        self.root_namespace = namespaces[0] if len(namespaces) > 1 else ""
         self.validate()
 
     def validate(self):

semantic-conventions/src/opentelemetry/semconv/templating/code.py

@@ -21,12 +21,15 @@
 from jinja2 import Environment, FileSystemLoader, select_autoescape
 
 from opentelemetry.semconv.model.semantic_attribute import (
+    AttributeType,
     RequirementLevel,
     SemanticAttribute,
+    StabilityLevel,
     TextWithLinks,
 )
 from opentelemetry.semconv.model.semantic_convention import (
     BaseSemanticConvention,
+    MetricSemanticConvention,
     SemanticConventionSet,
 )
 from opentelemetry.semconv.model.utils import ID_RE
@@ -160,8 +163,34 @@ def to_camelcase(name: str, first_upper=False) -> str:
     return first + "".join(word.capitalize() for word in rest)
 
 
+def first_up(name: str) -> str:
+    return name[0].upper() + name[1:]
+
+
+def is_stable(obj: typing.Union[SemanticAttribute, BaseSemanticConvention]) -> bool:
+    return obj.stability == StabilityLevel.STABLE
+
+
 def is_deprecated(obj: typing.Union[SemanticAttribute, BaseSemanticConvention]) -> bool:
-    return obj.deprecated is not None
+    return obj.stability == StabilityLevel.DEPRECATED
+
+
+def is_experimental(
+    obj: typing.Union[SemanticAttribute, BaseSemanticConvention]
+) -> bool:
+    return obj.stability is None or obj.stability == StabilityLevel.EXPERIMENTAL
+
+
+def is_definition(attribute: SemanticAttribute) -> bool:
+    return attribute.is_local and attribute.ref is None
+
+
+def is_template(attribute: SemanticAttribute) -> bool:
+    return AttributeType.is_template_type(str(attribute.attr_type))
+
+
+def is_metric(semconv: BaseSemanticConvention) -> bool:
+    return isinstance(semconv, MetricSemanticConvention)
 
 
 class CodeRenderer:
@@ -196,6 +225,7 @@ def get_data_single_file(
             "semconvs": semconvset.models,
             "attributes": semconvset.attributes(),
             "attribute_templates": semconvset.attribute_templates(),
+            "attributes_and_templates": self._grouped_attribute_definitions(semconvset),
         }
         data.update(self.parameters)
         return data
@@ -214,20 +244,30 @@ def setup_environment(env: Environment, trim_whitespace: bool):
         env.filters["to_const_name"] = to_const_name
         env.filters["merge"] = merge
         env.filters["to_camelcase"] = to_camelcase
+        env.filters["first_up"] = first_up
         env.filters["to_html_links"] = to_html_links
         env.filters["regex_replace"] = regex_replace
         env.filters["render_markdown"] = render_markdown
         env.filters["is_deprecated"] = is_deprecated
+        env.filters["is_definition"] = is_definition
+        env.filters["is_stable"] = is_stable
+        env.filters["is_experimental"] = is_experimental
+        env.filters["is_template"] = is_template
+        env.filters["is_metric"] = is_metric
+        env.tests["is_stable"] = is_stable
+        env.tests["is_experimental"] = is_experimental
         env.tests["is_deprecated"] = is_deprecated
+        env.tests["is_definition"] = is_definition
+        env.tests["is_template"] = is_template
+        env.tests["is_metric"] = is_metric
         env.trim_blocks = trim_whitespace
         env.lstrip_blocks = trim_whitespace
 
     @staticmethod
-    def prefix_output_file(file_name, pattern, semconv):
+    def prefix_output_file(file_name, prefix):
         basename = os.path.basename(file_name)
         dirname = os.path.dirname(file_name)
-        value = getattr(semconv, pattern)
-        return os.path.join(dirname, to_camelcase(value, True), basename)
+        return os.path.join(dirname, to_camelcase(prefix, True) + basename)
 
     def render(
         self,
@@ -243,19 +283,97 @@ def render(
             autoescape=select_autoescape([""]),
         )
         self.setup_environment(env, self.trim_whitespace)
-        if pattern:
-            for semconv in semconvset.models.values():
-                output_name = self.prefix_output_file(output_file, pattern, semconv)
-                data = self.get_data_multiple_files(semconv, template_path)
-                template = env.get_template(file_name, globals=data)
-                template.globals["now"] = datetime.datetime.utcnow()
-                template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
-                template.globals["RequirementLevel"] = RequirementLevel
-                template.stream(data).dump(output_name)
+        if pattern == "root_namespace":
+            self._render_group_by_root_namespace(
+                semconvset, template_path, file_name, output_file, env
+            )
+        elif pattern is not None:
+            self._render_by_pattern(
+                semconvset, template_path, file_name, output_file, pattern, env
+            )
         else:
             data = self.get_data_single_file(semconvset, template_path)
             template = env.get_template(file_name, globals=data)
-            template.globals["now"] = datetime.datetime.utcnow()
-            template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
-            template.globals["RequirementLevel"] = RequirementLevel
-            template.stream(data).dump(output_file)
+            self._write_template_to_file(template, data, output_file)
+
+    def _render_by_pattern(
+        self,
+        semconvset: SemanticConventionSet,
+        template_path: str,
+        file_name: str,
+        output_file: str,
+        pattern: str,
+        env: Environment,
+    ):
+        for semconv in semconvset.models.values():
+            prefix = getattr(semconv, pattern)
+            output_name = self.prefix_output_file(output_file, prefix)
+            data = self.get_data_multiple_files(semconv, template_path)
+            template = env.get_template(file_name, globals=data)
+            self._write_template_to_file(template, data, output_name)
+
+    def _render_group_by_root_namespace(
+        self,
+        semconvset: SemanticConventionSet,
+        template_path: str,
+        file_name: str,
+        output_file: str,
+        env: Environment,
+    ):
+        attribute_and_templates = self._grouped_attribute_definitions(semconvset)
+        metrics = self._grouped_metric_definitions(semconvset)
+        for ns, attribute_and_templates in attribute_and_templates.items():
+            sanitized_ns = ns if ns != "" else "other"
+            output_name = self.prefix_output_file(output_file, sanitized_ns)
+
+            data = {
+                "template": template_path,
+                "attributes_and_templates": attribute_and_templates,
+                "metrics": metrics.get(ns) or [],
+                "root_namespace": sanitized_ns,
+            }
+            data.update(self.parameters)
+
+            template = env.get_template(file_name, globals=data)
+            self._write_template_to_file(template, data, output_name)
+
+    def _grouped_attribute_definitions(self, semconvset):
+        grouped_attributes = {}
+        for semconv in semconvset.models.values():
+            for attr in semconv.attributes_and_templates:
+                if not is_definition(attr):  # skip references
+                    continue
+                if attr.root_namespace not in grouped_attributes:
+                    grouped_attributes[attr.root_namespace] = []
+                grouped_attributes[attr.root_namespace].append(attr)
+
+        for ns in grouped_attributes:
+            grouped_attributes[ns] = sorted(grouped_attributes[ns], key=lambda a: a.fqn)
+        return grouped_attributes
+
+    def _grouped_metric_definitions(self, semconvset):
+        grouped_metrics = {}
+        for semconv in semconvset.models.values():
+            if not is_metric(semconv):
+                continue
+
+            if semconv.root_namespace not in grouped_metrics:
+                grouped_metrics[semconv.root_namespace] = []
+
+            grouped_metrics[semconv.root_namespace].append(semconv)
+
+        for ns in grouped_metrics:
+            grouped_metrics[ns] = sorted(
+                grouped_metrics[ns], key=lambda a: a.metric_name
+            )
+        return grouped_metrics
+
+    def _write_template_to_file(self, template, data, output_name):
+        template.globals["now"] = datetime.datetime.utcnow()
+        template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
+        template.globals["RequirementLevel"] = RequirementLevel
+
+        content = template.render(data)
+        if content != "":
+            with open(output_name, "w", encoding="utf-8") as f:
+                f.write(content)