proposal for transformer annotations option

proposals/21-11-transformer-annotations.md

@@ -0,0 +1,259 @@
+<!--
+**Note:** When your proposal is complete, all of these comment blocks should be removed.
+
+To get started with this template:
+
+- [ ] **Make a copy of this file.**
+  Name it `YY-MM-short-descriptive-title.md` (where `YY-MM` is the current year and month).
+- [ ] **Fill out this file as best you can.**
+  At minimum, you should fill in the "Summary" and "Motivation" sections.
+- [ ] **Create a PR.**
+  Ping `@kubernetes-sigs/kustomize-admins` and `@kubernetes-sigs/kustomize-maintainers`.
+-->
+
+# Option for origin and transformer annotations
+
+**Authors**:
+- natasha41575
+
+**Reviewers**: 
+- monopole
+- KnVerey
+- yuwenma
+
+**Status**: implementable
+
+## Summary
+
+This proposal is to extend the `buildMetadata.originAnnotations` option in the kustomization to annotate
+generated resources with information about the source of their generators, and to add a new option 
+`buildMetadata.transformerAnnotations` that will add annotations to each resource with information about
+all the transformers that have processed it. These options will help users understand how their output was
+created and debug it accordingly. 
+
+## Motivation
+
+When a user is managing a large number of resources with kustomize, the output of `kustomize build` will be a 
+long stream of yaml resources, and it can be extremely tedious for the user to understand how each resource 
+was created and processed. However, we can provide the user with an option to annotate each resource with 
+origin and transformation data. This information will help kustomize users
+debug their large kustomize configurations. Because these annotations can be used for debugging purposes, we 
+would like them to be verbose and communicate as much information as possible to the user.
+
+Kustomize previously had a [feature request](https://github.com/kubernetes-sigs/kustomize/issues/3979) to add an 
+option to kustomize build to retain the origin of a resource in an annotation, and a final design was settled 
+on after some discussion. It was implemented by [this pull request](https://github.com/kubernetes-sigs/kustomize/pull/4065) 
+and released in kustomize v4.3.0. This feature added a new field to the kustomization file, `buildMetadata`, an 
+array of boolean options such as `originAnnotations`. If `originAnnotations` is set, `kustomize build` will add annotations 
+to each resource describing the resource's origin. 
+
+While the above feature gives us valuable information about where many resources originated from, there are two 
+pieces missing:
+
+- Origin annotations for generated resources. A comment in the original issue proposed an annotation for generated 
+resources. Though this was accepted as part of the proposal, it was never implemented. However, the original proposal 
+for generated resources doesn't account for resources generated from remote bases.
+
+- Everything in kustomize is a transformer, so we would like to have annotations about which transformers, builtin or 
+custom, touched each resource. Builtin transformers include those that are invoked through various kustomization 
+fields. Some of these transformers are: 
+  - AnnotationsTransformer
+  - HashTransformer
+  - ImageTagTransformer
+  - LabelTransformer
+  - LegacyOrderTransformer
+  - NamespaceTransformer
+  - PatchJson6902Transformer
+  - PatchStrategicMergeTransformer
+  - PatchTransformer
+  - PrefixSuffixTransformer
+  - ReplacementTransformer
+  - ReplicaCountTransformer 
+  - ValueAddTransformer
+
+  Custom transformers are invoked via the `transformers` field in the kustomization file.
+
+  For a user attempting to debug a large stream of output from `kustomize build`, knowing how each resource has been 
+  transformed through various layers will make it much easier to understand how the output was rendered, easing debugging 
+  and making changes to kustomization files.
+
+**Goals:**
+1. When `originAnnotations` is set, add annotations to generated resources that describe the source of the generator that 
+created them.
+2. Add a new option, `transformerAnnotations`, to `buildMetadata` that, when set, will annotate each resource with 
+information about which transformers, builtin or custom, touched each resource.
+
+**Non-goals:**
+1. Change the syntax of the existing `originAnnotations` option.
+2. Add these annotations to the output of kustomize by default.
+
+## Proposal
+
+### Generated Resources
+
+We should add the same set of annotations to generated resources as other resources when the `buildMetadata.originAnnotations` 
+option is set. If the source of the file points to a generator, we know that it is a generated resource.
+
+For local generators:
+
+```
+config.kubernetes.io/origin: |
+  path: path/to/generator.yaml
+```
+
+For remote generators:
+
+```
+config.kubernetes.io/origin: |
+  path: examples/multibases/base/generator.yaml
+  repo: https://github.com/kubernetes-sigs/kustomize
+  ref: v1.0.6 
+```
+
+
+### Retaining Transformation Data
+
+To retain the information about what transformers have acted on each resource, we can propose a new option 
+`transformerAnnotations` in the `buildMetadata` field of the kustomization:
+
+```yaml
+buildMetadata: [originAnnotations, transformerAnnotations]
+```
+
+When the `transformerAnnotations` option is set, kustomize will add annotations with information about what transformers 
+have acted on each resource. Transformers can be invoked either through various fields in the kustomization file 
+(e.g. the `replacements` field will invoke the ReplacementTransformer), or through the `transformers` field.
+
+#### Local transformers
+For transformers defined locally, kustomize will add the following annotation:
+
+```yaml
+config.kubernetes.io/transformations: |
+  - kind: TransformerKind 
+    path: filepathToTransformerConfig 
+    type: builtin/custom
+```
+
+`kind` is the kind from the transformer's GVK.
+
+`path` is the filepath to the file containing the transformer config, rooted at the directory upon which kustomize build was invoked.
+
+`type` has two legal values, custom or builtin, depending on whether the transformer is built into kustomize or is defined by the user.
+
+#### Remote transfoermers
+For transformers defined in a remote base, kustomize will an annotation like the following:
+
+```yaml
+config.kubernetes.io/transformations: |
+  - kind: NamespaceTransformer
+    path: examples/multibases/base/kustomization.yaml
+    repo: https://github.com/kubernetes-sigs/kustomize
+    ref: v1.0.6 
+    type: builtin
+```
+
+The only difference between the remote transformer and the local one is that the remote transformer will have `repo` and `ref` fields to store where the 
+transformer originated from. 
+
+A resource that has been through multiple transformers through several layers may be outputted by kustomize build with an annotation like the following:
+
+```yaml
+config.kubernetes.io/transformations: |
+  - kind: PrefixSuffixTransformer
+    path: ../overlays/kustomization.yaml
+    type: builtin
+  - kind: InflateHelmChart
+    path: inflatehelmchart.yaml
+    type: custom
+  - kind: NamespaceTransformer
+    path: examples/multibases/base/kustomization.yaml
+    repo: https://github.com/kubernetes-sigs/kustomize
+    ref: v1.0.6 
+    type: builtin
+```
+
+#### Wildcard match
+
+As a shortcut for all buildMetadata options, we can allow the user to use a wildcard match:
+
+```
+buildMetadata: [*]
+```
+
+### Kustomize edit
+
+We will want to provide convenient commands for users to edit the `buildMetadata` field in their kustomization.
+We should support the following commands:
+
+`kustomize edit add buildMetadata originAnnotations`
+
+`kustomize edit add buildMetadata transformerAnnotations`
+
+`kustomize edit add buildMetadata *`
+
+`kustomize edit remove buildMetadata originAnnotaitons`
+
+`kustomize edit remove buildMetadata transformerAnnotations`
+
+`kustomize edit remove buildMetadata *`
+
+
+### User Stories
+
+#### Story 1
+
+I am running kustomize build on some configuration that has already been created. This configuration has a large number
+of resources and patches through multiple overlays in different directories. After running `kustomize build`, I get a 
+stream of 100+ resources, but I notice that a few of them have a typo, likely originating from the same base or patch 
+file. However, because my directory structure is large and complex, it is difficult to narrow down which base these
+resources originated from and which patches have been applied to it. In the top-level kustomization file, I can add a
+new `buildMetadata` field:
+
+```yaml
+buildMetadata: [originAnnotations, transformerAnnotations]
+```
+
+The output of `kustomize build` will contain patched resources with annotations similar to the following:
+```
+annotations:
+  config.kubernetes.io/origin: |
+    path: deployment.yaml
+  config.kubernetes.io/transformations:
+  - kind: PatchTransformer
+    path: ../base/kustomization.yaml
+    type: builtin
+  - kind: PatchStrategicMergeTransformer
+    path: ../dev/patch.yaml
+    type: builtin
+  - kind: PatchesJson6902Transformer
+    path: patch.yaml
+    type: builtin
+```
+
+From these annotations, I can narrow down the set of files that contributed to the resource's final output and
+debug it accordingly. 
+
+### Risks and Mitigations
+N/A
+
+### Dependencies
+N/A
+
+### Scalability
+N/A
+
+## Drawbacks
+N/A
+
+## Alternatives
+We considered putting this data in the comments of the rendered resources, but this would be a large project because 
+kustomize doesn't currently support comments. 
+
+We also considered making this a flag to `kustomize build` rather than additional field, but we would like to align 
+with the kustomize way of avoiding build-time side effects and have everything declared explicitly in the kustomization.
+
+## Rollout Plan
+This is a fairly lightweight, minor and optional feature that can be implemented and
+released without needing a rollout plan, much like the `originAnnotations` option that
+we are extending. With sufficient testing, there is no need to hide this feature behind 
+an alpha flag.