Support for cloud resource metadata

[0x2b3bfa0]

Epic for iterative/cml#430 provider-side support; follow-up to #3

Some vendors rely heavily on key/value metadata for resource management and, in some specific cases, permissions and policy enforcing. Most production–grade policies on vendors like AWS won't allow users to manage resources without proper creation–time tags.

Moreover, data science teams might also be interested in adding metadata to their cloud experiments for cost management and traceability purposes. See Kubeflow Metadata for a similar concept.

Implementation

• Add support for generic metadata #193
  • Add metadata support tpi#10
  • Add support for AWS metadata (tags) #187, depends on Migrate to aws-sdk-go-v2 #183
  • Add support for Azure metadata (tags) #191
  • Add support for Google Cloud metadata (labels) #190
  • Add support for Kubernetes metadata (labels+annotations) #189

Interface

Naming

• metadata (subject to change)
• labels
• tags

Typing

• map[string]string

Defaults

The official AWS provider supports default tags for all the created resources, but implementing this approach in a multicloud setting doesn't look like a good idea: every vendor uses labels/tags in a different way, and providing sensible defaults could be very difficult.

• Add support for AWS metadata (tags) #187 (comment)

Example

resource "iterative_machine" "machine" {
  ···
  metadata = {
    project = "example"
    creator = "anybody"
  }
}

Requests

• Slack
• Discord

[DavidGOrtega]

@0x2b3bfa0 Are we missing docs?

[0x2b3bfa0]

@DavidGOrtega, yes, we're missing documentation and API propagation to:

• Tagging of instances on AWS cml#430
• Metadata support tpi#9

[casperdcl]

@0x2b3bfa0 does iterative/cml#750 need to be linked to this issue?

[dacbd]

@0x2b3bfa0 does iterative/cml#750 need to be linked to this issue?

After a new version is published?/ then it should be good to move forward.

[0x2b3bfa0]

Yes!

[0x2b3bfa0]

Terraform-side support is complete. The only missing part is CML, that you have already implemented.

[dacbd]

Terraform-side support is complete. The only missing part is CML, that you have already implemented.

I appreciate your help with all the PRs!