Support the pseudo dynamic type

[paultyng]

Tracking issue for adding DynamicPseudoType type support.

[paultyng]

An example that could be replaced with dynamic: https://www.terraform.io/docs/providers/aws/r/iam_policy.html#example-usage

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:Describe*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

Could be:

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = {
    Version = "2012-10-17"
    Statement = [
      {
        Action = ["ec2:Describe*"]
        Effect = "Allow"
        Resource = "*"
      }
    ]
}

Or even:

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = jsondecode(<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:Describe*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
)
}

[skylerto]

I am definitely interested in this great feature! I have a few requirements out of an internal provider I'm writing to house what can be generic JSON passed to an API, similar to the generality of IAM policies! 😄

I'm currently doing the opposite of the JSON decode here, encoding HCL any type into a String for the provider to pass to an API.

[MaciejKaras]

@paultyng So I suppose we have to stick to serializing nested content via e.g. jsonencode function or some other method that creates a string. Or is there other more proper workaround that you are aware of?

Do you have any idea approximately when milestone v2.0.0 containing discussed enhancement is going to be released? I need to decide whether apply a workaround for now or just wait for new SDK release.

[paultyng]

@MaciejKaras we are looking to exposing this in 2.x (not 2.0), but we are probably months out for a solution for it right now if that helps you to decide, until then, yes I think the best approach is still to use typed HCL and jsonencode it to a string. To prepare for this to exist, you could have like an object_json as TypeString with the plan to eventually just add object and deprecate the _json suffixed attribute at a later date.

[gdavison]

I have another usecase for this feature, also from the AWS Provider:

The resource type aws_codepipeline has an attribute, several levels of nesting down, where the type depends on the value of another attribute. It's currently implemented using a TypeMap, so we lose the ability to do validation, marking values as Sensitive, etc.

resource "aws_codepipeline" "example" {
  # ...

  stage {
   name = "Source"
   # ...
    action {
      # ...
      provider         = "GitHub"
      configuration = {
        Owner      = "lifesum-terraform"
        Repo       = "test"
        Branch     = "master"
        OAuthToken = var.github_token # This field will be settable starting in v3.0 of the AWS provider
      }
    }
  }
}

The values accepted by stage[x].action[y].configuration are controlled by stage[x].action[y].provider.

For v3.0 of the provider, we're making the whole configuration attribute as Sensitive because of the possibility of the GitHub OAuthToken being set. We're also adding a custom attribute configuration_github as an alternative, but since there are currently 29 action providers, exposing 29+ named attributes would clutter the interface.

We would still need to define the concrete types, but having them be selected by the value of provider would be much cleaner.

For example:

resource "aws_codepipeline" "example" {
  # ...

  stage {
    name = "Source"
    # ...
    action {
      # ...
      provider         = "GitHub"
      configuration = {
        owner      = "lifesum-terraform"
        repo       = "test"
        branch     = "master"
        oauth_token = var.github_token # This field would be Sensitive, the others wouldn't
      }
    }
  }

  stage {
    name = "Build"

    action {
      # ...
      provider        = "CodeBuild"
      configuration = {
        project_name = "test"
        environment_variable { # These could be structured in HCL, rather than collapsed as a string
          name  = "EXAMPLE_1"
          value = "VALUE_1"
          type  = "PLAINTEXT"
        }
        environment_variable {
          name  = "EXAMPLE_2"
          value = "example/parameter_2"
          type  = "PARAMETER_STORE"
        }
      }
    }
  }
}

[paddycarver]

@paultyng with the introduction of terraform-plugin-go, I'm inclined to call this complete. It's not easy to use, but it's possible. Making it easier to use (by creating higher-level abstractions) I think is work that is ongoing, but that work is not specifically about exposing DynamicPseudoType.

There's an argument to be made for leaving this open until it's surfaced in the helper/schema package, but I don't know that we have any intention of doing that.

With all that being said, I'm going to close the issue out, but people should feel free to comment if they think it needs to be reopened; just make your case in the comment. :)

[skylerto]

@paddycarver awesome, glad to hear it's possible! Is there any API docs, or code blocks that you could link here as an artifact?

I haven't gotten a chance yet to play around, but so far I've seen these:

• https://github.com/hashicorp/terraform-plugin-go/blob/main/tfprotov5/tftypes/primitive.go#L9
• https://github.com/hashicorp/terraform-plugin-go/blob/main/tfprotov5/dynamic_value_msgpack_test.go#L341-L345

[paddycarver]

Hey @skylerto,

Apologies, our examples are fairly lacking and the library assumes a lot of familiarity with Terraform's protocol. I can't give you a full end-to-end example, but let's see if I can give you the gist:

First, assume we're returning the schema for a resource. Let's go ahead and define that schema. In terraform-plugin-go, a schema is a map[string]tftypes.Type:

schema := map[string]tftypes.Type{
  "name": tftypes.String,
  "data": tftypes.DynamicPseudoType,
}

Then the user may specify a config like:

resource "my_resource" "foo" {
  name = "my-name"
  data = {
    "some": ["arbitrary", 123, "data", false, "here"],
  }
}

or whatever. Literally anything that can be assigned to a variable can be used as the value for data.

Then in the provider, maybe in the ApplyResourceChange RPC handler, you'll have code like this:

value, err := req.Config.Unmarshal(schema)
if err != nil {
  // return error
}
var conf map[string]tftypes.Value
err = value.As(&conf)
if err != nil {
  // return error
}

Now conf["data"] has a tftypes.Value for data. You can examine this to figure out what kind of data the user supplied. For example:

switch {
  case conf["data"].Is(tftypes.Object{}):
    var data map[string]tftypes.Value
    err := conf["data"].As(&data)
    if err != nil {
      // return error
    }
  case conf["data"].Is(tftypes.String):
    var data string
    err := conf["data"].As(&data)
    if err != nil {
        return err
    }
}

I don't know if that helps at all, but the fundamental idea is that you get the tftypes.Value with the type information from the config, and can inspect it using Value.Is and convert to Go types using Value.As.

When specifying provider-supplied data, a DynamicPseudoType just means any kind of value is acceptable in the response, just supply the value like any other, no need for special handling.

Does that help at all? Sorry, all that code was typed in a GitHub comment box, so I apologise for typos and errors.

[skylerto]

Extremely helpful @paddycarver, thank you so much!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.