Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for maps with non-primitive types (rich / object map support) #62

Open
radeksimko opened this issue Apr 18, 2016 · 9 comments
Assignees
Labels
enhancement New feature or request subsystem/types Issues and feature requests related to the type system of Terraform and our shims around it. terraform-plugin-framework Resolved in terraform-plugin-framework

Comments

@radeksimko
Copy link
Member

Terraform Version

Terraform v0.6.14

Terraform Configuration Files

resource "aws_api_gateway_stage" "boo" {
    rest_api_id = "${aws_api_gateway_rest_api.demo.id}"
    name = "test"

    method_setting "yada" {
        metrics_enabled = true
        logging_level = "DEBUG"
    }

    method_setting "bada" {
        metrics_enabled = false
        logging_level = "ERROR"
    }
}

Expected Behavior

+ aws_api_gateway_stage.boo
    rest_api_id:                         "" => "${aws_api_gateway_rest_api.demo.id}"
    name:                                "" => "test"
    method_setting.#:                    "" => "2"
    method_setting.yada.metrics_enabled: "" => "1"
    method_setting.yada.logging_level:   "" => "DEBUG"
    method_setting.bada.metrics_enabled: "" => "0"
    method_setting.bada.logging_level:   "" => "ERROR"

Actual Behavior

Error running plan: 1 error(s) occurred:

* method_setting: 2 error(s) decoding:

* '[bada]' expected type 'string', got unconvertible type '[]map[string]interface {}'
* '[yada]' expected type 'string', got unconvertible type '[]map[string]interface {}'

Steps to Reproduce

            "method_setting": &schema.Schema{
                Type:     schema.TypeMap,
                Optional: true,
                Elem: &schema.Resource{
                    Schema: map[string]*schema.Schema{
                        "metrics_enabled": &schema.Schema{
                            Type:     schema.TypeBool,
                            Optional: true,
                        },
                        "logging_level": &schema.Schema{
                            Type:     schema.TypeString,
                            Optional: true,
                        },
                        "data_trace_enabled": &schema.Schema{
                            Type:     schema.TypeBool,
                            Optional: true,
                        },
                    },
                },
            },
$ terraform plan

The current schema does support TypeMap which translates into map[string]interface{}. The interface is then decoded via mapstructure:
https://github.com/mitchellh/mapstructure/blob/master/mapstructure.go#L70

which seems to support slice of maps, but somehow expects string.


I'm creating this issue as I'm working around this by putting the key inside TypeSet as another field and I plan to link here.

@hugomd
Copy link

hugomd commented Mar 14, 2017

@radeksimko How did you end up working around this issue?

@glenjamin
Copy link

glenjamin commented Mar 22, 2017

I just ran into this when trying to pass list/hash types to the azurerm_template_deployment resource.

AzureRM templates and the go wrapper around them both support arbitrary types, but it looks like the diffMap function in schema.go assumes the values will be strings, and causes an error

resource "azurerm_template_deployment" "webapp" {
    name = "${var.app-name}"
    resource_group_name = "${azurerm_resource_group.group.name}"
    deployment_mode = "Incremental"
    template_body = "${file("../webapp.template.json")}"
    parameters {
        config {
            A = "1"
            B = "2"
        }
    }
}
* azurerm_template_deployment.webapp: parameters (config): '' expected type 'string', got unconvertible type '[]map[string]interface {}'

Is anyone aware of a workaround for this?

@radeksimko
Copy link
Member Author

@hugomd I think the easiest workaround - if it's a new resource - is to use Type: TypeList; MaxItems: 1. The only caveat is that addresses will then (have to) contain the index, i.e. resource_type.ref_name.list_attribute.0.nested_attr.

The other option is to just use TypeMap with no Elem which has the downside of relaxed restrictions on keys on the schema level - i.e. user can define any map keys and it would be then up to the CRUD code to do any validation. The upside is reasonable address formats (resource_type.ref_name.map_attribute.nested_attr)

@benjvi
Copy link

benjvi commented Feb 23, 2018

@radeksimko Do you know if this planned to be addressed as part of the hcl2 work? I've run into a couple of issues I've had to work around with maps now, one as you've described above, and also in trying to put heterogeneous items as values into a map which it looks like also can't work

@apparentlymart
Copy link
Contributor

Hi @benjvi!

The initial set of work addresses the low-level problems that prevented us from supporting this before, but we're not planning to include new features for the helper/schema in the first release just to minimize the number of subsystems changing (there are already a lot!). Once the core changes are in place to make this possible we plan to add new features to the helper/schema API that will make use of these new capabilities as a separate step.


I know some readers appreciate having some more details on why certain things in Terraform behave the way they do, so what follows is a little background on why TypeMap doesn't support complex-typed elements in the current release. Feel free to skip this entirely if you're not interested in Terraform's internals; none of this knowledge is required for Terraform users.

In Terraform Core's current internal model, the state for a resource is modeled as a flat map from strings to strings, with complex data structured flattened into dot-separated keys as you can see in the plan output, like in Radek's example above:

    rest_api_id:                         "" => "${aws_api_gateway_rest_api.demo.id}"
    name:                                "" => "test"
    method_setting.#:                    "" => "2"
    method_setting.yada.metrics_enabled: "" => "1"
    method_setting.yada.logging_level:   "" => "DEBUG"
    method_setting.bada.metrics_enabled: "" => "0"
    method_setting.bada.logging_level:   "" => "ERROR"

This technique dates back to the earliest versions of Terraform, where lists and maps were not supported yet at all. Support for complex types has been gradually improved, with the most notable changes happening in 0.7, but the physical model internally has been unchanged due to the fact that this is a cross-cutting concern that touches almost all of Terraform's subsystems.

This flattening technique is lossy due to limitations of the key format. If there were any key in the path containing a period then the result would be ambiguous:

    method_setting "ya.da" {
        metrics_enabled = true
        logging_level = "DEBUG"
    }
    method_setting.ya.da.metrics_enabled: "" => "1"
    method_setting.ya.da.logging_level:   "" => "DEBUG"

In this scenario helper/schema could in principle walk backwards from the deeper schema and infer that da must be part of the key because there is no da attribute in the nested resource schema, but this would quickly become complex and possibly undecidable with many nested maps that may have identical intermediate attribute names.

In all cases except maps we know that periods cannot appear in the segments, because periods are not valid in attribute names and indexes into lists and sets are numeric and thus cannot contain periods. Maps of primitive types are possible because we know that primitives cannot have attributes or elements and so any remaining characters in the flattened key must be the key into the map structure.

The lower-level change made by our configuration language work is to represent these data structures internally without flattening them, using the type system of the new language interpreter. Retaining the individual collection objects removes the ambiguity, and avoids the need for helper/schema to infer the original structure itself based on the period-separated path.

@kilokahn
Copy link

kilokahn commented Feb 6, 2019

hey @apparentlymart - I think you may have already explained this above, but wanted to check in if this support is available in the v0.12 alpha builds of the provider-sdk? Maybe not yet (since you said we'd not change it in the first release), but wanted to know. Thanks! 😄

@apparentlymart
Copy link
Contributor

Indeed, the current state is that Terraform Core supports this but the SDK cannot, because the SDK is still working with the old-style representation. Our goal for the initial v0.12.0 release is to ensure that the existing SDK functionality is all still broadly working in spite of the type system changes, which has been tricky enough as it is, so we're not going to try to introduce any new functionality at the same time.

Once we have v0.12.0 final released, we'll begin discussions about how to improve the SDK itself, which should include making it use the new native Terraform type system instead of its own legacy representation, so that it can make use of all of the new capabilities.

platinummonkey referenced this issue in platinummonkey/terraform-provider-datadog Aug 13, 2019
nmuesch referenced this issue in DataDog/terraform-provider-datadog Aug 15, 2019
* add SLO support

* fix docs

* fix docs

* small tweaks to formatting

* PR review fixes

* add clarifying sentence

* fix test

* fix provider to Optional fields

make syntax backwards compatible

suppress *_display changes

* because of https://github.com/hashicorp/terraform/issues/6215/ move from TypeMap to TypeList and validate

* add validation to query options

* Revert "add validation to query options"

This reverts commit 6338ed2.

* fix docs

address possible difference in state value for thresholds

* apply same logic to query since it is now TypeList

* fix

* ValueType casts as []interface{} for TypeList

* fix dangling character
@yann-soubeyrand
Copy link

Hi,

What is the state of the discussions on upgrading the SDK?

@hashibot hashibot transferred this issue from hashicorp/terraform Sep 26, 2019
@hashibot hashibot added the enhancement New feature or request label Oct 2, 2019
@radeksimko
Copy link
Member Author

The problem described in my initial post could be solved either by #220 or #155 either of which may actually be better suited solution for that problem.

Both problems describe the need for complex/nested data structures, they just aim to solve it in different ways.

There may still be value in allowing more/variadic types in TypeMap though, so I'll keep this open for now.

@paultyng paultyng added this to the v2.x (next minor) milestone Apr 30, 2020
@paddycarver paddycarver removed this from the v2.x (next minor) milestone Dec 17, 2020
@paddycarver paddycarver added the subsystem/types Issues and feature requests related to the type system of Terraform and our shims around it. label Jan 6, 2021
0gajun added a commit to 0gajun/terraform-provider-cloudflare that referenced this issue Mar 18, 2022
Currently, TypeMap doesn't support non-primitive type values.
ref : hashicorp/terraform-plugin-sdk#62

When using non-primitive values with TypeMap, terraform crashes at
runtime with a following error message.

```
panic: Unknown validation type: 6
```

This error is originated at
https://github.com/hashicorp/terraform-plugin-sdk/blob/v2.11.0/helper/schema/schema.go#L2017
Only TypeBool, TypeInt, TypeFloat and TypeString are supported.

So, in this change, use TypeList with new custom resource to store
non-primitive values.
0gajun added a commit to 0gajun/terraform-provider-cloudflare that referenced this issue Mar 18, 2022
Currently, TypeMap doesn't support non-primitive type values.
ref : hashicorp/terraform-plugin-sdk#62

When using non-primitive values with TypeMap, terraform crashes at
runtime with a following error message.

```
panic: Unknown validation type: 6
```

This error is originated at
https://github.com/hashicorp/terraform-plugin-sdk/blob/v2.11.0/helper/schema/schema.go#L2017
Only TypeBool, TypeInt, TypeFloat and TypeString are supported.

So, in this change, use TypeList with new custom resource to store
non-primitive values.
0gajun added a commit to 0gajun/terraform-provider-cloudflare that referenced this issue Mar 21, 2022
Currently, TypeMap doesn't support non-primitive type values.
ref : hashicorp/terraform-plugin-sdk#62

When using non-primitive values with TypeMap, terraform crashes at
runtime with a following error message.

```
panic: Unknown validation type: 6
```

This error is originated at
https://github.com/hashicorp/terraform-plugin-sdk/blob/v2.11.0/helper/schema/schema.go#L2017
Only TypeBool, TypeInt, TypeFloat and TypeString are supported.

So, in this change, use TypeList with new custom resource to store
non-primitive values.
@bflad bflad added terraform-plugin-framework Resolved in terraform-plugin-framework and removed protocol-5-only labels Mar 30, 2022
iwahbe added a commit to pulumi/pulumi-terraform-bridge that referenced this issue Apr 12, 2024
~This fixes a bug in path traversals with nested `MaxItems: 1` values, but I don't know if
that has ever effected customers.~ This would fix a bug in path traversals through maps,
but SDKv2 does not currently support complex types under
maps (hashicorp/terraform-plugin-sdk#62), so this is a pure
refactor.
iwahbe added a commit to pulumi/pulumi-terraform-bridge that referenced this issue Apr 13, 2024
~This fixes a bug in path traversals with nested `MaxItems: 1` values, but I don't know if
that has ever effected customers.~ This would fix a bug in path traversals through maps,
but SDKv2 does not currently support complex types under
maps (hashicorp/terraform-plugin-sdk#62), so this is a pure
refactor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request subsystem/types Issues and feature requests related to the type system of Terraform and our shims around it. terraform-plugin-framework Resolved in terraform-plugin-framework
Projects
None yet
Development

No branches or pull requests