Parsing api_data output

[luiscachog]

Hi again,

As mentioned on #43, I'm creating some LB's using an API call, and once the API is made I'm trying to use the output on terraform.

Right now, when I use:

output "LB-IP" {
  value = ["${restapi_object.create_lb.api_data}"]
}

This is the output:

LB-IP = [
  {
    "loadBalancer" = "map[cluster:map[name:ztm-n17.iad3.lbaas.rackspace.net] httpsRedirect:false protocol:HTTP port:80 nodes:[map[type:PRIMARY port:80 status:ONLINE condition:ENABLED weight:1 address:10.176.160.39 id:964073] map[condition:ENABLED weight:1 address:10.176.160.34 id:964075 type:PRIMARY port:80 status:ONLINE] map[condition:ENABLED weight:1 address:10.176.160.56 id:964077 type:PRIMARY port:80 status:ONLINE]] contentCaching:map[enabled:false] name:terra-lb updated:map[time:2019-07-03T21:56:34Z] halfClosed:false sourceAddresses:map[ipv6Public:2001:4802:7903:100::5/64 ipv4Public:146.20.52.5 ipv4Servicenet:10.187.186.5] timeout:120 id:250983 connectionLogging:map[enabled:false] created:map[time:2019-07-03T21:56:34Z] virtualIps:[map[id:14393 type:PUBLIC ipVersion:IPV4 address:146.20.52.137] map[ipVersion:IPV6 address:2001:4802:7903:0100:18f8:8715:0000:001a id:9.086273e+06 type:PUBLIC]] algorithm:ROUND_ROBIN status:BUILD]"
  },
]

When I use:

output "LB-IP" {
  value = ["${restapi_object.create_lb.api_data.loadBalancer}"]
}

The output is:

  "map[nodes:[map[condition:ENABLED weight:1 address:10.176.160.56 id:964085 type:PRIMARY port:80 status:ONLINE] map[address:10.176.160.34 id:964087 type:PRIMARY port:80 status:ONLINE condition:ENABLED weight:1] map[type:PRIMARY port:80 status:ONLINE condition:ENABLED weight:1 address:10.176.160.39 id:964089]] contentCaching:map[enabled:false] connectionLogging:map[enabled:false] httpsRedirect:false sourceAddresses:map[ipv6Public:2001:4802:7901::8/64 ipv4Public:162.209.114.8 ipv4Servicenet:10.189.254.8] timeout:120 name:terra-lb port:80 virtualIps:[map[ipVersion:IPV4 address:23.253.147.48 id:4583 type:PUBLIC] map[ipVersion:IPV6 address:2001:4802:7901:0000:18f8:8715:0000:0012 id:9.086277e+06 type:PUBLIC]] updated:map[time:2019-07-03T22:53:15Z] id:250987 protocol:HTTP cluster:map[name:ztm-n05.iad3.lbaas.rackspace.net] created:map[time:2019-07-03T22:53:15Z] halfClosed:false status:BUILD algorithm:ROUND_ROBIN]",
]

And finally, when I use:

output "LB-IP" {
  value = ["${lookup(restapi_object.create_lb.api_data.loadBalancer, "ipv4Public}"]
}

Does not even run correctly, the output is:

Error: Invalid character

  on terraform/outputs.tf line 17, in output "LB-IP":
  17: #   value = "${formatlist(

Please help!!

[DRuggeri]

The purpose of api_data was so you could reference one of the items that came back in the response directly. Unfortunately, it has a limitation. It can only provide that for "root" keys. So, if I parsed your example correctly, the response looks like what follows, which means only "loadBalancer" is something that can be referenced in this way. The result of that lookup will be what you are seeing: a golang representation of that key coaxed to a string.

{
  "loadBalancer": {
    "algorithm": "ROUND_ROBIN",
    "cluster": {
      "name": "ztm-n17.iad3.lbaas.rackspace.net"
    },
    "connectionLogging": {
      "enabled": false
    },
    "contentCaching": {
      "enabled": false
    },
    "created": {
      "time": "2019-07-03T21:56:34Z"
    },
    "halfClosed": false,
    "httpsRedirect": false,
    "id": 250983,
    "name": "terra-lb",
    "nodes": [
      {
        "address": "10.176.160.39",
        "condition": "ENABLED",
        "id": 964073,
        "port": 80,
        "status": "ONLINE",
        "type": "PRIMARY",
        "weight": 1
      },
      {
        "address": "10.176.160.34",
        "condition": "ENABLED",
        "id": 964075,
        "port": 80,
        "status": "ONLINE",
        "type": "PRIMARY",
        "weight": 1
      },
      {
        "address": "10.176.160.56",
        "condition": "ENABLED",
        "id": 964077,
        "port": 80,
        "status": "ONLINE",
        "type": "PRIMARY",
        "weight": 1
      }
    ],
    "port": 80,
    "protocol": "HTTP",
    "sourceAddresses": {
      "ipv4Public": "146.20.52.5",
      "ipv4Servicenet": "10.187.186.5",
      "ipv6Public": "2001:4802:7903:100::5/64"
    },
    "status": "BUILD",
    "timeout": 120,
    "updated": {
      "time": "2019-07-03T21:56:34Z"
    },
    "virtualIps": [
      {
        "address": "146.20.52.137",
        "id": 14393,
        "ipVersion": "IPV4",
        "type": "PUBLIC"
      },
      {
        "address": "2001:4802:7903:0100:18f8:8715:0000:001a",
        "id": 9086273,
        "ipVersion": "IPV6",
        "type": "PUBLIC"
      }
    ]
  }
}

[DRuggeri]

I think having a way to deeper reference the data of the API result is a very useful feature, but I'm not sure how to go about doing this right now. I'll keep this issue open with the help wanted flag to see if any proposals come up.

[gazoakley]

Hi @DRuggeri - I just saw this provider and it looks awesome! 🎉

Terraform 0.12 now has a jsondecode function. If this provider had some kind of raw output attribute (maybe api_data_raw?) something like this could work:

locals {
    json_raw = "{\"foo\":[{\"bar\":456}]}"
    json_parsed = jsondecode(local.json_raw)
}

output "json_test" {
    value = local.json_parsed.foo[0].bar
}

Alternatively, if api_data returned a JSON string of the data for that specific key it could parsed in the same way. This suggestion was made for the http provider: hashicorp/terraform#10363 (comment)

[DRuggeri]

Now that is something I can get behind! I've been thinking about also making an api_data_string type of attribute as well... jsondecode makes that a nobrainer!

[gazoakley]

That seems to work OK. I just added an attribute named api_body. This is what terraform state show looks like:

$ terraform state show module.opsgenie_database.restapi_object.default
# module.opsgenie_database.restapi_object.default:
resource "restapi_object" "default" {
    api_body = jsonencode(
        {
            data      = {
                apiKey  = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
                enabled = true
                id      = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
                name    = "RDS CloudWatch Events"
                teamId  = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
                type    = "CloudWatchEvents"
            }
            requestId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
            took      = 0.348
        }
    )
    api_data = {
        "data"      = "xxxxxxxxxxxx"
        "took"      = "0.348"
    }
    data     = jsonencode(
        {
            enabled   = true
            name      = "RDS CloudWatch Events"
            ownerTeam = {
                id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
            }
            type      = "CloudWatchEvents"
        }
    )
    id       = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    path     = "/integrations"
}

I can access the apiKey attribute like so:

output "api_key" {
  value = lookup(lookup(jsondecode(restapi_object.default.api_body), "data"), "apiKey")
}

Terraform recognises the string looks like a JSON object and nicely formats it. I can send you a PR if you're OK with it. I ended up using nested lookup calls to get the data I wanted out, but it works (and is nicer than the hacky regex based approach I was thinking of using...)

[gazoakley]

Actually, dot notation works with functions:

output "api_key" {
  value = jsondecode(restapi_object.default.api_body).data.apiKey
}

[cvalentin-adeo]

@gazoakley, need your PR :p
I have same problem, my api_data output is unusable... :/

[gazoakley]

@cvalentin-adeo: PR #54 is waiting for review.

[cvalentin-adeo]

Good news ;)

[luiscachog]

Indeed! Waiting for the approval on the PR #54

[DRuggeri]

I've reviewed #54 - and am totally on board! I have just one question about expanding scope to include the datasource, but once we settle a path forward, I'll merge and release soon. This is an awesome feature!

[DRuggeri]

With #54 having been merged and released, I think we can close this guy out!