Support for workload profiles in Terraform

[arshadazeem]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Workload profiles are in preview as part of the Consumption + Dedicated plan structure. It would helpful to create and manage workload profiles using Terraform

New or Affected Resource(s)/Data Source(s)

azurerm_container_app_environment

Potential Terraform Configuration

No response

References

No response

[thuneby]

Currently only the plan with workload profiles work within an enterprise setting with ingress and egress limitations by the firewall. This feature is thus very desired by enterprise customers using Terraform on Azure.

[RSwarnkar]

May I know if this is available yet in Terraform 3.x.x ?

[RSwarnkar]

For my case, temporary workaround was to use AzApi provider with Preview version of the ARM Microsoft.App/managedEnvironments@2022-11-01-preview :

resource "azapi_resource" "aca_env" {
  for_each = { for aca_env in var.aca_envs : aca_env.name => aca_env} 
  type      = "Microsoft.App/managedEnvironments@2022-11-01-preview"
  name      = each.value.name
  parent_id = azurerm_resource_group.rg.id
  location  = each.value.location
  tags      = merge(var.default_tags, each.value.tags)
  
  body   = jsonencode({
    properties = {
      appLogsConfiguration = {
        destination               = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = azurerm_log_analytics_workspace.law["${each.value.name}-law"].workspace_id
          sharedKey  = azurerm_log_analytics_workspace.law["${each.value.name}-law"].primary_shared_key
        }
      }
      vnetConfiguration = {
        "internal" = true
        "infrastructureSubnetId" = data.azurerm_subnet.subnets[each.value.subnet_id].id
      }
      workloadProfiles = [
        {
          name = "Consumption"
          workloadProfileType = "Consumption"
        }
      ]
    }
 })
}

[DuncanCasteleyn]

Workload profiles has left preview and is now GA.

[vermegi]

Is there any update on this one, has work started, and is there a potential date by which this will be available?

[johofer]

Enterprise customers are waiting on this! Do we have a statement by hashicorp?
The azapi workaround keeps me pulling my hair out for our complex infrastructure ...

[tim-chaffin]

Enterprise customers are waiting on this! Do we have a statement by hashicorp? The azapi workaround keeps me pulling my hair out for our complex infrastructure ...

Same here... I'm considering a "ClickOps" approach for the CAE, and then referencing it as a data object until the desired attributes become available.

[jsheetzmt]

Same here... I'm considering a "ClickOps" approach for the CAE, and then referencing it as a data object until the desired attributes become available.

@tlchaffi I converted our environment over to the AzApi approach but also had to update our container app creation to AzApi as well. Otherwise, you can't set the workloadProfileName on the container app.

https://learn.microsoft.com/en-us/azure/templates/microsoft.app/2022-11-01-preview/containerapps?pivots=deployment-language-terraform#containerappproperties-2

[tim-chaffin]

Gah, good call out. I didn't even think about that. You're right AzAPI is the only way to go right now.

Any chance you'd be willing to share your AzAPI block for both the container app environment, and a container app? OpSec stuff redacted, of course.

[jsheetzmt]

@tlchaffi hope this helps.

Env

resource "azapi_resource" "aca_env" {
  type                      = "Microsoft.App/managedEnvironments@2022-11-01-preview"
  schema_validation_enabled = false
  name                      = "${var.project_abbreviation != "" ? substr(lower(var.project_abbreviation), 0, 4) : substr(lower(var.project), 0, 4)}-${lookup(var.environment_map, var.environment)}-${lookup(var.region_map, var.region)}-${var.instance}-cae"
  parent_id                 = var.resource_group_id
  location                  = var.region
  tags                      = merge(local.tags, var.tags)
  
  body = jsonencode({
    properties = {
      daprAIConnectionString = var.app_insights_connection_string
      appLogsConfiguration = {
        destination               = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = var.log_analytics_workspace_customer_id
          sharedKey  = var.log_analytics_workspace_primary_shared_key
        }
      }
      vnetConfiguration = {
        "internal" = false
        "infrastructureSubnetId" = var.infrastructure_subnet_id
      }
      workloadProfiles = [
        {
          name = "Consumption"
          workloadProfileType = "Consumption"
        }
      ]
    }
 })
}

Container App

resource "azapi_resource" "container_app" {
  type                          = "Microsoft.App/containerApps@2022-11-01-preview"
  schema_validation_enabled     = false
  name                          = "${local.name}-ca-${var.container_name}"
  location                      = var.region
  parent_id                     = var.resource_group_id
  tags                          = merge(local.tags, var.tags)
  identity {
    type = var.user_identity_resource_id == "" ? "SystemAssigned" : "SystemAssigned, UserAssigned"
    identity_ids = var.user_identity_resource_id == "" ? [] : [var.user_identity_resource_id]
  }
  body = jsonencode({
    properties = {
      environmentId = var.container_app_environment_id
      configuration = {
        activeRevisionsMode = var.revision_mode
        dapr = var.dapr != null ? {
          appId       = var.dapr.app_id
          appPort     = var.dapr.app_port
          appProtocol = var.dapr.app_protocol
        } : null
        ingress = var.ingress != null ? {
          allowInsecure = var.ingress.allow_insecure_connections
          external      = var.ingress.external_enabled
          targetPort    = var.ingress.target_port
          transport     = var.ingress.transport
        } : null
        registries = var.registry != null ? [{
          server    = var.registry.server
          identity  = var.registry.identity
        }] : []
      }
      workloadProfileName = "Consumption"
      template = {
        scale = {
          minReplicas = var.min_replicas
          maxReplicas = var.max_replicas
        }
        containers = [{
          name = "${local.name}-ca-${var.container_name}"
          env = [for k, v in var.env_variables : {
            name = k
            value = v.value
          }]
          image = var.container_image
          probes = var.ingress != null ? [{
            httpGet = {
              port = var.liveness_probe.port
              path = var.liveness_probe.path
            }
            type = "Liveness"
          }, {
            httpGet = {
              port = var.startup_probe.port
              path = var.startup_probe.path
            }
            type = "Startup"
          }] : []
          resources = {
            cpu    = lookup(var.resource_map, var.resources)[0]
            memory = lookup(var.resource_map, var.resources)[1]
          }
        }]
      }
    }
  })
  
  response_export_values = ["identity"]
}

[tim-chaffin]

@jsheetzmt thank you! I sincerely appreciate this!

[thuneby]

@mybayern1974 please remove the preview tag, so we can get this done

[ZarakiKenpachi7]

@jsheetzmt thank you for your comment - I want to create a container app environment [ Workloads Profile ] and a container app that uses a managed identity to pull images from ACR.

My Container app environment code works but when I try to create a container app - it fails with a error that says that the managed identity is unable to pull images from ACR. I don't know if the error means to say that there's something wrong with my azapi resource block. Does anyone has any idea why this is failing?

My CAE block :

resource "azapi_resource" "app_environment" {
  type                      = "Microsoft.App/managedEnvironments@2023-05-01"
  schema_validation_enabled = false
  name                      = var.container_app_environment_name
  parent_id                 = var.application_rg_id
  location                  = var.location


  body = jsonencode({
    properties = {

      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = "XXXXXXXXXXXXXXXXXXXX"
          sharedKey  = var.law_shared_key
        }
      }
      vnetConfiguration = {
        "internal"               = true
        "infrastructureSubnetId" = var.app_env_subnet_id
      }
      workloadProfiles = [
        {
          name                = "Consumption"
          workloadProfileType = "Consumption"
        }
      ]
    }
  })
  depends_on = [azurerm_log_analytics_workspace.log_analytics_workspace]
}

Here's my container app block :

resource "azapi_resource" "container_app" {
  type      = "Microsoft.App/containerApps@2023-05-01"
  name      = "${var.name_prefix}-${var.app_name}"
  location  = "UKSouth"
  parent_id = data.azurerm_resource_group.rg.id

  identity {
    type         = "UserAssigned"
    identity_ids = [var.identity_id]
  }
  body = jsonencode({
    properties = {
      environmentId       = var.app_environment_id
      workloadProfileName = "Consumption"
      configuration = {
        activeRevisionsMode = "Single"
        ingress = {
          allowInsecure = false
          external      = false
          targetPort    = var.target_port
          transport     = "http"
          traffic = [
            {
              latestRevision = true
              weight         = 100
            }
          ]
        }
        registries = [
          {
            server   = split("/", var.container_image)[0]
            identity = var.identity_id
          }
        ]
      }
      template = {
        containers = [
          {
            name  = var.app_name
            image = var.container_image
            resources = {
              cpu    = var.container_cpu
              memory = var.container_memory
            }
            volumeMounts = [
              for volume_key, volume_value in var.volumes : {
                volumeName = volume_key
                mountPath  = volume_value.path
              }
            ]
          }
        ]
        scale = {
          maxReplicas = var.max_replicas
          minReplicas = var.min_replicas
        }
        volumes = [
          for volume_key, volume_value in var.volumes : {
            name        = volume_key
            storageName = volume_value.storage_name
            storageType = volume_value.storage_type
          }
        ]
      }
    }
  })
}

This is the error I am getting :

Error: creating/updating "Resource: (ResourceId \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-uks-u-application-01/providers/Microsoft.App/containerApps/app-uks-u-com-app01\" / Api Version \"2023-05-01\")": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-uks-u-application-01/providers/Microsoft.App/containerApps/app-uks-u-com-app01 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidParameterValueInContainerTemplate -------------------------------------------------------------------------------- { "error": { "code": "InvalidParameterValueInContainerTemplate", "message": "The following field(s) are either invalid or missing. Field 'template.containers.app01.image' is invalid with details: 'Invalid value: \"acruksucom01.azurecr.io/app01:staging_594d\": unable to pull image using Managed identity /subscriptions/XXXXXXXXXXXXXXXXX/resourceGroups/rg-uks-u-security-01/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mi-uks-u-com-01 for registry acruksucom01.azurecr.io';." } } --------------------------------------------------------------------------------
with module.app01.azapi_resource.container_app
on container_app/main.tf line 73, in resource "azapi_resource" "container_app":
resource "azapi_resource" "container_app" {

I have assigned my user-assigned managed identity ACRpull role on the subscription.

[RSwarnkar]

@ZarakiKenpachi7 did you check RBAC permissions (ACR Pull) on the managed identity?

[ZarakiKenpachi7]

@RSwarnkar yes, my uami has ACRpull on the subscription level.

[dss010101]

tbh...sometimes i think its just better to use python + azcli than terraform.

[ZarakiKenpachi7]

Alright, found the issue - my dns A records were pointing to the wrong fqdn. DNS Resolution was the issue while calling the image.

[kraduk]

Then you remember the dependency tree hell you can get the into

…

On Thu, 5 Oct 2023, 23:28 dss010101, ***@***.***> wrote: tbh...sometimes i think its just better to use python + azcli than terraform. — Reply to this email directly, view it on GitHub <#21747 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADE23XNO4YJXLPE4XFZDHTDX54X7TAVCNFSM6AAAAAAX5DOH5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZG42DCNZUGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[Ancitik]

Could we remove the preview tag in order to have this implemented please?

[Mounsseff]

Any update please ?

[jackofallops]

Closed by #23478

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.