Lambda in Docker publishing error

[IaroslavR]

$ terraform -v
Terraform v0.14.3
+ provider registry.terraform.io/hashicorp/aws v3.20.0
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0

Module code:

module "lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "1.31.0"

  function_name  = "${var.application}-${var.environment}-docker-lambda"
  create_package = false
  publish        = true

  image_uri                         = "${var.ecr}/${var.image_name}:${var.image_tag}"
  package_type                      = "Image"
  attach_cloudwatch_logs_policy     = true
  cloudwatch_logs_retention_in_days = var.cloudwatch_logs_retention_in_days

  memory_size                    = var.memory_size
  reserved_concurrent_executions = var.reserved_concurrent_executions
  timeout                        = var.timeout

  vpc_subnet_ids         = var.vpc_subnet_ids
  vpc_security_group_ids = var.vpc_security_group_ids
  attach_network_policy  = true

  tags = var.tags
}

Error log:

module.docker_lambda.module.lambda.aws_lambda_function.this[0]: Modifying... [id=lambdas-test-docker-lambda]
Error: Error publishing Lambda Function (lambdas-test-docker-lambda) version: ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:0000000000:function:lambdas-test-docker-lambda
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "7f03a622-e059-46f1-804b-a1975f753eb5"
  },
  Message_: "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:0000000000:function:lambdas-test-docker-lambda",
  Type: "User"
}

Workaround - run terragrunt apply again after update will be done.

[samkirton]

I am also experiencing this issue

[antonbabenko]

I can't reproduce this issue by running examples/container-image combined with examples/with-vpc.

Probably, this issue is related - hashicorp/terraform-provider-aws#5154

Can you see if your execution log looks similar to this one?

random_pet.this: Creating...
random_pet.this: Creation complete after 0s [id=game-ladybird]
aws_ecr_repository.this: Creating...
module.vpc.aws_vpc.this[0]: Creating...
module.lambda_function_from_container_image.aws_cloudwatch_log_group.lambda[0]: Creating...
module.lambda_function_from_container_image.aws_iam_role.lambda[0]: Creating...
module.lambda_function_from_container_image.aws_iam_policy.vpc[0]: Creating...
aws_ecr_repository.this: Creation complete after 1s [id=game-ladybird]
module.lambda_function_from_container_image.aws_cloudwatch_log_group.lambda[0]: Creation complete after 1s [id=/aws/lambda/game-ladybird-lambda-from-container-image]
module.lambda_function_from_container_image.data.aws_iam_policy_document.logs[0]: Reading...
module.lambda_function_from_container_image.data.aws_iam_policy_document.logs[0]: Read complete after 0s [id=2180372558]
docker_registry_image.app: Creating...
module.lambda_function_from_container_image.aws_iam_policy.logs[0]: Creating...
module.lambda_function_from_container_image.aws_iam_role.lambda[0]: Creation complete after 1s [id=game-ladybird-lambda-from-container-image]
module.lambda_function_from_container_image.aws_iam_policy.vpc[0]: Creation complete after 2s [id=arn:aws:iam::835367859851:policy/game-ladybird-lambda-from-container-image-vpc]
module.lambda_function_from_container_image.aws_iam_policy_attachment.vpc[0]: Creating...
module.lambda_function_from_container_image.aws_iam_policy.logs[0]: Creation complete after 1s [id=arn:aws:iam::835367859851:policy/game-ladybird-lambda-from-container-image-logs]
module.lambda_function_from_container_image.aws_iam_policy_attachment.logs[0]: Creating...
module.lambda_function_from_container_image.aws_iam_policy_attachment.vpc[0]: Creation complete after 1s [id=game-ladybird-lambda-from-container-image-vpc]
module.vpc.aws_vpc.this[0]: Creation complete after 3s [id=vpc-0257b1131001b6a6d]
module.vpc.aws_route_table.intra[0]: Creating...
module.vpc.aws_subnet.intra[1]: Creating...
module.vpc.aws_subnet.intra[0]: Creating...
module.vpc.aws_subnet.intra[2]: Creating...
module.lambda_function_from_container_image.aws_iam_policy_attachment.logs[0]: Creation complete after 2s [id=game-ladybird-lambda-from-container-image-logs]
module.vpc.aws_route_table.intra[0]: Creation complete after 1s [id=rtb-0043ebf348cd99326]
module.vpc.aws_subnet.intra[2]: Creation complete after 1s [id=subnet-0a1c5f5fcccf6e3aa]
module.vpc.aws_subnet.intra[0]: Creation complete after 2s [id=subnet-0b16426f650f7cdfe]
module.vpc.aws_subnet.intra[1]: Creation complete after 2s [id=subnet-04db8476b6c80a227]
module.vpc.aws_route_table_association.intra[2]: Creating...
module.vpc.aws_route_table_association.intra[1]: Creating...
module.vpc.aws_route_table_association.intra[0]: Creating...
module.vpc.aws_route_table_association.intra[0]: Creation complete after 0s [id=rtbassoc-04dd01c4a386d0193]
module.vpc.aws_route_table_association.intra[1]: Creation complete after 0s [id=rtbassoc-0a714530273925cfc]
module.vpc.aws_route_table_association.intra[2]: Creation complete after 0s [id=rtbassoc-06865117e6874d05b]
docker_registry_image.app: Creation complete after 4s [id=sha256:45a5a55f2e9b79a6ea15c1efea93b08c231e746a13741d027d7a142d8326ad92]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Creating...
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [10s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [20s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [30s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [40s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [50s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m0s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m10s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m20s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m30s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m40s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [1m50s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m0s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m10s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m20s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m30s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m40s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [2m50s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [3m0s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Still creating... [3m10s elapsed]
module.lambda_function_from_container_image.aws_lambda_function.this[0]: Creation complete after 3m13s [id=game-ladybird-lambda-from-container-image]

Apply complete! Resources: 18 added, 0 changed, 0 destroyed.

[mauriciocm9]

I was able to reproduce the problem by setting the publish argument to true and the source_code_hash argument with the latest digest from the ECR repository (so the latest change of the image is used). Since source_code_hash has changed, terraform will trigger the UpdateFunctionCode method causing the lambda to move to pending state while the image undergoes an optimization process for running on Lambda.
Once the image is updated, terraform will try to publish the version. However it will fail because the lambda is still in pending state.
Here are some logs of the plan:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.lambda_function_from_container_image.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        arn                            = "arn:aws:lambda:us-east-1:xxx:function:bursting-deer-lambda-from-container-image"
        description                    = "My awesome lambda function from container image"
        function_name                  = "bursting-deer-lambda-from-container-image"
        id                             = "bursting-deer-lambda-from-container-image"
        image_uri                      = "xxx.dkr.ecr.us-east-1.amazonaws.com/bursting-deer:1.0"
        invoke_arn                     = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:xxx:function:bursting-deer-lambda-from-container-image/invocations"
      ~ last_modified                  = "2021-01-18T00:08:53.000+0000" -> (known after apply)
        layers                         = []
        memory_size                    = 128
        package_type                   = "Image"
        publish                        = true
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:xxx:function:bursting-deer-lambda-from-container-image:$LATEST" -> (known after apply)
        reserved_concurrent_executions = -1
        role                           = "arn:aws:iam::xxx:role/bursting-deer-lambda-from-container-image"
      ~ source_code_hash               = "6c4f42f959e3d6471b4128a4f3806e43a8aaa36dc1ca7e67b30ecf994a0439f8" -> "d09bd32a66fb7747aa2d180c4326011822cf1195704f2ca24c07f6ea8351bc96"
        source_code_size               = 0
        tags                           = {}
        timeout                        = 3
      ~ version                        = "$LATEST" -> (known after apply)

        tracing_config {
            mode = "PassThrough"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Once the apply has finished (with Error publishing Lambda Function message), you can check that the lambda status is still in pending:
aws lambda get-function-configuration --function-name bursting-deer-lambda-from-container-image

{
    "FunctionName": "bursting-deer-lambda-from-container-image",
    "FunctionArn": "arn:aws:lambda:us-east-1:xxxx:function:bursting-deer-lambda-from-container-image",
    "Role": "arn:aws:iam::xxx:role/bursting-deer-lambda-from-container-image",
    "CodeSize": 0,
    "Description": "My awesome lambda function from container image",
    "Timeout": 3,
    "MemorySize": 128,
    "LastModified": "2021-01-18T00:13:05.000+0000",
    "CodeSha256": "6c4f42f959e3d6471b4128a4f3806e43a8aaa36dc1ca7e67b30ecf994a0439f8",
    "Version": "$LATEST",
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "bc691819-b517-42a4-85ad-ba9b2e828449",
    "State": "Active",
    "LastUpdateStatus": "InProgress",
    "LastUpdateStatusReason": "The function is being created.",
    "LastUpdateStatusReasonCode": "Creating"
}

[antonbabenko]

@mauriciocm9 Thank you that you have also opened an issue in the AWS provider - hashicorp/terraform-provider-aws#17153

[antonbabenko]

This issue is fixed in Terraform AWS provider v3.29.0 (ref - hashicorp/terraform-provider-aws#17153).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.