aws_lambda_function ResourceConflictException due to a concurrent update operation

[jannikko]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

• provider.aws v1.26.0
• provider.template v1.0.0

Affected Resource(s)

aws_lambda_function

Terraform Configuration Files

This is an extract of my main terraform file, it shows all the resources relevant to the issue:

resource "aws_lambda_function" "voucher-validation-lambda" {
  filename = "${var.artifact}"
  function_name = "${var.prefix}voucher-validation"
  publish = true
  role = "${lookup(var.s3_lambda_role, var.target)}"
  handler = "vouchers.validation.VoucherValidationHandler"
  source_code_hash = "${var.artifact}"
  runtime = "java8"
  memory_size = "256"
  timeout = "60"

  vpc_config {
    subnet_ids = ["${data.terraform_remote_state.global.subnet-business-services-1a.id}", "${data.terraform_remote_state.global.subnet-business-services-1b.id}"]
    security_group_ids = ["${data.aws_security_group.default.id}"]
  }
}

resource "aws_lambda_permission" "voucher-validation-lambda-permission" {
  depends_on = [
    "aws_lambda_function.voucher-validation-lambda",
    "aws_api_gateway_rest_api.voucher-validation-api",
    "aws_api_gateway_method.voucher-validation-api-post"
  ]
  statement_id = "AllowExecutionFromAPIGateway"
  action = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.voucher-validation-lambda.function_name}"
  principal = "apigateway.amazonaws.com"
}

resource "aws_lambda_permission" "voucher-validation-lambda-permission-method" {
  depends_on = [
    "aws_lambda_function.voucher-validation-lambda",
    "aws_lambda_permission.voucher-validation-lambda-permission",
    "aws_api_gateway_rest_api.voucher-validation-api",
    "aws_api_gateway_method.voucher-validation-api-post"
  ]
  statement_id = "AllowExecutionFromAPIGatewayMethod"
  action = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.voucher-validation-lambda.function_name}"
  principal = "apigateway.amazonaws.com"
  source_arn = "arn:aws:execute-api:${var.region}:${lookup(var.aws_account_id, var.target)}:${aws_api_gateway_rest_api.voucher-validation-api.id}/*/*"
}

resource "aws_lambda_permission" "voucher-validation-lambda-cloudwatch-permission" {
  depends_on = [
    "aws_lambda_function.voucher-validation-lambda",
    "aws_lambda_permission.voucher-validation-lambda-permission-method",
    "aws_cloudwatch_event_target.keep_lambda_hot",
    "aws_cloudwatch_event_rule.every_15_minutes"
  ]
  action = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.voucher-validation-lambda.function_name}"
  principal = "events.amazonaws.com"
  statement_id = "AllowExecutionFromCloudWatch"
  source_arn = "${aws_cloudwatch_event_rule.every_15_minutes.arn}"

resource "aws_cloudwatch_event_rule" "every_15_minutes" {
  name = "keep-voucher-validation-hot-${uuid()}"
  is_enabled = "true"
  schedule_expression = "rate(15 minutes)"
}

resource "aws_cloudwatch_event_target" "keep_lambda_hot" {
  depends_on = ["aws_lambda_function.voucher-validation-lambda"]
  rule = "${aws_cloudwatch_event_rule.every_15_minutes.name}"
  target_id = "voucher-validation-lambda"
  arn = "${aws_lambda_function.voucher-validation-lambda.arn}"
  input = "{\"keepLambdaHot\": true}"
}

A lot of the depends_on statements were added later to try to solve the issue but it did not really make a difference.

Debug Output

data.terraform_remote_state.global: Refreshing state...
data.terraform_remote_state.logstash: Refreshing state...
aws_cloudwatch_event_rule.every_15_minutes: Refreshing state... (ID: keep-voucher-validation-hot-34c34825-b9b6-6c64-df43-3db1a3f61432)
aws_api_gateway_rest_api.voucher-validation-api: Refreshing state... (ID: ka71rldd07)
aws_api_gateway_resource.voucher-validation-api-resource: Refreshing state... (ID: zn8sfr)
data.aws_security_group.default: Refreshing state...
aws_api_gateway_method.voucher-validation-api-post: Refreshing state... (ID: agm-ka71rldd07-zn8sfr-POST)
aws_lambda_function.voucher-validation-lambda: Refreshing state... (ID: voucher-validation)
aws_api_gateway_integration.voucher-validation-integration: Refreshing state... (ID: agi-ka71rldd07-zn8sfr-POST)
aws_lambda_permission.voucher-validation-lambda-permission: Refreshing state... (ID: AllowExecutionFromAPIGateway)
aws_cloudwatch_event_target.keep_lambda_hot: Refreshing state... (ID: keep-voucher-validation-hot-34c34825-b9...3db1a3f61432-voucher-validation-lambda)
aws_lambda_permission.voucher-validation-lambda-permission-method: Refreshing state... (ID: AllowExecutionFromAPIGatewayMethod)
aws_api_gateway_deployment.voucher-validation-api-deployment: Refreshing state... (ID: k1311i)
aws_lambda_permission.voucher-validation-lambda-cloudwatch-permission: Refreshing state... (ID: AllowExecutionFromCloudWatch)
aws_api_gateway_base_path_mapping.voucher-validation-api-mapping: Refreshing state... (ID: XXX/voucher-validation)
aws_lambda_permission.voucher-validation-lambda-cloudwatch-permission: Destroying... (ID: AllowExecutionFromCloudWatch)
aws_lambda_function.voucher-validation-lambda: Modifying... (ID: voucher-validation)
  filename:         "../voucher-validation-build-20180711-145700.zip" => "../voucher-validation-build-20180711-152240.zip"
  last_modified:    "2018-07-11T13:01:17.858+0000" => "<computed>"
  qualified_arn:    "" => "<computed>"
  source_code_hash: "hmDwZyNt9s62X0Bgihg4sItVG620pssjYblnJykT97c=" => "../voucher-validation-build-20180711-152240.zip"
  version:          "39" => "<computed>"
aws_lambda_permission.voucher-validation-lambda-cloudwatch-permission: Destruction complete after 0s
aws_cloudwatch_event_target.keep_lambda_hot: Destroying... (ID: keep-voucher-validation-hot-34c34825-b9...3db1a3f61432-voucher-validation-lambda)
aws_cloudwatch_event_target.keep_lambda_hot: Destruction complete after 0s
aws_cloudwatch_event_rule.every_15_minutes: Destroying... (ID: keep-voucher-validation-hot-34c34825-b9b6-6c64-df43-3db1a3f61432)
aws_cloudwatch_event_rule.every_15_minutes: Destruction complete after 0s
aws_cloudwatch_event_rule.every_15_minutes: Creating...
  arn:                 "" => "<computed>"
  is_enabled:          "" => "true"
  name:                "" => "keep-voucher-validation-hot-4521cd11-fe17-bcae-f741-9a9b2e167cec"
  schedule_expression: "" => "rate(15 minutes)"
aws_cloudwatch_event_rule.every_15_minutes: Creation complete after 1s (ID: keep-voucher-validation-hot-4521cd11-fe17-bcae-f741-9a9b2e167cec)

Error: Error applying plan:

1 error(s) occurred:

* aws_lambda_function.voucher-validation-lambda: 1 error(s) occurred:

* aws_lambda_function.voucher-validation-lambda: Error modifying Lambda Function Configuration voucher-validation: ResourceConflictException: The function could not be updated due to a concurrent update operation.
	status code: 409, request id: 9280e5d8-850d-11e8-a03f-5d871c0e8cce

Expected Behavior

Should modify the Lambda function with the updated source code without crashing.

Actual Behavior

Throws an error and only works on the second try

[brikis98]

Has anyone found a workaround for this? It has shown up for some of our Lambda functions and we're now unable to deploy them using Terraform.

[brikis98]

Looks like the serverless folks are hitting this issue too, so perhaps an AWS bug? serverless/serverless#4964

[brikis98]

Running destroy to delete the Lambda function and apply to recreate it from scratch works to fix this... But then next time you go to update the Lambda function, you get the same error again.

[brikis98]

OK, I've figured out what's happening here based on a comment here: AWS has some sort of limit on how many concurrent modifications you can make to a Lambda function. In our code, we had a bunch of resources like this:

resource "aws_lambda_permission" "foo" {
  statement_id  = "foo"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"
}

resource "aws_lambda_permission" "bar" {
  statement_id  = "bar"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"
}

resource "aws_lambda_permission" "baz" {
  statement_id  = "baz"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"
}

It turns out that each of these modifies the Lambda function in the function_name parameter, and by default, Terraform will try to do all these modifications concurrently. This triggers an error. The workaround for now is ugly: you force Terraform to make these changes sequentially by daisy-chaining depends_on calls:

resource "aws_lambda_permission" "foo" {
  statement_id  = "foo"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"
}

resource "aws_lambda_permission" "bar" {
  statement_id  = "bar"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"

  depends_on = ["aws_lambda_permission.foo"]
}

resource "aws_lambda_permission" "baz" {
  statement_id  = "baz"
  action        = "lambda:InvokeFunction"
  function_name = "${var.lambda_function_name}"
  principal     = "apigateway.amazonaws.com"

  depends_on = ["aws_lambda_permission.bar"]
}

[jannikko]

Well, if you take a look at my postе, this is what we tried as well. Even though there might be a workaround like this, I think it's still a bug.

[brikis98]

I saw you mentioned:

A lot of the depends_on statements were added later to try to solve the issue but it did not really make a difference.

But for me, it definitely fixed it.

Even though there might be a workaround like this, I think it's still a bug.

Yes, 100% a bug.

[brikis98]

And today the bug is back. The depends_on fix I mentioned earlier worked reliably for a few days and now, despite no significant code changes, I get the concurrent update operation error every single time.

I've now resorted to a crappy workaround: run apply with the -parallelism=1 flag 😢

[kyleian]

This cropped up for me today creating two aws_lambda_function resources, but only appears when targeting us-east-1; Been using these same resource definitions for a variety of projects in us-west-2 for months and have never seen it come up until today; doesn't make much sense to me why region would make a difference here but perhaps worth calling out.

[maulik887]

This is happening for me as well (same terraform stuff, Lambda function + CW event target + CW event + permission) . I have tried in both region us-east-1 and us-west-2 and it's happening in both the regions.

[vinod386]

Running into the same issue here. @apparentlymart Do we know if someone's working on this?

[Botono]

I am seeing this error today with a very simple setup. API Gateway with Lambda Proxy. I added code to force Terraform to re-deploy the API on configuration changes (by updating the stage_description parameter of the aws_api_gateway_deployment resource with a hash). I believe this causes a concurrency conflict error when updating the Lambda function code at the same time. According to CloudTrail logs, the AddPermission20150331v2 and the UpdateFunctionCode20150331v2 events on the Lambda function happen within 1 second of each other.

Seeing as the deployment setup I'm attempting is a workaround I will revert it and go back to manually deploying the API after changes.

I am still using v1.60 of the AWS provider. Terraform v0.11.13.

[Botono]

Adding a create_before_destroy lifecycle rule to my aws_lambda_permission resources seems to have solved this bug for me. The deployment gets updated, then the Lambda function code is updated, then the new Lambda permission is created and the old one destroyed.

[Aerendel]

This is definitely still happening, been running into this exact issue for a few days now.

[vinod386]

@Aerendel fyi, daisy chaining the resources with depends_on worked for me

[kmaid]

I have found that terraform tries to create two aws_lambda_permission in the same region concurrently it fails with the error:
Error modifying Lambda Function Configuration XXXXX: ResourceConflictException: The function could not be updated due to a concurrent update operation.

I found that that I had the source_arn of my aws_lambda_permission set to my aws_api_gateway_deployment. This resulted in two aws_lambda_permission being destroyed and recreated with every aws_api_gateway_deployment deployment. The solution was to set it to the aws_api_gateway_rest_api so these permissions aren't updated each time.

The terraform AWS provider should probably take the no more than 1 AWS permission being set at once into account however.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.