lambda acceptance tests failing randomly on master

[Puneeth-n]

Tests fail randomly. In this case TestAccAWSLambdaFunction_encryptedEnvVariables

blocks: #2942 #2964

commit tested on: ced7b06

AWS_REGION=us-east-1 make testacc TESTARGS='-run=TestAccAWSLambdaFunction_'

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -run=TestAccAWSLambdaFunction_ -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccAWSLambdaFunction_importLocalFile
--- PASS: TestAccAWSLambdaFunction_importLocalFile (42.35s)
=== RUN   TestAccAWSLambdaFunction_importLocalFile_VPC
--- PASS: TestAccAWSLambdaFunction_importLocalFile_VPC (45.99s)
=== RUN   TestAccAWSLambdaFunction_importS3
--- PASS: TestAccAWSLambdaFunction_importS3 (49.20s)
=== RUN   TestAccAWSLambdaFunction_basic
--- PASS: TestAccAWSLambdaFunction_basic (40.87s)
=== RUN   TestAccAWSLambdaFunction_concurrency
--- PASS: TestAccAWSLambdaFunction_concurrency (65.67s)
=== RUN   TestAccAWSLambdaFunction_concurrencyCycle
--- PASS: TestAccAWSLambdaFunction_concurrencyCycle (86.00s)
=== RUN   TestAccAWSLambdaFunction_updateRuntime
--- PASS: TestAccAWSLambdaFunction_updateRuntime (62.04s)
=== RUN   TestAccAWSLambdaFunction_expectFilenameAndS3Attributes
--- PASS: TestAccAWSLambdaFunction_expectFilenameAndS3Attributes (27.67s)
=== RUN   TestAccAWSLambdaFunction_envVariables
--- PASS: TestAccAWSLambdaFunction_envVariables (116.19s)
=== RUN   TestAccAWSLambdaFunction_encryptedEnvVariables
--- FAIL: TestAccAWSLambdaFunction_encryptedEnvVariables (76.59s)
	testing.go:513: Step 1 error: Check failed: Check 4/5 error: aws_lambda_function.lambda_function_test: Attribute 'environment.0.variables.foo' not found
=== RUN   TestAccAWSLambdaFunction_versioned
--- PASS: TestAccAWSLambdaFunction_versioned (39.04s)
=== RUN   TestAccAWSLambdaFunction_DeadLetterConfig
--- PASS: TestAccAWSLambdaFunction_DeadLetterConfig (48.64s)
=== RUN   TestAccAWSLambdaFunction_DeadLetterConfigUpdated
--- PASS: TestAccAWSLambdaFunction_DeadLetterConfigUpdated (63.92s)
=== RUN   TestAccAWSLambdaFunction_nilDeadLetterConfig
--- PASS: TestAccAWSLambdaFunction_nilDeadLetterConfig (28.14s)
=== RUN   TestAccAWSLambdaFunction_tracingConfig
--- PASS: TestAccAWSLambdaFunction_tracingConfig (62.50s)
=== RUN   TestAccAWSLambdaFunction_VPC
--- PASS: TestAccAWSLambdaFunction_VPC (42.97s)
=== RUN   TestAccAWSLambdaFunction_VPCUpdate
--- PASS: TestAccAWSLambdaFunction_VPCUpdate (74.86s)
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
--- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (70.64s)
=== RUN   TestAccAWSLambdaFunction_s3
--- PASS: TestAccAWSLambdaFunction_s3 (47.36s)
=== RUN   TestAccAWSLambdaFunction_localUpdate
--- PASS: TestAccAWSLambdaFunction_localUpdate (47.79s)
=== RUN   TestAccAWSLambdaFunction_localUpdate_nameOnly
--- PASS: TestAccAWSLambdaFunction_localUpdate_nameOnly (48.32s)
=== RUN   TestAccAWSLambdaFunction_s3Update_basic
--- PASS: TestAccAWSLambdaFunction_s3Update_basic (106.45s)
=== RUN   TestAccAWSLambdaFunction_s3Update_unversioned
--- PASS: TestAccAWSLambdaFunction_s3Update_unversioned (77.96s)
=== RUN   TestAccAWSLambdaFunction_runtimeValidation_noRuntime
--- PASS: TestAccAWSLambdaFunction_runtimeValidation_noRuntime (1.93s)
=== RUN   TestAccAWSLambdaFunction_runtimeValidation_nodeJs43
--- PASS: TestAccAWSLambdaFunction_runtimeValidation_nodeJs43 (39.32s)
=== RUN   TestAccAWSLambdaFunction_runtimeValidation_python27
--- PASS: TestAccAWSLambdaFunction_runtimeValidation_python27 (39.41s)
=== RUN   TestAccAWSLambdaFunction_runtimeValidation_java8
--- PASS: TestAccAWSLambdaFunction_runtimeValidation_java8 (40.09s)
=== RUN   TestAccAWSLambdaFunction_tags
--- PASS: TestAccAWSLambdaFunction_tags (86.83s)
=== RUN   TestAccAWSLambdaFunction_runtimeValidation_python36
--- PASS: TestAccAWSLambdaFunction_runtimeValidation_python36 (40.90s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	1619.664s
GNUmakefile:18: recipe for target 'testacc' failed
make: *** [testacc] Error 1

[Puneeth-n]

cc: @bflad

[bflad]

Thanks for creating an issue for this! We've been noticing this in our daily acceptance test runs as well. This seems to be related to the following events executed by that test currently:

1. Create lambda function with environment variable and KMS CMK
2. Delete KMS CMK and update lambda function to remove KMS CMK

On read it then gets:

2018/01/17 07:04:19 [DEBUG] [aws-sdk-go] {..."Environment":{"Error":{"ErrorCode":"KMSInvalidStateException","Message":"Lambda was unable to decrypt your environment variables because the KMS key used is an invalid state. Please check your KMS key settings.KMS Exception: KMSInvalidStateException KMS Message: arn:aws:kms:us-west-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx is pending deletion."},"Variables":null},

Which now returns the empty environment back. I'm not sure if the right fix is re-passing the environment variables back when updating the KMS attribute or something else yet, but I'll give that a shot first.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!