Support VPC config for Amazon Kinesis Data Firehose #13269

rajholla · 2020-05-11T21:32:54Z

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Release note for CHANGELOG:

Amazon Kinesis Data Firehose can now deliver streaming data to an Amazon Elasticsearch Service domain in an Amazon VPC.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1677.81s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1677.859s

terraform-provider-aws git:(firehose-es-vpc-support) ✗ make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream -timeout 120m=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (121.13s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (123.65s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (127.99s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (140.90s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (156.63s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (158.61s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (162.78s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (163.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (166.46s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (167.18s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (171.60s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (186.54s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (193.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (223.99s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (226.55s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (229.93s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (139.98s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (121.53s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (160.25s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (156.87s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (302.40s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (180.29s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (557.15s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (852.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1830.37s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1830.450s
...

DrFaust92

A few minor changes.

can you also add an example in the docs for this?

aws/resource_aws_kinesis_firehose_delivery_stream.go

DrFaust92 · 2020-05-11T21:54:21Z

aws/resource_aws_kinesis_firehose_delivery_stream.go

+		return nil
+	}
+	vpcConfig := config[0].(map[string]interface{})
+	s := vpcConfig["subnet_ids"].(*schema.Set).List()


helper func expandStringSet() can be used on vpcConfig["subnet_ids"].(*schema.Set) instead

aws/resource_aws_kinesis_firehose_delivery_stream.go

rajholla · 2020-05-12T00:58:10Z

I have addressed the PR comments.

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1605.30s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1605.359s

ewbankkit · 2020-05-18T22:18:27Z

Verified acceptance tests:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_basic -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (110.48s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	110.514s

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (2.45s)
    resource_aws_elasticsearch_domain_test.go:991: missing IAM Service Linked Role (es.amazonaws.com), please create it in the AWS account and retry
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	2.487s

$ aws iam create-service-linked-role --aws-service-name es.amazonaws.com

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1444.38s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1444.406s

ssreechandra · 2020-05-22T14:23:29Z

when could I expect this PR to be merged ?

bramalingam81 · 2020-05-26T10:55:42Z

@ewbankkit , Could you please let us know by when the PR would be merged ?

rajholla · 2020-06-02T02:41:34Z

Hello @ewbankkit @DrFaust92
Can we get some eyes on this PR ? Please let me know if there is anything pending.

jeffmachado · 2020-06-09T18:09:58Z

Anxiously waiting for the merge too! :D

rajholla · 2020-06-20T02:36:43Z

Hello @breathingdust
Can you please let me know the pending items here? It has been several weeks since the initial review and looking forward to getting this released.

breathingdust · 2020-06-24T23:49:39Z

Hi @rajholla 👋

This is in our backlog of items to review and we are hoping to get to it soon. We are currently working through roadmap items for this quarter but once that has been completed we'll be in a better position to give feedback and hopefully merge.

At the moment we can't give you a timeline, but stay tuned for an update. We appreciate your contributions and your patience!

walterd1969 · 2020-06-26T14:30:36Z

Hello,
Hopefully this is the right place...but I noticed that trying to modify the vpc_arn seems NOT to work.

vpc_config {
  role_arn           = "${module.FlowlogsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}

change to:

vpc_config {
  role_arn           = "${module.CloudTrailsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}

The "plan" shows that it will be modified, but after "apply", the change doesn't happen.

rajholla · 2020-06-26T21:20:29Z

Hello,
Hopefully this is the right place...but I noticed that trying to modify the vpc_arn seems NOT to work.
vpc_config {
  role_arn           = "${module.FlowlogsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  
change to:
vpc_config {
  role_arn           = "${module.CloudTrailsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  
The "plan" shows that it will be modified, but after "apply", the change doesn't happen.

Hello @walterd1969
I believe you meant changing role_arn is not working.
Unfortunately, AWS API's doesn't allow updating destination VPC connectivity with in ES configuration.

dmccaffery · 2020-07-01T15:18:13Z

Per documentation:

The resource should handle this appropriately with a taint (delete/create) when the vpc_config changes.

According to the code, this should happen:

https://github.com/terraform-providers/terraform-provider-aws/pull/13269/files#diff-6bd9774171ef581867b57e81348d87d8R1297

Not sure why it didn't work as expected for @walterd1969

lucaschain · 2020-07-16T17:03:42Z

Hey folks, any updates on this? We've been using the compiled version of this branch and it seems to be working just fine, but we didn't test the resource recreation handling pointed by @dmccaffery

rajholla · 2020-07-20T13:20:30Z

@walterd1969 Thanks for reporting the issue. It should be fixed now. Can you rebuild and test again?
cc @dmccaffery @lucaschain

DrFaust92

some changes to tests

aws/resource_aws_kinesis_firehose_delivery_stream_test.go

website/docs/r/kinesis_firehose_delivery_stream.html.markdown

aws/resource_aws_kinesis_firehose_delivery_stream.go

aws/resource_aws_kinesis_firehose_delivery_stream_test.go

rajholla · 2020-07-21T01:07:15Z

@DrFaust92 thank you for the review. All comments are addressed.

valorl · 2020-07-21T13:43:29Z

@breathingdust Since the roadmap you linked is for August - October and does not include this MR, is it safe to assume that this is not coming before end of October ?

breathingdust · 2020-07-22T18:37:39Z

Hey @valorl 👋. The roadmap doesn't represent all that we will do in a quarter, but does represent what we are able to commit to. That said, this issue is on our radar and we hope to have an update for you soon.

brunorochadasilva · 2020-08-24T12:37:37Z

Hey guys, any updates about this?

ewbankkit · 2020-08-24T12:46:53Z

aws/resource_aws_kinesis_firehose_delivery_stream.go

@@ -1275,6 +1291,39 @@ func resourceAwsKinesisFirehoseDeliveryStream() *schema.Resource {
 							},
 						},

+						"vpc_config": {


@rajholla Could you rename this to vpc_configuration to match the AWS API? Thanks.

@ewbankkit vpc_config is consistent with other resources like eks and lambda
Is there any advantage changing this ?

For Lambda the AWS API names the field VpcConfig but for EKS it's named ResourceVpcConfig in the API so we are not consistently sticking to the API naming 😄.
Staying with vpc_config is fine.

ewbankkit · 2020-08-24T14:42:30Z

aws/resource_aws_kinesis_firehose_delivery_stream.go

+		"subnet_ids":         flattenStringList(description.SubnetIds),
+		"security_group_ids": flattenStringList(description.SecurityGroupIds),


Can use flattenStringSet here for consistency with the expandStringSet below.

rajholla · 2020-08-25T18:56:03Z

@ewbankkit requested changes are in now.

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (2073.09s)
PASS

ewbankkit · 2020-08-25T22:18:35Z

@DrFaust92 Could you try verifying the TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates acceptance test?
It times out on creation for me in us-west-2.
Thanks.

ewbankkit · 2020-08-26T19:59:02Z

Verified in us-east-1:

$ AWS_DEFAULT_REGION=us-east-1 make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1548.08s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1548.124s

ewbankkit

LGTM.

DrFaust92 · 2020-08-26T20:39:20Z

ah, im actually getting the following:

    resource_aws_elasticsearch_domain_test.go:1203: missing IAM Service Linked Role (es.amazonaws.com), please create it in the AWS account and retry
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (3.61s)

does that even make sense?

ewbankkit · 2020-08-26T20:49:54Z

@DrFaust92 Yes, I got that originally.

aws iam create-service-linked-role --aws-service-name es.amazonaws.com

will fix.

DrFaust92 · 2020-08-26T21:20:16Z

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1609.14s)

us-west-2

DrFaust92

LGTM

bflad

Looks good 🚀

Output from acceptance testing:

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (102.19s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (106.75s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (118.90s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (123.17s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (133.80s)
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_basic (146.83s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (148.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (149.37s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (149.70s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (152.19s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (163.95s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (167.63s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (168.05s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (173.94s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (175.01s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (176.11s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (180.04s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (187.17s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (197.56s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (98.68s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (253.21s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (107.01s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (139.39s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (163.46s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (340.62s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (991.76s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1574.83s)

ghost · 2020-09-03T19:30:34Z

This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2020-09-30T17:10:32Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

rajholla requested a review from a team May 11, 2020 21:32

DrFaust92 suggested changes May 11, 2020

View reviewed changes

ewbankkit mentioned this pull request May 18, 2020

Firehose kinesis & elaastic support VpcConfiguration #13379

Closed

ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Jun 9, 2020

DrFaust92 suggested changes Jul 20, 2020

View reviewed changes

anGie44 force-pushed the master branch from 54db70d to 48ba2f7 Compare July 21, 2020 18:26

ewbankkit reviewed Aug 24, 2020

View reviewed changes

ghost added size/XXL Managed by automation to categorize the size of a PR. and removed size/XL Managed by automation to categorize the size of a PR. labels Aug 25, 2020

rajholla added 6 commits August 25, 2020 12:39

support vpc config

5be14f1

address PR comments

3cd4a3c

taint when resource change

9713904

fix failing test with bad permission

e6d6578

addressing review comments

a4d658c

flattenStringList to flattenStringSet for consistency

2033379

rajholla force-pushed the firehose-es-vpc-support branch from da0deef to e09a126 Compare August 25, 2020 17:40

ghost added size/XL Managed by automation to categorize the size of a PR. and removed size/XXL Managed by automation to categorize the size of a PR. labels Aug 25, 2020

update to terraform 0.12 syntax

3f8303b

rajholla force-pushed the firehose-es-vpc-support branch from e09a126 to 3f8303b Compare August 25, 2020 17:53

ewbankkit approved these changes Aug 26, 2020

View reviewed changes

DrFaust92 approved these changes Aug 26, 2020

View reviewed changes

bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Aug 28, 2020

bflad added this to the v3.5.0 milestone Aug 28, 2020

bflad approved these changes Aug 31, 2020

View reviewed changes

bflad merged commit 4d3799e into hashicorp:master Aug 31, 2020

bflad added a commit that referenced this pull request Aug 31, 2020

Update CHANGELOG for #13269

4df0c68

ewbankkit mentioned this pull request Sep 2, 2020

Support for Elasticsearch domain in a VPC #14973

Closed

ghost locked as resolved and limited conversation to collaborators Sep 30, 2020

		"subnet_ids": flattenStringList(description.SubnetIds),
		"security_group_ids": flattenStringList(description.SecurityGroupIds),

Support VPC config for Amazon Kinesis Data Firehose #13269

Support VPC config for Amazon Kinesis Data Firehose #13269

Conversation

rajholla commented May 11, 2020

Community Note

DrFaust92 left a comment

Choose a reason for hiding this comment

DrFaust92 May 11, 2020

Choose a reason for hiding this comment

rajholla commented May 12, 2020

ewbankkit commented May 18, 2020

ssreechandra commented May 22, 2020

bramalingam81 commented May 26, 2020

rajholla commented Jun 2, 2020

jeffmachado commented Jun 9, 2020

rajholla commented Jun 20, 2020

breathingdust commented Jun 24, 2020 • edited Loading

walterd1969 commented Jun 26, 2020

rajholla commented Jun 26, 2020

dmccaffery commented Jul 1, 2020 • edited Loading

lucaschain commented Jul 16, 2020

rajholla commented Jul 20, 2020

DrFaust92 left a comment

Choose a reason for hiding this comment

rajholla commented Jul 21, 2020

valorl commented Jul 21, 2020

breathingdust commented Jul 22, 2020

brunorochadasilva commented Aug 24, 2020

ewbankkit Aug 24, 2020

Choose a reason for hiding this comment

rajholla Aug 24, 2020

Choose a reason for hiding this comment

ewbankkit Aug 24, 2020

Choose a reason for hiding this comment

ewbankkit Aug 24, 2020

Choose a reason for hiding this comment

rajholla commented Aug 25, 2020

ewbankkit commented Aug 25, 2020

ewbankkit commented Aug 26, 2020

ewbankkit left a comment

Choose a reason for hiding this comment

DrFaust92 commented Aug 26, 2020

ewbankkit commented Aug 26, 2020

DrFaust92 commented Aug 26, 2020

DrFaust92 left a comment

Choose a reason for hiding this comment

bflad left a comment

Choose a reason for hiding this comment

ghost commented Sep 3, 2020

ghost commented Sep 30, 2020

breathingdust commented Jun 24, 2020 •

edited

Loading

dmccaffery commented Jul 1, 2020 •

edited

Loading