Incorrect reporting of processing_configuration using AWS Provider 1.16.0 #4392

si-robinson · 2018-04-30T08:11:03Z

Hi

We have an AWS firehose that uses elasticsearch as an endpoint under terraform with AWS provider 1.10.0 that does not use transformation:

resource "aws_kinesis_firehose_delivery_stream" "agevents-es-stream" {
  name        = "agevents-es"
  destination = "elasticsearch"

  s3_configuration {
    role_arn           = "${aws_iam_role.agevents-firehose-role.arn}"
    bucket_arn         = "${aws_s3_bucket.agevents-bucket.arn}"
    buffer_size        = 5
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  elasticsearch_configuration {
    domain_arn     = "${aws_elasticsearch_domain.events.arn}"
    role_arn       = "${aws_iam_role.agevents-firehose-role.arn}"
    index_name     = "${aws_elasticsearch_domain.events.domain_name}"
    type_name      = "event"
    s3_backup_mode = "AllDocuments"
  }
}

Last week we upgraded the AWS Provider to 1.16.0 in order to use lambda transformations in a new firehose with redshift as the endpoint.
Since doing this, running plan always returns:
elasticsearch_configuration.0.processing_configuration.#: "1" => "0"
for the original firehose.
If I apply the change and run plan again, it gives the same change it wants to make.

Terraform Version

0.11.7

Affected Resource(s)

elasticsearch_configuration.0.processing_configuration.#

Debug Output

no errors thrown

Panic Output

no panic

Expected Behavior

a) Running plan should not have seen that elasticsearch_configuration.0.processing_configuration.# was 1 as there was never any transformation included in the spec.
b) After applying the change I would expect not to see it surfaced again by running plan.

Actual Behavior

terraform indicated there was a processing_configuration when there wasn't and seems not to have updated it's state on applying the change and still sees empty config as containing a processing config.

Steps to Reproduce

import a pre-existing pipeline with no transformation into terraform using AWS provider 1.10.0. Upgrade provider to 1.16.0.
Run plan
See the above indicated.
Run apply
Run plan
See the above still indicated

The text was updated successfully, but these errors were encountered:

bflad · 2018-05-01T17:52:07Z

Hi @si-robinson 👋 Sorry you're running into trouble here.

Is this the only thing in the terraform plan output or is there more?

elasticsearch_configuration.0.processing_configuration.#: "1" => "0"

albrechtsimon · 2018-05-14T16:04:31Z

I'm experiencing the same issue. According to the Git history of the resource declaration, we never defined a processing_configuration for the delivery stream. Nonetheless, plan runs output the following: elasticsearch_configuration.0.processing_configuration.#: "1" => "0"

saliceti · 2018-06-13T14:34:34Z

Similar issue here with processing_configuration

bflad · 2018-06-13T14:43:29Z

@saliceti is the plan output strictly only the XXX.0.processing_configuration.#: "1" => "0" line? Also it would be really helpful to see the API response coming back from AWS in the Terraform debug logs so we can try to replicate this behavior in unit/acceptance testing.

si-robinson · 2018-06-13T15:24:43Z

Hey @bflad - apologies for not getting back sooner.
So, the only output from plan is
elasticsearch_configuration.0.processing_configuration.#: "1" => "0"

Enabling TF_LOG=TRACE got me this response from AWS:

{
	"DeliveryStreamDescription": {
		"CreateTimestamp": 1.473949127336E9,
		"DeliveryStreamARN": "arn:aws:firehose:eu-west-1:<acc-no>:deliverystream/<stream-name>",
		"DeliveryStreamName": "<stream-name>",
		"DeliveryStreamStatus": "ACTIVE",
		"DeliveryStreamType": "DirectPut",
		"Destinations": [{
			"DestinationId": "destinationId-000000000001",
			"ElasticsearchDestinationDescription": {
				"BufferingHints": {
					"IntervalInSeconds": 300,
					"SizeInMBs": 5
				},
				"CloudWatchLoggingOptions": {
					"Enabled": true,
					"LogGroupName": "/aws/kinesisfirehose/<stream-name>",
					"LogStreamName": "ElasticsearchDelivery"
				},
				"DomainARN": "arn:aws:es:eu-west-1:<acc-no>:domain/<es-domain>",
				"IndexName": "events",
				"IndexRotationPeriod": "OneDay",
				"ProcessingConfiguration": {
					"Enabled": false,
					"Processors": []
				},
				"RetryOptions": {
					"DurationInSeconds": 300
				},
				"RoleARN": "arn:aws:iam::<acc-no>:role/<firehose-role>",
				"S3BackupMode": "AllDocuments",
				"S3DestinationDescription": {
					"BucketARN": "arn:aws:s3:::prod-eu-west-1-agevents",
					"BufferingHints": {
						"IntervalInSeconds": 60,
						"SizeInMBs": 5
					},
					"CloudWatchLoggingOptions": {
						"Enabled": true,
						"LogGroupName": "/aws/kinesisfirehose/<stream-name>",
						"LogStreamName": "S3Delivery"
					},
					"CompressionFormat": "GZIP",
					"EncryptionConfiguration": {
						"NoEncryptionConfig": "NoEncryption"
					},
					"Prefix": "",
					"RoleARN": "arn:aws:iam::<acc-no>:role/<firehose-role>"
				},
				"TypeName": "event"
			}
		}],
		"HasMoreDestinations": false,
		"LastUpdateTimestamp": 1.526641707191E9,
		"VersionId": "7"
	}
}

Shout if you need any more :)

saliceti · 2018-06-14T13:19:25Z

The issue here is with the buffer size in the lambda processor, in extended_s3_configuration.
The default size is 1MB which is fine for us, but terraform wants to replace it with empty values, which is equivalent to default.

resource "aws_kinesis_firehose_delivery_stream" "xxx" {
  name        = "${local.stream_name}"
  destination = "extended_s3"

 extended_s3_configuration {
    role_arn   = "xxx"
    bucket_arn = "${aws_s3_bucket.bucket.arn}"
    buffer_size        = 1
    buffer_interval    = 60
    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = "/aws/kinesisfirehose/${local.stream_name}"
      log_stream_name = "S3Delivery"
    }

    processing_configuration = [
      {
        enabled = "true"
        processors = [
          {
            type = "Lambda"
            parameters = [
              {
                parameter_name = "LambdaArn"
                parameter_value = "xxx"
              }
            ]
          }
        ]
      }
    ]
  }
}

~ aws_kinesis_firehose_delivery_stream.xxx
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.#:                 "2" => "1"
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.1.parameter_name:  "BufferSizeInMBs" => ""
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.1.parameter_value: "1" => ""

2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 2018/06/14 09:09:25 [DEBUG] [aws-sdk-go] DEBUG: Request firehose/DescribeDeliveryStream Details:
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: POST / HTTP/1.1
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Host: firehose.eu-west-1.amazonaws.com
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: User-Agent: aws-sdk-go/1.14.5 (go1.9.2; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Content-Length: 54
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Authorization: xxx
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Content-Type: application/x-amz-json-1.1
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: X-Amz-Date: 20180614T090925Z
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: X-Amz-Security-Token: xxx
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: X-Amz-Target: Firehose_20150804.DescribeDeliveryStream
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Accept-Encoding: gzip
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 
2018-06-14T09:09:25.823Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: -----------------------------------------------------
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 2018/06/14 09:09:25 [DEBUG] [aws-sdk-go] DEBUG: Response firehose/DescribeDeliveryStream Details:
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: ---[ RESPONSE ]--------------------------------------
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: HTTP/1.1 200 OK
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Connection: close
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Content-Length: 1921
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Content-Type: application/x-amz-json-1.1
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: Date: Thu, 14 Jun 2018 09:09:25 GMT
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: X-Amz-Id-2: xxx
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: X-Amzn-Requestid: xxx
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: -----------------------------------------------------
2018-06-14T09:09:25.963Z [DEBUG] plugin.terraform-provider-aws_v1.23.0_x4: 2018/06/14 09:09:25 [DEBUG] [aws-sdk-go] {"DeliveryStreamDescription":{"CreateTimestamp":1.52353585233E9,"DeliveryStreamARN":"xxx","DeliveryStreamName":"xxx","DeliveryStreamStatus":"ACTIVE","DeliveryStreamType":"DirectPut","Destinations":[{"DestinationId":"destinationId-000000000001","ExtendedS3DestinationDescription":{"BucketARN":"xxx","BufferingHints":{"IntervalInSeconds":60,"SizeInMBs":1},"CloudWatchLoggingOptions":{"Enabled":true,"LogGroupName":"/aws/kinesisfirehose/xxx","LogStreamName":"S3Delivery"},"CompressionFormat":"UNCOMPRESSED","EncryptionConfiguration":{"NoEncryptionConfig":"NoEncryption"},"Prefix":"","ProcessingConfiguration":{"Enabled":true,"Processors":[{"Parameters":[{"ParameterName":"LambdaArn","ParameterValue":"xxx"},{"ParameterName":"NumberOfRetries","ParameterValue":"3"},{"ParameterName":"RoleArn","ParameterValue":"xxx"},{"ParameterName":"BufferSizeInMBs","ParameterValue":"1"},{"ParameterName":"BufferIntervalInSeconds","ParameterValue":"60"}],"Type":"Lambda"}]},"RoleARN":"xxx","S3BackupMode":"Disabled"},"S3DestinationDescription":{"BucketARN":"xxx","BufferingHints":{"IntervalInSeconds":60,"SizeInMBs":1},"CloudWatchLoggingOptions":{"Enabled":true,"LogGroupName":"/aws/kinesisfirehose/xxx","LogStreamName":"S3Delivery"},"CompressionFormat":"UNCOMPRESSED","EncryptionConfiguration":{"NoEncryptionConfig":"NoEncryption"},"Prefix":"","RoleARN":"xxx/xxx"}}],"HasMoreDestinations":false,"LastUpdateTimestamp":1.5289006975E9,"VersionId":"68"}}

aarnone · 2018-06-14T15:14:30Z

@saliceti I'm not sure your is the same issue as @si-robinson

As far as I can see, the problem is that AWS api doesn't allow to remove ProcessingConfiguration from any destination after it's set the first time, it only allow to disable the the processing configuration like this:

"ProcessingConfiguration": {
    "Enabled": false,
    "Processors": []
}

The effect is that terraform will see a difference between an absent ProcessingConfiguration and a disabled ProcessingConfiguration. It will try to remove the existing ProcessingConfiguration, AWS will silently ignore it, and therefore the next plan will still be the same.

I've reproduced this in aarnone@10f0c21 for extended s3 destination, but it applies to elasticsearch as well.

I see few possible solutions:

Change the tf code and add the disabled processing_configuration
Set a default to disabled processing_configuration in the schema
Ignore the diff between absent and disabled processing_configuration.

I'm not sure options 2 and 3 are easily implementable.
@bflad have you got any thought?

saliceti · 2018-06-14T20:49:52Z

So should I open a new issue? Or is this related somehow?

batzel · 2018-07-24T11:50:18Z

Similar issue here:
extended_s3_configuration.0.data_format_conversion_configuration.#: "1" => "0"
extended_s3_configuration.0.data_format_conversion_configuration.0.enabled: "false" => "true"
even after applying the change, the plan does this again.
I didn't find a way to change my .tf file to prevent this. Seems to be a somewhat common problem throughout the firehose code?

jychen7 · 2018-08-28T15:56:07Z

$ terraform version
Terraform v0.11.8
+ provider.aws v1.33.0

my issue is similar, but in reverse case

I keep getting following when plan or apply

~ aws_kinesis_firehose_delivery_stream.firehose
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.#:                 "2" => "3"
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_name:  "" => "BufferIntervalInSeconds"
      extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_value: "" => "60"

even I already apply multiple times.

aws_kinesis_firehose_delivery_stream.firehose: Modifying... (ID: arn:aws:firehose:ap-southeast-1:xxx:deliverystream/yyy)
  extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.#:                 "2" => "3"
  extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_name:  "" => "BufferIntervalInSeconds"
  extended_s3_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_value: "" => "60"
aws_kinesis_firehose_delivery_stream.firehose: Modifications complete after 1s (ID: arn:aws:firehose:ap-southeast-1:xxx:deliverystream/yyy)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

when using TF_LOG=DEBUG terraform plan, the response of firehose/DescribeDeliveryStream shows

          "ProcessingConfiguration": {
            "Enabled": true,
            "Processors": [
              {
                "Parameters": [
                  {
                    "ParameterName": "LambdaArn",
                    "ParameterValue": "arn:aws:lambda:ap-southeast-1:xxx:function:yyy:$LATEST"
                  },
                  {
                    "ParameterName": "NumberOfRetries",
                    "ParameterValue": "3"
                  },
                  {
                    "ParameterName": "RoleArn",
                    "ParameterValue": "arn:aws:iam::xxx:role/yyy"
                  },
                  {
                    "ParameterName": "BufferSizeInMBs",
                    "ParameterValue": "1"
                  },
                  {
                    "ParameterName": "BufferIntervalInSeconds",
                    "ParameterValue": "60"
                  }
                ],
                "Type": "Lambda"
              }
            ]

but looks like terraform fail to store BufferIntervalInSeconds in the .tfstate file.

Here is part of my tf file

    processing_configuration = [
      {
        enabled = "true"

        processors = [
          {
            type = "Lambda"

            parameters = [
              {
                parameter_name  = "LambdaArn"
                parameter_value = "${data.aws_lambda_function.yyy.arn}"
              },
              {
                parameter_name  = "BufferSizeInMBs"
                parameter_value = 1
              },
              {
                parameter_name  = "BufferIntervalInSeconds"
                parameter_value = 60
              },
            ]
          },
        ]
      },
    ]

saurabh-hirani · 2018-09-26T10:09:38Z

I can confirm the same. Terraform fails to store BufferIntervalInSeconds - this leads to plan showing the following output post Firehose creation:

      elasticsearch_configuration.0.processing_configuration.0.processors.0.parameters.#:                 "2" => "3"
      elasticsearch_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_name:  "" => "BufferIntervalInSeconds"
      elasticsearch_configuration.0.processing_configuration.0.processors.0.parameters.2.parameter_value: "" => "10"

ewbankkit · 2018-11-21T23:08:28Z

Similar to #6053?

limitusus · 2019-04-26T13:21:07Z

Looks similar to #6053.
I faced the same issues and the workaround mitigates:

  lifecycle {
    ignore_changes = [
      "extended_s3_configuration.0.processing_configuration",
    ]
  }

si-robinson · 2019-04-28T13:33:20Z

Nice. Thanks for flagging that - will give it a go in the week On Friday, 26 April 2019, 14:21:21 BST, Tomoya Kabe <[email protected]> wrote: Looks similar to #6053. I faced the same issues and the workaround mitigates: lifecycle { ignore_changes = [ "extended_s3_configuration.0.processing_configuration", ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

si-robinson · 2019-06-18T12:02:52Z

Thanks! Worked a treat :) On Friday, 26 April 2019, 14:21:21 BST, Tomoya Kabe <[email protected]> wrote: Looks similar to #6053. I faced the same issues and the workaround mitigates: lifecycle { ignore_changes = [ "extended_s3_configuration.0.processing_configuration", ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bflad · 2019-08-20T02:49:40Z

The fix for the original issue has been merged and will release with version 2.25.0 of the Terraform AWS Provider, later this week.

ghost · 2019-08-23T01:11:14Z

This has been released in version 2.25.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

alexdachin · 2019-10-18T11:12:08Z

We updated to 2.29.0, but still have issues with BufferIntervalInSeconds not being persisted in the tfstate and being dirty every time

ghost · 2019-11-01T14:51:52Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

bflad added bug Addresses a defect in current functionality. service/firehose Issues and PRs that pertain to the firehose service. labels May 1, 2018

bflad added the waiting-response Maintainers are waiting on response from community or contributor. label May 1, 2018

aarnone pushed a commit to aarnone/terraform-provider-aws that referenced this issue Jun 13, 2018

Add test to reproduce issue hashicorp#4392

10f0c21

bflad removed the waiting-response Maintainers are waiting on response from community or contributor. label Jun 14, 2018

This was referenced Jun 13, 2019

Error on plan refreshing firehose state: insufficient items for attribute output_format_configuration #8982

Closed

resource_aws_kinesis_firehose_delivery_stream: Avoid invalid state on refresh #9016

Closed

staffan-einarsson mentioned this issue Jun 24, 2019

resource_aws_kinesis_firehose_delivery_stream: Avoid invalid state on refresh (cont.) #9103

Merged

pracucci mentioned this issue Jun 25, 2019

lifecycle / ignore_changes blocking upgrade from 0.11 to 0.12 hashicorp/terraform#21433

Closed

bflad added this to the v2.25.0 milestone Aug 20, 2019

bflad closed this as completed in #9103 Aug 20, 2019

ghost locked and limited conversation to collaborators Nov 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect reporting of processing_configuration using AWS Provider 1.16.0 #4392

Incorrect reporting of processing_configuration using AWS Provider 1.16.0 #4392

si-robinson commented Apr 30, 2018

bflad commented May 1, 2018

albrechtsimon commented May 14, 2018 •

edited

Loading

saliceti commented Jun 13, 2018

bflad commented Jun 13, 2018

si-robinson commented Jun 13, 2018

saliceti commented Jun 14, 2018

aarnone commented Jun 14, 2018 •

edited

Loading

saliceti commented Jun 14, 2018

batzel commented Jul 24, 2018

jychen7 commented Aug 28, 2018 •

edited

Loading

saurabh-hirani commented Sep 26, 2018

ewbankkit commented Nov 21, 2018

limitusus commented Apr 26, 2019

si-robinson commented Apr 28, 2019 via email

si-robinson commented Jun 18, 2019 via email

bflad commented Aug 20, 2019

ghost commented Aug 23, 2019

alexdachin commented Oct 18, 2019

ghost commented Nov 1, 2019

Incorrect reporting of processing_configuration using AWS Provider 1.16.0 #4392

Incorrect reporting of processing_configuration using AWS Provider 1.16.0 #4392

Comments

si-robinson commented Apr 30, 2018

Terraform Version

Affected Resource(s)

Debug Output

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

bflad commented May 1, 2018

albrechtsimon commented May 14, 2018 • edited Loading

saliceti commented Jun 13, 2018

bflad commented Jun 13, 2018

si-robinson commented Jun 13, 2018

saliceti commented Jun 14, 2018

aarnone commented Jun 14, 2018 • edited Loading

saliceti commented Jun 14, 2018

batzel commented Jul 24, 2018

jychen7 commented Aug 28, 2018 • edited Loading

saurabh-hirani commented Sep 26, 2018

ewbankkit commented Nov 21, 2018

limitusus commented Apr 26, 2019

si-robinson commented Apr 28, 2019 via email

si-robinson commented Jun 18, 2019 via email

bflad commented Aug 20, 2019

ghost commented Aug 23, 2019

alexdachin commented Oct 18, 2019

ghost commented Nov 1, 2019

albrechtsimon commented May 14, 2018 •

edited

Loading

aarnone commented Jun 14, 2018 •

edited

Loading

jychen7 commented Aug 28, 2018 •

edited

Loading