leftover module module.metric_filters_Classifier... (likely a terraform version issue)

[jack1902]

Background

Ran into issues when running ./manage.py destroy. The version of terraform in use:

terraform 0.12.9 states:

Error: leftover module module.metric_filters_Classifier_S3DownloadTime_PROD_AGGREGATE in state that should have been removed; this is a bug in Terraform and should be reported

... truncated to reduce noise ...

Description

When i run ./manage.py destroy it causes the above error (truncated) to happen. Having spoken with another streamalert user on slack, he pointed out: hashicorp/terraform#21313 which has a similar error message

Steps to Reproduce

• Currently trying to reproduce this on my side

1. ./manage.py init
2. ./manage.py destroy

Desired Change

• Increase the version of the terraform provider to at least 0.12.11

[jack1902]

tl;dr hashicorp/terraform#22579

Update

So i changed

streamalert/streamalert_cli/terraform/generate.py

Lines 56 to 57 in ff3aa21

	TERRAFORM_VERSION = '~> 0.12.9'
	TERRAFORM_PROVIDER_VERSION = '~> 2.28.1'

to:

TERRAFORM_VERSION = '~> 0.12.19'
TERRAFORM_PROVIDER_VERSION = '~> 2.44.0'

This removed the above issue but i believe i could have found a bug in terraform itself -_- (the bugs never end)

Error: leftover module module.classifier_prod_lambda in state that should have been removed; this is a bug in Terraform and should be reported

Error: error deleting S3 Bucket (PREFIX.streamalert.athena-results): BucketNotEmpty: The bucket you tried to delete is not empty. You must delete all versions in the bucket.
        status code: 409, request id: REQUEST_ID, host id: HOST_ID

Error: leftover module module.athena_monitoring in state that should have been removed; this is a bug in Terraform and should be reported

Error: leftover module module.classifier_prod_iam in state that should have been removed; this is a bug in Terraform and should be reported

Error: leftover module module.cloudwatch_monitoring_prod in state that should have been removed; this is a bug in Terraform and should be reported

Although the bucket did have stuff remaining in it so not sure if that caused the error in the first place

[ryandeivert]

sadly we saw this a lot internally too and concluded it was some trash that happens during terraform (apparently poor) implementation of state file version migration from 11 --> 12. we had to manually removed some of these references in order to get back in a good place internally post-migration.

Additionally, it's worth noting that they are, in fact, NOT errors and should more appropriately be logged as a warning by terraform, as it does not result in failures during terraform apply