Can't plan or apply kubernetes_manifest resources unless Kubernetes namespace already exists

[tgeoghegan]

in terraform/modules/kubernetes/kubernetes.tf, we create a couple of kubernetes_manifest resources. Those manifests get deployed into a Kubernetes namespace. terraform plan-ning the creation of those resources evidently entails (the equivalent of) kubectl apply --dry-run. If the namespace in question doesn't exist, then the manifest application dry run fails because the Kubernetes cluster doesn't know about the namespace we propose to deploy the manifest into.

This breaks deploying new localities, because that operation involves creating a namespace and creating a kubernetes_manifest in one apply operation. Curiously, make apply-bootstrap doesn't work around this: in spite of apply-bootstrap threatening to create namespaces during its planning phase, it doesn't actually do it. Presumably this is because of the weird behavior around -target that terraform warns us about.

[tgeoghegan]

The workaround is to comment out the kubernetes_manifest resources, do make apply, then uncomment the manifests and apply again.

[tgeoghegan]

Possibly relates to #1046, in that we might want to consider formalizing the notion of multi-stage Terraform application, so that we don't run afoul of surprising behavior in TF providers or around the -target option.

[tgeoghegan]

I wound up resolving this by adopting the gavinbunney/kubectl provider just for manifests, which doesn't have the same problematic behavior as the hashicorp/kubernetes provider. One final note here: terraform applying this change will destroy the existing ExternalMetrics for SQS queue depth and replace them with new, identical ones. This should cause nothing more than a momentary interruption of pod autoscaling while the corresponding HPAs rediscover the ExternalMetric, after which normal operation will resume, so I didn't think it was worth the hassle of doing terraform imports across all the EKS environments we have.