Kubernetes failed to configure tls

[gaba-xyz]

Terraform Version

Terraform v0.9.3

Affected Resource(s)

• google_container_cluster
• google_compute_disk
• kubernetes_secret

Terraform Configuration Files

gce-container-cluster/main.tf

provider "google" {
    credentials = "${file(var.credentials_path)}"
    project     = "${var.project_name}"
    region      = "${var.region}"
}

variable "disk_name" {
    default = [
        "mysql-disk",
        "wordpress-disk"
    ]
}

resource "google_compute_disk" "disk" {
    count = 2
    name  = "${var.disk_name[count.index]}"
    zone  = "${var.cluster_zone}"
    size  = "${var.disk_size}"
}

resource "google_container_cluster" "primary" {
    name               = "${var.cluster_name}"
    zone               = "${var.cluster_zone}"
    initial_node_count = 3

    master_auth {
        username = "${var.master_auth_username}"
        password = "${var.master_auth_password}"
    }

    node_config {
        oauth_scopes = ["${var.oauth_scopes}"]
    }
}

gce-container-cluster/outputs.tf

output "ip" {
    value = "${google_container_cluster.primary.endpoint}"
}

output "client_cert" {
    value = "${google_container_cluster.primary.master_auth.client_certificate}"
}

output "client_key" {
    value = "${google_container_cluster.primary.master_auth.client_key}"
}

output "ca_cert" {
    value = "${google_container_cluster.primary.master_auth.cluster_ca_certificate}"
}

kubernetes-secret/main.tf

provider "kubernetes" {
    host                   = "https://${var.endpoint_ip}"
    username               = "${var.username}"
    password               = "${var.password}"
    client_certificate     = "${var.client_cert}"
    client_key             = "${var.client_key}"
    cluster_ca_certificate = "${var.cluster_ca_certificate}"
}

resource "kubernetes_secret" "mysql_pwd" {

    metadata {
        name = "mysql"
    }

    data {
        password = "${var.kubernetes_secret_pwd}"
    }
}

main.tf

module "container_cluster" {
    source               = "./gce-container-cluster"
    credentials_path     = "${var.credentials_path}"
    project_name         = "${var.project_name}"
    region               = "${var.region}"
    cluster_name         = "${var.cluster_name}"
    cluster_zone         = "${var.cluster_zone}"
    master_auth_username = "${var.master_auth_username}"
    master_auth_password = "${var.master_auth_password}"
}

module "kubernetes_secret" {
    source                 = "./kubernetes-secret"
    endpoint_ip            = "${module.container_cluster.ip}"
    username               = "${var.master_auth_username}"
    password               = "${var.master_auth_password}"
    client_cert            = "${module.container_cluster.client_cert}"
    client_key             = "${module.container_cluster.client_key}"
    cluster_ca_certificate = "${module.container_cluster.ca_cert}"
    kubernetes_secret_pwd  = "${var.kubernetes_secret_pwd}"
}

Steps to Reproduce

terraform plan

Output

module.kubernetes_secret.provider.kubernetes: Failed to load config (/Users/ubuntu/.kube/config; default context): invalid configuration: no configuration has been provided.

[radeksimko]

Hi @OverlyExcessive
I think there is a few things happening here.

Firstly the outputs need to reference the certs/key with the appropriate index as master_auth is list (note the zero):

output "client_cert" {
    value = "${google_container_cluster.primary.master_auth.0.client_certificate}"
}

output "client_key" {
    value = "${google_container_cluster.primary.master_auth.0.client_key}"
}

output "ca_cert" {
    value = "${google_container_cluster.primary.master_auth.0.cluster_ca_certificate}"
}

It may look confusing, but it's due to the current limitations of HCL.

Secondly the certificates and key come from the GKE resource base64-encoded so the correct way of binding the modules would be e.g.

module "container_cluster" {
    source               = "./gce-container-cluster"
    credentials_path     = "${var.credentials_path}"
    project_name         = "${var.project_name}"
    region               = "${var.region}"
    cluster_name         = "${var.cluster_name}"
    cluster_zone         = "${var.cluster_zone}"
    master_auth_username = "${var.master_auth_username}"
    master_auth_password = "${var.master_auth_password}"
}

module "kubernetes_secret" {
    source                 = "./kubernetes-secret"
    endpoint_ip            = "${module.container_cluster.ip}"
    username               = "${var.master_auth_username}"
    password               = "${var.master_auth_password}"
    client_cert            = "${base64decode(module.container_cluster.client_cert)}"
    client_key             = "${base64decode(module.container_cluster.client_key)}"
    cluster_ca_certificate = "${base64decode(module.container_cluster.ca_cert)}"
    kubernetes_secret_pwd  = "${var.kubernetes_secret_pwd}"
}

There's a plan to solve this by automatically decoding anything that looks as "base64-decodable". See #12869

Thirdly there's a known core bug which breaks relationships between resource in one provider and another provider. This is tracked in #12393 and #4149 respectively.

Do you agree we can close this issue in favour of the mentioned issues?

Thanks.

[gaba-xyz]

@radeksimko
Absolutely, thank you for the clarification.

[rberlind]

The advice to use base64decode() was very helpful to me in creating a configuration that used the Kubernetes provider against a k8s cluster provisioned by Terraform to GKE. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.