U of Toronto new cluster terraform config

[GeorgianaElena]

Part of #853 (comment)

( The config of the original cluster is here )

[sgibson91]

LGTM @GeorgianaElena! Do you have the output of terraform plan?

[GeorgianaElena]

Do you have the output of terraform plan?

Sorry for missing this question @sgibson91. No output yet, still at the init terraform step. Will come back with details.

[GeorgianaElena]

@sgibson91, turning global_container_registry_name and global_storage_account_name to all lowercase letters, made terraform plan happy.

I'm pasting the output here:

Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_container_registry.container_registry will be created
  + resource "azurerm_container_registry" "container_registry" {
      + admin_enabled                 = true
      + admin_password                = (sensitive value)
      + admin_username                = (known after apply)
      + encryption                    = (known after apply)
      + georeplication_locations      = (known after apply)
      + georeplications               = (known after apply)
      + id                            = (known after apply)
      + location                      = "canadacentral"
      + login_server                  = (known after apply)
      + name                          = "2i2cutorontohubregistry"
      + network_rule_bypass_option    = "AzureServices"
      + network_rule_set              = (known after apply)
      + public_network_access_enabled = true
      + resource_group_name           = "2i2c-utoronto-cluster"
      + retention_policy              = (known after apply)
      + sku                           = "premium"
      + storage_account_id            = (known after apply)
      + trust_policy                  = (known after apply)
      + zone_redundancy_enabled       = false

      + identity {
          + identity_ids = (known after apply)
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }
    }

  # azurerm_kubernetes_cluster.jupyterhub will be created
  + resource "azurerm_kubernetes_cluster" "jupyterhub" {
      + dns_prefix                          = "k8s"
      + fqdn                                = (known after apply)
      + id                                  = (known after apply)
      + kube_admin_config                   = (known after apply)
      + kube_admin_config_raw               = (sensitive value)
      + kube_config                         = (known after apply)
      + kube_config_raw                     = (sensitive value)
      + kubernetes_version                  = "1.20.7"
      + location                            = "canadacentral"
      + name                                = "hub-cluster"
      + node_resource_group                 = (known after apply)
      + portal_fqdn                         = (known after apply)
      + private_cluster_enabled             = (known after apply)
      + private_cluster_public_fqdn_enabled = false
      + private_dns_zone_id                 = (known after apply)
      + private_fqdn                        = (known after apply)
      + private_link_enabled                = (known after apply)
      + resource_group_name                 = "2i2c-utoronto-cluster"
      + sku_tier                            = "Free"

      + addon_profile {
          + aci_connector_linux {
              + enabled     = (known after apply)
              + subnet_name = (known after apply)
            }

          + azure_policy {
              + enabled = (known after apply)
            }

          + http_application_routing {
              + enabled                            = (known after apply)
              + http_application_routing_zone_name = (known after apply)
            }

          + ingress_application_gateway {
              + effective_gateway_id                 = (known after apply)
              + enabled                              = (known after apply)
              + gateway_id                           = (known after apply)
              + gateway_name                         = (known after apply)
              + ingress_application_gateway_identity = (known after apply)
              + subnet_cidr                          = (known after apply)
              + subnet_id                            = (known after apply)
            }

          + kube_dashboard {
              + enabled = (known after apply)
            }

          + oms_agent {
              + enabled                    = (known after apply)
              + log_analytics_workspace_id = (known after apply)
              + oms_agent_identity         = (known after apply)
            }

          + open_service_mesh {
              + enabled = (known after apply)
            }
        }

      + auto_scaler_profile {
          + balance_similar_node_groups      = false
          + empty_bulk_delete_max            = (known after apply)
          + expander                         = (known after apply)
          + max_graceful_termination_sec     = (known after apply)
          + max_node_provisioning_time       = "15m"
          + max_unready_nodes                = 3
          + max_unready_percentage           = 45
          + new_pod_scale_up_delay           = (known after apply)
          + scale_down_delay_after_add       = (known after apply)
          + scale_down_delay_after_delete    = (known after apply)
          + scale_down_delay_after_failure   = (known after apply)
          + scale_down_unneeded              = (known after apply)
          + scale_down_unready               = (known after apply)
          + scale_down_utilization_threshold = (known after apply)
          + scan_interval                    = (known after apply)
          + skip_nodes_with_local_storage    = true
          + skip_nodes_with_system_pods      = true
        }

      + default_node_pool {
          + enable_auto_scaling  = true
          + kubelet_disk_type    = (known after apply)
          + max_count            = 10
          + max_pods             = (known after apply)
          + min_count            = 1
          + name                 = "core"
          + node_count           = 1
          + node_labels          = {
              + "hub.jupyter.org/node-purpose" = "core"
              + "k8s.dask.org/node-purpose"    = "core"
            }
          + orchestrator_version = "1.20.7"
          + os_disk_size_gb      = 40
          + os_disk_type         = "Managed"
          + os_sku               = (known after apply)
          + type                 = "VirtualMachineScaleSets"
          + ultra_ssd_enabled    = false
          + vm_size              = "Standard_E4s_v3"
          + vnet_subnet_id       = (known after apply)
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = "SystemAssigned"
        }

      + kubelet_identity {
          + client_id                 = (known after apply)
          + object_id                 = (known after apply)
          + user_assigned_identity_id = (known after apply)
        }

      + linux_profile {
          + admin_username = "hub-admin"

          + ssh_key {
              + key_data = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQJ4h39UYNi1wybxAH+jCFkNK2aqRcuhDkQSMx0Hak5xkbt3KnT3cOwAgUP1Vt/SjhltSTuxpOHxiAKCRnjwRk60SxKhUNzPHih2nkfYTmBBjmLfdepDPSke/E0VWvTDIEXz/L8vW8aI0QGPXnXyqzEDO9+U1buheBlxB0diFAD3vEp2SqBOw+z7UgrGxXPdP+2b3AV+X6sOtd6uSzpV8Qvdh+QAkd4r7h9JrkFvkrUzNFAGMjlTb0Lz7qAlo4ynjEwzVN2I1i7cVDKgsGz9ZG/8yZfXXx+INr9jYtYogNZ63ajKR/dfjNPovydhuz5zQvQyxpokJNsTqt1CiWEUNj georgiana@georgiana"
            }
        }

      + network_profile {
          + dns_service_ip     = (known after apply)
          + docker_bridge_cidr = (known after apply)
          + load_balancer_sku  = "standard"
          + network_mode       = (known after apply)
          + network_plugin     = "kubenet"
          + network_policy     = "calico"
          + outbound_type      = "loadBalancer"
          + pod_cidr           = (known after apply)
          + service_cidr       = (known after apply)

          + load_balancer_profile {
              + effective_outbound_ips    = (known after apply)
              + idle_timeout_in_minutes   = (known after apply)
              + managed_outbound_ip_count = (known after apply)
              + outbound_ip_address_ids   = (known after apply)
              + outbound_ip_prefix_ids    = (known after apply)
              + outbound_ports_allocated  = (known after apply)
            }

          + nat_gateway_profile {
              + effective_outbound_ips    = (known after apply)
              + idle_timeout_in_minutes   = (known after apply)
              + managed_outbound_ip_count = (known after apply)
            }
        }

      + role_based_access_control {
          + enabled = (known after apply)

          + azure_active_directory {
              + admin_group_object_ids = (known after apply)
              + azure_rbac_enabled     = (known after apply)
              + client_app_id          = (known after apply)
              + managed                = (known after apply)
              + server_app_id          = (known after apply)
              + server_app_secret      = (sensitive value)
              + tenant_id              = (known after apply)
            }
        }

      + windows_profile {
          + admin_password = (sensitive value)
          + admin_username = (known after apply)
          + license        = (known after apply)
        }
    }

  # azurerm_kubernetes_cluster_node_pool.user_pool["default"] will be created
  + resource "azurerm_kubernetes_cluster_node_pool" "user_pool" {
      + enable_auto_scaling   = true
      + eviction_policy       = (known after apply)
      + id                    = (known after apply)
      + kubelet_disk_type     = (known after apply)
      + kubernetes_cluster_id = (known after apply)
      + max_count             = 100
      + max_pods              = (known after apply)
      + min_count             = 1
      + mode                  = "User"
      + name                  = "nbdefault"
      + node_count            = (known after apply)
      + node_labels           = {
          + "hub.jupyter.org/node-purpose" = "user"
          + "hub.jupyter.org/node-size"    = "Standard_E8s_v3"
          + "k8s.dask.org/node-purpose"    = "scheduler"
        }
      + node_taints           = [
          + "hub.jupyter.org_dedicated=user:NoSchedule",
        ]
      + orchestrator_version  = "1.20.7"
      + os_disk_size_gb       = 200
      + os_disk_type          = "Managed"
      + os_sku                = (known after apply)
      + os_type               = "Linux"
      + priority              = "Regular"
      + spot_max_price        = -1
      + ultra_ssd_enabled     = false
      + vm_size               = "Standard_E8s_v3"
      + vnet_subnet_id        = (known after apply)
    }

  # azurerm_resource_group.jupyterhub will be created
  + resource "azurerm_resource_group" "jupyterhub" {
      + id       = (known after apply)
      + location = "canadacentral"
      + name     = "2i2c-utoronto-cluster"
    }

  # azurerm_storage_account.homes will be created
  + resource "azurerm_storage_account" "homes" {
      + access_tier                      = (known after apply)
      + account_kind                     = "StorageV2"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + allow_blob_public_access         = false
      + enable_https_traffic_only        = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + large_file_share_enabled         = (known after apply)
      + location                         = "canadacentral"
      + min_tls_version                  = "TLS1_0"
      + name                             = "2i2cutorontohubstorage"
      + nfsv3_enabled                    = false
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + queue_encryption_key_type        = "Service"
      + resource_group_name              = "2i2c-utoronto-cluster"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)
      + shared_access_key_enabled        = true
      + table_encryption_key_type        = "Service"

      + blob_properties {
          + change_feed_enabled      = (known after apply)
          + default_service_version  = (known after apply)
          + last_access_time_enabled = (known after apply)
          + versioning_enabled       = (known after apply)

          + container_delete_retention_policy {
              + days = (known after apply)
            }

          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + identity {
          + identity_ids = (known after apply)
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }

      + network_rules {
          + bypass                     = (known after apply)
          + default_action             = (known after apply)
          + ip_rules                   = (known after apply)
          + virtual_network_subnet_ids = (known after apply)

          + private_link_access {
              + endpoint_resource_id = (known after apply)
              + endpoint_tenant_id   = (known after apply)
            }
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }

      + routing {
          + choice                      = (known after apply)
          + publish_internet_endpoints  = (known after apply)
          + publish_microsoft_endpoints = (known after apply)
        }

      + share_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + retention_policy {
              + days = (known after apply)
            }

          + smb {
              + authentication_types            = (known after apply)
              + channel_encryption_type         = (known after apply)
              + kerberos_ticket_encryption_type = (known after apply)
              + versions                        = (known after apply)
            }
        }
    }

  # azurerm_storage_share.homes will be created
  + resource "azurerm_storage_share" "homes" {
      + enabled_protocol     = "SMB"
      + id                   = (known after apply)
      + metadata             = (known after apply)
      + name                 = "homes"
      + quota                = 100
      + resource_manager_id  = (known after apply)
      + storage_account_name = "2i2cutorontohubstorage"
      + url                  = (known after apply)
    }

  # azurerm_subnet.node_subnet will be created
  + resource "azurerm_subnet" "node_subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.1.0.0/16",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "k8s-nodes-subnet"
      + resource_group_name                            = "2i2c-utoronto-cluster"
      + virtual_network_name                           = "k8s-network"
    }

  # azurerm_virtual_network.jupyterhub will be created
  + resource "azurerm_virtual_network" "jupyterhub" {
      + address_space         = [
          + "10.0.0.0/8",
        ]
      + dns_servers           = (known after apply)
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "canadacentral"
      + name                  = "k8s-network"
      + resource_group_name   = "2i2c-utoronto-cluster"
      + subnet                = (known after apply)
      + vm_protection_enabled = false
    }

  # kubernetes_namespace.homes will be created
  + resource "kubernetes_namespace" "homes" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "azure-file"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # kubernetes_secret.homes will be created
  + resource "kubernetes_secret" "homes" {
      + data = (sensitive value)
      + id   = (known after apply)
      + type = "Opaque"

      + metadata {
          + generation       = (known after apply)
          + name             = "access-credentials"
          + namespace        = "azure-file"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

Plan: 10 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + kubeconfig            = (sensitive value)
  + registry_creds_config = (sensitive value)

[sgibson91]

@GeorgianaElena The lowercase thing is an Azure requirement, not a terraform one, and I misled you because I didn't read this doc properly. Sorry about that!

[sgibson91]

LGTM!

[GeorgianaElena]

Thanks @sgibson91! The steps would then be, merge and then ➡️ terraform apply? 🤞

[sgibson91]

@GeorgianaElena i think we terraform apply and then merge. In case something happens that's not revealed by the plan step :)

[GeorgianaElena]

Just ran terraform apply successfully and added the cluster access credentials! There is a new cluster 🎉

Merging this 🚀