Enable https and ambassador routing

[ghost]

Switches from nginx-ingress to Ambassador
Adds cert-manager for handling Let's Encrypt
Adds TLS termination
Grafana accessible with https://domain/grafana/

[ghost]

FIRST
change email in tls.yaml to a real email
if multiple people are running clusters at the same time each has to have a unique DNS name label
change "smaddis" in mentions of "smaddis.westeurope.cloudapp.azure.com" in tls.yaml, ambassador_mappings.yaml and prom_values.yaml to something unique "foo"

Run terraform apply
When terraform is done run az aks get-credentials --resource-group barkuksatrng-k8stest-rg --name barkuksatrng-k8stest-cluster if needed
Run "kubectl get all"
Note external IP of Ambassador service
Navigate to Azure Portal -> Resource groups -> MC_barkuksatrng-k8stest-rg
-> Go through kubernetes-aXXXXXXXXXXXXXXXXXXXX until you find the one that has the same IP
as ambassador service
-> Select Configuration
-> Write "foo" (=unique thing you replaced it with) into DNS name label field and hit save
Run kubectl apply -f ambassador_mappings.yaml
Run kubectl apply -f tls.yaml
-> You should now be able to access grafana through URL https://foo.westeurope.cloudapp.azure.com/grafana/
jaeger through https://foo.westeurope.cloudapp.azure.com:16686

[ghost]

1. Enter a real email in place of "[email protected]" in tls.yaml

2. Ensure that you have a unique DNS name label, only one cluster can use the same name. Replace each "smaddis" in "smaddis.westeurope.cloudapp.azure.com" in files tls.yaml, ambassador_values.yaml and modules/container_deployment/prom_values.yaml with something unique "foo"

---

Set DNS name

After the cluster is ready,
0. Run

$ az aks get-credentials --resource-group barkuksatrng-k8stest-rg --name barkuksatrng-k8stest-cluster

1. Run

$ kubectl get all

2. Note the EXTERNAL-IP for service/ambassador
3. Head over to Azure Portal -> Resource Groups
4. Select your resource group MC_barkuksatrng-k8stest-...
5. Check each "kubernetes-aXXXXXXXXXXXXXX" Public IP until you find the one for ambassador
6. Select Configuration on the left menu
7. Write your unique DNS name label foo and Save
8. Perform

$ Run kubectl apply -f ambassador_mappings.yaml
$ Run kubectl apply -f tls.yaml

9. Grafana should be available in https://foo.westeurope.cloudapp.azure.com/grafana/ Jaeger https://foo.westeurope.cloudapp.azure.com:16686

Known issues

Rate limits for Let's encrypt

The current setup uses production let's encrypt which limits the amounts of time a certificate can be issued to a certain email, IP or domain name

Jaeger routing

Jaeger doesn't work through https://domain/jaeger/
Research query.base-path to configure it

Parameterize domain name and email

Changing mentions of "foo.westeurope.cloudapp.azure.com" is cumbersome and parametrizing these would be nice

Setting DNS manually

Using Azure Portal to set "foo" for Ambassador IP domain name is annoying. Research how to do this automatically.

[hjhsalo]

I would like to see the following done with Terraform.

Head over to Azure Portal -> Resource Groups
Select your resource group MC_barkuksatrng-k8stest-...
Check each "kubernetes-aXXXXXXXXXXXXXX" Public IP until you find the one for ambassador
Select Configuration on the left menu
Write your unique DNS name label foo and Save

Create a Public IP address with something like following:
(adapted from https://stackoverflow.com/q/64740298)

resource "azurerm_public_ip" "ambassador-ingress" {
  name                = "ambassador-ingress-pip"
  location            = azurerm_kubernetes_cluster.aks.location
  resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group
  
  ## or allocation_method = "Dynamic" is sku is "Basic":
  ## https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/public_ip#sku
  sku                 = "Standard"
  allocation_method   = "Static" 
  
  domain_name_label   = var.dns_name_prefix
}

Push created IP-address to Ambassador's Helm chart config.
There is a service.loadBalancerIP in Ambassador Helm chart configuration which seems to be used to propagate loadBalancerIP to Ambassador's K8S Service definition.
(https://github.com/datawire/ambassador-chart/tree/c540b0d9e91f7def8a7d9b99217cb62cfe3014fb#configuration)

Note that Azure specific service.beta.kubernetes.io/azure-load-balancer-resource-group might be needed for Ambassador's K8S Service. (https://docs.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address)

Azure specific service.beta.kubernetes.io/azure-dns-label-name annotation might be all that is needed to create a DNS entry for Ambassador's K8S Service. I.e. no public ip needs to be created with Terraform. (https://docs.microsoft.com/en-us/azure/aks/static-ip#apply-a-dns-label-to-the-service)

Annotations for Ambassador's K8S Service seem to be defined by 'service.annotations' key of Ambassador's Helm chart config.
(Not sure what should go inside the string for multiple annotations) https://github.com/datawire/ambassador-chart/blob/c540b0d9e91f7def8a7d9b99217cb62cfe3014fb/templates/service.yaml#L31-L35)

Related documentation and information available here:

• https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/public_ip#sku
• https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/public_ip
• https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip
• https://docs.microsoft.com/en-us/azure/aks/static-ip
• https://stackoverflow.com/a/60038748
• https://stackoverflow.com/a/56118653
• https://kubernetes.io/docs/concepts/services-networking/service/

[hjhsalo]

Since DNS name, smaddis.westeurope.cloudapp.azure.com hardcoded in the PR, would need to be parametrized (e.g. to support multiple, workspaced named deployments), tls.yaml could

• be rewritten in HCL for Terraform Kubernetes provider (https://github.com/hashicorp/terraform-provider-kubernetes). This doesn't support custom resource definitions, so it might not even work for this use case.
• OR be split and template for 3rd party Terraform extensions, e.g. https://github.com/banzaicloud/terraform-provider-k8s or https://github.com/gavinbunney/terraform-provider-kubectl
• OR be transformed with https://github.com/jrhouston/tfk8s to be used with https://github.com/hashicorp/terraform-provider-kubernetes-alpha
  Note that https://github.com/hashicorp/terraform-provider-kubernetes-alpha has quite alarming disclaimer and might be worth waiting on it is out of alpha.
  (See Feature Request: Custom Resource hashicorp/terraform-provider-kubernetes#215 for more info/speculation on GA schedule)

[hjhsalo]

DNS name parametrization should also be done for various Helm values.yaml files.
Terraform Helm provider mentions merging of values and set keys for helm_release resources: https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release#set

I'm not sure if 'set' is the best/correct way to parameterize values.yaml or does it have some drawbacks.
One alternative is to use 'yamlencode' available in Terraform: hashicorp/terraform-provider-helm#669 (comment)