Add carbonplan cluster + hubs

[yuvipanda]

• staging and prod hubs that are exactly the same,
  with just domain differences
• Uses traditional autohttps + LoadBalancer to get traffic
  into the cluster. Could be nginx-ingress later on if necessary.
• Manual DNS entries for staging.carbonplan.2i2c.cloud and
  carbonplan.2i2c.cloud. Initial manual deploy with
  proxy.https.enabled set to false to complete deployment,
  fetch externalIP of proxy-public service, setup DNS,
  then re-deploy with proxy.https.enabled set to true.
• Simple hacky script to setup EFS properly for a cluster
• Use image automatically built at https://github.com/carbonplan/trace/tree/main/envs/python-notebook
• Add ssh keys used to build the cluster
• Standardize labels used for our nodes - see Standardize on nodepool labeling conventions #136

Depends on #389 and #390

Ref #291

[yuvipanda]

I've deleted everything and recreating it from scratch.

0. cd into the kops directory

1. Set a 'state' bucket for kops - it will store current cluster state here. export KOPS_STATE_STORE=s3://<2i2c>-<hub-name>-kops-state

2. Create this bucket: aws s3 mb $KOPS_STATE_STORE --region <region>. (FIXME: Make this versioned?)

3. Render the kops config file: jsonnet carbonplan.jsonnet -y > carbonplan.kops.yaml

4. Delete last line of the yaml file (...)

5. Create the cluster config in s3: kops create -f carbonplan.kops.yaml

6. Create an ssh key ssh-keygen -f ssh-key

7. Create the cluster itself: kops update cluster carbonplanhub.k8s.local --yes --ssh-public-key ssh-key.pub --admin. The --admin will modify your ~/.kube/config file to point to the new cluster.

8. Wait for kops cluster to create kops validate cluster --wait 10m

9. In another terminal, workaround Ability to run CoreDNS on master node kubernetes/kops#11199 with k -n kube-system patch deployment kube-dns --type json --patch '[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "node-role.kubernetes.io/master", "effect": "NoSchedule"}]}]' and k -n kube-system patch deployment kube-dns-autoscaler --type json --patch '[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "node-role.kubernetes.io/master", "effect": "NoSchedule"}]}]'. Validate will not pass until this is done.

10. Create an EFS file system for this hub, with python3 setup-efs.py <cluster-name> <region>. You need boto3 python package installed. This will output an fs- id, which you should use in basehub.nfsPVC.nfs.serverIP. It should be something like fs-<id>.efs.<region>.amazonaws.com

11. cd back to the root of the repo

12. Generate kubeconfig for deployer: KUBECONFIG=secrets/carbonplan.yaml kops export kubecfg --admin=730h carbonplanhub.k8s.local. If you already have a file there you need to remove it.

13. Encrypt generated kubeconfig: sops -i -e secrets/carbonplan.yaml

14. Set proxy.https.enabled to false in carbonplan.clusters.yaml. This creates the hubs without trying to give them HTTPS, so we can create DNS entries for them appropriately.

15. Deploy hubs, with python3 deployer deploy carbonplan --skip-hub-health-test, and wait! It should hopefully complete successfully

16. Get external IP for staging with kubectl -n staging get svc proxy-public, and make a DNS CNAME record for staging.carbonplan.2i2c.cloud pointing to that.

17. Get external IP for staging with kubectl -n prod get svc proxy-public, and make a DNS CNAME record for carbonplan.2i2c.cloud pointing to that.

18. Wait for about 5 minutes to make sure the DNS records actually resolve

19. Set proxy.https.enabled to true in carbonplan.clusters.yaml, so we can get HTTPS!

20. Run python3 deployer deploy carbonplan again, and this should setup HTTPS + run a test!

[yuvipanda]

When rotating the master, you might run into kube-dns not evicting since that would violate its pdb. You can temporarily work around this by reducing the availability guarantee:

k -n kube-system patch pdb kube-dns --type json -p '[{"op": "replace", "path": "/spec/minAvailable", "value": 0}]'

However, once done, you should undo the change

k -n kube-system patch pdb kube-dns --type json -p '[{"op": "replace", "path": "/spec/minAvailable", "value": 1}]'

[damianavila]

OK, I left some comments, I will try to deploy this one from scratch tomorrow and report back how it goes!

[damianavila]

When rotating the master, you might run into kube-dns not evicting since that would violate its pdb.

I am pretty sure that was the problem preventing my kops rolling-update to succeed...

[yuvipanda]

I'll need to deploy and merge #397 first before merging this, so as to not break existing clusters.

[yuvipanda]

With #397 all GKE clusters have new labels. I've removed Farallon from CI/CD in #400

[yuvipanda]

Ok this one is ready to merge :)

[damianavila]

I have been reviewing this one as it was evolving.
Last final check LGTM
This is great @yuvipanda!
Feel free to push the green button!

[yuvipanda]

@damianavila done!