Helm chart for Flyte

[rstanevich]

This PR contains Helm chart for Flyte with sandbox and EKS configurations.

The configuration for sandbox (values-sandbox.yaml) is ready for deploying in Minikube. But EKS config (values-eks.yaml) should be edited before installation in the cloud: s3 bucket, RDS hosts, iam roles, secrets and etc need to be configured and modified.

[kumare3]

@rstanevich i have been looking at Helm now and I am liking it. I will review your PR and we can build on it I feel. One of the problem seems to be how to use from remote configurations like - pytorch operator etc

[kumare3]

on EKS now you can use alb

[rstanevich]

oh, yes I've just found this announce https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/
looks like for now it is possible, also it requires aws-load-balancer-controller 2.0+ installed in kubernetes.
thank you!

[kumare3]

Ya I did set it up already on a personal
Account and works really well. I will be updating the eks manifests

[sbrunk]

Since we're stuck with Helm for the time being we'd like to contribute here. We could help test the chart and perhaps work on the GKE config.

[rstanevich]

So, I still didn't try new feature of AWS ALB with gRPC support. For provisioning it in K8s it requires new AWS loadbalancer controller for Kubernetes. I need some time to setup own devbox with new controller for testing this stuff.

[kumare3]

Since we're stuck with Helm for the time being we'd like to contribute here. We could help test the chart and perhaps work on the GKE config.

@sbrunk i would love to help you with some testing as well

[sbrunk]

@kumare3 @rstanevich what do you think about an approach that minimizes the diff between kustomize output and helm template output first for core Flyte w.o. dependencies, and then iterate from there?

That way we can make sure the helm installation is on par with what we have right now and it could also provide a smoother upgrade path.

My first crude try looks like this:

gh pr checkout 550
kustomize build kustomize/base/single_cluster/complete > base_deployment.yaml
kubectl apply -f base_deployment.yaml
helm template . -f values-sandbox.yaml | kubectl diff -f - 

Then incrementally work through the errors (and the diff later), change the helm chart accordingly to minimize the diff and run helm template again.

A slightly better approach could be using a structural diff of the rendered yaml output. That's because kubectl diff will check some API constraints (immutable fields etc.) that can slow us down here. I just havn't tried that yet because I couldn't find good tooling on first sight.

[rstanevich]

@kumare3 @rstanevich what do you think about an approach that minimizes the diff between kustomize output and helm template output first for core Flyte w.o. dependencies, and then iterate from there?

That way we can make sure the helm installation is on par with what we have right now and it could also provide a smoother upgrade path.

My first crude try looks like this:

gh pr checkout 550
kustomize build kustomize/base/single_cluster/complete > base_deployment.yaml
kubectl apply -f base_deployment.yaml
helm template . -f values-sandbox.yaml | kubectl diff -f - 

Then incrementally work through the errors (and the diff later), change the helm chart accordingly to minimize the diff and run helm template again.

A slightly better approach could be using a structural diff of the rendered yaml output. That's because kubectl diff will check some API constraints (immutable fields etc.) that can slow us down here. I just havn't tried that yet because I couldn't find good tooling on first sight.

@sbrunk, do you mean we just need to compare the generated helm manifest and flyte_generated.yaml? Do we need run this once for this PR or some script to check it regularly? So, at first glance, I see one evident problem:

• ConfigMap generated by Kustomize has a hash suffix in the name, but helm does not. So diff for ConfigMap won't work.

If the main goal is just to check smooth update from kustomize to helm installation I can check it out.

And an obvious note: If we'd like using helm install (I don't like this option :) ) - we cannot override existent k8s resources, we'll get smth like resource already exists.

[sbrunk]

@rstanevich yes I meant to use the diff only to help during development of the chart. It actually came up when I was looking into this PR to see how far you got compared with the kustomize based deployment, i.e. is the sandbox on par. I guess this is something you can answer, too. 😉

For us the upgrade path is actually not important because we don't run Flyte in prod yet but I guess for most people running Flyte in prod it's quite important.

[rstanevich]

resolved in #916