flux_bootstrap_git optionnaly commit files

[sereinity]

Hello,

I suggest the following improvement: add a flag to disable the commit to the git repository and only push resources to the kubernetes cluster.

Regards,

[stefanprodan]

This goes against what we've designed bootstrap for, bootstrap means that Flux manages itself from Git, if you don't push the manifests to Git than you don't need to use the flux_bootstrap_git provider.

[sereinity]

Alright, as long as flux_sync and flux_system are kept.

Another resource is planed to bootstrap without creating a commit?

(I feel like I've reached the https://xkcd.com/1172/ point 🤣)

[phillebaba]

So I actually started working on a flux_install resource before I built the flux_bootstrap resource thinking it would be easier. There are some challenges building flux_install resource in regards to tracking diffs.

I do not think it is impossible but it is not something I am giving priority to currently. Can I ask what your use case is?

[sereinity]

I do not think it is impossible but it is not something I am giving priority to currently. Can I ask what your use case is?

Example workflow:

I want to test and maybe create a merge request for the new ✨ flux_bootstrap_git.

I create a new terraform workspace for one of my cluster recipe. I setup this workspace to deploy from my working branch (in terraform cloud + add some providers variables).
It will:

• create the cluster and every resources at various places, by example it will also create a service account for flux in gitlab
• create a new workspace, on another tree from the same git repository, bound to the current branch. This workspace is only used for resources inside the kubernetes instance (making easier the removal of the whole − without having to deal with the proper order to remove such namespace/crd …)
• the new workspace will then run and bootstrap flux (and other k8s resources) without commiting anything and especially, it will reuse the root manifests from the previous existing cluster

Then I can work on my new feature, upgrade … at the end a do a destroy plan on my first workspace to remove everything.

Why did we designed it like that?

I had a bad experiment with the management of a single tree with multiple clusters.
At the time it felt strange and maybe dangerous to have two ways to push flux upgrades; one by terraform, one by the files in git. So I decided to only keep the management of flux by terraform (which is managed in git).

Today and regarding the current issue, I wonder if will create a manifest directory isolated by cluster, even if they are the same, even if it's temporary, or something else.

Some questions that should go to the doc

On bootstrap flux_bootstrap_git will commit and push files in the repository and it creates resources in kubernetes. But what will happens on upgrade?
Will it only commit manifest upgrades and then flux to upgrade itself? Or the terraform provider will also push them directly to the kubernetes API?

[phillebaba]

So this would not work with the current resource. It cheats by only making changes to the git repository, as it is a lot easier to track files in git than it is to track resources in Kubernetes. So it is not possible to have it only update Kubernetes.

This is why I said that it would be better to implement a new resource called flux_install which would only install Flux into the cluster. It would require end users to both commit the files to git and apply the required bootstrap reconcile resources in the cluster. From my perspective this is taking a step back to how it was before with the datasource method. Which is why I chose to not spend more time implementing that initially.

[amrecio]

For what is worth, we have another use case for the flux_install and flux_sync Terraform datasources at the company I work for (yes, XKCD 1172 strikes again).

We use the Weave GitOps Terraform Controller to provision and manage Kubernetes clusters on several cloud providers. Each Terraform resource for TF-controller applies an instance of a Terraform module that we have developed that takes care not only of creating the Kubernetes cluster, but also of creating a Git repository, populating it, and, crucially, bootstrapping Flux.

Unfortunally, Terraform doesn't support creating a Kubernetes cluster and using it in a single apply operation. Quoting from the documentation:

WARNING
When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources. Please refer to this section of Terraform docs for further explanation.

To work around this limitation and keep everything in a single Terraform module so that TF-controller can apply it atomically, our Terraform module performs roughly the following steps:

1. Create the Git repository for Flux.
2. Get the Flux bootstrapping manifests from the flux_install and flux_sync data sources.
3. Populate the Git using the bootstrapping manifests from step 2.
4. Create the Kubernetes cluster VMs, passing a cloud-init script to deploy Kubernetes and apply the same Flux manifests from step 2.

We don't currently have an use for the new flux_bootstrap_git, but the eventual removal of the flux_install and flux_sync data sources will leave us without a convenient way of generating the Flux manifests from Terraform.