Install argocd-vault-plugin

[mathis-marcotte]

In relation to https://github.com/StatCan/aaw-private/issues/149

If we want to handle secrets coming from Azure Key Vaults coming through ArgoCD, we will need to install the argocd-vault-plugin.

Link to the plugin (It has doc on how to install it in there)

Will also gave us some CNS documentation on installing it here , but basically it seems that they have a dockerfile for their argocd install that has the plugin in there

I also noticed that we have our own fork of the argocd-vault-plugin. Might be useful to quickly look into if that is actually used anywhere

[mathis-marcotte]

Looking at the fork, it was last updated 2 years ago, and the CNS dockerfile doesn't use it, they just use the public plugin repo. So I don't really know why that fork was created, but we can probably avoid using it.

[mathis-marcotte]

Might be worth looking into this repository from CNS on how to setup argo for handling secrets coming from AKV

[mathis-marcotte]

I tried installing it following the installation documentation.

1. I first created the config map cmp-plugin in the daaas-system namespace to define the ConfigManagementPlugin.
2. I then applied a patch to the daaas-system-repo-server deployment to add the vault plugin sidecar install.
3. I then created a vault-configuration secret for the configurations for the plugin, just defining the backend type.
4. Then I followed the usage guide, following the kustomize instructions. I went with kustomize just because we already use it for our manifests.

And then to test, I added the secret we want managed to the manifest. But it is failing to sync in argocd(generic error saying the value of the secret CLIENT_SECRET isnt encoded which is a normal error message that makes sense since the placeholder path didnt get replaced by the plugin).

So i need to investigate which part of the install didn't work, or even if i was updating it in the correct deployment.

From what I can tell, it's the patch to the deployment that seemed to not have worked as there was no sidecar for the plugin running, plus the init container to download the avp plugin also didnt seem to have run.
I tried patching the deployment manually and then running kubectl apply the deployment, but that also didnt seem to add the sidecar installation. So I'm thinking that the deployment is being managed by something else, and that is preventing me from updating it by just re-updating back to its default after i try to add stuff.

[mathis-marcotte]

I added the deployment yaml to aaw-argocd-manifests and that made it work and read that deployment resource for the avp install.

But the issue I am now encountering is that the initcontainer to download the plugin isn't working. It is getting an error when trying to download from github, saying that the connection is refused when connection to github port 443. But, we already have a netpol in place to allow for the argocd-repo-server pod to egress to the internet(or at least that what it says it should do). So I'm not sure what the issue is actually, will need to investigate further.

[mathis-marcotte]

With the networking issues, i decided to switch how the install is done and instead use a custom image saved in our dev acr called argocd-vault-plugin-dev. Here is the gist for the dockerfile of the image we used.

With using this new custom image, the pod is running successfully, and the plugin seems to be installed in the sidecar. So now, it will be a matter of testing to see if the plugin is properly working.

[mathis-marcotte]

Still slowing resolving the issues from this install. Right now, all the pods sucessfully boot up for argocd without errors, although it seems like the deployment gets overridden on the cluster sometimes, ignoring the argocd manifest deployment override I have defined (plus the argocd deployment isnt on autosync, so that a reason why it doesnt re-apply my deployment definition). This might be something with were I defined those resources (in the argocd manifest github instead of the gitlab somewhere), so just tweaking that. But that is a minor issue for now.

The major issue I am currently looking into is this one, which i think is because of the permissions that the pod needs to connect to the key vault

[mathis-marcotte]

I talked with @justbert from CNS about this issue a little bit. He suggested that we move away from trying to connect to the vault we already had (aaw-dev-cc-00-kv-vault) in favor of a new key vault that we can create specifically for passing secrets to argocd.
The reason for this is for a better security and separation of concerns (and as Justin put it, "we wouldn't use the same database for two different applications").
That key vault we already have is actually Vault's backing key vault, which is in use by "Vault" to store Certificate/Private key pairs so that vault can auto-unseal itself.

I do agree with his suggestion and I will probably move forward with the work needed to create a new key vault for the purpose of being used by the plugin.

[justbert]

Yeah, it's used for this Vault feature: https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-azure-keyvault

[mathis-marcotte]

On a side note, I found this terraform definition example from CNS for an argocd installation with the azure-vault-plugin.
I've been doing the install in a bit of a less organized fashion to just get it running and tested, but it would be a very good idea to mirror a bit more how it's setup there.

[mathis-marcotte]

It looks like I was able to get a proof of concept up and running with a secret resource managed by the plugin.

I have been testing with this application definition

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-plugin
  namespace: daaas-system
spec:
  destination:
    name: in-cluster
    namespace: daaas-system
  project: default
  source:
    repoURL: https://github.com/mathis-marcotte/argocd-example-apps.git
    targetRevision: master
    path: guestbook
    plugin: {}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The repo it points to is a fork I made of the argocd examples repo. It contains a secret definition with a value that the plugin reads and replaces with the secret it finds in the azure key vault from the path in the placeholder.

[mathis-marcotte]

Trying to write everything down that was put in place for this install:

I created 3 files in the argocd-manifests repo: deployment.yaml, configmanagementplugin.yaml and vaultconfiguration.yaml

vaultconfiguration.yaml creates a secret resource that contains the configurations for the plugin. These are the type of vault we are using(azurekeyvault) as well as the variables used for authentication against azure. I decided to use the client credentials for this first install since I was more familiar with that type of authentication. CNS used azure managed identities for this authentication, which I think we should follow in the future.
configmanagementplugin.yaml creates a configmap with instructions for the plugins functions. I decided to define instructions for both the normal plugin as well as the kustomize instructions since I knew we used kustomize. I honestly don't know the full extent of the use of the kustomize instructions, but I assume they are ran when we kustomize add a secret. This is also where the vaultconfiguration secret is used. CNS instead defined the helm instructions in their implementation. This is something we could look at in the future for our implementation.
The deployment.yaml file is used to override the deployment resource of our argocd-repo-server instance to include the sidecars for both the normal plugin install as well as the kustomize plugin sidecar. For both sidecars, we used a custom image, as mentioned above. That was the solution we used to fix the networking issues I was having with the initContainer example defined in the plugin docs.

After those 3 files were commited to the main branch, they could then be managed through ArgoCD. The reason I went with the solution with files in the manifests repo was because I was more familiar with that repo, and I don't have rights to run the tf apply command in the gitlab repo. So it was a lot more easy for testing for me. In the future, I think we should add these components where our ArgoCD instance is initially defined in gitlab. This would keep our argocd install all in one place. This would also mirror how CNS did the install. Plus, I have seen some issues in my testing where the argo instance would get re-created, and it would then ignore the deployment override defined in the manfest repo and therefore wouldn't have the plugin installed. So having everything in one place would solve this potential issue

To then test the install, I applied the application definition pasted in the comment above. The important part of that application definition is the plugin: {} line. This is where you can usually define some environment variables for the application to be used in the plugins, and even if there is none needed in this example, we still need this line to let argocd know that this application wants to use the installed plugin.
Inside the repo that this application is calling, I added this secret definition

apiVersion: v1
kind: Secret
metadata: 
  name: super-duper-secret
  namespace: daaas-system
type: Opaque
data:
  CLIENT_ID: YmYyOWE3M2UtY2M5Yy00MmY0LTg1NDctYzYwMmFmZGQ5ZmY1
  #CLIENT_SECRET: <path:aaw-dev-cc-00-kv-argocd#kubeflow-client-secret>
  CLIENT_SECRET: PHBhdGg6YWF3LWRldi1jYy0wMC1rdi1hcmdvY2Qja3ViZWZsb3ctY2xpZW50LXNlY3JldD4=

The client_id is just some random value. The client_secret is the one that will be watching for the Azure key vault secret, and I wrote it following the examples from the plugin docs.
One issue I had was that since this was a v1 secret, the values defined in the data section had to be encoded with base64. So that is why the secret looks like a bunch of random characters. The issue with that is that the plugin is configured so that it looks for that <path> annotation. BUT! With the secret needing encoded values, that hides that path annotation. So I actually had to add that commented line so that the plugin could pick up this secret as something that needed the plugin. Now, this is definitely a problem, or at least an annoyance, that I needed a commented line for the secret to be successfully picked up by the plugin. This is something that we can look more into in the future on either if we can refine the discover rules in the configmanagementplugin file, or if we can have secrets that dont need to be encoded first.

Once everything is up and running, then it is very easy to update the secret resource that gets created. Just update the secret in the key vault to a new version, and then do a hard refresh in the argocd application that contains the secret.