Configure Vault connection details in Provider pod

[tomhjp]

Currently, Vault connection details are specified in the SecretProviderClass, e.g.:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-kv
spec:
  provider: vault
  parameters:
    roleName: "kv-role"
    vaultAddress: https://vault:8200
    vaultCACertPath: /mnt/tls/ca.crt
    vaultTLSClientCertPath: /mnt/tls/client.crt
    vaultTLSClientKeyPath: /mnt/tls/client.key
    objects: |
      - objectName: "secret-1"
        secretPath: "secret/data/kv1"
        secretKey: "bar1"

However, this awkwardly punctures the separation of concerns between the application namespace and the CSI provider's namespace. The application-namespaced SPC has to know about the contents of the provider pod's file system in order to configure which TLS certificates to use. The argument for vaultAddress being out of place is less clear, but would also arguably make more sense to be configured on the Vault provider side rather than on the application side.

If we move Vault connection details into the provider pod's configuration, in addition to a better separation of concerns, it will also allow us to safely deploy Vault Agent as a side car for the provider. That would then give us lots of nice features like caching and automatic lease renewal almost for free.

[isugimpy]

In this vein, the defaultVaultAddress and defaultVaultKubernetesMountPath are hardcoded constants. By changing both to be configurable at the Provider level, a sane default could be set for the Provider to use if the SecretProviderClass does not set them. This would be a helpful abstraction for users, and allow them to still override if they want to. Functionally, all that would be changing here is adding a configuration value with defaults for these two variables, instead of making them consts.

[ambis]

I would love to see the client tls cert configs to be moved to the provider config as well.

The client TLS cert configs (eg. vaultTLSClientKeyPath) are tightly tied to the vault instance (ie. defaultVaultAddress), and having to define them on application level feels just wrong, especially when the vault address has been defaulted.