[BUG] Certificate injection doesn't work as expected in 1.4.0.

[pyrxm]

Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting

Components and versions
Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag)
[x] Env-Injector (webhook), version: 1.4.0 (docker image tag)
[ ] Other

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Install v1.3.0
2. Follow Inject Certificate tutorial from documentation.
3. Inspect the logs for created pod, expect to see public key for PUBLIC_KEY and private key for PRIVATE_KEY

   # Temporarily using 1.3.0 release
   ➜  ~ kgp -n akv2k8s -o jsonpath="{.items[*].spec.containers[*].image}"
   spvest/azure-keyvault-webhook:1.3.0
   spvest/azure-keyvault-webhook:1.3.0
   
   # Certificates injected
   ➜  ~ kl akvs-certificate-app-7f54c6d78d-h67vd
   Defaulted container "akv2k8s-env-test" out of: akv2k8s-env-test, copy-azurekeyvault-env (init)
   I0606 09:21:37.662786       1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv"
   I0606 09:21:37.662860       1 main.go:176] "azure key vault env injector initializing"
   I0606 09:21:37.662892       1 main.go:227] "akv2k8s auth service not enabled - will look for azure key vault credentials locally"
   I0606 09:21:37.663253       1 main.go:245] "found original container command" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh PUBLIC_KEY PRIVATE_KEY]
   I0606 09:21:37.817793       1 main.go:342] "secret injected into env var" azurekeyvaultsecret="xm-test/certificate-inject" env="PUBLIC_KEY"
   I0606 09:21:37.874004       1 main.go:342] "secret injected into env var" azurekeyvaultsecret="xm-test/certificate-inject" env="PRIVATE_KEY"
   I0606 09:21:37.874049       1 main.go:348] "starting process with secrets in env vars" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh PUBLIC_KEY PRIVATE_KEY]
   -----BEGIN CERTIFICATE-----
   MIIDOjCCAiKgAwIBAgIQFXKGVyJPTK2yjuFc5dBY2TANBgkqhkiG9w0BAQsFADAa
   .... SNIP ....
   -----END CERTIFICATE-----
   
   -----BEGIN RSA PRIVATE KEY-----
   MIIEowIBAAKCAQEA5eucv+XQfk624PC5/UX/6/8XTg5YevtpH/E/qnNImrqdtR+w
   .... SNIP ....
   -----END RSA PRIVATE KEY-----
   
   waiting forever...
   ➜  ~

4. Upgrade to 1.4.0
5. Restart deployment kubectl rollout restart deployment akvs-certificate-app
6. Inspect the logs for created pod, PUBLIC_KEY and PRIVATE_KEY now show the reference to the secret.

   # Using 1.4.0
   ➜  ~ kgp -n akv2k8s -o jsonpath="{.items[*].spec.containers[*].image}"
   spvest/azure-keyvault-webhook:1.4.0
   spvest/azure-keyvault-webhook:1.4.0
   
   # Restarting deployment
   ➜  ~ k rollout restart deployment akvs-certificate-app
   deployment.apps/akvs-certificate-app restarted
   
   # Certificates failing to inject again.
   ➜  ~ kl akvs-certificate-app-76dd968cbb-c5stk
   Defaulted container "akv2k8s-env-test" out of: akv2k8s-env-test, copy-azurekeyvault-env (init)
   I0606 09:33:03.548159       1 version.go:31] "version info" version="" commit="15d87b2" buildDate="2022-12-08T21:20:01Z" component="vaultenv"
   I0606 09:33:03.548331       1 main.go:184] "azure key vault env injector initializing"
   I0606 09:33:03.548381       1 main.go:235] "akv2k8s auth service not enabled - will look for azure key vault credentials locally"
   I0606 09:33:03.548596       1 main.go:253] "found original container command" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh PUBLIC_KEY PRIVATE_KEY]
   I0606 09:33:03.548933       1 main.go:361] "starting process with secrets in env vars" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh PUBLIC_KEY PRIVATE_KEY]
   certificate-inject@azurekeyvault?tls.crt
   certificate-inject@azurekeyvault?tls.key
   waiting forever...
   ➜  ~

Expected behavior

Expected certificates to be injected in version 1.4.0.

Logs
If applicable, add logs to help explain your problem.

They don't look applicable, but including anyway.

env-injector Logs for 1.3.0:

➜  ~ kl akv2k8s-envinjector-5fb97d9864-tg4bd
I0606 10:02:49.578858       1 version.go:31] "version info" version="1.3.0" commit="a375982" buildDate="2021-08-06T06:51:44Z" component="webhook"
I0606 10:02:49.578967       1 main.go:289] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="azureCloudConfig" useAuthService=false dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="2"
I0606 10:02:49.580450       1 main.go:321] "auth service disabled - azure key vault credentials must be provided manually for each pod" useAuthService=false
I0606 10:02:49.580516       1 main.go:396] "serving health endpoint" path=":8080/healthz"
2023/06/06 10:02:49 [WARN] no tracer active
I0606 10:02:49.580658       1 main.go:419] "serving encrypted webhook endpoint" path=":8443/pods"
I0606 10:02:49.580682       1 main.go:422] "serving encrypted healthz endpoint" path=":8443/healthz"
I0606 10:03:16.427400       1 main.go:143] "found pod to mutate" pod="xm-test/"
I0606 10:03:16.427430       1 pod.go:292] "mutate init-containers" xm-test/="(MISSING)"
I0606 10:03:16.427437       1 pod.go:298] "mutate containers" xm-test/="(MISSING)"
I0606 10:03:16.427445       1 pod.go:116] "found container to mutate" container="xm-test/akv2k8s-env-test"
I0606 10:03:16.427452       1 pod.go:119] "checking for env vars to inject" container="xm-test/akv2k8s-env-test"
I0606 10:03:16.427465       1 pod.go:122] "found env var to inject" env="certificate-inject@azurekeyvault?tls.crt" container="xm-test/akv2k8s-env-test"
I0606 10:03:16.427480       1 pod.go:122] "found env var to inject" env="certificate-inject@azurekeyvault?tls.key" container="xm-test/akv2k8s-env-test"
E0606 10:03:16.604260       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I0606 10:03:17.322962       1 pod.go:149] "found container arguments to use for env-injector" cmd="entrypoint.sh PUBLIC_KEY PRIVATE_KEY" container="xm-test/akv2k8s-env-test"
I0606 10:03:17.473836       1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="xm-test/"

env-injector Logs for 1.4.0

➜  ~ kl akv2k8s-envinjector-686547446-9q7tz
I0606 09:32:44.438553       1 version.go:31] "version info" version="1.4.0" commit="15d87b2" buildDate="2022-12-08T21:19:10Z" component="webhook"
I0606 09:32:44.438658       1 main.go:290] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="azureCloudConfig" useAuthService=false dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="2"
I0606 09:32:44.440441       1 main.go:322] "auth service disabled - azure key vault credentials must be provided manually for each pod" useAuthService=false
I0606 09:32:44.440494       1 main.go:397] "serving health endpoint" path=":8080/healthz"
2023/06/06 09:32:44 [WARN] no tracer active
I0606 09:32:44.440598       1 main.go:420] "serving encrypted webhook endpoint" path=":8443/pods"
I0606 09:32:44.440613       1 main.go:423] "serving encrypted healthz endpoint" path=":8443/healthz"
I0606 09:33:00.568537       1 main.go:142] "found pod to mutate" pod="xm-test/"
I0606 09:33:00.568567       1 pod.go:292] "mutate init-containers" xm-test/="(MISSING)"
I0606 09:33:00.568574       1 pod.go:298] "mutate containers" xm-test/="(MISSING)"
I0606 09:33:00.568581       1 pod.go:116] "found container to mutate" container="xm-test/akv2k8s-env-test"
I0606 09:33:00.568587       1 pod.go:119] "checking for env vars to inject" container="xm-test/akv2k8s-env-test"
I0606 09:33:00.568598       1 pod.go:122] "found env var to inject" env="certificate-inject@azurekeyvault?tls.crt" container="xm-test/akv2k8s-env-test"
I0606 09:33:00.568609       1 pod.go:122] "found env var to inject" env="certificate-inject@azurekeyvault?tls.key" container="xm-test/akv2k8s-env-test"
I0606 09:33:01.376324       1 pod.go:149] "found container arguments to use for env-injector" cmd="entrypoint.sh PUBLIC_KEY PRIVATE_KEY" container="xm-test/akv2k8s-env-test"
I0606 09:33:01.707557       1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="xm-test/"

Additional context

• Controller is disabled, we only use Env-Injector
• authService is disabled, each pod uses aad-pod-identity

[Treehopper]

I have the same, or a very similar problem. It already occurs with the update to 1.3.1 (and persists with 1.4.0).
In the init container I maintain, "foo@azurekeyvault?tls.crt" is mapped to the environment variable TLS_CRT.

After some research about what has changed between 1.3.0 and 1.3.1, I believe the regex here might be responsible:
https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/pull/320/files/445cec25db675d7c6df347e7793805c84c33eeed
I don't speak Go, but I assume the regex should include the . character in order to accept the string tls.crt.

If so, is there any workaround to this?

[tspearconquest]

I was able to confirm this with regexpal.com

Doesn't work:

• Regex:^([a-z-\d]*)@azurekeyvault([\?]?[a-z-\d]*)$
• Input: @azurekeyvault?tls-01.crt

Works:

• Regex: ^([a-z-\d]*)@azurekeyvault([\?]?[\w\d\.-]*)$
• Input: @azurekeyvault?tls-01.crt

In master, the line was moved to: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/blob/master/cmd/azure-keyvault-env/environment.go#L12 and enhanced to:

envLookupRegex = `^([a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)@azurekeyvault(\?([a-zA-Z_][a-zA-Z0-9_\.]*)?)?$`

This change was made 4 months ago and released with v1.5.0-beta.3 of the env-injector.

It will go out with the stable release of v1.5.0.

[tspearconquest]

/close

[Treehopper]

Thanks! Any chance this could be backported to a 1.3.x or 1.4.x release?

[Speeddymon]

Unfortunately I'm not on the dev team so I don't know that they would accept a patch to backport, but if you're willing to build it from source, I could generate a diff for you to patch in just before the build.

[Treehopper]

I can certainly create a branch where I patch the 1.3.1 version with the new regex, but I wouldn't be able to build it myself.

[181192]

Closing, 1.5.0 is released with fixes

[Treehopper]

See #475