Skip to content
This repository has been archived by the owner on Jul 26, 2022. It is now read-only.

Application stops functioning after a few days #763

Closed
jlact opened this issue May 26, 2021 · 2 comments
Closed

Application stops functioning after a few days #763

jlact opened this issue May 26, 2021 · 2 comments

Comments

@jlact
Copy link

jlact commented May 26, 2021

Application seems to just stop functioning, usually after a few days. No action is taken to stop or terminate deployment. As far as kubernetes is concerned the pod is still running, and it does indeed respond to calls on the metrics endpoint, just doesn't manage the secrets anymore. Issuing a rolling restart on the deployment resolves it, but it's a manual fix.

version: 7.2.0
k8s: 1.19
backend: Azure Keyvault

Last entries in log files look like:

{"level":"info","message_time":"2021-05-21T22:12:49.695Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"Stopping watch stream for namespace * due to event: END"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/notifications-secret"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/private-key"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns2/dns-config"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns3/external-dns-config"}

Metrics from kubernetes show CPU (and network) usage as 0 once it occurs, memory usage is also static. Green being the 'failed' pod and orange a new one from restarting the deployment:
image

@anderius
Copy link

anderius commented Jun 2, 2021

Same problem here:

helm chart version: 8.0.1
k8s (AKS): 1.20.5
backend: Azure Keyvault

@Flydiverny
Copy link
Member

Should be resolved in 8.0.2 by #765

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants