Retry failed namespace informer creation in promOperator CRD watcher

[davidhaja]

Description:
Added retry mechanism for namespace informer creation in promOperator CRD watcher.
In case of the namespace informer creation still fails, then exit the target allocator with error.

Link to tracking Issue(s): 3216

• Resolves: Temporary network issue causes Target Allocator to stuck #3216

Testing:
Done some manual testing
If I changed the context to have timeout, and configured the timeout to 1 nanosecond:

$ go run cmd/otel-allocator/main.go
{"level":"info","ts":"2024-08-29T23:17:08+02:00","msg":"Starting the Target Allocator"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Retrying namespace informer creation in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher.func1\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:111\nk8s.io/client-go/util/retry.OnError.func1\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:55\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50\ngithub.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:109\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Retrying namespace informer creation in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher.func1\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:111\nk8s.io/client-go/util/retry.OnError.func1\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:55\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50\ngithub.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:109\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Retrying namespace informer creation in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher.func1\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:111\nk8s.io/client-go/util/retry.OnError.func1\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:55\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50\ngithub.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:109\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Retrying namespace informer creation in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher.func1\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:111\nk8s.io/client-go/util/retry.OnError.func1\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:55\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50\ngithub.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:109\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Retrying namespace informer creation in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher.func1\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:111\nk8s.io/client-go/util/retry.OnError.func1\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:55\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/Users/dhaja/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50\ngithub.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:109\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup.prometheus-cr-watcher","msg":"Failed to create namespace informer in promOperator CRD watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/watcher.NewPrometheusCRWatcher\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/watcher/promOperator.go:118\nmain.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:119\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
{"level":"error","ts":"2024-08-29T23:17:08+02:00","logger":"setup","msg":"Can't start the prometheus watcher","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"main.main\n\t/Users/dhaja/Projects/external/forked/opentelemetry-operator/cmd/otel-allocator/main.go:121\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.0/libexec/src/runtime/proc.go:272"}
exit status 1

Without any "network issue"/failure:

$ go run cmd/otel-allocator/main.go
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Starting the Target Allocator"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","logger":"allocator","msg":"Starting server..."}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Waiting for caches to sync for namespace"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Caches are synced for namespace"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Waiting for caches to sync for servicemonitors"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Caches are synced for servicemonitors"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Waiting for caches to sync for podmonitors"}
{"level":"info","ts":"2024-08-29T23:12:54+02:00","msg":"Caches are synced for podmonitors"}

Documentation:
N/A