Automatically collect metrics, logs from K8s Control Plane

[jvoravong]

Overview

• Added support for collecting the k8s control plane apiserver metrics via the smartagent/kubernetes-apiserver receiver.
• Collection of metrics from other control plane components will be in a future PR.
• Collection of logs from control plane components will be in a future PR.

Notes:

• I'm open to suggestions and changing how I feed the apisever.config value from values.yaml to the _otel-agent file durring the render process. This was the best solution I could come up with. I wanted to support having different default config values based on k8s distribution and also support allowing the user to supply their own custom config.

[dmitryax]

Just to confirm. Since these are plain prometheus metrics, this is a temporary solution, and will be replaced with receiver_creator / prometheus going forward, right?

Prior to switching to prometheus, we probably need to setup all the filtering of the non-default metrics in signalfx exporter https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/signalfxexporter/internal/translation/default_metrics.go#L22 @mstumpfx can correct me if I'm wrong

[dmitryax]

also don't forget to run make render or setup pre-commit

[dmitryax]

I'm getting this error trying it locally:

2022-01-13T07:45:30.626Z	error	[email protected]/observerhandler.go:136	failed to start receiver	{"kind": "receiver", "name": "receiver_creator", "receiver": "smartagent/kubernetes-apiserver", "error": "failed to load template config: error reading receivers configuration for \"smartagent/kubernetes-apiserver\": you must specify a \"type\" for a smartagent receiver"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).OnAdd
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/observerhandler.go:136
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/k8sobserver.(*handler).OnAdd
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/[email protected]/handler.go:41
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:777
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73

[mstumpfx]

Just to confirm. Since these are plain prometheus metrics, this is a temporary solution, and will be replaced with receiver_creator / prometheus going forward, right?

Prior to switching to prometheus, we probably need to setup all the filtering of the non-default metrics in signalfx exporter https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/signalfxexporter/internal/translation/default_metrics.go#L22 @mstumpfx can correct me if I'm wrong

The benefit of this approach is that the filtering is already built-in to the smart agent monitor. But yeah, they're just prometheus metrics and we could do exactly what you said.

[jvoravong]

Looks like I am still running into some render issues, looking into them.

[jvoravong]

I'm getting this error trying it locally:

2022-01-13T07:45:30.626Z	error	[email protected]/observerhandler.go:136	failed to start receiver	

I started this PR earlier then intended and my first commit had config errors in it. The sa apiserver configs have been updated and work properly now. The git actions are passing now.

Using the auto discovery rule with type==port made the configs a lot cleaner, thanks for the input @dmitryax and @mstumpfx. Can you guys look at this PR again?

[mstumpfx]

LGTM from discovery perspective. Glad the port rule works.

[dmitryax]

LGTM. Please rebase and add a line to CHANGELOG