Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kubernetes Module] Add missing metrics for replicaset and pod owners #36746

Closed
wants to merge 12 commits into from
Closed

[Kubernetes Module] Add missing metrics for replicaset and pod owners #36746

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
cbe4e56
Adding missing info for replicaset and pod owners
gizas Oct 4, 2023
81fee2f
Fixing fields docs
gizas Oct 5, 2023
3ca416c
Fixing fields docs
gizas Oct 5, 2023
f7a14d9
Fixing fields docs
gizas Oct 5, 2023
eeaa34b
Fixing fields docs
gizas Oct 5, 2023
3f7bc37
Fixing conflicts
gizas Oct 5, 2023
1008174
Fixing tests
gizas Oct 5, 2023
0213476
Fixing tests
gizas Oct 5, 2023
390a311
Fixing tests for replicaset
gizas Oct 5, 2023
3656949
Adding Changelog
gizas Oct 5, 2023
e30bf1a
Update metricbeat/module/kubernetes/state_pod/_meta/fields.yml
gizas Oct 6, 2023
62079d6
Update metricbeat/module/kubernetes/state_pod/_meta/fields.yml
gizas Oct 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,8 @@ is collected by it.
- Add remaining dimensions for azure storage account to make them available for tsdb enablement. {pull}36331[36331]
- Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413]
- Add log error when statsd server fails to start {pull}36477[36477]
- [Kubernetes Module] Add missing metrics for replicaset and pod owners {pull}36746[36746]


*Osquerybeat*

Expand Down
74 changes: 74 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46370,6 +46370,43 @@ type: keyword
Kubernetes pod scheduled status (true, false, unknown)


type: keyword

--

[float]
=== owner

Kubernetes pob owner information



*`kubernetes.pod.owner.name`*::
+
--
The name of the resource that owns this pob


type: keyword

--

*`kubernetes.pod.owner.kind`*::
+
--
The kind of resource that owns this pob (eg. "Replicaset")


type: keyword

--

*`kubernetes.pod.owner.is_controller`*::
+
--
Owner is controller ("true", "false", or "<none>")


type: keyword

--
Expand Down Expand Up @@ -46438,6 +46475,43 @@ type: long

--

[float]
=== owner

Kubernetes replicaset owner information



*`kubernetes.replicaset.replicas.owner.name`*::
+
--
The name of the resource that owns this replicaset


type: keyword

--

*`kubernetes.replicaset.replicas.owner.kind`*::
+
--
The kind of resource that owns this replicaset (eg. "Deployment")


type: keyword

--

*`kubernetes.replicaset.replicas.owner.is_controller`*::
+
--
Owner is controller ("true", "false", or "<none>")


type: keyword

--

[float]
=== resourcequota

Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/kubernetes/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

15 changes: 10 additions & 5 deletions metricbeat/module/kubernetes/state_pod/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,22 @@
"module": "kubernetes"
},
"kubernetes": {
"namespace": "default",
"namespace": "kube-system",
"node": {
"name": "kind-control-plane"
},
"pod": {
"host_ip": "172.18.0.2",
"ip": "10.244.0.22",
"name": "hello-28192809-jbn5z",
"ip": "10.244.0.3",
"name": "coredns-787d4945fb-kx4f4",
"owner": {
"is_controller": "true",
"kind": "ReplicaSet",
"name": "coredns-787d4945fb"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How this would replace the disabled addition of the kubernetes.deployment field? This only adds the first level owner so in a case of Deploymenyt->ReplicaSet->Pod it won't add the Deployment information, right?

Also this is happening already supported through the Metadata Enricher -> https://github.com/elastic/elastic-agent-autodiscover/blob/2cc3dcb075fe437467387d36096f554c13c6d144/kubernetes/metadata/resource.go#L138-L152

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not add the deployment name correct only 1st level, you are right.

I was thinking that we can justify that this is enough for the users in order to figure out the reference. Of course we need to document this. Otherwise even if we can try in code or with ingest pipelines to trim the strings and produce ancestors we might fall in mistakes because we are not sure for the existance.

Also if we integrate this elastic/elastic-agent-autodiscover#62, this means that also the enrichers will loose those metadata is not it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Through the Enricher every metrics' document for Pod's like pod.cpu.usage will be enriched with Pod's metadata. In this metadata the Enricher adds the owner name like kubernetes.cronjob.name, kubernetes.replicaset.name etc. So this is not related to the settings you change at elastic/elastic-agent-autodiscover#62. That's a diffrent feature takes place at https://github.com/elastic/elastic-agent-autodiscover/blob/2cc3dcb075fe437467387d36096f554c13c6d144/kubernetes/metadata/pod.go#L96-L122.

So still this patch seems to be redundant.
Could you try to run Metricbeat with k8s module enabled and see that Pod's metrics are enriched with the owner name (disable the deployment and cronjob metadata as usually to isolate the functionality)? This will help you understand how the feature works and that this patch actually overlaps already existent information.

},
"status": {
"phase": "succeeded",
"ready": "false",
"phase": "running",
"ready": "true",
"scheduled": "true"
}
}
Expand Down
17 changes: 17 additions & 0 deletions metricbeat/module/kubernetes/state_pod/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,20 @@
type: keyword
description: >
Kubernetes pod scheduled status (true, false, unknown)
- name: owner
type: group
description: >
Kubernetes pob owner information
fields:
- name: name
type: keyword
description: >
The name of the resource that owns this Pod
- name: kind
type: keyword
description: >
The kind of resource that owns this Pob (eg. "Replicaset")
- name: is_controller
type: keyword
description: >
Owner is controller ("true", "false", or "<none>")
Loading