Event.dataset is set differently for KSM between beats and Elastic Agent

[gizas]

Investigate why event.dataset is set differently for kube-state-metrics metricsets between Elastic agent and Beats.

See below images for:

Beats:

For Elastic Agent:

So for the same kube-state-metrics node metricset we can see that event.dataset is state_node in Elastic Agent vs node in Beats.

We should investigate where this change came from and see if we need to backport any changes in beats code.

Related discussion:

elastic/integrations#7144 (comment)

[constanca-m]

This description is not totally correct, we see the same behavior using metricbeat: elastic/integrations#7144 (comment).

[gizas]

One thing also to mention here is that in elastic-agent:

state_pod: https://github.com/elastic/integrations/blob/main/packages/kubernetes/data_stream/state_pod/sample_event.json#L105
state_node: https://github.com/elastic/integrations/blob/main/packages/kubernetes/data_stream/state_node/sample_event.json#L84

So we need to understand why this was not picked in our tests

[ChrsMark]

@constanca-m I think what you indicate is a different issue compared to what @gizas described:

1. When KSM is reachable we get node from Beats while state_node from Agent. This might be because of [Elastic Agent] Add event.dataset to all events #20076.
2. When KSM is not reachable we get state_node from both. I guess this is because Metricbeat module is failing early and does not reach the code line that set the value to node. To me this is different to the above issue. Ofc we can fix it as well.

In any case we need to figure out which values are set and why. Also check which one is the correct one to use. And last but not least to document our findings.

[gizas]

In order to be sure we can test only with kube-state-metrics datastream enabled.

Also my tests with elastic-package

Only if I delete the sample_event.json file are failing :)

PACKAGE="kubernetes" elastic-package test -vvv static
2023/08/04 16:42:39 DEBUG Enable verbose logging
Run static tests for the package
--- Test results for package: kubernetes - START ---
╭────────────┬─────────────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM                 │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ kubernetes │ apiserver                   │ static    │ Verify sample_event.json │ PASS   │   28.15525ms │
│ kubernetes │ audit_logs                  │ static    │ Verify sample_event.json │ PASS   │  27.145375ms │
│ kubernetes │ container                   │ static    │ Verify sample_event.json │ PASS   │  29.038625ms │
│ kubernetes │ container_logs              │ static    │ Verify sample_event.json │ PASS   │  27.649291ms │
│ kubernetes │ controllermanager           │ static    │ Verify sample_event.json │ PASS   │  26.838125ms │
│ kubernetes │ event                       │ static    │ Verify sample_event.json │ PASS   │  27.290958ms │
│ kubernetes │ node                        │ static    │ Verify sample_event.json │ PASS   │  27.363083ms │
│ kubernetes │ pod                         │ static    │ Verify sample_event.json │ PASS   │  28.844959ms │
│ kubernetes │ proxy                       │ static    │ Verify sample_event.json │ PASS   │  29.012625ms │
│ kubernetes │ scheduler                   │ static    │ Verify sample_event.json │ PASS   │   27.03975ms │
│ kubernetes │ state_container             │ static    │ Verify sample_event.json │ PASS   │  26.760958ms │
│ kubernetes │ state_cronjob               │ static    │ Verify sample_event.json │ PASS   │  26.938208ms │
│ kubernetes │ state_daemonset             │ static    │ Verify sample_event.json │ PASS   │  26.615958ms │
│ kubernetes │ state_deployment            │ static    │ Verify sample_event.json │ PASS   │  25.856583ms │
│ kubernetes │ state_job                   │ static    │ Verify sample_event.json │ PASS   │  27.160208ms │
│ kubernetes │ state_node                  │ static    │ Verify sample_event.json │ PASS   │  26.102334ms │
│ kubernetes │ state_persistentvolume      │ static    │ Verify sample_event.json │ PASS   │  25.917791ms │
│ kubernetes │ state_persistentvolumeclaim │ static    │ Verify sample_event.json │ PASS   │  26.550083ms │
│ kubernetes │ state_pod                   │ static    │ Verify sample_event.json │ PASS   │  26.650875ms │
│ kubernetes │ state_replicaset            │ static    │ Verify sample_event.json │ PASS   │    27.2635ms │
│ kubernetes │ state_resourcequota         │ static    │ Verify sample_event.json │ PASS   │  26.446584ms │
│ kubernetes │ state_service               │ static    │ Verify sample_event.json │ PASS   │  26.637625ms │
│ kubernetes │ state_statefulset           │ static    │ Verify sample_event.json │ PASS   │  26.783625ms │
│ kubernetes │ state_storageclass          │ static    │ Verify sample_event.json │ PASS   │  24.991459ms │
│ kubernetes │ system                      │ static    │ Verify sample_event.json │ PASS   │   25.75625ms │
│ kubernetes │ volume                      │ static    │ Verify sample_event.json │ PASS   │  25.846417ms │
╰────────────┴─────────────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: kubernetes - END   ---
Done

[zmoog]

Only if I delete the sample_event.json file are failing :)

Which version of format_version are you using in the manifest.yml file?

[gizas]

@zmoog you were correct, I was testing in an old branch where
❯ grep format_version manifest.yml format_version: 1.0.0

Now I pulled latest one,

❯ grep format_version manifest.yml
format_version: 2.9.0

and when I change

"event": {
        "dataset": "kubernetes.node",

(it was "dataset": "kubernetes.state_node",) test correctly fails

PACKAGE="kubernetes" elastic-package test -vvv static
2023/08/17 17:06:52 DEBUG Enable verbose logging
2023/08/17 17:06:53  INFO New version is available - v0.85.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.85.0
Run static tests for the package
--- Test results for package: kubernetes - START ---
FAILURE DETAILS:
kubernetes/state_node Verify sample_event.json:
[0] field "event.dataset" should have value "kubernetes.state_node", it has "kubernetes.node"

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!