Disable Deployment and Replicaset enrichment by default

[gizas]

Disabling AddResourceMetadataConfig.Deployment: false and AddResourceMetadataConfig.Cronjob: false as per discussion here #31 (comment)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 3b2630a

History

• 💔 Build #39 failed 295cd10

cc @gizas

[ChrsMark]

The change looks sane to me. This would need a changelog entry in the proper section as a breaking change.

Since this is a breaking change (with valid reasoning) please wait for validation from @tommyers-elastic and @mlunadia before merging.

[tommyers-elastic]

+1 from me

[bartoszcisek]

Can you please make it at least configurable? We use kubernetes.deployment do distinguish between pod types while configuring elastic-agent. Hint based discovery is still limited so we rely on condition based.

[gizas]

@bartoszcisek you can still enable it in your manifest

You need to enable the kubernetes provider

data:
  agent.yml: |-
    ...
    providers.kubernetes:
      add_resource_metadata:
        deployment: true
        cronjob: true

I am testing with 8.11.0-SNAPSHOT
I exec in my elastic agent pod:

elastic-agent inspect -v --variables --variables-wait 2s
....
- data_stream:
    namespace: default
  id: filestream-container-logs-d105fc5a-36b3-4d5a-b97a-288d141781de-kubernetes-7107f2fe-3ae6-49fb-9aa6-5de2a5e23259.local-path-provisioner
  meta:
    package:
      name: kubernetes
      version: 1.46.0
  name: kubernetes-1
  original_id: filestream-container-logs-d105fc5a-36b3-4d5a-b97a-288d141781de
  - add_fields:
      fields:
        container:
          name: local-path-provisioner
        deployment:   < -------This is present
          name: local-path-provisioner

And if you need also your metrics to be enriched, you can change the state_pod and state_container datasets with the add_resource_metadata block

streams:
          - id: >-
              kubernetes/metrics-kubernetes.state_container-d105fc5a-36b3-4d5a-b97a-288d141781de
            data_stream:
              type: metrics
              dataset: kubernetes.state_container
           ...
            add_resource_metadata:
              deployment: true
              cronjob: true

         - id: >-
              kubernetes/metrics-kubernetes.state_pod-d105fc5a-36b3-4d5a-b97a-288d141781de
            data_stream:
              type: metrics
              dataset: kubernetes.state_pod
           ...
            add_metadata: true
            add_resource_metadata:
              deployment: true
              cronjob: true

There is also the option to use ingest pipelines as described in this new doc https://www.elastic.co/guide/en/fleet/master/ingest-pipeline-kubernetes.html

[mlunadia]

Since this is a breaking change (with valid reasoning) please wait for validation from @tommyers-elastic and @mlunadia before merging.

@ChrsMark what does this mean for users, what do they have to do when they upgrade?
Are there other mechanisms we currently/could communicate breaking changes to them besides release notes? eg. manifest comments, notifications on upgrade, etc...

[ChrsMark]

Since this is a breaking change (with valid reasoning) please wait for validation from @tommyers-elastic and @mlunadia before merging.

@ChrsMark what does this mean for users, what do they have to do when they upgrade?

Hey @mlunadia. So on upgrade I expect that users:

1. will stop seeing by default the kubernetes.deployment.name field being populated as part of Pod's metadata whenever the Pod is owned by a ReplicaSet and a Deployment.
2. will stop seeing by default the kubernetes.cronjob.name field being populated as part of Pod's metadata whenever the Pod is owned by a Job and a Cronjob.

Are there other mechanisms we currently/could communicate breaking changes to them besides release notes? eg. manifest comments, notifications on upgrade, etc...

Maybe we can add a commented section in the manifests to note these settings so as users interested into these to be able to easily renable them. Also in our docs we can mention it somehow as well. @gizas what do you think?