Update kubernetes provider documentation and 8.6 release notes #2552

MichaelKatsoulis · 2023-01-26T14:24:51Z

Update kubernetes provider documentation to clearly state the correct usage of kubernetes labels in autodiscover conditions and 8.6 release notes with a breaking change.

This is due to changes done in elastic/elastic-agent#1398 effective in 8.6 release onwards.

Release notes have been updated, because even if this is considered a bug fix, there may be clients that have already set some templates in standalone agent or conditions in packages (the few that support it).
If in those conditions they make use of dedoted labels then when upgrading to agent 8.6, the data sets affected won't collect logs/metrics as the conditions won't match.
They need to manually update their configuration.

Updates of this PR can be found in #2552 (comment) Fleet and Elastic Agent Guide in three pages.

Release notes
Configure standalone Elastic Agents --> Providers --> Kubernetes Provider
Configure standalone Elastic Agents --> Kubernetes autodiscovery with Elastic Agent --> Conditions based autodiscover

…usage of kubernetes labels in autodiscover conditions

github-actions · 2023-01-26T14:25:04Z

A documentation preview will be available soon:

mergify · 2023-01-26T14:25:27Z

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

gizas · 2023-01-26T15:25:45Z

...gement/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

@@ -84,6 +86,13 @@ To set the target host dynamically for a targeted Pod based on its labels, use a
  condition: ${kubernetes.labels.component} == 'kube-scheduler'
 ----

+NOTE: Pods labels and annotations can be used in autodiscover conditions. In the case of labels or annotations that include dots(`.`), they can be used in conditions exactly as
+they are defined in the pods. For example `condition: ${kubernetes.labels.app.kubernetes.io/name} == 'ingress-nginx'`. This should not be mistaken with optional dedoted labels and annotations


Do you think can we enhance the examples here?
We like users to understand what is the actual label and what is the condition.

Also an example with annotation?

I think we should so a condition example before and after 8.6.0

To me, there is already an example with the usage of labels in conditions. It is mentioned above that The condition ${kubernetes.labels.app} == 'redis' will make the Elastic Agent look for a Pod with the label app:redis within the scope defined in its manifest.
The more complex labels that are not dedoted any more is not a new case. It is just another label to be used. I believe that this note explains it.
@ChrsMark what do you think ?

From my perspective this paragraph achieves the goal to explain how labels should be used in conditions specially regarding the dedoting concpet. Maybe we can add one extra example here for an annotation or provide a bunch of examples in a list like:

1. To target a Pod labeled with `app: apache` use the condition X 2. To target a Pod with the annotation .... use ....

It's up to you to decide how detailed you want to make it.

I can add some examples of conditions with complex labels, annotations and host.ip without having to create detailed examples that do not offer a lot.

@gizas , @ChrsMark see 23a4382

gizas · 2023-01-26T15:28:48Z

...gement/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

@@ -84,6 +86,13 @@ To set the target host dynamically for a targeted Pod based on its labels, use a
  condition: ${kubernetes.labels.component} == 'kube-scheduler'
 ----

+NOTE: Pods labels and annotations can be used in autodiscover conditions. In the case of labels or annotations that include dots(`.`), they can be used in conditions exactly as
+they are defined in the pods. For example `condition: ${kubernetes.labels.app.kubernetes.io/name} == 'ingress-nginx'`. This should not be mistaken with optional dedoted labels and annotations


This should not be mistaken with optional dedoted labels and annotations stored into Elasticsearch([kubernetes-provider]).

This confused me more here. Do you think shall we remove it?

Which line confused you?

See above updated.

I mean do users need this link there.

you mean this link Elasticsearch(<<kubernetes-provider>>). I have added that as it is the page that includes all the info regarding label and annotations dedoting

ChrsMark

Overall looks good. I've left some suggestions, feel free to include as many of them you find useful.

...gement/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc

…overy/kubernetes-conditions-autodiscover.asciidoc Co-authored-by: Chris Mark <[email protected]>

…s/kubernetes-provider.asciidoc Co-authored-by: Chris Mark <[email protected]>

tetianakravchenko · 2023-01-27T09:41:32Z

docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc

@@ -189,6 +189,9 @@ The available keys are:


 These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event.
+`kubernetes.labels.*` and `kubernetes.annotations.*` used in config templating are not dedoted and should not be confused with


this sentence is in the same paragraph with the previous one - https://observability-docs_2552.docs-preview.app.elstc.co/guide/en/fleet/master/kubernetes-provider.html

can we split it? and maybe highlight with NOTE: similar to https://github.com/elastic/observability-docs/pull/2552/files#diff-62a97949a501942a1b602f35ca893cebef309cffa0e97dd233dc6c7492a5d8f3R93?

Yes you are right. I will highlight with NOTE

tetianakravchenko · 2023-01-27T09:55:12Z

...gement/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

@@ -67,7 +71,9 @@ You should now be able to see Redis data flowing in on index `metrics-redis.info

 NOTE: All assets (dashboards, ingest pipelines, and so on) related to the Redis integration are not installed. You need to explicitly <<install-uninstall-integration-assets,install them through {kib}>>.

-To set the target host dynamically for a targeted Pod based on its labels, use a variable in the {agent} policy to return path information from the provider:
+Conditions can also be used in the `Kubernetes` package in order to set the target host dynamically for a targeted Pod based on its labels.


by Kubernetes package are you referring to the https://docs.elastic.co/en/integrations/kubernetes ?

for standalone we usually use in documentation input instead of package, for example - here or here

can we keep it consistent?

The input can have whatever name the user wants so kubernetes input would not be accurate wording. Also in each input we have the package information inside the meta. And although not used anywhere, the name of the package is the same used in integrations.
Also I initially had written input. But I changed it after @ChrsMark comment (#2552 (comment)).
Chris are your thoughts the same as mine in this?

thank you for clarifications, I don't have a strong opinion on it

No strong opinion either :)

Ok as long you don't have a strong opinion I decided to use input instead of package to keep consistency because I am not sure if the user at this point is 100% familiar with packages.

gizas · 2023-01-27T12:43:34Z

...gement/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

@@ -37,6 +37,10 @@ To automatically identify a Redis Pod and monitor it with the Redis integration,
 The condition `${kubernetes.labels.app} == 'redis'` will make the {agent} look for a Pod with the  label `app:redis` within the scope defined in its manifest.

 For a list of provider fields that you can use in conditions, refer to <<kubernetes-provider>>.
+Some examples of conditions usage are:
+1. For a pod with label `app.kubernetes.io/name=ingress-nginx` the condition should be `condition: ${kubernetes.labels.app.kubernetes.io/name} == "ingress-nginx"`


Also the numeric list here seems broken in the view https://github.com/elastic/observability-docs/blob/e7771ca322ef8a0a7b1c8cf1cc2493aa62c951d4/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

Can you have a look to update it

…ic#2552) * Update kubernetes provder documentation to clearly state the correct usage of kubernetes labels in autodiscover conditions

Update kubernetes provder documentation to clearly state the correct …

05f3fb8

…usage of kubernetes labels in autodiscover conditions

MichaelKatsoulis requested a review from dedemorton as a code owner January 26, 2023 14:24

MichaelKatsoulis marked this pull request as draft January 26, 2023 14:24

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 26, 2023

Reorder warnings and notes

de2602f

MichaelKatsoulis requested review from gizas, tetianakravchenko, gsantoro and ChrsMark January 26, 2023 14:39

Fix typo

0c11433

MichaelKatsoulis added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Jan 26, 2023

MichaelKatsoulis marked this pull request as ready for review January 26, 2023 14:57

gizas reviewed Jan 26, 2023

View reviewed changes

ChrsMark reviewed Jan 27, 2023

View reviewed changes

MichaelKatsoulis and others added 7 commits January 27, 2023 10:12

Update docs/en/ingest-management/elastic-agent/configuration/autodisc…

37053b7

…overy/kubernetes-conditions-autodiscover.asciidoc Co-authored-by: Chris Mark <[email protected]>

Update docs/en/ingest-management/elastic-agent/configuration/autodisc…

f599a50

…overy/kubernetes-conditions-autodiscover.asciidoc Co-authored-by: Chris Mark <[email protected]>

Update docs/en/ingest-management/elastic-agent/configuration/autodisc…

49604a3

…overy/kubernetes-conditions-autodiscover.asciidoc Co-authored-by: Chris Mark <[email protected]>

Update docs/en/ingest-management/elastic-agent/configuration/autodisc…

37c7132

…overy/kubernetes-conditions-autodiscover.asciidoc Co-authored-by: Chris Mark <[email protected]>

Update docs/en/ingest-management/elastic-agent/configuration/provider…

56e395a

…s/kubernetes-provider.asciidoc Co-authored-by: Chris Mark <[email protected]>

Update docs/en/ingest-management/elastic-agent/configuration/provider…

d11720c

…s/kubernetes-provider.asciidoc Co-authored-by: Chris Mark <[email protected]>

Add more conditions examples

23a4382

tetianakravchenko reviewed Jan 27, 2023

View reviewed changes

Highlight labels and annotations comment

e7771ca

gizas reviewed Jan 27, 2023

View reviewed changes

MichaelKatsoulis added 3 commits January 27, 2023 15:22

Fix list

329d1e9

Update kubernetes-conditions-autodiscover.asciidoc

e40babb

Update kubernetes-conditions-autodiscover.asciidoc

48ccaf1

gizas approved these changes Jan 27, 2023

View reviewed changes

tetianakravchenko approved these changes Jan 27, 2023

View reviewed changes

Update kubernetes-conditions-autodiscover.asciidoc

f73c366

MichaelKatsoulis merged commit 829c303 into elastic:main Jan 31, 2023

MichaelKatsoulis self-assigned this Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update kubernetes provider documentation and 8.6 release notes #2552

Update kubernetes provider documentation and 8.6 release notes #2552

MichaelKatsoulis commented Jan 26, 2023 •

edited

Loading

github-actions bot commented Jan 26, 2023

mergify bot commented Jan 26, 2023

gizas Jan 26, 2023

MichaelKatsoulis Jan 27, 2023

ChrsMark Jan 27, 2023 •

edited

Loading

MichaelKatsoulis Jan 27, 2023

MichaelKatsoulis Jan 27, 2023

gizas Jan 27, 2023

gizas Jan 26, 2023 •

edited

Loading

MichaelKatsoulis Jan 27, 2023

gizas Jan 27, 2023

MichaelKatsoulis Jan 30, 2023

ChrsMark left a comment •

edited

Loading

tetianakravchenko Jan 27, 2023

MichaelKatsoulis Jan 27, 2023

MichaelKatsoulis Jan 27, 2023

tetianakravchenko Jan 27, 2023

MichaelKatsoulis Jan 27, 2023

tetianakravchenko Jan 27, 2023

ChrsMark Jan 30, 2023

MichaelKatsoulis Jan 30, 2023

gizas Jan 27, 2023

MichaelKatsoulis Jan 27, 2023

		@@ -189,6 +189,9 @@ The available keys are:


		These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event.
		`kubernetes.labels.` and `kubernetes.annotations.` used in config templating are not dedoted and should not be confused with

Update kubernetes provider documentation and 8.6 release notes #2552

Update kubernetes provider documentation and 8.6 release notes #2552

Conversation

MichaelKatsoulis commented Jan 26, 2023 • edited Loading

github-actions bot commented Jan 26, 2023

mergify bot commented Jan 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark Jan 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gizas Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelKatsoulis commented Jan 26, 2023 •

edited

Loading

ChrsMark Jan 27, 2023 •

edited

Loading

gizas Jan 26, 2023 •

edited

Loading

ChrsMark left a comment •

edited

Loading