Adding config variables to leader_election provider

[gizas]

• Enhancement

What does this PR do?

WHAT: Adding config options for the kubernetes provider leader_election
WHY: We need to make the values configurable in order to adjust the number of API calls we need for the Leader Election process

Why is it important?

In big Kubernetes cluster, the number of API calls is proportional to number of agents that take part in the leader election. We have seen that this number of API calls towards kubernetes control API can be significant. With this effort we can offer to our users the ability to increase the period of leader election requests

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

1. Checkout this PR
2. Build you custom agent image by following https://github.com/elastic/elastic-agent#testing-elastic-agent-on-kubernetes
3. Sample standalone agent.yml config to provide:

providers.kubernetes_leaderelection:
      enabled: true
      leader_retryperiod: 15
      leader_leaseduration: 20
      leader_renewdeadline: 19

Related issues

• Relates https://github.com/elastic/sdh-beats/issues/3863
• Relates Adding docs for leader election timings ingest-docs#635

Screenshot

With retry period every 15 sec

The kube-state-metrics processing every 10 sec works as expected:

Logs

kubectl logs -n kube-system elastic-agent-gj7zt | grep leader

I1101 14:29:15.495697      11 leaderelection.go:258] successfully acquired lease kube-system/agent-k8s-leader-lock2
{"log.level":"debug","@timestamp":"2023-11-01T14:29:15.495Z","log.logger":"composable","log.origin":{"file.name":"kubernetesleaderelection/kubernetes_leaderelection.go","file.line":91},"message":"leader election lock GAINED, id elastic-agent-leader-elastic-agent-gj7zt","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-01T14:29:15.496Z","log.logger":"composable","log.origin":{"file.name":"kubernetesleaderelection/kubernetes_leaderelection.go","file.line":92},"message":"leader configuration timings: LeaseDuration: 20s , RenewDeadline: 19s, RetryPeriod: 15s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

[mergify]

This pull request does not have a backport label. Could you fix it @gizas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-03T11:24:22.344+0000

• Duration: 17 min 21 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	98.81% (83/84)	👍
Files	67.105% (204/304)	👍
Classes	65.719% (370/563)	👍
Methods	53.043% (1168/2202)	👍
Lines	39.333% (13632/34658)	👎 -0.029
Conditionals	100.0% (0/0)	💚

[gizas]

Sample manifest used:

data:
  agent.yml: |-
   
    providers.kubernetes_leaderelection:
      enabled: true
      leader_lease: agent-k8s-leader-lock3
      leader_retryperiod: 15
      leader_leaseduration: 20
      leader_renewdeadline: 19

Verification:

kubectl describe leases -n kube-system agent-k8s-leader-lock3
Name:         agent-k8s-leader-lock3
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2023-10-30T14:42:38Z
  Managed Fields:
    API Version:  coordination.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:acquireTime:
        f:holderIdentity:
        f:leaseDurationSeconds:
        f:leaseTransitions:
        f:renewTime:
    Manager:         elastic-agent
    Operation:       Update
    Time:            2023-10-30T14:42:38Z
  Resource Version:  70559
  UID:               a024c0c4-8d90-4776-81a2-a37b1c768b7a
Spec:
  Acquire Time:            2023-10-30T14:42:38.346903Z
  Holder Identity:         elastic-agent-leader-elastic-agent-7nh9q
  Lease Duration Seconds:  20
  Lease Transitions:       0
  Renew Time:              2023-10-30T14:42:38.372912Z
Events:                    <none>

[gizas]

@elastic/elastic-agent-control-plane team can I have a review here please?

[tetianakravchenko]

@gizas I think you need to add changelog record for this changes - https://github.com/elastic/elastic-agent#changelog?

[gizas]

@gizas I think you need to add changelog record for this changes - https://github.com/elastic/elastic-agent#changelog?

Thanks added one Tania (had the impression that we add it only for customer phasing issues, bugs or features. Does not harm to have it :))

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication