Add kubernetes_leaderelection provider

[ChrsMark]

What does this PR do?

Adds support for leaderelection provider in Agent based on #24267.

After this one is merged I plan to tune standalone manifest in follow-up PR so as to remove deployment section and only deploy as Deamonset with leaderelection "on".

Why is it important?

To support cluster wide metrics collection by only one Agent at a time between a group of multiple Agents (ie Deamonset).

On a second step we can change the standalone manifest, completely remove the Deployment of Agent and go on with only the Daemonset leveraging the leaderelection feature.

How to test this PR locally

1. Enable the provider and an input based on the leader condition:

providers:
  kubernetes_leaderelection:
    kube_config: /Users/chrismark/.kube/config
    leader_lease: elastic-agent
  
inputs:
  - type: logfile
    condition: ${kubernetes_leaderelection.leader} == true
    streams:
      - paths: /var/log/leaderlog/another.log

2. run inspect command to verify the output: ./elastic-agent -e -d "*" -c /elastic-agent.yml inspect output -o default
3. Verify the conditional input is enabled:

filebeat:
  inputs:
  - index: logs-generic-default
    paths: /var/log/leaderlog/another.log
    processors:
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    - add_fields:
        fields:
          id: 32806c0a-4f04-499c-9427-e6e24e5f6035
          snapshot: false
          version: 8.0.0
        target: elastic_agent
    type: log

4. (Optional:) Perform same test from different Agents and on a running mode so as to verify that only one of them picks the lock and becomes the leader

Related issues

• Closes [Agent] Implement leader election on k8s for Elastic Agent #24267

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Started by user Chris Mark

• Start Time: 2021-04-07T14:12:19.936+0000

• Duration: 79 min 26 sec

• Commit: 8f84f23

Test stats 🧪

Test	Results
Failed	0
Passed	6612
Skipped	16
Total	6628

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6612
Skipped	16
Total	6628

[ChrsMark]

@blakerouse could you have a look into this one please when you have the time? I mostly want to get feedback from Agent's team point of view.

[blakerouse]

So by default running Elastic Agent this is off. Is that the behavior we want? Being that will require a specific setting in the providers top-level key, and not something that could then be used by Fleet at the moment.

I think that we should have this enabled by default, by having it have a default value.

[ChrsMark]

Agree, I will add it.

[blakerouse]

Should there also be a configuration value for this? Might want to run it on a different namespace?

[ChrsMark]

The convention here is that we will create the lease object under the same namespace with one where Agent is running. We also add role so as Agent to have access only to leases under the same namespace, see #24913 (comment)

So I think we should keep it as is and not expose this as an option to the user.

[blakerouse]

+1

[blakerouse]

I question if this should be a dynamic provider. I think this should be a context provider. That way leader election can effect all the other vars even dynamic ones discovered by the main kubernetes provider.

[ChrsMark]

👍🏼 ok I can change it

[ChrsMark]

@jsoriano @blakerouse thanks for your review/comments. I think have I covered your comments so you can consider it ready for another review round.

[blakerouse]

Looks great!

[blakerouse]

+1

[jsoriano]

👍