Cherry-pick #20512 to 7.x: Add k8s manifest leveraging leaderelection

[ChrsMark]

Cherry-pick of PR #20512 to 7.x branch. Original message:

What does this PR do?

This PR proposes new Kubernetes manifests for shake of #19731, leveraging unique Autodiscover provider implemented at #20281.

With these manifests, only Metricbeat's Deamonset will be able to monitor whole k8s cluster since one Deamonset Pod each time will hold the leadership being responsible to coordinate metricsets that collect cluster wide metrics.

We will might need a meta issue to keep track of deprecating Deployment manifests (if needed).

• Add an extra section within https://www.elastic.co/guide/en/beats/metricbeat/7.x/running-on-kubernetes.html

Why is it important?

To get rid of the requirement to maintain/handle two different deployment strategies.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Test the manifests similarly to the testing steps of #20281 (comment).

1. Prepare a multinode cluster (ie on GKE)
2. Edit metricbeat-leaderelection-kubernetes.yml properly to set the proper image (ie docker.elastic.co/beats/metricbeat:7.10.0-SNAPSHOT) and the proper ES output (ie on Elastic Cloud)
3. Deploy the metricbeat-leaderelection-kubernetes.yml manifest and make sure that all the desired metricsets are shipping events and that k8s related Dashboards are populated with data correctly.
4. kubectl delete the leader pod and make sure that the leadership is either transfered to another Pod or it is gained again by the new replacement Pod.
5. Add static or hints-based autodiscovery configs/providers and make sure that work all together.
   Example:

metricbeat.autodiscover:
      providers:
        # To enable hints based autodiscover uncomment this:
        #- type: kubernetes
        #  node: ${NODE_NAME}
        #  hints.enabled: true
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                contains:
                  kubernetes.pod.name: "nats"
              config:
                - module: nats
                  hosts: "${data.host}:${data.port}"
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          identifier: gke-lease
          templates:
            - config:
                - module: kubernetes
                  hosts: ["kube-state-metrics:8080"]
                  period: 10s
                  add_metadata: true
                  metricsets:
                    - state_node
                    - state_deployment
                    - state_replicaset

Related issues

• Closes Implement kubernetes agent cluster scope leader election #19731

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #20600 opened]

• Start Time: 2020-08-14T09:39:08.237+0000

• Duration: 61 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2694
Skipped	725
Total	3419