Add autodiscover for aws_ec2

[kaiyan-sheng]

This PR is to add aws_ec2 as a new provider for autodiscover. It allows autodiscover to react to EC2 instances status, information currently included in autodiscover events, to be used in autodiscover conditions are:

cloud.availability_zone
cloud.instance.id
cloud.machine.type
cloud.provider
cloud.region

and

aws.ec2.architecture
aws.ec2.image.id
aws.ec2.kernel.id
aws.ec2.monitoring.state
aws.ec2.private.dns_name
aws.ec2.private.ip
aws.ec2.public.dns_name
aws.ec2.public.ip
aws.ec2.root_device_name
aws.ec2.state.code
aws.ec2.state.name
aws.ec2.subnet.id
aws.ec2.tags
aws.ec2.vpc.id

Autodiscover config looks like:

logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: kaiyan
              password: kaiyan

TODO List

• Documentation
• Unit tests
• Test with mysql module in Metricbeat

Event generated by autodiscovered mysql:

{
  "_index": "metricbeat-8.0.0-2020.02.05-000001",
  "_type": "_doc",
  "_id": "y5crGHABZy5rQOjN9Rqq",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-02-06T01:44:48.040Z",
    "metricset": {
      "period": 10000,
      "name": "status"
    },
    "error": {
      "message": "dial tcp 3.86.223.141:3306: i/o timeout"
    },
    "aws": {
      "ec2": {
        "subnet": {
          "id": "subnet-76404958"
        },
        "public": {
          "dns_name": "ec2-3-86-223-141.compute-1.amazonaws.com",
          "ip": "3.86.223.141"
        },
        "state": {
          "name": "running",
          "code": 16
        },
        "vpc": {
          "id": "vpc-e8722d92"
        },
        "private": {
          "ip": "172.31.81.156",
          "dns_name": "ip-172-31-81-156.ec2.internal"
        },
        "tags": {
          "service": "mysql",
          "Name": "mysql-test",
          "created-by": "ks"
        },
        "root_device_name": "/dev/xvda",
        "monitoring": {
          "state": "disabled"
        },
        "kernel": {
          "id": ""
        },
        "architecture": "x86_64",
        "image": {
          "id": "ami-00068cd7555f543d5"
        }
      }
    },
    "cloud": {
      "region": "us-east-1",
      "instance": {
        "id": "i-05381e495aa0f95ef"
      },
      "machine": {
        "type": "t2.micro"
      },
      "availability_zone": "us-east-1b",
      "provider": "aws"
    },
    "host": {
      "architecture": "x86_64",
      "name": "KaiyanMacBookPro",
      "os": {
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0",
      "type": "metricbeat",
      "ephemeral_id": "5d697e16-910b-4363-a375-f36e8e362ffc"
    },
    "ecs": {
      "version": "1.4.0"
    },
    "event": {
      "dataset": "mysql.status",
      "module": "mysql",
      "duration": 10004769210
    },
    "service": {
      "address": "3.86.223.141:3306",
      "type": "mysql"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-02-06T01:44:48.040Z"
    ]
  },
  "sort": [
    1580953488040
  ]
}

How to test it

• Start an EC2 instance in AWS
• Add created-by: ks as tag for this EC2 instance
• Install and start mysql service in the EC2 instance
• Change default password of mysql
• Now you can try autodiscover with provider aws_ec2 by attach config below to metricbeat.yml under x-pack/metricbeat:

logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: kaiyan
              password: kaiyan

closes #12518

[exekias]

This is looking great! left a few comments but they are mostly about data format

[kaiyan-sheng]

Also tested with both mysql and mongodb by using metricbeat.yml:

logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 30s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: root
              password: root
            - module: mongodb
              metricsets:
                - dbstats
                - status
                - collstats
                - metrics
              period: 30s
              hosts: ["${data.aws.ec2.public.ip}:27017"]