Generate configuration from mage for all Beats

[kvch]

From now on all Beats use mage when generating configuration files. This comes with the following changes:

• Heartbeat does not use post_process_config.py to modify processors, instead it is added using the templating
• Functionbeat does not include unsupported outputs
• All Beats can alter the common configuration coming from libbeat.

What does it mean for Community Beats?

When adopting this version of libbeat, you need to make sure that you have a target named Config in your magefile.go. An example target which generates a sample configuration, a reference configuration and a Docker configuration from {yourbeat}/_meta/beat.yml and the common configuration from libbeat is the following:

import devtools "github.com/elastic/beats/dev-tools/mage"

func Config() error {
    return devtools.Config(devtools.AllConfigTypes, devtools.ConfigFileParams{}, ".")
}

If the configuration of your Beat consists of multiple files, you can pass your own instance of ConfigFileParams:

func ConfigFileParams() devtools.ConfigFileParams {
    return devtools.ConfigFileParams{
        ShortParts: []string{
            devtools.OSSBeatDir("_meta/beat.yml"),
            devtools.OSSBeatDir("_meta/beat.custom.yml"),
            devtools.LibbeatDir("_meta/config.yml.tmpl"),
        },
        ReferenceParts: []string{
            devtools.OSSBeatDir("_meta/beat.reference.yml"),
            devtools.OSSBeatDir("_meta/beat.custom.referenceyml"),
            devtools.LibbeatDir("_meta/config.reference.yml.tmpl"),
        },
        DockerParts: []string{
            devtools.OSSBeatDir("_meta/beat.docker.yml"),
            devtools.LibbeatDir("_meta/config.docker.yml"),
        },
    }
}

It is also possible to pass variables to the templates using the ExtraVars attribute of ConfigFileParams struct.

[ycombinator]

LGTM — tested that nothing broken from a Stack Monitoring metricsets/filesets config generation POV. Nice change!

[jsoriano]

LGTM, only a last small detail

[andrewkroh]

Lovely 👍, no more cat + sed + bash + make magic.

[kvch]

Failing tests are unrelated.