Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow a Beat to override monitoring-related fields #7850

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented Aug 2, 2018

By default all beats will use the beats_system user to talk to the Elasticsearch Monitoring bulk API. End-users can, of course, override this via the xpack.monitoring.elasticsearch.username configuration setting but sometimes a beat might want to specify a default other than beats_system. APM server is one such example wanting to use the apm_system user as it's default.

This PR makes it so a beat can override the Monitoring default username. This override should be done when the beat exports the cmd.RootCmd variable like so:

import (
    "github.com/elastic/beats/libbeat/cmd/instance"
    "github.com/elastic/beats/libbeat/monitoring/report"
)

var settings = instance.Settings{
	Name: "apm_server",
	Version: "",
	Monitoring: report.Settings{
          DefaultUsername: "apm_system",
        },
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)

Deprecation notice

This PR introduces a new function, cmd.GenRootCmdWithSettings. See the code snippet above for sample usage. This new function should be preferred over now-deprecated functions, cmd.GenRootCmd, cmd.GenRootCmdWithRunFlags, and cmd.GenRootCmdWithIndexPrefixWithRunFlags. See examples below on how to upgrade from the using the deprecated functions to using the new function.

Upgrading from using cmd.GenRootCmd

// Before
var RootCmd = cmd.GenRootCmd("countbeat", "", beater.New)

// After
import "github.com/elastic/beats/libbeat/cmd/instance"
var settings = instance.Settings{
    Name: "countbeat",
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)

Upgrading from using cmd.GenRootCmdWithRunFlags

// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithRunFlags("countbeat", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)

Upgrading from using cmd.GenRootCmdWithIndexPrefixWithRunFlags

// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithIndexPrefixWithRunFlags("countbeat", "cb", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    IndexPrefix: "cb",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)

@andrewkroh
Copy link
Member

That’s a good PR description.

@ruflin
Copy link
Member

ruflin commented Aug 6, 2018

Implementation looks good to me. It feels a bit odd to have these configs inside the Info struct and I would prefer to keep it out but I understand why it's there.

@urso you have some ideas on a better way on how this could be put somewhere else?

Copy link

@urso urso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach looks quite hacky. The beat.Info structure is created by libbeat. It's not supposed to be used to set/configure libbeat. Let me check if we can find a more clean solution.

E.g. the cmd/instance is very minimal, with positional parameters. We could replace positional parameters with a struct providing some additional configs/constants to libbeat.

@urso
Copy link

urso commented Aug 6, 2018

The cmd/instance is somewhat involved and requires some interaction with the beater code, working on partially initialized state only. With the introduction of features, we will try to cleanup the beat setup (rewrite it). Therefore I'd prefer to apply changes, which do not rely on any call-order between the beater and the beat setup code. Otherwise we're more likely to introduce bugs by accident or refactoring might become somewhat more complicated (depends if test code points out potential bugs).

The idea would be to pass init-information from top to bottom only:

Update cmd.GenRootCmd<X> function to accept a struct instead of positional parameters (we might add more settings like features and config schema in the future):

type Settings struct { 
  Name string
  Version string
  ...
}

Circular dependencies might be a problem. Let's put the struct in libbeat/beats (name: InstanceSettings?) or libbeat/cmd/instance package.

Update cmd.GenRootCmd<X> to accept the settings struct:

func GenRootCmd(creator beat.Creator, settings instance.Settings) *BeatsRootCmd {...}

Note: libbeat/beats package is supposed to be mostly stable/public API. Do not apply any change if possible. Only add structs if we are sure they are not going to change. That's why cmd/instance package takes no parameters from libbeat/beats package. The cmd/instance package requires cleanups or might even be replaced in the future.

We pass beats.Info to the monitoring reporters, so to allow us to create a local publisher pipeline and create the event meta-data. We can change the reporter factory(package 'reporter') to accept more rich settings:

type Factory func(settings Settings, cfg *common.Config) (Reporter, error)

type Settings struct {
  Beat beat.Info

  SystemID string
  ...
}

The report.Settings can be easily constructed in (cmd/instance).Run and passed to the (reporter).New and (log).MakeLogReporter.

@ycombinator
Copy link
Contributor Author

Thanks for the detailed feedback, @urso. Agreed that the original implementation is hacky. I will look into updating this PR with your suggestions now.

@elastic elastic deleted a comment from houndci-bot Aug 21, 2018
@elastic elastic deleted a comment from houndci-bot Aug 21, 2018
@ycombinator ycombinator force-pushed the libbeat/parameterize-monitoring-defaults branch from 9754381 to 449e2d2 Compare August 21, 2018 02:17
type Reporter interface {
Stop()
}

type ReporterFactory func(beat.Info, *common.Config) (Reporter, error)
type ReporterFactory func(Settings, *common.Config) (Reporter, error)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type ReporterFactory should have comment or be unexported

@@ -30,11 +30,16 @@ type config struct {
Reporter common.ConfigNamespace `config:",inline"`
}

type Settings struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Settings should have comment or be unexported

@ycombinator
Copy link
Contributor Author

@urso I've tried to reimplement this PR as per your guidance. Please check if I did the right thing.

Note that originally this PR was allowing beat authors to override the monitoring system_id in addition to the default username. Requirements have since changed and now we only need to allow the default username to be overridden.

Name string
IndexPrefix string
Version string
MonitoringDefaultUsername string
Copy link

@urso urso Aug 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

phew, that's a long/descriptive name. Is there a chance we will add more fields?

What if MonitoringDefaultUsername == ""?

How about:

type Settings struct {
  ...
  Monitoring MonitoringConfig
}

type MonitoringConfig struct {
  Username string  // default username
}

I see we already transform this parameter to report.Settings.

How about:

type Settings struct {
  ...
  Monitoring report.Settings
}

and in code:

monSettings := settings.Monitoring // this is a copy, as Monitoring is no pointer
if monSettings.Beat == "" { // fill in more defaults if not configured
  monSettings.Beat = b.Info 
}
reporter, err := report.New(monSettings, ...)

config := defaultConfig
overrideConfigFromBeat(&config, settings)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's replace these with:

config := defaultConfig(settings)

with:

func defaultConfig(settings report.Settings) config {
   ...
}

We started with creating default configs as struct and use them as kinda templates by copying the struct. Rethinking this approach I'm in favor of functions producing the default configs.

// GenRootCmdWithSettingsWithRunFlags returns the root command to use for your beat. It takes the
// run command, which will be called if no args are given (for backwards compatibility),
// beat settings, and runFlags. runFlags parameter must the flagset used by run command
func GenRootCmdWithSettingsWithRunFlags(beatCreator beat.Creator, settings instance.Settings, runFlags *pflag.FlagSet) *BeatsRootCmd {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By intruducing 'settings', I wonder if we can unify all these constructors by adding runFlags (or some other means to compute runFlags` from settings.

Copy link

@urso urso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for making this change backwards-compatible. Still, this change warrants an update in the developers changelog + explanation(upgrade info) in PR.

@ycombinator ycombinator force-pushed the libbeat/parameterize-monitoring-defaults branch from 265bb26 to 6c2be56 Compare August 21, 2018 17:51
@ycombinator
Copy link
Contributor Author

@urso I made most of the changes from your latest round of review. I didn't quite understand the second half of this comment so perhaps you could take a look at the latest changes and clarify your comment w.r.t. the changed code? Thanks!

// MonitoringConfig contains monitoring-specific settings
type MonitoringConfig struct {
DefaultUsername string
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ycombinator the idea is to reuse reporter.Settings here.

Like change instance.Settings type to:

type Settings struct {
	Name        string
	IndexPrefix string
	Version     string
	Monitoring  reporter.Settings
	RunFlags    *pflag.FlagSet
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, duh! Thanks for holding my hand through this one 😊.

// GenRootCmdWithIndexPrefixWithRunFlags returns the root command to use for your beat. It takes
// beat name, index prefix, version, run command, and runFlags. runFlags parameter must the flagset used by
// run command
// DEPRECATED! Use GenRootCmdWithSettings instead.
Copy link

@urso urso Aug 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no valid note marker. See: https://blog.golang.org/godoc-documenting-go-code

It's some badly documented convention, but godoc can parse and filter out/display notes. The syntax for notes is: // Marker(id): description.

In this case the comment should say:

// GenRootCmdWithIndexPrefixWithRunFlags returns the root command to use for your beat. It takes
// beat name, index prefix, version, run command, and runFlags. runFlags parameter must the flagset used by
// run command
//
// Deprecated: Use GenRootCmdWithSettings instead.

For conventions on Deprecation: use see: golang/go#10909 (comment)

Also see this stackoverflow answer with even more links for tooling around Deprecated:: https://stackoverflow.com/a/36360323

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graphaelli Any chance you could test this branch with apm-server?

@ycombinator Please check comments from Steffen.

@ycombinator
Copy link
Contributor Author

@ruflin said:

@ycombinator Please check comments from Steffen.

Yes, I've seen them. Just getting back from holiday and plan to address them today.

@ycombinator
Copy link
Contributor Author

ycombinator commented Aug 27, 2018

@urso I've addressed the latest round of feedback in c2b867a and bc6574125de6720319f171bae94f5ccf20ee1780. My one "concern" with reusing report.Settings is that that struct also contains a beat.Info field which we aren't really reusing. Seems a bit dirty, no?

@urso
Copy link

urso commented Aug 30, 2018

My one "concern" with reusing report.Settings is that that struct also contains a beat.Info field which we aren't really reusing. Seems a bit dirty, no?

Oh, didn't consider this. Yeah, you are right. 2 Options:

  1. revert the change and have monitor configs as before
  2. reduce reporter.Settings to common public non-redundant fields and pass the internal beat.Info as separate parameter to the reports constructor in addition to reporter.Settings.

Option 2 might gives us some flexibility if we decide we need more reporter.Settings (if you can think of any), but both solutions are fine with me.

@ycombinator ycombinator force-pushed the libbeat/parameterize-monitoring-defaults branch from 93c84e9 to a04b9eb Compare September 5, 2018 16:15
@ycombinator
Copy link
Contributor Author

The only CI failure I'm seeing is this flaky test: #8208.

So I'm ignoring it and merging this PR now.

@ycombinator ycombinator merged commit 74cbb79 into elastic:master Sep 5, 2018
@ycombinator ycombinator added the needs_backport PR is waiting to be backported to other branches. label Sep 5, 2018
@ycombinator ycombinator added v6.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 5, 2018
ycombinator added a commit that referenced this pull request Sep 13, 2018
* Adding TODOs

* Allow a beat to override monitoring-related fields

* Disallowing override of system_api_version (until we need it)

* Fixing datatype

* Create Settings structs for monitoring-specific info

* Changes from review

* Make runFlags part of settings struct

* Adding comments

* Adding developer changelog entry

* Adding default for index prefix

* Fixing deprecated note markers

* Reuse report.Settings

* Extract beat.Info back out of Settings struct

* Updating generator template

* Tabs, not spaces :eyeroll:

(cherry picked from commit 74cbb79)
ph pushed a commit to ph/beats that referenced this pull request Oct 29, 2018
* Adding TODOs

* Allow a beat to override monitoring-related fields

* Disallowing override of system_api_version (until we need it)

* Fixing datatype

* Create Settings structs for monitoring-specific info

* Changes from review

* Make runFlags part of settings struct

* Adding comments

* Adding developer changelog entry

* Adding default for index prefix

* Fixing deprecated note markers

* Reuse report.Settings

* Extract beat.Info back out of Settings struct

* Updating generator template

* Tabs, not spaces :eyeroll:

(cherry picked from commit 74cbb79)
@ph ph added the v6.4.3 label Oct 29, 2018
ph added a commit that referenced this pull request Oct 30, 2018
… fields (#8802)

Cherry-pick of PR #7850 to 6.4 branch. Original message: 

By default all beats will use the `beats_system` user to talk to the Elasticsearch Monitoring bulk API. End-users can, of course, override this via the `xpack.monitoring.elasticsearch.username` configuration setting but sometimes a beat might want to specify a default other than `beats_system`. APM server is one such example wanting to use the `apm_system` user as it's default.

This PR makes it so a beat can override the Monitoring default username. This override should be done when the beat exports the `cmd.RootCmd` variable like so:

```golang
import (
    "github.com/elastic/beats/libbeat/cmd/instance"
    "github.com/elastic/beats/libbeat/monitoring/report"
)

var settings = instance.Settings{
	Name: "apm_server",
	Version: "",
	Monitoring: report.Settings{
          DefaultUsername: "apm_system",
        },
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

---

### Deprecation notice
This PR introduces a new function, `cmd.GenRootCmdWithSettings`. See the code snippet above for sample usage. This new function should be preferred over now-deprecated functions, `cmd.GenRootCmd`, `cmd.GenRootCmdWithRunFlags`, and `cmd.GenRootCmdWithIndexPrefixWithRunFlags`. See examples below on how to upgrade from the using the deprecated functions to using the new function.

#### Upgrading from using `cmd.GenRootCmd`

```golang
// Before
var RootCmd = cmd.GenRootCmd("countbeat", "", beater.New)

// After
import "github.com/elastic/beats/libbeat/cmd/instance"
var settings = instance.Settings{
    Name: "countbeat",
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

#### Upgrading from using `cmd.GenRootCmdWithRunFlags`

```golang
// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithRunFlags("countbeat", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

#### Upgrading from using `cmd.GenRootCmdWithIndexPrefixWithRunFlags`

```golang
// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithIndexPrefixWithRunFlags("countbeat", "cb", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    IndexPrefix: "cb",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```
@ycombinator ycombinator deleted the libbeat/parameterize-monitoring-defaults branch December 25, 2019 11:12
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…related fields (elastic#8802)

Cherry-pick of PR elastic#7850 to 6.4 branch. Original message: 

By default all beats will use the `beats_system` user to talk to the Elasticsearch Monitoring bulk API. End-users can, of course, override this via the `xpack.monitoring.elasticsearch.username` configuration setting but sometimes a beat might want to specify a default other than `beats_system`. APM server is one such example wanting to use the `apm_system` user as it's default.

This PR makes it so a beat can override the Monitoring default username. This override should be done when the beat exports the `cmd.RootCmd` variable like so:

```golang
import (
    "github.com/elastic/beats/libbeat/cmd/instance"
    "github.com/elastic/beats/libbeat/monitoring/report"
)

var settings = instance.Settings{
	Name: "apm_server",
	Version: "",
	Monitoring: report.Settings{
          DefaultUsername: "apm_system",
        },
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

---

### Deprecation notice
This PR introduces a new function, `cmd.GenRootCmdWithSettings`. See the code snippet above for sample usage. This new function should be preferred over now-deprecated functions, `cmd.GenRootCmd`, `cmd.GenRootCmdWithRunFlags`, and `cmd.GenRootCmdWithIndexPrefixWithRunFlags`. See examples below on how to upgrade from the using the deprecated functions to using the new function.

#### Upgrading from using `cmd.GenRootCmd`

```golang
// Before
var RootCmd = cmd.GenRootCmd("countbeat", "", beater.New)

// After
import "github.com/elastic/beats/libbeat/cmd/instance"
var settings = instance.Settings{
    Name: "countbeat",
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

#### Upgrading from using `cmd.GenRootCmdWithRunFlags`

```golang
// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithRunFlags("countbeat", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```

#### Upgrading from using `cmd.GenRootCmdWithIndexPrefixWithRunFlags`

```golang
// Before
import "github.com/spf13/pflag"

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var RootCmd = cmd.GenRootCmdWithIndexPrefixWithRunFlags("countbeat", "cb", "", beater.New, runFlags)

// After
import (
    "github.com/spf13/pflag"
    "github.com/elastic/beats/libbeat/cmd/instance"
)

var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
runFlags.AddGoFlag(flag.CommandLine.Lookup("I"))
runFlags.AddGoFlag(flag.CommandLine.Lookup("dump"))

var settings = instance.Settings{
    Name: "countbeat",
    IndexPrefix: "cb",
    RunFlags: runFlags,
}
var RootCmd = cmd.GenRootCmdWithSettings(beater.New, settings)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants