Add new --setup-and-exit flag for Logstash module setup

[ycombinator]

Currently, in order to setup Logstash modules (e.g. netflow), there is a first-time setup command that must be run, like so:

bin/logstash --modules MODULE_NAME --setup [-M "CONFIG_SETTING=VALUE"]

This command creates the necessary artifacts in Elasticsearch and Kibana such as Elasticsearch templates, Kibana visualizations, and Kibana dashboards. It also starts up a Logstash pipeline for the given module, i.e. the Logstash process keeps running.

While keeping the --setup flag as-is, we also want to introduce a --setup-and-exit flag that will create the necessary artifacts in Elasticsearch and Kibana and then exit. This puts us on the path towards decoupling module setup from running the module's pipeline, as detailed in #8709 (comment).

Concretely, this would mean that users would run two commands instead of the one shown above:

bin/logstash --modules MODULE_NAME --setup-and-exit

bin/logstash --modules MODULE_NAME [-M "CONFIG_SETTING=VALUE"]

If the user accidentally specifies both --setup and --setup-and-exit, Logstash should fail and let the user know that these flags are mutually exclusive.

[ycombinator]

@jordansissel Based on our out-of-issue discussion with @tbragin and @alexfrancoeur, where do we stand on the changes proposed in this issue?

My take away from our meeting was that:

• We do want Kibana module tutorials to have instructions for Debian- and RPM-based installs, but this is not an immediate priority.

• Given that, at some point, we do want modules to work in Debian- and RPM-based installs, we should go ahead with the proposal in this issue and introduce the --setup-and-exit flag.

Does this match your understanding as well?

[jordansissel]

++

I think it will be weird for us to deliver incomplete instructions (missing rpm/deb), but my sense from the discussion is that it is not a deal breaker. We should maybe add rpm/deb tabs and say "instructions are coming soon" to avoid folks asking the obvious question "Where is rpm instructions?"

[jordansissel]

I think you can move forward with just tar/zip instructions, and I would prefer a placeholder for rpm/deb where we openly state "These are missing, we know, and we will provide this instruction soon" or similar.

[ycombinator]

Thanks @jordansissel. I will add RPM/Deb tabs and say that instructions are coming soon.

[tbragin]

Regarding the plan to introduce the --setup-and-exit flag in Logstash and eventually change the meaning of the --setup flag to mean "setup and exit" (according to #8709 (comment))... is that still the way we would like to proceed? Or is that still under discussion?

Would it make sense to connect with the Beats team about semantics around these flags? @jordansissel @ycombinator happy to set that up if it would help.

I know the Beats team had to wrestle with the decision of what to do with the --setup flag (which means do the setup but continue running the Beat and send data) once they introduced the setup command (which means run the setup and exit). As far as I know they decided to keep the --setup flag, at least for now, so in Beats 6.0 the --setup flag is not deprecated and is going to continue to mean what it does today. I think it would be confusing to have different meaning for the same flags between two products.

cc: @brandonmensing @tsg @monicasarbu @acchen97

[jordansissel]

@tbragin my thought is that long term, users shoudl not necessarily be doing anything other than "run filebeat" and "run logstash" as the deployment step. Everything else can be clicks in Kibana.

Proposed future workflow:

Scenario: User wants to use netflow module and uses the netflow module tutorial in Kibana to do it.

Assumption 1: Logstash central configuration is enabled.
Assumption 2: Logstash central config supports role-based configuration (certain Logstash nodes can be labeled similar to Elasticsearch node attributes). This feature is not implemented today.

1. Step 1: Choose nodes (by attributes, perhaps) to deploy netflow module on
   • If there are no known nodes, or no attributes are set, provide instructions for configuring and deploying Logstash with such attributes
2. Step 2: Click button to activate Netflow module
   • This will automatically install Kibana dashboards, Elasticsearch templates, and Logstash pipelines via central config.

The overall goal of my proposal is aimed at minimizing the number of human steps required to accomplish "Deploy the netflow module". It should just be a single-step for most users.

This means:

1. The notion of setup from filebeat/logstash goes away as Kibana becomes responsible for this step
2. The notion of 'enable' in filebeat goes away as central configuration management takes ownership of what is enabled.

There can be a hybrid approach, but this is the direction I am thinking long-term. This approach works for both existing and new users. New users will not have filebeat/logstash deployed, and we can offer download-and-basic-setup instructions for users without these things deployed.

[jordansissel]

@tbragin, @andrewvc, @jsvd, @monicasarbu, and @tsg met today to discuss consistency/roadmap of module setup experience.

Summary of decisions (please advise/edit if incorrect):

1. Short term, implement setup-and-exit in Logstash as a flag.
2. Medium term, consider adding a subcommand "setup" like beats has for module deployment: bin/logstash setup
3. Medium term, make "setup" an implicit action enabled by default. This will require changes in Kibana to make it possible to skip setup if setup has already been done previously.
4. Long term, move module setup into Kibana as a button.

Impact on this issue: Move forward with --setup-and-exit as planned. Target is Logstash 6.2

[Securitybits-io]

Whats the status on this feature. As of right now there is to my knowledge no suitible way of deploying modules in logstash with code (IaaC). And this is a really cruicial part in order to be able to seamlessly deploy the modules as well! The target was 6.2, current version is 6.7?