Add Data UI - Metricbeat

[Bargs]

Related to #5974

The goal here is to create an additional Add Data wizard that makes it easier to install Topbeat along with its index template and Kibana dashboards.

Questions

• How should we keep the index template and example dashboards in sync between the Kibana and Topbeat projects?
• Should we include an ingest pipeline creation step so that users have the option of manipulating the data coming in from topbeat? If so, should we automatically create an example input object with topbeat fields and if so, how do we keep those in sync with changes in topbeat?
• Should we give the user the option of configuring the index and index pattern topbeat is sending data to? I think the answer is yes, but we'll need to add an additional step to gather this input before creating everything in the background.

Mockups and Workflow

Step 0 - New option on the Add Data landing page

The Topbeat wizard will appear as an additional option on the Add Data landing page, describing it as a way to gather CPU/Memory metrics. When the user selects this option, Kibana will take a number of actions in the background:

• Load index template in Elasticsearch
• Create kibana index pattern for topbeat
• Upload kibana dashboards

Step 1 - Install Topbeat

This is the first and only step of the wizard. The page includes links to docs and/or embedded instructions to help the user install Filebeat on their servers. This page will also poll the topbeat index pattern and report to the user when documents are successfully being index.

[Bargs]

In regards to the question about keeping topbeat and Kibana in sync, I have a few thoughts. I'd love to get feedback from the Beats team as well. This discussion will set a precedent for other beats in the future. First of all, here's a list of all of the resources I know of that will need to be synchronized:

• Topbeat index template
• Topbeat Kibana documents (dashboard, viz, searches, index pattern)
• Topbeat default source fields (if we add a pipeline creation step)
• ???

Potential solutions:

• Keep copies of each resource in both Kibana and Beats and keep them in sync manually.

Pro: Simple
Con: Relies of word of mouth, it's not a matter of if, but when we'll forget to make an update in both places.

• Store shared resources in a separate repo that can be consumed easily by both projects.

Pro: Really easy to consume from Kibana's side if we publish it as an NPM module. It's mostly DRY.
Con: I don't know how easy it is to consume from Go. The default topbeat fields would still essentially be a copy that needs to be maintained.

• Create the topbeat wizard as a Kibana plugin, include the correct topbeat distributable inside the plugin pack.

Pro: It's DRY. We could probably figure out a way for Kibana to get the default fields without copying them.
Con: Now we have a separate plugin to maintain and keep in lock step with Kibana's versioning.

• Topbeat could expose an API with this information.

Pro: It's DRY. It's sort of the Elastic way.
Con: Kibana needs to have access to a topbeat instance, which complicates setup for the user.

Any other ideas?

[tbragin]

Should we include an ingest pipeline creation step so that users have the option of manipulating the data coming in from topbeat? If so, should we automatically create an example input object with topbeat fields and if so, how do we keep those in sync with changes in topbeat?

I'd say -- no. The idea with Topbeat is that the data is sufficiently structured and we know that structure ahead of time to create a reasonable indexing template. I'm not aware of users having to modify that data once it's in ES.

Should we give the user the option of configuring the index and index pattern topbeat is sending data to? I think the answer is yes, but we'll need to add an additional step to gather this input before creating everything in the background.

By default, topbeat sends to the topbeat-* index pattern. It can be adjusted in the config file, so it's probably helpful to provide an option to create an index pattern for a different indes.

[tsg]

The overall flow LGTM. Trying to answer some of the questions / add more info:

How should we keep the index template and example dashboards in sync between the Kibana and Topbeat projects?

At the moment we keep the dashboards in a separate repository (https://github.com/elastic/beats-dashboards) with a script to load them. We're not terribly happy with that and because with Metricbeat we expect a lot more dashboards we're thinking of storing them closer to the source code. I'm not sure about the solution but it's the right time for the question :-).

These dashboards don't necessarily need to be loaded from Go (they currently aren't) but whatever loads them shouldn't require new dependencies.

The dashboards can potentially change with every topbeat release and they should be included / tested with the Kibana having the same version (yay synchronized version numbers).

I think my favorite is your number 3, if I understand it correctly, in which you bundle a topbeat tar.gz containing the required objects in a Kibana plugin. What I don't like about it is introducing a release time dependency (topbeat must be built when Kibana is build). I don't know enough about Kibana plugins to know if this is doable with reasonable effort.

Pinging @kimchy as I think he also had some thoughts around this.

Should we include an ingest pipeline creation step so that users have the option of manipulating the data coming in from topbeat? If so, should we automatically create an example input object with topbeat fields and if so, how do we keep those in sync with changes in topbeat?

In the interest of keeping the wizard simple I'd also say no.

Should we give the user the option of configuring the index and index pattern topbeat is sending data to? I think the answer is yes, but we'll need to add an additional step to gather this input before creating everything in the background.

It could be an option pre-filled with topbeat-*. But it might complicate creating the dashboards creation?

[Bargs]

Thanks for the feedback so far.

Not including a pipeline creation step sounds good to me, seems like it wouldn't make much sense and it would only create additional coupling between Kibana and Topbeat by requiring Kibana to know about Topbeat's default document fields.

It could be an option pre-filled with topbeat-*. But it might complicate creating the dashboards creation?

@tsg that's a really good point. What do you do today if a user configures a different index name? Are they not able to use the default dashboards at all? How common is it for users to change the index name? I'm not particularly familiar with how Topbeat is used in the wild.

I think my favorite is your number 3, if I understand it correctly, in which you bundle a topbeat tar.gz containing the required objects in a Kibana plugin. What I don't like about it is introducing a release time dependency (topbeat must be built when Kibana is build). I don't know enough about Kibana plugins to know if this is doable with reasonable effort.

Yeah my idea was to literally bundle the matching Topbeat distribution (probably expanded, since the plugin itself will be compressed) in a Kibana plugin so that we can pull the template/dashboards from the same place a user would. But as you said, this makes releases much more complicated. It would also mean we need to create a Kibana plugin for any future beats we want a setup UI for, so maybe it's not such a good idea.

I had no idea the dashboards were already in a separate repo, I thought they were stored here. Could you elaborate a bit more on the troubles you're having with using a separate repo?

[tsg]

@tsg that's a really good point. What do you do today if a user configures a different index name? Are they not able to use the default dashboards at all? How common is it for users to change the index name? I'm not particularly familiar with how Topbeat is used in the wild.

At the moment they wouldn't be able to use the sample dashboards at all, but there is an open PR to make renaming the indices possible: elastic/beats-dashboards#85

I had no idea the dashboards were already in a separate repo, I thought they were stored here.

We're in the process of moving them there, but currently all releases have been using the beats-dashboards repo.

Could you elaborate a bit more on the troubles you're having with using a separate repo?

There are a few things:

• Because they are in a separate repository, it's more likely that they get out of sync with the exported fields.
• With Metricbeat we want to have one dashboard for most of the modules, meaning that there will be a lot more dashboards. We think that the community contributors will be more likely to contribute dashboards if they are in the same place with the code.
• The current beats dashboards script loads the templates for all Beats, so it's either you take all dashboards or none. Since not everyone is running all the Beats, they would prefer to be able to load only selected dashboards.
• The dashboards have a different release process, so that gives us another thing to worry when making releases.

[Bargs]

This is blocked until we can come up with a good solution for managing the Topbeat template and dashboards/visualizations/searches. We talked about how the Kibana REST refactor could expose endpoints that would allow Beats to manage their Kibana dashboards directly without having to mess with the .kibana index directly.

[tsg]

Small update also from our side: it looks like we'll have the Kibana dashboards together with a loading script inside the Topbeat packages by alpha2.

[Bargs]

@tsg I noticed you guys enabled autoloading of index templates as well. Will the dashboards get auto loaded, or is there still separate script to run for those?

Also, semi-related question, what does the user do about the index template if they've configured a different index other than the default filebeat-*? Do they have to manually edit topbeat.template.json file?

[ruflin]

@Bargs Currently there is still a separate script. I would like to also automate this part.

For the second question: Good point. At the moment manual editing is needed. @tsg Should we perhaps make it configurable and parse the file? Also a good job for beat-helper ;-)

[monicasarbu]

@Bargs We have a bash script for Unix systems and a powershell script for Windows systems to import the dashboards in Kibana. We are planning to include this script together with the Beat dashboards in the Beat packages, so the user can import the sample dashboards only if he/she wants.

[epixa]

While this would be a nice feature to have, we've put our add data efforts on hold for the time being to focus on more impactful enhancements for Kibana.

[RedCloudDC]

add data as you had originally envisioned would be the most critical enhancement. I would not assume that all users are ninjas with grok and using conf files for ingest. Copy/paste of sample data, perhaps a headers preview, upload a csv, map field types and pipeline wizard would be a huge feature. The ingestion and transform components are not very intuitive. I hope this doesn't remain on hold for much longer. It looked fantastic.

[epixa]

@RedCloudDC The value is immense, for sure! One of our biggest problems with getting the add data feature out the door was its sheer scale. There are really tons of features that tie together to form what we were previously lumping into a single "add data" effort.

We have not abandoned this effort as a whole, we've just focussed our energy on individual features until enough pieces are in place to form the broader vision. We've been working on a Pipelines UI, for example, which deals with iterating on sample data to create and edit ingest pipelines through Kibana. Another feature that is high on our list is a UI for managing indices.

I also strongly encourage everyone to look into the awesome ingestion workflows coming from the beats team around metricbeat and filebeat. With those, you can kickstart an ingestion pipeline and have it wired up into Kibana automatically with default visualizations/dashboards. Those ingestion features will harmonize with the features we're building into Kibana rather than be completely replaced/reimplemented at the UI level.

[chrisronline]

Closing this out. This idea is still valuable, but we've started moving in a different direction, specifically with a new home page