[Discuss] Dashboard Creation Guidelines

[sorantis]

The goal of this issue is to compile a set of best practices that can be applied for creating a dashboard for any target system.

Here's my proposal, feel free to add your thoughts.

Level of detail

Many integrations cover more than one component of a target system. For instance, the RabitMQ module provides several metricsets covering connection, exchange, node, queue. In such cases it makes sense to break this information down into several, interconnected dashboards (see the Navigation section), with the default one being an Overview of a target system and the others providing deep-dives in the various parts of the target system. The content of the Overview dashboard should be cherry-picked from all datasets and individually compiled for every such integration.

Metrics

Always check the type of a metric and ensure that the correct transformation is applied where applicable. For instance, in the majority of cases for cumulative counters it makes sense to apply the rate function (see this great article for more details).

Drill-downs

TBD

Visualization types

For new visualizations try using Lens first. If what you're trying to achieve cannot be accomplished with the current capabilities of Lens log the identified gap here and switch to TSVB. Do not use other visualizations.

Filters

When building a dashboard always consider adding a filter dropdown. Why? In most cases the integrations are used to monitor multiple instances of a target system, so we need to provide a way to switch between them.

Use the Controls visualization to build a filter dropdown. Here's an example of a host name dropdown that can be added the System dashboard:

And the result:

Navigation

If an integration has several dashboards ensure that all of them can be easily navigated. To build dashboard navigation use the Markdown visualization type.

For example, the System dashboard provides the following navigation:

source:

[System Overview](#/dashboard/system-Metrics-system-overview-ecs)  | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) |
[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)

While this can work, it doesn't highlight the selected dashboard. Unfortunately the Markdown control is not optimized for navigation, which makes it cumbersome to build navigation with highlighted links because each link should be highlighted separately. This means that the navigation control you're building has to be cloned as many times as there are dashboard to ensure proper link highlighting. E.g.

**[System Overview](#/dashboard/system-Metrics-system-overview-ecs)**  | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) |
[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)

[System Overview](#/dashboard/system-Metrics-system-overview-ecs)  | **[Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs)** |
[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)

[System Overview](#/dashboard/system-Metrics-system-overview-ecs)  | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) |
**[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)**

Target System Name

Currently we don't make it a rule to show on a dashboard what system it's designed to monitor. The only way to see it is through the dashboard name.

When using multiple dashboard on bigger screens, it makes it hard to distinguish between the dashboards. This can be improved by using the Markdown control to show what target system the dashboard is used for.

Layout

TBD

Naming

Use the following naming convention when building dashboards

Visualizations:

<NAME> [<Metrics | Logs> <PACKAGE NAME>]

Examples:

• Memory Usage Gauge [Metrics System]
• New groups [Logs System]

Rename all visualizations added to a dashboard to only show the <NAME> part.

Examples:

• Memory Usage Gauge
• New groups

Dashboards:

[<Metrics | Logs> <PACKAGE NAME>] <Name>

Examples:

• [Metrics System] Host overview
• [Metrics MongoDB] Overview

Screenshots

There are many many nuances to taking screenshots, some of which I didn't have the time to go into detail about. This wiki article has some great information about taking screenshots, but it is not specific to Ingest Manager and not all information is relevant.

Here's a quick demo video that shows @hbharding walking through these steps.

Download these Chrome Extensions

• Awesome Screenshot & Screen Recorder
• Window Resizer

Requirements:

• Hide your scrollbars. In Mac, go to System > General > Show Scroll bars and choose "When Scrolling"
• Use a retina display to take screenshots. Macbook pro or a 4k monitor will do. We require retina because we allow the user to view screenshots at full size. A non-retina screenshot will appear blurry for users who have retina monitors.
• Screenshots should use a 16:10 aspect ratio. Resize your viewport so that it is 1440 x 900.
• Screenshots should include the entire visible part of the page. Do not crop screenshots to focus on a particular area. Do not add annotations. Your screenshot should be of the viewport only. Do not include browser chrome at the top.
• Avoid capturing any personally identifiable information. This includes profile pictures or any public IP addresses, even in demo data.

Steps:

• Choose the dashboard in Kibana you want to capture
• Make sure the dashboard shows data close to real-world. Don't take a screenshot of empty charts.
• Use Window Resizer extension to set the viewport size to 1440x900. The viewport is the visible area of a web page you intend to screenshot, not to be confused with window size. You can create a preset for this size in Window Resizer options.
  • If you've recently downloaded anything (like a screenshot, which you'll do in a later step), be sure to close the download bar at the bottom of the browser's window. The download bar affects the size of the viewport, so it's important that it is not visible when taking a screenshot.
• Use Awesome Screenshot to capture the "Visible part" of the page. This will open a new tab. Click "done" in the top right and then choose "Download as image". When saving the image, choose a unique name that begins with the name of the integration. For example: "nginx-dashboard-summary" or "system-metrics-inventory" Do not use spaces or capital letters.

Exporting

See this overview of how to export dashboards in Metricbeat/Filebeats.

We'll need a similar information for Elastic Agent.

Review

When making a PR try to always include a screenshot and a link to a test system with live or simulated data where the dashboard can be tested.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ChrsMark]

Maybe slightly out-of-topic, but maybe we also need to define/update the dev guide (with specific commands, environment etc) of creating/updating/publishing dashboards. To illustrate the point, I have my own process of creating dashboards (based on Elastic Cloud), which is also a little bit hacky, and I miss a complete step-by-step cheatsheet both for internal but also for external contributors.

[exekias]

Maybe slightly out-of-topic, but maybe we also need to define/update the dev guide (with specific commands, environment etc) of creating/updating/publishing dashboards. To illustrate the point, I have my own process of creating dashboards (based on Elastic Cloud), which is also a little bit hacky, and I miss a complete step-by-step cheatsheet both for internal but also for external contributors.

++, I also know that @mtojek has plans to add this workflow to elastic-package, depending on timelines, prioritizing it could solve this part

[exekias]

About naming, now that we will stop installing all dashboards by default, the amount of dashboards and visualizations should be reduced to the ones that users request.

Perhaps we can simplify naming a little bit? For instance, removing the ECS suffix (let's say we take that for granted), and trim visualization names even more?

[sorantis]

Perhaps we can simplify naming a little bit? For instance, removing the ECS suffix (let's say we take that for granted), and trim visualization names even more?

Agreed, I don't see the value of adding ECS to every visualization/dashboard, unless it provides an indicator that the dashboard is supported by us.

[jsoriano]

+1 to remove ECS suffix from guidelines. It was added during the migration to ECS to be able to differentiate 6.x from 7.x dashboards, but it shouldn't be needed in dashboards that didn't exist before 7.0. Automatic checks support titles with and without ECS.

[jsoriano]

• Use a retina display to take screenshots. Macbook pro or a 4k monitor will do. We require retina because we allow the user to view screenshots at full size. A non-retina screenshot will appear blurry for users who have retina monitors.

This requirement can be limiting, very little people have 4K monitors (even between gamers). For example I don't have a 4K monitor. Do non-resized screenshots appear blurry too? Is there any alternative to this?

[masci]

Maybe slightly out-of-topic, but maybe we also need to define/update the dev guide (with specific commands, environment etc) of creating/updating/publishing dashboards. To illustrate the point, I have my own process of creating dashboards (based on Elastic Cloud), which is also a little bit hacky, and I miss a complete step-by-step cheatsheet both for internal but also for external contributors.

++, I also know that @mtojek has plans to add this workflow to elastic-package, depending on timelines, prioritizing it could solve this part

Feature is scheduled for 7.11 (elastic/elastic-package#12), it would be cool to prioritise the dashboard part of that task as we don't need to wait for 7.11 to use the tool and the sooner the better.

[kaiyan-sheng]

+1 on removing ECS from naming.

I've been using --decode parameter when trying to export the dashboard. Maybe we should revisit this export dashboard readme and make sure it's up-to-date.

For review/testing, where should we put the simulated data? That would be very helpful for testing the dashboard itself and also let users try out the dashboard with sample data.

[sorantis]

Removed ECS from the Dashboard/Visualization naming

[mostlyjason]

I'd like to propose adding the "Integration" to our assets to distinguish them from Beats modules. This will help prevent confusion identified here elastic/beats#23390. I'm open to other ideas as well.

Visualizations:

<NAME> [<Metrics | Logs> <PACKAGE NAME> Integration]

Examples:

• Memory Usage Gauge [Metrics System Integration]
• New groups [Logs System Integration]

Dashboards:

[<Metrics | Logs> <PACKAGE NAME> Integration] <Name>

Examples:

• [Metrics System Integration] Host overview
• [Metrics MongoDB Integration] Overview

[tbragin]

@mostlyjason We now have tags in Kibana, and my understanding is that we are going to leverage that to differentiate and organize assets we install via modules and integrations going forward. @masci @sorantis i've seen some threads on this - with the target of 7.12, is that accurate? #327 (comment)

[mostlyjason]

Thanks tags would be even better. I just checked the latest build and the tags are present on the dashboard and visualization list pages, so the user will have visibility to them.

[alexfrancoeur]

In the Fleet Package Lifecycle Management working group the topic of dashboard best practices has come up a number of times and @mostlyjason had asked for me to share my thoughts in this issue. My thoughts and feedback come not only as Kibana product manager, but also as someone who has spent a lot of time in Kibana over the years. I think there is a ton of opportunity to provide an amazing dashboarding experience with our current capabilities. Given that we plan to lead with integrations for both onboarding and as the primary way to ingest data in the not too distant future, it would be good for Kibana assets to align with current and future best practices. First impressions are important, and these integrations will be the starting point for many folks using Elastic for the first time.

Lens

As Lens begins to introduce formulas and calculations, it seems that most TSVB visualizations will can likely migrate over. Given that we're investing the majority of our time into Lens, any new capabilities derived from Lens will simply be available when these dashboards are added to Kibana. There is also an opportunity to start leveraging the new visualization types. I believe there are already ongoing discussions here, but probably one of the more important best practices to highlight.

By value

A number of these integrations have many saved objects, cluttering end-users Kibana space as soon as the dashboards have been installed. In recent versions of Kibana, we've added a concept of "by value" which essentially persists all visualizations within the dashboard JSON blob itself. This simplifies content management significantly, especially for an out of the box dashboard.

Context

Context is important for new consumers of a dashboard, and this can be handled in a few ways. Dashboards can have a description, and if we don't already provide one for each integration we should probably add them. In a similar fashion each dashboard panel can also have a description. Clicking into the information icon allows you to view that description as a dashboard consumer. Ever wonder exactly what a visualization is showing you? This is an opportunity to provide that context to users so they don't have to click into and edit a visualization to understand. We also have markdown capabilities, that will be evolving more as part of the analytics roadmap. While not necessary, Markdown can be used as a good way to segment a dashboard into meaningful chunks.

Workflows

Over the last few releases we introduced support for drilldowns to Dashboards, Discover and URLs. Dashboard & Discover are in the basic tier, URLs are a commercial feature. These drilldowns can be an option available as you click into a data point or available as part of the panel to navigate directly. They are a powerful tool for how you can rethink your dashboards and reduce the perceived slowness for loading a large dashboard. In order to incorporate these flows appropriately, you really need to take a step back think through where it's useful to view the data at a high level and drilldown further for details, taking that context with you. For what's available today, I'd focus on dashboard to dashboard workflows and probably add a link to Discover where it'd be useful to view the raw data (everywhere?).

When I demo drilldowns, I tend to create a URL back to solution apps (view in logs, view in metrics, etc.). Unfortunately, this is hack right now using the URL drilldown capabilities in the Gold tier. I think we have an opportunity here to provide "solution drilldowns" that link back to solution applications, with context, filtering UI's based on the data they were viewing in a dashboard. This is a great way to provide additional entry points back into solutions from ad-hoc analytics.

Out of the box queries

Not too long ago we introduce saved queries to dashboards. This is an exciting feature because it allows you to save a query, filters and a description to re-use common and expected filters. I think there might be an opportunity to pre-package these dashboards with filters we know might be useful for a use case. Since we are the experts in analyzing this data, I'm sure there are plenty we could use here.

Aesthetics

Aesthetics is important when viewing a dashboard. Colors, layout and panel size make a difference. One of the more recent lens capabilities introduced allow you to style a series the same across multiple visualizations. This is extremely powerful for visual correlation and something you get out of the box when using lens as the primary visualization tool in your dashboards.

Content management

Content management is a theme we've started leaning into and would like to invest in more in the future. With recent releases, we've added tags that allow you to tag saved objects in Kibana with a name, color and description. These tags can then be searched for in navigational search for quick navigation. I know there have been a few comments about standardizing on tags here and other issues, but worth reiterating.

Input controls

Input controls are currently listed as an experimental feature, but I believe there are plans to GA and lean into this concept way more than we do today. As a dashboard curator, input controls are your best friend. They allow you to control the experience for your end users. Coming from products like Grafana, customers tend to ask for dashboard templates. To an extent, we have them. Simply introduce an input control for host and filter that dashboard on a particular host. This not only provides a more curated dashboard experience, but also highlights the power of our dashboards. You can imagine a dashboard flow that starts at a meta dashboard, drills down to a host dashboard, and then provides the opportunity to swtich host with a simple drilldown. All the building blocks are available to provide this experience today

Solution embeddables

Teams are starting to introduce embeddables for dashboards, with the Observability team leading the charge 👏 . Now that we no longer have to worry about these integrations (previously modules) being installed in OSS deployments, we can use all Elastic licensed features available at our finger tips. Why not have a logs embeddable available in out of the box integrations? I've seen some recent POC's around APM embeddables as well. Let's show our users they exist when they spin up Kibana for the first time, providing another entry point back to solutions.

Shareable SO's

In 8.0, we'll be laying the groundwork for making saved objects shareable. Meaning, you'll be able to share a dashboard across one, multiple or all spaces. Thinking into 8.x, we'll have to figure out what our strategy is for adding Kibana assets from an integration. Should they default to all spaces? Should the user installing choose? If it's already installed in the deployment, should there be a button to quickly add to space? There's lots to discuss here and no best practices defined at the moment, but something worth keeping an eye on and thinking through.

Maps

I'm not sure to what extent we ship geo data as part of the default ingest pipeline with our integrations, but if we do (and there are a number of use cases today), we should start pushing Maps more. New capabilities are introduced with every release and it is a great entry point to surfacing the fact that geo data isn't special, it's just data, and you can do it to.

I'm a bit further removed from the analytics use case and plans these days, so I'm sure there is plenty I'm forgetting here. Adding @AlonaNadler & @VijayDoshi to the discussion for some additional thoughts around best practices. Hope this is helpful and adds some value to the discussion! Always happy to chat live and discuss further if it'd be helpful.

[sorantis]

I cam across this great article on how to build dashboards for monitoring. I think some of the points on how to structure visualizations are very useful.

[AlonaNadler]

Thanks @alexfrancoeur for raising all these areas, I think you are spot on and there are several ways to level up some of the dashboard packages with recent enhancements.

Few other updates:

• input control is being worked on at the moment would be good to join one of your meetings and provide an update
• new dashboard flow, we heard it from users a lot over the years, packages create hundreds of objects in their Kibana. With the new dashboard improvement. If you create the panels of the dashboard from the dashboard it doesn't create a separate object for each visualization, only one object is created for the dashboard. Please let's see how to move the dashboard to be a single object.
• Lens we have this issue Gaps in moving to Lens #265 with gaps in Lens. The lens team is about to close all gaps. Please let us know what needs to happen to move some of these panels into Lens

@sorantis this is an important topic for us what would be a good next step for this issue?

cc: @kmartastic @clintandrewhall @jb1b84

[monfera]

Scrolling (report) vs no scrolling (dashboard)

Would be great to shift toward the standard meaning of dashboards, which is a scroll-free, single screen experience with all information visible simultaneously. While users can create "dashboards" of any length, something that scrolls would be called something like report, notebook, article or page. Meanwhile, we Elasticians (SAs, consultants, folks working on sample dashboards) are at liberty of sticking to a single, non-scrolling page worth of information. The benefits of not scrolling are several:

• show all information
• the user can spot patterns in the data, because the only thing that changes is the data
• doesn't require interaction
• can be projected or used on shared displays for all to see

Drawbacks are not too numerous; for example, getting more detail for the dashboard is usually better achieved via drilldown, even if the dashboard was meant to be interacted with.

Some other design items

There is input to dashboard design here, such as, considering the use of shared color mapping, so that colors map to the same categorical variables if multiple charts are split by these categories.

There were also other suggestions here, such as aligning time series vertically for a shared horizontal mapping of the time, or preferably avoiding more obsolete visualization types such as gauge/goal charts.

[jamiehynds]

@P1llus is currently working on dashboard guidelines for integrations and can close this issue out once the guidelines have been finalised.

[jamiehynds]

Closing as dashboard guidelines have been published for some time: https://github.com/elastic/integrations/blob/main/docs/dashboard_guidelines.md