[Infrastructure UI] inital hosts page #138173

neptunian · 2022-08-04T20:13:13Z

Basic implementation of Hosts page that includes a Lens table and Unified Search. The plan is to put this behind a Technical Preview until complete. This approach of using Lens instead of a EuiTable and creating a custom endpoint is a bit unusual and I'd be interested in hearing any feedback for/against it. We'd continue adding more Lens components for other visualizations. The biggest upside is not having to manage data fetching (custom api) or creating custom visualizations / tables. One limitation is Lens only returns at the most Top 10k items. I think we could add a switcher at the top so thats more clear and the user can limit how many Hosts they see. We've always questioned how useful or common it is in Inventory to look at that many hosts at once.

You can read more about pros/cons here https://github.com/elastic/kibana/tree/main/x-pack/plugins/lens#use-lens-embeddable

checks that a Data View exists equal to source.configuration.metricAlias, else creates it because the Lens table requires it
Embeds a lens table with supported metrics
uses the Unified Search component to create/update the table, currently only supporting a query filter and date range
metric for services on host has a separate issue
metric for rx and tx have a separate issue

elasticmachine · 2022-08-23T18:44:17Z

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

matschaffer · 2022-08-24T02:24:59Z

x-pack/plugins/infra/public/pages/metrics/hosts/components/hosts_list_table.tsx

+  query: Query
+): TypedLensByValueInput['attributes'] =>
+  ({
+    visualizationType: 'lnsDatatable',


This is cool.

Did you find a nice way to get from a lens viz to this code or was there a lot of formatting/replacement involved?

I created the table in Lens and then used the "Open in Playground" feature that gives you all the attributes and just basically copied it into here https://github.com/elastic/kibana/tree/main/x-pack/plugins/lens#getting-started. Vice versa would work as well. I didn't have to do any direct editing of this code other than handing it a Data View. Everything was done in the Lens UI. I was telling Milton in another comment (#138173 (comment)) that the idea would be we'd store this data somewhere as models and generate the lens attributes ourselves once we've got most of the data from the Lens Playground. You can also always paste this back into Lens and view your viz and edit it there.

matschaffer · 2022-08-24T04:55:14Z

I get the impression this is due to the infra plugin field query issue but this loads really slowly for me. Seems to be ~30s from a basically warm tab.

The initial load seemed to create the data view which caused the whole load to be more like ~90s.

Maybe the speed isn't something to fix in this PR though.

Also, can we add refresh and recent ranges to this selector?

That'd make it match what's on dashboards usually:

matschaffer · 2022-08-24T05:00:40Z

A few other things I noticed:

The filter +/- doesn't seem to do anything. I would have expected it to add filters under the search bar similar to dashbaord behavior

Not sure how to share the query contents. Seems like we should either be able to copy the URL or have a "share" button.

This could be followup issues as well if you're trying to keep this MVP.

miltonhultgren · 2022-08-24T07:33:41Z

x-pack/plugins/infra/public/pages/metrics/hosts/hooks/use_data_view.ts

+    services: { dataViews },
+  } = useKibana<InfraClientStartDeps>();
+  const loadDataView = useCallback(async () => {
+    let view = (await dataViews.find(metricAlias, 1))[0];


@matschaffer I'm fairly certain this call and the creteAndSave call below try to resolve all the fields in the data view.
I'm running into the same issue when trying to optimize the source loading for the Logs UI.
I'm making a follow up issue to see if we can make field resolution optional.

x-pack/plugins/infra/public/plugin.ts

miltonhultgren

I'm anxious about how we'll maintain the Lens attributes JSON blob but the rest looks good to me 👍🏼

x-pack/plugins/infra/public/pages/metrics/hosts/index.tsx

miltonhultgren · 2022-08-24T07:39:44Z

x-pack/plugins/infra/public/pages/metrics/hosts/hosts_content.tsx

+  const {
+    services: { data },
+  } = useKibana<InfraClientStartDeps>();
+  const [dateRange, setDateRange] = useState<TimeRange>({ from: 'now-15m', to: 'now' });


Should we look to use the global Kibana timerange service here instead?

I'm getting the time range from the Unified Search component and storing it as state so I can pass it to the lens component. Would it make sense to use that here? Could you point me to an example?

miltonhultgren · 2022-08-24T07:41:25Z

x-pack/plugins/infra/public/pages/metrics/hosts/hosts_content.tsx

+        <InfraLoadingPanel
+          height="100vh"
+          width="auto"
+          text={i18n.translate('xpack.infra.waffle.loadingDataText', {
+            defaultMessage: 'Loading data',
+          })}
+        />


I wonder if we can re-use some of the state screens created for the InfrastructureMetricsTable where there is an animation and the text "loading metrics"?

Sounds good, though technically this is loading the Data View. The lens component has its own loader which is a vertical bar on the top of the table (try clicking refresh) that is used for loading the actual metrics data.

miltonhultgren · 2022-08-24T07:47:50Z

x-pack/plugins/infra/public/pages/metrics/hosts/components/hosts_list_table.tsx

+import type { DataView } from '@kbn/data-views-plugin/public';
+import { InfraClientStartDeps } from '../../../../types';
+
+const getLensHostsTable = (


I'm guessing this isn't really to be reviewed because it's generated but I think we'll need to document how to re-generate this when needed.
This is why I'd wish we could use a Lens-as-Saved-Object instead and just edit it in the Lens UI.

If we decide to do it this way, I think all these formulas and fields for calculating the metrics will be stored somewhere, like the inventory model metrics. So we would be generating the lens attributes based on these. Similar to how we do with TSVB https://github.com/elastic/kibana/tree/main/x-pack/plugins/infra/common/inventory_models/host/metrics/tsvb. If you want to view it in Lens to edit it there you can take the generated attributes and paste it into the Lens Playground, save it, and start editing it in the Lens UI.

See https://github.com/elastic/kibana/tree/main/x-pack/plugins/lens#getting-started

The saved object might be the better way to go, but I thought this might be easier then having to ship them with packages or metricbeat and worrying about the user editing them, but I need to check the status on read only Kibana assets. Also being able to store the fields and formulas for possibly reusing. If we need to update a field, it would be easy to replace it in the code in one place. Once we get the metrics right, I don't suppose they'll be changing very often.

Yeah, if we can write some (strongly typed) code to generate these attributes and combine them then that would feel a lot better!

What would storing this configuration as a saved object enable ? Sounds like we want this config to be readonly for users, and if we don't have to update it at runtime neither it would be an unnecessary network call

neptunian · 2022-08-24T12:34:40Z

I'm anxious about how we'll maintain the Lens attributes JSON blob but the rest looks good to me 👍🏼

Let me know if this comment answers that #138173 (comment). That was my general thought.

smith · 2022-08-24T20:51:38Z

/oblt-deploy

smith · 2022-08-24T21:02:02Z

x-pack/plugins/infra/public/pages/metrics/hosts/hosts_content.tsx

+        <>
+          <SearchBar
+            showQueryBar={true}
+            showFilterBar={false}


Any reason to be hiding the filter bar?

Supporting filtering is a bit more complicated so I just hid it for now to do in a follow up

klacabane · 2022-08-26T12:10:20Z

The biggest upside is not having to manage data fetching (custom api)

Sounds like a big win if we can delegate this to reusable components. otoh we would lack customization capabilities and we're subject to breaking changes from the lens API ?

x-pack/plugins/infra/public/pages/metrics/hosts/hooks/use_data_view.ts

neptunian · 2022-08-26T13:16:21Z

I get the impression this is due to the infra plugin field query issue but this loads really slowly for me. Seems to be ~30s from a basically warm tab.

The initial load seemed to create the data view which caused the whole load to be more like ~90s.

Maybe the speed isn't something to fix in this PR though.

@matschaffer Are you connected to your edge cluster? Loading the source configuration in order to get the fields used for auto complete takes a long time, I think due to the fact that there are remote clusters. What is your index pattern to in Settings? If you're on edge you probably changed it to add the remote_cluster: prefix.

The Data View that gets auto created uses this same index pattern value in Settings. On the edge cluster this takes me about 10 seconds using remote_cluster:metricbeat-*,metricbeat-*,remote_cluster:metrics-*,metrics-* with the Data View containg 6,672 fields. If the Data View is already created it takes a few seconds. Locally, while running metricbeat, it takes about 2 seconds to create a Data View of metrics-*,metricbeat-* containing 5,901 fields.

Part of the bottleneck seems to be connecting to the remote clusters in order to complete these tasks. Milton has opened an issue to see if we can defer having to resolve these fields. #138173 (comment)

Note:
You would see this during the time it takes to load the source fields:

This loading sign would be while the Data View is being created/fetched (changed it from Loading to Loading Data View):

neptunian · 2022-08-26T14:44:35Z

Also, can we add refresh and recent ranges to this selector?

That'd make it match what's on dashboards usually:

@matschaffer I don't see any information in the Unified Search storybook on how to do this but looking into the Lens code, they are using a component provided by navigation.ui called TopNavMenu which uses a stateful version of Unified Search AggregateQuerySearchBar. I'm assuming we would need to use this and can do that in a follow up. I think we need to do use this to handle filters anyway.

neptunian · 2022-08-26T15:50:02Z

A few other things I noticed:

The filter +/- doesn't seem to do anything. I would have expected it to add filters under the search bar similar to dashbaord behavior

@matschaffer
Yea, I had turned off the filters for the Unified Search because it was a bit more complicated and wanted to separate it out into a different issue. Didn't realize that the table had this when hovering over cells. Doesn't seem to be a way to disable the feature for now but asking Lens.

Not sure how to share the query contents. Seems like we should either be able to copy the URL or have a "share" button.

This could be followup issues as well if you're trying to keep this MVP.

I had opened this issue #138518, but will open another issue just to get the basic url state working. Hoping to get this in first so others could pick up tasks and this PR isn't so big.

neptunian · 2022-08-26T18:44:21Z

The biggest upside is not having to manage data fetching (custom api)

Sounds like a big win if we can delegate this to reusable components. otoh we would lack customization capabilities and we're subject to breaking changes from the lens API ?

@klacabane Lens is pretty customizable but already I've found some things they don't support. They've been pretty responsive and I think if more of Kibana starts using Lens like this they would build the necessary features. There is still a risk of not being able to do something, at least in the short term. One thing they are adding next week for us is the ability to get the data back from the underlying fetch so we could access the data being returned because currently we can't. This arose from this issue #139506 where we need to pass in the list of hosts to get Alerts back until they have better data and capability for filtering. Another limitation is it only returns at the most Top 10k items. I think we could add a switcher at the top so thats more clear and the user can limit how many Hosts they see. We've always questioned how useful or common it is in Inventory to look at that many hosts at once.

Here are some other pros/cons https://github.com/elastic/kibana/tree/main/x-pack/plugins/lens#use-lens-embeddable

kibana-ci · 2022-08-26T19:38:28Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`infra`	950	954	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`infra`	1016.5KB	1.0MB	+7.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`infra`	84.2KB	84.4KB	+172.0B

History

💔 Build #67766 failed e5bf383
💚 Build #66849 succeeded c3c22a8
💚 Build #66757 succeeded 37f2103
💚 Build #66167 succeeded 9f881b5c3f8efb33595f2873778b14c8f96a0966
💚 Build #63332 succeeded 53d234e7d167c12acc84869ec1bbc92c1e32a0f9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* inital empty hosts page and navigation item * lens table and unified search * add data view id * cleanup * clear searchs session when unmounting * cleanup * add some basic error handling for Data View * change back loading text for now because of breaking test

neptunian changed the title ~~inital empty hosts page~~ inital hosts page Aug 4, 2022

neptunian force-pushed the 138111-initial-hosts-page branch from 53d234e to 9f881b5 Compare August 18, 2022 21:08

neptunian added 4 commits August 23, 2022 09:36

inital empty hosts page and navigation item

804e669

lens table and unified search

86b9c80

add data view id

afa6524

cleanup

37f2103

neptunian force-pushed the 138111-initial-hosts-page branch from 9f881b5 to 37f2103 Compare August 23, 2022 13:37

clear searchs session when unmounting

c3c22a8

neptunian added Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services release_note:feature Makes this part of the condensed release notes labels Aug 23, 2022

neptunian marked this pull request as ready for review August 23, 2022 18:44

neptunian requested a review from a team as a code owner August 23, 2022 18:44

matschaffer reviewed Aug 24, 2022

View reviewed changes

miltonhultgren reviewed Aug 24, 2022

View reviewed changes

x-pack/plugins/infra/public/plugin.ts Outdated Show resolved Hide resolved

miltonhultgren approved these changes Aug 24, 2022

View reviewed changes

miltonhultgren mentioned this pull request Aug 24, 2022

[Infra UI] Make call to _field_caps non-blocking for UI rendering #127616

Closed

smith reviewed Aug 24, 2022

View reviewed changes

matschaffer added the ci:cloud-deploy Create or update a Cloud deployment label Aug 25, 2022

neptunian changed the title ~~inital hosts page~~ [Infrastructure UI] inital hosts page Aug 26, 2022

klacabane reviewed Aug 26, 2022

View reviewed changes

x-pack/plugins/infra/public/pages/metrics/hosts/hooks/use_data_view.ts Outdated Show resolved Hide resolved

neptunian added 4 commits August 26, 2022 13:21

cleanup

461baed

Merge branch 'main' into 138111-initial-hosts-page

e5bf383

add some basic error handling for Data View

a3a81ff

change back loading text for now because of breaking test

e4b013b

neptunian mentioned this pull request Aug 26, 2022

[Infrastructure UI] put hosts view behind technical preview #139589

Closed

neptunian merged commit 7e1b60f into elastic:main Aug 26, 2022

kibanamachine added v8.5.0 backport:skip This commit does not require backporting labels Aug 26, 2022

ruflin mentioned this pull request Sep 8, 2022

Schema for metrics elastic/ecs#474

Open

neptunian added the Epic: Host Observability label Nov 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Infrastructure UI] inital hosts page #138173

[Infrastructure UI] inital hosts page #138173

neptunian commented Aug 4, 2022 •

edited

Loading

elasticmachine commented Aug 23, 2022

matschaffer Aug 24, 2022

neptunian Aug 24, 2022

matschaffer commented Aug 24, 2022

matschaffer commented Aug 24, 2022 •

edited

Loading

miltonhultgren Aug 24, 2022

miltonhultgren Aug 24, 2022

miltonhultgren left a comment

miltonhultgren Aug 24, 2022

neptunian Aug 26, 2022

miltonhultgren Aug 24, 2022

neptunian Aug 26, 2022

miltonhultgren Aug 24, 2022

neptunian Aug 24, 2022 •

edited

Loading

miltonhultgren Aug 24, 2022

klacabane Aug 26, 2022 •

edited

Loading

neptunian commented Aug 24, 2022

smith commented Aug 24, 2022

smith Aug 24, 2022

neptunian Aug 24, 2022 •

edited

Loading

klacabane commented Aug 26, 2022

neptunian commented Aug 26, 2022

neptunian commented Aug 26, 2022 •

edited

Loading

neptunian commented Aug 26, 2022 •

edited

Loading

neptunian commented Aug 26, 2022 •

edited

Loading

kibana-ci commented Aug 26, 2022

[Infrastructure UI] inital hosts page #138173

[Infrastructure UI] inital hosts page #138173

Conversation

neptunian commented Aug 4, 2022 • edited Loading

elasticmachine commented Aug 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matschaffer commented Aug 24, 2022

matschaffer commented Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miltonhultgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neptunian Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klacabane Aug 26, 2022 • edited Loading

Choose a reason for hiding this comment

neptunian commented Aug 24, 2022

smith commented Aug 24, 2022

Choose a reason for hiding this comment

neptunian Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

klacabane commented Aug 26, 2022

neptunian commented Aug 26, 2022

neptunian commented Aug 26, 2022 • edited Loading

neptunian commented Aug 26, 2022 • edited Loading

neptunian commented Aug 26, 2022 • edited Loading

kibana-ci commented Aug 26, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

neptunian commented Aug 4, 2022 •

edited

Loading

matschaffer commented Aug 24, 2022 •

edited

Loading

neptunian Aug 24, 2022 •

edited

Loading

klacabane Aug 26, 2022 •

edited

Loading

neptunian Aug 24, 2022 •

edited

Loading

neptunian commented Aug 26, 2022 •

edited

Loading

neptunian commented Aug 26, 2022 •

edited

Loading

neptunian commented Aug 26, 2022 •

edited

Loading