Make viewing documents in the Discover datatable human readable

[timductive]

In addition to the current document flyout view, we should be able to format the documents in the actual table rows with a more readable structure.

When viewing Documents in Discover:

• Turn on auto-size for the row
• Represent the full document in a hierarchical structure
• Collapse objects in the document by default
• Introduce more colors to denote different field types and values

Example:

{ [ - ]
  metadata: { [ - ]
    some: field,
    foo: bar
  },
  timestamp: '2024-05-05',
  name: 'a log or something',
  id: 123456789456
  status: { [ + ] },
}

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[timductive]

@andreadelrio @stratoula @ninoslavmiskovic Something for us to consider in the document view for ESQL

[kertal]

Worth to note, that even if we enable auto-size to view it, there needs to be a limit what can be rendered, and inline scrolling / virtual rendering is needed to show what's outside the limit. Just to prevent performance issues, due to a very large number of fields

[stratoula]

I am really in favor of this change, I find the current Document view very difficult to read. We can not show everything of course but the idea here @kertal is to prioritize the important fields and not show everything of course

[kertal]

I am really in favor of this change, I find the current Document view very difficult to read. We can not show everything of course but the idea here @kertal is to prioritize the important fields and not show everything of course

@stratoula yes, this makes lot of sense. Given we work on ECS data this can be used in a nice way to identify important fields, currently we just display the fields up to the number configured in discover:maxDocFieldsDisplayed, with a default of 200

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[timductive]

@kertal @stratoula I was reading about EUI virtualization of rows in the datagrid. If the user expands a json document, does the row height auto-adjust in the auto-fit mode? Or does it auto-fit based on the initial content and any expanding/collapsing will be tied to the current height of the row?

[kertal]

@timductive I've checked by manually editing the dom content, and it looks like when the content is changed, the height is auto adjusted:

[stratoula]

Yes this should work just fine 🤞

[andreadelrio]

Sharing the first pass at design. Please let me know what you think. Note that, unlike the Unified Doc Viewer flyout, I've opted for no line numbers which I think is better suited for a table display.

Default

Expanding coordinates

[stratoula]

Looks great Andrea, thanx!

[ninoslavmiskovic]

Looks great @andreadelrio !

[kertal]

Linking the one related component for this kind of UI in EUI https://eui.elastic.co/#/navigation/tree-view it would need to be extended of course. I don't think we have a similar tree rendering in Kibana, do we? Which means if we are not using EUI it would need to be a custom built component

[stratoula]

@kertal I had in my mind something different, use the monaco editor in the view mode (not sure how it works and possiblly is not performant, I just know that it has the capability to render json and group them

[kertal]

Feedback after a chat with @stratoula:

• Yes Monaco could be an option, given the number of first level fields is limited
• Performance of having multiple Monaco components on one page should be evaluated
• Open question (just came to my mind), how much of the Monaco Expand Collapse UI can be adapted?

[stratoula]

Open question (just came to my mind), how much of the Monaco Expand Collapse UI can be adapted?

You don't adjust the UI in monaco, you just go with what it gives you 😄

I think the most performant way is to create this component on our own. But of course it increases the scope.

Update: I synced with @timductive, we agreed to do a PoC with creating this component on our own and see if we encounter any problem. The EUI tree is not ideal for this usage.

[ninoslavmiskovic]

Question. When we can see the _source field in the table with other fields/columns (when this is completed).

I guess it would make sense to also improve the experience of the _source ?

See here (I know it shows as "Document" currently, but in the future you would be able to see columns like e.g. _source, @timestamp, host etc.:

Skaermoptagelse.2024-04-11.kl.12.59.52.mov

[stratoula]

I have another issue tracking _source Nino, check here #175437

[ninoslavmiskovic]

@stratoula That is the one I am linking to that needs to get done 👍

If we do both will the _source get this new readable UX ?

[stratoula]

oh sorry I missed that. Possibly yes, although the _source issue is also about the sidebar in the left, I am not going to deal with this in this issue.

[ninoslavmiskovic]

++ the _source is great , your issue should be fixed for sure 👌

[ryankeairns]

@stratoula @MichaelMarcialis and I discussed how to unblock, UX-wise, what Stratoula has in-flight with regards to row height settings that exist on the doc table.

TL;DR - we proposed having a 'Format as JSON' (wording TBD) option either in the header as a custom action on the Document column or in the table settings menu alongside/below the row height setting and defaulting to a height of 10.

The longer version is that we currently default to 3 (this can be overridden in advanced settings) which is not ideal for the tree view layout (i.e. you don't see much). Also, using the Auto height setting could be unpredictably long if a doc has hundreds or thousands of fields. By defaulting to 10 when the 'Format as JSON' option is enabled, you see a bit more and still have the option to change that value or use Auto.

As for placement, my understanding is that this is only for the Document field to start, so placing it directly in the header as a custom action alongside the sort values makes sense, contextually. Using the settings menu keeps it alongside the row height settings which means you discover both simultaneously, but this may not make as much sense considering it only affects Document to start.

Perhaps we start with it in the Document column header and consider also adding/moving it to the settings menu if we are able to affect other JSON-like fields. Then this becomes a more global setting - try and use the JSON format wherever we detect it makes sense.

I don't know that either of these are extremely discoverable. We may also consider making this the default in time, on ES|QL mode, on cell hover actions, etc.

[ryankeairns]

Beyond this initial addition, we are curious to see how this impacts use of the document flyout. Does increased use of this decrease use of that? Things to keep an eye on.

[stratoula]

I have very briefly discussed it with @timductive , he says: We need fewer settings and fewer user decisions and I personally agree with him. I think that:

• we should go with this view (and only) on the ES|QL mode. I want to keep the existing document view in the dataview mode as it is a bigger change for us and also if we are moving towards making ES|QL the default experience it doesnt make sense to give a lot of effort in the data view mode

• if we agree on the statement above then the only problem is the existence of the advanced setting (which defaults to 3). I understand that this setting is very old. I don't like the advanced settings a lot tbh as they are not very discoverable. We already have settings for the row height in Discover so I think that this setting complicates the code and the experience in general.

With that being said, does it make sense to give a slightly different experience in ES|QL mode i.e.:

• Not use the advanced setting for changing the defaults row height in ES|QL mode (we can add a note on the advanced setting explaining that it doesnt apply for ES|QL)
• On ES|QL default always the custom in 10 lines
• Use a different local storage variable for ES|QL (because atm we are storing the selections in the local storage which means the selections a user does in dataview mode, affects the esql mode and vice versa).

[stratoula]

Btw from the telemetry it seems that a super small portion of users are changing this setting

[kertal]

I have very briefly discussed it with @timductive , he says: We need fewer settings and fewer user decisions and I personally agree with him. I think that:

yes, this make sense

• we should go with this view (and only) on the ES|QL mode. I want to keep the existing document view in the dataview mode as it is a bigger change for us and also if we are moving towards making ES|QL the default experience it doesnt make sense to give a lot of effort in the data view mode
• if we agree on the statement above then the only problem is the existence of the advanced setting (which defaults to 3). I understand that this setting is very old. I don't like the advanced settings a lot tbh as they are not very discoverable. We already have settings for the row height in Discover so I think that this setting complicates the code and the experience in general.

yes, I see this setting very linked to the Legacy table, that we aim to deprecate, so it make sense to not use it in ESQL if it's blocking

With that being said, does it make sense to give a slightly different experience in ES|QL mode i.e.:

• Not use the advanced setting for changing the defaults row height in ES|QL mode (we can add a note on the advanced setting explaining that it doesnt apply for ES|QL)

Makes sense IMO

• On ES|QL default always the custom in 10 lines
• Use a different local storage variable for ES|QL (because atm we are storing the selections in the local storage which means the selections a user does in dataview mode, affects the esql mode and vice versa).

👍 , yes this needs to be seperated

[stratoula]

ok if @timductive and @ryankeairns are also ok with this I can do the necessary changes on my PR 👍

[stratoula]

Synced with @timductive and agrees with the above plan. We also decided that:

• clicking on a column will change the default to 3 (or the value the user has set for this mode)
• clicking on the document view will change the default to 10 (or the value the user has set for this mode)

I will make the necessary changes to my PR

[ryankeairns]

For historical context, line heights were a sensitive change at the time we moved to data grid. iirc, the longstanding behavior of Discover was 5 (?) lines in the default state (@timestamp + Document columns). Also, I don't believe we had the line height settings, initially, so getting that dialed in - without expanding rows in a data grid - was the challenge.

With that in mind, making this the default behavior / focusing on ES|QL feels reasonable. It also has the added benefit of avoiding or mitigating potential sensitivity by not disturbing long-time Discover (KQL) users.

[stratoula]

This got deprioritzed so I removed it from our current sprint and changed the impact to low. There are other plans on making Discover more useful to our users (default columns for example) so it seems that this enhancement might not be needed. (or at least it needs more discussion to make it more useful to our users.

[IanLee1521]

@stratoula - finding this now, but for what it's worth I think that this sort of a change would be really useful. The default document view is rarely what I want to see, and I'm usually always having to pop open the single document view to find what I want.

On that note, I think this would be a separate issue, but I'm not finding it atm... is anyone trackign that when viewing a single document, that all fields for all documents possible are shown, even when they are blank / missing for the particular document I have open? I'd much rather hide the blank fields that don't apply. -- currently I end up with a view of 900+ fields and it's very hard to find what I'm looking for. (If this isn't already an issue that I just can't find, I'm willing to open a new one).

[davismcphee]

is anyone trackign that when viewing a single document, that all fields for all documents possible are shown, even when they are blank / missing for the particular document I have open?

Hi @IanLee1521, thanks for the feedback. If I'm understanding this request correctly, yes we are tracking this and planning to soon implement an enhancement that will allow hiding blank/null values from the doc viewer table. Issue here: #188846.

[IanLee1521]

@davismcphee - yes, that looks exactly like what I'm thinking. Just added a comment there with my example case. Thanks for tracking that down!

[stratoula]

@IanLee1521 thanx, I personally agree with you, it is on our backlog but not on our immediate roadmap for the time being. I hope it will get prioritized soon yet, I have a PoC ready 😄 #182557