Add Data field descriptions

[rayafratkina]

Describe a specific use case for the feature:
In complex deployments of Elastic stack, there is often a need to annotate fields for end users beyond just the name and type since the user who create visualization and explore data in discover may not be deeply familiar with the schema or the schema is too complex and even knowledgeable users may benefit from reminders.

Describe the feature:
Idea is to add a Description field to Index pattern management. This will be optional.
There are 2 ways we can use it

1. Add a toooltip to the field list in Lens/Discover.
2. Create a read-only "Data dictionary" page that would be available to anyone with view Discover/Dashboard permissions.
   User will be able to pick the index pattern and see a table showing field name, type, and description. Ideally the table will be searchable and filterable based on keyword and type.

Open questions/other considerations:

• What about localization? Assuming we would not support translating the descriptions explicitly, but it would be possible to have different descriptions per space if we implement this on the Kibana index patterns.

cc @alexfrancoeur @linyaru @timductive @alexh97 @VijayDoshi

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[VijayDoshi]

+1 to this. It is very common for users to need more information about a field than what is possible in the name to give them confidence in their analysis. Suggest limiting the description to a reasonable number of characters - perhaps 255.

[alexfrancoeur]

++ I think this would be useful for a number of reasons. At some point, we probably want to connect with the Elasticsearch team to see if Elasticsearch admins can ship standardized descriptions independent of index patterns, but having the framework in place to start adding descriptions to tooltips, stand alone views and other areas within Kibana will be necessary to make these data dictionaries more prominent.

@elastic-jb maybe this is something we can sync with the team on? I imagine a number of edge cases will pop up, but overall, it feels like lower hanging fruit to augment index patterns.

[alexfrancoeur]

@stevewritescode you might be interested in following this feature

[timroes]

Linking this since very slightly related: #17087 This issue here is about allowing free form semantical descriptions on fields, the linked issue about structured semantic data. No duplicate just linking for better findability.

[rschirin]

hey there, I saw that my request has been closed and "merged" with this one. I thought to a tooltip since I got feedback from analysts in my office

[rayafratkina]

@rschirin sorry, looks like 2 different changes to your issue conflated: in this comment @kertal mentioned that we have recently added ability to customize field names in Management.

This issue will be used going forward to track adding an additional Descriptions field. It is not done yet and something we are talking about doing in the future.

[rschirin]

@rschirin sorry, looks like 2 different changes to your issue conflated: in this comment @kertal mentioned that we have recently added ability to customize field names in Management.

This issue will be used going forward to track adding an additional Descriptions field. It is not done yet and something we are talking about doing in the future.

Personally, I think that just the capacity to change/add/modify/customize a field name is not what I was looking for.
The description of a field's name could not be summarized with other 2 words. That is the reason I think that currently Kibana is missing of a tooltip or glossary.
As I described in my scenario, if we use an acronyn we cannot explain it in a short field name.

[elastic-jb]

@cjcenizal this sounds very similar to the idea we were brainstorming late last year. I think there is a ton of value especially in context of runtime fields. Who created it? When was the formula changed? Even just the acronym expansion idea is very useful.

[cjcenizal]

Yes @elastic-jb, if I recall correctly we were thinking that it'd be useful to support extensive annotations on runtime fields so that you could record the reason behind your changes to any given field. This would help multiple analysts collaborating on the same index patterns communicate as they iterate on the data together.

[shaunmcgough]

@m-adams and I just spoke, and this issue came up. He has a logged ER somewhere for this as well, so it's something that customers are requesting (of course... right?) This will help the getting started experience. We cannot know what people will call their fields, but if the description can be added as a tooltip when hovering over the field, or as an indicator elsewhere, this will really advance the understanding of the data and what it is for our community.

[streamich]

Maybe the description could be in Markdown format. If we wanted to show the description in a short tooltip, we could take just the first markdown block to show in tooltip.

[elastic-jb]

I really like that idea. JB Jason Burns Principal Product Manager, Kibana ***@***.***

…

On Tue, May 4, 2021 at 2:21 AM Vadim Dalecky ***@***.***> wrote: Maybe the description could be in Markdown format. If we wanted to show the description in a short tooltip, we could take just the first markdown block to show in tooltip. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#89726 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO3AA4G6NVFG24FAEYNKEFTTL64APANCNFSM4WY7XHXA> .

[michaelkeevildown]

I agree this would be a really nice feature. It is becoming more and more apparent when using Elastic that the same data can have a different meaning depending on the user, providing a way for a user to talk in their own language but access a common data model will be imperative as we start to expand into the business lines.

I am seeing the same thing happen with new standards that are being created where they decouple the Business Model from the actual data Elements. You can see this with standards like ISO 20022.

https://www.iso20022.org/iso20022-repository/business-model

[odmby]

Is this feature planned in the near future?
At least having the option to add and view the description in the Index Patterns, but also to have a hover possibility in discover.

[elastic-jb]

This is something we are actively discussing, albeit at a more general level. We are scoping an initiative related to Saved Object Metadata and Governance. The ability to annotate anything from a field to a dashboard is in scope. This effort will not kick off in earnest until 8.0, but it's part of a quickly developing roadmap in this area.

[rschirin]

well....cross finger! it will funny to say to our colleague that this feature will see the light

[cjcenizal]

@elastic-jb This topic came up in the Stack Management sync today, and a few considerations arose from the discussion:

• If we provide the ability to add descriptions to runtime fields, users will expect to be able to add descriptions to all fields, runtime or not. We'll also need to preserve these descriptions when a user promotes a runtime field to be an indexed field.
• This implies that descriptions are a generalized service, that can be associated with arbitrary objects.
• If this is the case, what's the best path forward for supporting descriptions on Elasticsearch primitives, such as index templates and ILM policies? Should this service support associating descriptions with those? Or should Elasticsearch implement a convention to add support for description fields on all of their primitives? If the former, then will we need ES to provide UUIDs for all of their primitives (an idea @sebelga has raised in the past)? In either case, it seems like some coordination with ES should happen.
• Have we considered supporting comments on objects, either instead of descriptions or to complement them? This seems like another generalized service that will support collaboration.

[odmby]

in my eyes it's at least very natural to have an ability to save a description in the Management section of the index patterns, and to begin with show it just there. Will be useful for big teams of data engineering/analysis who need to see and understand what they are using (especially for new people)

[m-adams]

I'm not sure we can assume analysts (rather than admins) will have access to the management section. For me the basic level that would make it broadly applicable would be to set descriptions for fields in the index pattern and have that info available when you are using a field e.g. Lens/Discover/Maps and later in the solutions.
I can see the argument for documenting the reasoning behind ILM policies and index templates but as these are admin functions IMO they are of much lower priority because admin functions are more likely to be touched by fewer people and be officially documented. Having said that, there are normally long term advantages of implementing in the most general way but just my opinion from the field on which bits of this feature will have the biggest impact.

[rayafratkina]

+1 to Matthew's comment: I think we are expanding the scope and complexity of this request. We should also consider adding descriptions for ILM policies/index templates and other admin structures but it's a separate issue and is a lower priority.
I also think we will benefit much from a generalized solution - we are talking about a text field, there is not much inherent functionality there!

[sebelga]

I also think we will benefit much from a generalized solution - we are talking about a text field, there is not much inherent functionality there!

It all depends on the vision we have. I am all in to build incrementally and start small with a text field to add a description. But maybe this is the first stone for a collaborative feature in Kibana where user can see a history of comments for saved object or ES primitives (instead of the author adding a description to a field maybe it is a consumer who needs to ask "what is this field for?" and the author get pinged to answer).

Thinking ahead on how that might look like and have a generalised solution even for a text field is a good first step towards that goal (if we see value in it). Basically it means decoupling the "adding a description" task from "data fields".

[rschirin]

Could be an idea to start from a tooltip for the field's name column in Kibana and then add a scroll icon for every field to see the history of comments?

[shaunmcgough]

@qn895 just FYI. This is an issues about adding a description field and then having it available for use as a popup or elsewhere. I thought it would also make sense in Field statistics.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[gbamparop]

This would also be useful for our telemetry indices which might contain hundreds of fields.