Conflicting fields in data views with data stream naming scheme

[ruflin]

Elastic Agent and other shippers are following the data stream naming scheme. This means that all that is shipped into data streams named as following: {type}-{dataset}-{namespace}. Many of the fields used are ECS but each data stream can also contain its own fields. This means for example logs-foo-default and logs-bar-default can both contain the field field1 and in one data stream it is keyword, in the other data stream it is float. This is ok and doesn't provide any problems for visualisations as these are normally built based on {type}-{dataset}-*. And when querying on Elasticsearch on logs-* for the field field1: 0.1, it works.

The problem is that we use a global logs-* data view for easy selection for our users as we don't want to create a data view for each data stream. But data views do not like conflicting fields as the type can't be fully detected. As described above, the conflict is expected so showing users a conflict indicates an error. There are cases where the data should be fixed like in #73797 but still data views should keep working.

Long term I think we should fix the more fundamental issue on how we use data views. Short term one idea to overcome the problem would be to be able to create data views with a flag, that such conflicts should not be shown as an error.

Related issues:

• host.ip and source.ip have conflicting data types in the metrics-* index pattern #73797
• Conflicting fields with object fields are wrongly detected as multi-fields #105238
• Display more information about conflicts in index pattern fields #74218

[ruflin]

@mattkime Original discussion happened here #73797 (comment) and #73797 (comment)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesSv)

[mattkime]

@ruflin Is this a problem in places aside from data view management?

[ruflin]

@elastic/integrations @ChrsMark Did you ever see this in visualisations.

AFAIK it is mainly in the data view management and in discovery where also an error is shown for the field.

[mattkime]

AFAIK it is mainly in the data view management and in discovery where also an error is shown for the field.

I'm still lacking a full user story - is the problem that users see the conflict and then think it needs resolving?

Maybe this is more about communication through the UI than anything else. Perhaps we just need to de-emphasize conflicting fields. It would be nice if we could highlight the problem only where users are affected and be more specific about what functionality is missing.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.