Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify that lookup runtime sub-fields can't be queried nor aggregated on #88296

Closed
javanna opened this issue Jul 5, 2022 · 3 comments · Fixed by #91410
Closed

Clarify that lookup runtime sub-fields can't be queried nor aggregated on #88296

javanna opened this issue Jul 5, 2022 · 3 comments · Fixed by #91410
Assignees
Labels
>docs General docs changes >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Docs Meta label for docs team Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@javanna
Copy link
Member

javanna commented Jul 5, 2022

The lookup runtime field enriches the top hits with fields retrieved from a secondary index. This is purely a fetch phase enrichment at the moment, and users may get surprised that the sub-fields that are fetched from the secondary index are not searchable nor aggregatable. Effectively those are treated like unmapped fields. Would it make sense to throw an error when they get queried or aggregated on?

@javanna javanna added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types labels Jul 5, 2022
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Jul 5, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@javanna
Copy link
Member Author

javanna commented Jul 8, 2022

Posting thoughts from a discussion with @dnhatn : the lookup field does throw error when you try to aggregate/query it. What users may stumble upon is trying to query/aggregate on the sub-fields that are fetched from the secondary index. Those are not in the mappings and will be treated like unmapped fields. Also, these are not returned by field_caps, which is good for consumers that do look at field_caps, but some users could be lead to believe that the lookup runtime field performs a join hence they try to query the field that they are joining on.

The fields fetched from the secondary index could be expressed with a wildcard expression, hence the only way we could throw an error when these are queried/aggregated on would be to make the lookup runtime field own its entire namespace (a la flattened field) and have a dynamic mapped field type for its sub-fields whose only purpose is to throw errors. These would shadow any field with same name already present in the mappings, and it seems overkill if the only purpose is to throw an error. We will look at improving the docs and making it even more explicit that these sub-fields can't be queried nor aggregated on.

elasticsearchmachine pushed a commit that referenced this issue Jul 8, 2022
Adds tests to ensure that querying and aggregating on lookup runtimes
aren't supported.

Relates #88296
@dnhatn dnhatn removed their assignment Aug 8, 2022
@javanna javanna added the >docs General docs changes label Aug 25, 2022
@javanna javanna changed the title Lookup runtime sub-fields to throw error when queried or aggregated on Clarify that lookup runtime sub-fields can't be queried nor aggregated on Aug 25, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@elasticsearchmachine elasticsearchmachine added the Team:Docs Meta label for docs team label Aug 25, 2022
@abdonpijpelink abdonpijpelink self-assigned this Sep 5, 2022
@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>docs General docs changes >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Docs Meta label for docs team Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
5 participants