Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove on-heap fielddata. #64612

Closed
jtibshirani opened this issue Nov 4, 2020 · 4 comments
Closed

Remove on-heap fielddata. #64612

jtibshirani opened this issue Nov 4, 2020 · 4 comments
Labels
:Analytics/Aggregations Aggregations >enhancement :Search/Search Search-related issues that do not fall into other categories Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team

Comments

@jtibshirani
Copy link
Contributor

jtibshirani commented Nov 4, 2020

Fielddata is a collection of on-heap data structures that support aggregating and sorting on certain field types. Since fielddata is expensive to build, it's held in a dedicated cache. The cache can consume significant heap memory and must be limited by a circuit breaker. Fielddata is a source of complexity and can make Elasticsearch harder to manage.

We'd like to explore removing the concept of on-heap fielddata entirely. This would include the following:

  • Remove the need for on-heap global ordinals. This could involve moving global ordinals off-heap, or reducing the need for ordinals in aggregations. (needs research)
  • Remove support for fielddata on _id. (Remove support for fielddata loading on _id. #64511)
  • Remove support for fielddata on text fields. (needs research)

This meta issue is meant to clarify the plans around fielddata. Some individual changes require more research -- we'll link to sub-issues when there are more details.

@jtibshirani jtibshirani added >enhancement :Analytics/Aggregations Aggregations :Search/Search Search-related issues that do not fall into other categories labels Nov 4, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

@elasticmachine elasticmachine added Team:Search Meta label for search team Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Nov 4, 2020
@nik9000
Copy link
Member

nik9000 commented Nov 17, 2020

* reducing the need for ordinals in aggregations

We keep bumping into more and more cases where it is useful to have global ordinals, sadly. I think we're stuck getting them off the heap.

@javanna
Copy link
Member

javanna commented Jun 17, 2024

We have no concrete plans to tackle this issue, hence I am closing it.

@javanna javanna closed this as not planned Won't fix, can't repro, duplicate, stale Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Aggregations Aggregations >enhancement :Search/Search Search-related issues that do not fall into other categories Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team
Projects
None yet
Development

No branches or pull requests

4 participants