Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New queryable "_tier" metadata field #69288

Merged
merged 2 commits into from
Mar 31, 2021
Merged

New queryable "_tier" metadata field #69288

merged 2 commits into from
Mar 31, 2021

Conversation

markharwood
Copy link
Contributor

New metadata field, like the "_index" field that supports term, terms and wildcard queries on the roles defined (explicitly or implicitly)for a node.

Will add tests if we think this is the way forward

Closes #68135

@markharwood markharwood self-assigned this Feb 19, 2021
@markharwood markharwood added :Search Foundations/Mapping Index mappings, including merging and defining field types >new-field-mapper Added when a new field type / mapper is being introduced labels Feb 19, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Feb 19, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@markharwood markharwood force-pushed the fix/68135 branch 2 times, most recently from 2a6fedc to e86181e Compare February 22, 2021 13:41
@markharwood markharwood force-pushed the fix/68135 branch 2 times, most recently from b5570c7 to d5c838d Compare February 23, 2021 14:44
@markharwood
Copy link
Contributor Author

@jpountz I commented out the familyTypeName support for now - was breaking some SQL test expectations

@henningandersen
Copy link
Contributor

I wonder if the approach taken here allows for sorting by "_tier" too? If so, it would be sort of nice to let the order be "content", "hot", "warm", "cold", "frozen", regardless of alphabetical ordering. Not sure I have a killer case for this though.

@markharwood
Copy link
Contributor Author

This PR has two dependencies:

  1. sorting/aggregations - can't sort or aggregate until ConstantFieldType.fielddataBuilder() is changed to pass a SearchExecutionContext object (or a new interface that also spans AggregationExecutionContext)
  2. some work Jim is doing on stripping metadata fields like _tier from the field caps API (we suspect this is behind the SQL test failures currently on this PR).

With regards to 1) I suspect we can merge this PR minus sort/aggs support and save for another issue.

jimczi
jimczi reviewed Mar 5, 2021
View reviewed changes
Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments but the logic looks good to me.

...ta-streams/src/main/java/org/elasticsearch/xpack/datastreams/mapper/DataTierFieldMapper.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/index/mapper/ConstantFieldType.java Outdated Show resolved Hide resolved
...ta-streams/src/main/java/org/elasticsearch/xpack/datastreams/mapper/DataTierFieldMapper.java Outdated Show resolved Hide resolved
...ta-streams/src/main/java/org/elasticsearch/xpack/datastreams/mapper/DataTierFieldMapper.java Outdated Show resolved Hide resolved
...ta-streams/src/main/java/org/elasticsearch/xpack/datastreams/mapper/DataTierFieldMapper.java Outdated Show resolved Hide resolved
@markharwood markharwood force-pushed the fix/68135 branch 3 times, most recently from 75b416b to dca2ded Compare March 10, 2021 10:08
@dakrone dakrone mentioned this pull request Mar 10, 2021
Closed
jimczi
jimczi approved these changes Mar 31, 2021
View reviewed changes
Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left minor comments, LGTM otherwise.

...main/java/org/elasticsearch/xpack/cluster/routing/allocation/mapper/DataTierFieldMapper.java Show resolved Hide resolved
...main/java/org/elasticsearch/xpack/cluster/routing/allocation/mapper/DataTierFieldMapper.java Show resolved Hide resolved
@markharwood markharwood merged commit 3aee4c1 into elastic:master Mar 31, 2021
@spalger spalger mentioned this pull request Mar 31, 2021
Closed
@alisonelizabeth alisonelizabeth mentioned this pull request Apr 1, 2021
Merged
markharwood added a commit that referenced this pull request Apr 7, 2021
@markharwood
New queryable "_tier" metadata field (#69288) (#71123)
754cda8 
* New _tier metadata field that supports term, terms, exists and wildcard queries on the first data tier preference stated for an index.

Backport of 3aee4c1

Closes #68135
@stevejgordon stevejgordon mentioned this pull request Apr 21, 2021
Closed
62 tasks
@droberts195 droberts195 mentioned this pull request Nov 5, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@henningandersen henningandersen henningandersen left review comments

@cbuescher cbuescher cbuescher left review comments

@jpountz jpountz jpountz left review comments

@jimczi jimczi jimczi approved these changes

Assignees

@markharwood markharwood

Labels
>enhancement >new-field-mapper Added when a new field type / mapper is being introduced :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team v7.13.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add ability to filter query based on data tier
7 participants
@markharwood @elasticmachine @henningandersen @jpountz @cbuescher @jimczi @jakelandis