Record timestamp field range in index metadata #65689
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DaveCTurner
added
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
DaveCTurner
force-pushed
the
2020-12-01-record-timestamp-field-range-in-index-metadata-7x
branch
from
d93986d to
96eadfb
Compare
Queries including a filter by timestamp range are common in time-series data. Moreover older time-series indices are typically made read-only so that the timestamp range becomes immutable. By recording in the index metadata the range of timestamps covered by each index we can very efficiently skip shards on the coordinating node, even if those shards are not assigned. This commit computes the timestamp range of immutable indices and records it in the index metadata as the shards start for the first time. Note that the only indices it considers immutable today are ones using the `ReadOnlyEngine`, which includes frozen indices and searchable snapshots but not regular indices with a write block. Backport of elastic#65564 and elastic#65720
DaveCTurner
force-pushed
the
2020-12-01-record-timestamp-field-range-in-index-metadata-7x
branch
from
96eadfb to
2c8957c
Compare
|
D'oh I didn't realise I already opened a PR for this branch, sorry for the force-pushing. |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Dec 2, 2020
DaveCTurner
deleted the
2020-12-01-record-timestamp-field-range-in-index-metadata-7x
branch
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Dec 2, 2020
Relates elastic#65564 Relates elastic#65689
DaveCTurner
added a commit
that referenced
this pull request
Dec 2, 2020
fcofdez
added a commit
to fcofdez/elasticsearch
that referenced
this pull request
Dec 9, 2020
This commit introduces an optimization that allows skipping shardsthat are not necessary directly on the coordinator for time based indices. This is possible for frozen and searchable snapshots since those store their min/max timestamp range in their IndexMetadata (introduced in elastic#65689). For indices that don't have that information available, the behaviour is the sameas it used to be.
fcofdez
added a commit
that referenced
this pull request
Dec 15, 2020
…vailable at the coordinator. (#65583) This commit introduces an optimization that allows skipping shards that are not necessary directly on the coordinator for time based indices. This is possible for frozen and searchable snapshots since those store their min/max timestamp range in their IndexMetadata (introduced in #65689). For indices that don't have that information available, the behaviour is the same as it used to be.
fcofdez
added a commit
to fcofdez/elasticsearch
that referenced
this pull request
Dec 15, 2020
…vailable at the coordinator. This commit introduces an optimization that allows skipping shards that are not necessary directly on the coordinator for time based indices. This is possible for frozen and searchable snapshots since those store their min/max timestamp range in their IndexMetadata (introduced in elastic#65689). For indices that don't have that information available, the behaviour is the same as it used to be. Backport of elastic#65583
fcofdez
added a commit
that referenced
this pull request
Dec 15, 2020
…a is available at the coordinator. (#66319) This commit introduces an optimization that allows skipping shards that are not necessary directly on the coordinator for time based indices. This is possible for frozen and searchable snapshots since those store their min/max timestamp range in their IndexMetadata (introduced in #65689). For indices that don't have that information available, the behaviour is the same as it used to be. Backport of #65583
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Queries including a filter by timestamp range are common in time-series
data. Moreover older time-series indices are typically made read-only so
that the timestamp range becomes immutable. By recording in the index
metadata the range of timestamps covered by each index we can very
efficiently skip shards on the coordinating node, even if those shards
are not assigned.
This commit computes the timestamp range of immutable indices and
records it in the index metadata as the shards start for the first time.
Note that the only indices it considers immutable today are ones using
the
ReadOnlyEngine, which includes frozen indices and searchablesnapshots but not regular indices with a write block.
Backport of #65564