[Profiling] Query in parallel only if beneficial #103061
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this commit we check index allocation before we do key-value lookups. To reduce latency, key-value lookups are done in parallel for multiple slices of data. However, on nodes with spinning disks, parallel accesses are harmful. Therefore, we check whether any index is allocated either to the warm or cold tier (which are usually on spinning disks) and disable parallel key-value lookups. This has improved latency on the warm tier by about 10% in our experiments.
danielmitterdorfer
added
>bug
auto-backport-and-merge
:UniversalProfiling/Application
elasticsearchmachine
added
the
Team:obs-knowledge
|
Pinging @elastic/obs-knowledge-team (Team:obs-knowledge) |
|
Hi @danielmitterdorfer, I've created a changelog YAML for you. |
|
@elasticmachine merge upstream |
rockdaboot
reviewed
Dec 7, 2023
| ClusterState clusterState = clusterService.state(); | ||
| List<Index> indices = resolver.resolve(clusterState, "profiling-stacktraces", responseBuilder.getStart(), responseBuilder.getEnd()); | ||
| // Avoid parallelism if there is potential we are on spinning disks (frozen tier uses searchable snapshots) | ||
| int sliceCount = IndexAllocation.isAnyOnWarmOrColdTier(clusterState, indices) ? 1 : desiredSlices; |
There was a problem hiding this comment.
Just a question. Did you test with 2 slices on the warm tier as well? If yes, what was the ~ impact?
There was a problem hiding this comment.
I just dug through my notes. I believe I did such a test but the superior alternative in all cases was just using a single slice (apparently I did not even bother to write down the results for anything except the default value of 16 and a single slice...).
💔 Backport failedThe backport operation could not be completed due to the following error: You can use sqren/backport to manually backport by running |
danielmitterdorfer
added a commit
to danielmitterdorfer/elasticsearch
that referenced
this pull request
Dec 7, 2023
With this commit we check index allocation before we do key-value lookups. To reduce latency, key-value lookups are done in parallel for multiple slices of data. However, on nodes with spinning disks, parallel accesses are harmful. Therefore, we check whether any index is allocated either to the warm or cold tier (which are usually on spinning disks) and disable parallel key-value lookups. This has improved latency on the warm tier by about 10% in our experiments.
|
I'm manually backporting this in #103144. |
elasticsearchmachine
pushed a commit
that referenced
this pull request
Dec 8, 2023
With this commit we check index allocation before we do key-value lookups. To reduce latency, key-value lookups are done in parallel for multiple slices of data. However, on nodes with spinning disks, parallel accesses are harmful. Therefore, we check whether any index is allocated either to the warm or cold tier (which are usually on spinning disks) and disable parallel key-value lookups. This has improved latency on the warm tier by about 10% in our experiments. Co-authored-by: Elastic Machine <[email protected]>
danielmitterdorfer
added a commit
to danielmitterdorfer/elasticsearch
that referenced
this pull request
Jan 22, 2024
In order to take advantage of inherent parallelism of modern SSDs, we slice keys and issue multiple mgets concurrently. In elastic#103061 we have introduced an additional heuristic to disable that behavior on the warm and cold tier which usually use spinning disks. We have unintentionally also disabled the behavior on content nodes, i.e. on any clusters which do not use data tiers. With this commit we explicitly exclude content nodes from the heuristic so they can benefit from speedups due to concurrent mgets. Relates elastic#103061
danielmitterdorfer
added a commit
that referenced
this pull request
Jan 22, 2024
In order to take advantage of inherent parallelism of modern SSDs, we slice keys and issue multiple mgets concurrently. In #103061 we have introduced an additional heuristic to disable that behavior on the warm and cold tier which usually use spinning disks. We have unintentionally also disabled the behavior on content nodes, i.e. on any clusters which do not use data tiers. With this commit we explicitly exclude content nodes from the heuristic so they can benefit from speedups due to concurrent mgets. Relates #103061
danielmitterdorfer
added a commit
to danielmitterdorfer/elasticsearch
that referenced
this pull request
Jan 22, 2024
In order to take advantage of inherent parallelism of modern SSDs, we slice keys and issue multiple mgets concurrently. In elastic#103061 we have introduced an additional heuristic to disable that behavior on the warm and cold tier which usually use spinning disks. We have unintentionally also disabled the behavior on content nodes, i.e. on any clusters which do not use data tiers. With this commit we explicitly exclude content nodes from the heuristic so they can benefit from speedups due to concurrent mgets. Relates elastic#103061
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jan 22, 2024
In order to take advantage of inherent parallelism of modern SSDs, we slice keys and issue multiple mgets concurrently. In #103061 we have introduced an additional heuristic to disable that behavior on the warm and cold tier which usually use spinning disks. We have unintentionally also disabled the behavior on content nodes, i.e. on any clusters which do not use data tiers. With this commit we explicitly exclude content nodes from the heuristic so they can benefit from speedups due to concurrent mgets. Relates #103061
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this commit we check index allocation before we do key-value lookups. To reduce latency, key-value lookups are done in parallel for multiple slices of data. However, on nodes with spinning disks, parallel accesses are harmful. Therefore, we check whether any index is allocated either to the warm or cold tier (which are usually on spinning disks) and disable parallel key-value lookups. This has improved latency on the warm tier by about 10% in our experiments.