-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deduplicate RepositoryData on a Best Effort Basis #67947
Changes from all commits
9dc8f8c
c58dc1c
130216c
19d2169
8e0c76f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -121,6 +121,7 @@ | |
import org.elasticsearch.snapshots.SnapshotMissingException; | ||
import org.elasticsearch.snapshots.SnapshotsService; | ||
import org.elasticsearch.threadpool.ThreadPool; | ||
import org.elasticsearch.action.ResultDeduplicator; | ||
|
||
import java.io.FilterInputStream; | ||
import java.io.IOException; | ||
|
@@ -1361,11 +1362,7 @@ public void getRepositoryData(ActionListener<RepositoryData> listener) { | |
// Fast path loading repository data directly from cache if we're in fully consistent mode and the cache matches up with | ||
// the latest known repository generation | ||
if (bestEffortConsistency == false && cached.generation() == latestKnownRepoGen.get() && cached.hasData()) { | ||
try { | ||
listener.onResponse(cached.repositoryData()); | ||
} catch (Exception e) { | ||
listener.onFailure(e); | ||
} | ||
repoDataDeduplicator.executeOnce(metadata, listener, (metadata, l) -> l.onResponse(cached.repositoryData())); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a little less significant in savings relative to the other spot but I think it's worth it.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I fail to see how this change could act as a rate limit for that scenario, it's a good improvement but the requests would accumulate anyway, right? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yea, I guess "rate-limiter" was a poor choice of wording. I guess we "rate limit" the rate at which we dispatch the actions after loading repo data to a single thread at a time but requests still pile up. In practice that probably means that things run much quicker than before effectively but with requests queuing and waiting for the first one to finish instead of executing in parallel it's more of a "thread-limiter" I suppose :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That makes sense, thanks for the clarification! |
||
return; | ||
} | ||
if (metadata.generation() == RepositoryData.UNKNOWN_REPO_GEN && isReadOnly() == false) { | ||
|
@@ -1375,7 +1372,12 @@ public void getRepositoryData(ActionListener<RepositoryData> listener) { | |
} else { | ||
logger.trace("[{}] loading un-cached repository data with best known repository generation [{}]", metadata.name(), | ||
latestKnownRepoGen); | ||
threadPool.generic().execute(ActionRunnable.wrap(listener, this::doGetRepositoryData)); | ||
if (bestEffortConsistency) { | ||
threadPool.generic().execute(ActionRunnable.wrap(listener, this::doGetRepositoryData)); | ||
} else { | ||
repoDataDeduplicator.executeOnce(metadata, listener, (metadata, l) -> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This would completely resolve the broken situations we observed where there were 50+ concurrent GENERIC threads fetching |
||
threadPool.generic().execute(ActionRunnable.wrap(l, this::doGetRepositoryData))); | ||
} | ||
} | ||
} | ||
|
||
|
@@ -1461,10 +1463,18 @@ public void clusterStateProcessed(String source, ClusterState oldState, ClusterS | |
} | ||
} | ||
|
||
/** | ||
* {@link RepositoryData} loading deduplicator. This may only be used with consistent generation repositories, meaning | ||
* {@link #bestEffortConsistency} must be {@code false}, in which case we can assume that the {@link RepositoryData} loaded is | ||
* unique for a given value of {@link #metadata} at any point in time. | ||
*/ | ||
private final ResultDeduplicator<RepositoryMetadata, RepositoryData> repoDataDeduplicator = new ResultDeduplicator<>(); | ||
|
||
private void doGetRepositoryData(ActionListener<RepositoryData> listener) { | ||
// Retry loading RepositoryData in a loop in case we run into concurrent modifications of the repository. | ||
// Keep track of the most recent generation we failed to load so we can break out of the loop if we fail to load the same | ||
// generation repeatedly. | ||
|
||
long lastFailedGeneration = RepositoryData.UNKNOWN_REPO_GEN; | ||
while (true) { | ||
final long genToLoad; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I audited all usages of this method and I couldn't find any remaining spot where this is a problem (at least in
master
, might need some adjustments in7.x
).