Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[stacked] Proof of concept retaining endpoint concurrency limits on refresh #2418

Draft
wants to merge 1 commit into
base: blaub/retain-state
Choose a base branch
from
Draft

[stacked] Proof of concept retaining endpoint concurrency limits on refresh #2418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions dialogue-core/src/main/java/com/palantir/dialogue/core/ConcurrencyLimitedChannel.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,12 @@ final class ConcurrencyLimitedChannel implements LimitedChannel {
private static final ChannelState.Key<CautiousIncreaseAggressiveDecreaseConcurrencyLimiter>
HOST_SPECIFIC_STATE_KEY = new ChannelState.Key<>(
CautiousIncreaseAggressiveDecreaseConcurrencyLimiter.class,
ConcurrencyLimitedChannel::createHostSpecificState);
() -> new CautiousIncreaseAggressiveDecreaseConcurrencyLimiter(Behavior.HOST_LEVEL));

private static final ChannelState.Key<CautiousIncreaseAggressiveDecreaseConcurrencyLimiter>
ENDPOINT_SPECIFIC_STATE_KEY = new ChannelState.Key<>(
CautiousIncreaseAggressiveDecreaseConcurrencyLimiter.class,
() -> new CautiousIncreaseAggressiveDecreaseConcurrencyLimiter(Behavior.ENDPOINT_LEVEL));

private final NeverThrowChannel delegate;
private final CautiousIncreaseAggressiveDecreaseConcurrencyLimiter limiter;
Expand All @@ -61,10 +66,11 @@ static LimitedChannel createForHost(Config cf, Channel channel, int uriIndex, Ch
* Creates a concurrency limited channel for per-endpoint limiting.
* Metrics are not reported by this component per-endpoint, only by the per-endpoint queue.
*/
static LimitedChannel createForEndpoint(Channel channel, String channelName, int uriIndex, Endpoint endpoint) {
static LimitedChannel createForEndpoint(
Channel channel, String channelName, int uriIndex, Endpoint endpoint, ChannelState endpointChannelState) {
return new ConcurrencyLimitedChannel(
channel,
new CautiousIncreaseAggressiveDecreaseConcurrencyLimiter(Behavior.ENDPOINT_LEVEL),
endpointChannelState.getState(ENDPOINT_SPECIFIC_STATE_KEY),
new EndpointConcurrencyLimitedChannelInstrumentation(channelName, uriIndex, endpoint));
}

Expand All @@ -77,10 +83,6 @@ static LimitedChannel createForEndpoint(Channel channel, String channelName, int
this.channelNameForLogging = instrumentation.channelNameForLogging();
}

static CautiousIncreaseAggressiveDecreaseConcurrencyLimiter createHostSpecificState() {
return new CautiousIncreaseAggressiveDecreaseConcurrencyLimiter(Behavior.HOST_LEVEL);
}

@Override
public Optional<ListenableFuture<Response>> maybeExecute(
Endpoint endpoint, Request request, LimitEnforcement limitEnforcement) {
Expand Down
36 changes: 35 additions & 1 deletion dialogue-core/src/main/java/com/palantir/dialogue/core/DialogueChannel.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
package com.palantir.dialogue.core;

import com.codahale.metrics.Meter;
import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.LoadingCache;
import com.github.benmanes.caffeine.cache.Ticker;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
Expand Down Expand Up @@ -252,12 +254,17 @@ private static ImmutableList<LimitedChannel> createHostChannels(
LimitedChannel limitedChannel;
if (cf.isConcurrencyLimitingEnabled()) {
Channel unlimited = channel;
EndpointChannelState endpointChannelState = channelState.getState(EndpointChannelState.KEY);
channel = new ChannelToEndpointChannel(endpoint -> {
if (endpoint.tags().contains("dialogue-disable-endpoint-concurrency-limiting")) {
return unlimited;
}
LimitedChannel limited = ConcurrencyLimitedChannel.createForEndpoint(
unlimited, cf.channelName(), uriIndexForInstrumentation, endpoint);
unlimited,
cf.channelName(),
uriIndexForInstrumentation,
endpoint,
endpointChannelState.get(endpoint));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will fail in an edge case where the endpoint limiter is completely full from a previous refresh, so the queued channel created on the next line cannot sense the boundary condition to begin allowing requests through

return QueuedChannel.create(cf, endpoint, limited);
});
limitedChannel = ConcurrencyLimitedChannel.createForHost(
Expand All @@ -271,6 +278,33 @@ private static ImmutableList<LimitedChannel> createHostChannels(
return perUriChannels.build();
}

/**
* {@link ChannelState} provider for per-endpoint channels like the endpoint concurrency limiter.
* This object is held in the per-host state, and can be used to look up a {@link ChannelState}
* scoped to an individual {@link Endpoint}.
* {@link Endpoint} state is held in a weak-keyed cache, equivalent to the one used in
* {@link ChannelToEndpointChannel}.
* {@link Endpoint} objects are usually enums, which will never be garbage collected, however it's possible
* that callers may build an endpoint instances on a per-call basis, so the weak-keyed map is defensive
* against short-lived endpoints.
* We don't use the same map because the {@link ChannelToEndpointChannel} retains full channel state which
* may or may not be designed to be reused across reloads, and we aim to be more precise with state that is
* kept across uri changes.
*/
private record EndpointChannelState(LoadingCache<Endpoint, ChannelState> cache) {
private static final ChannelState.Key<EndpointChannelState> KEY =
new ChannelState.Key<>(EndpointChannelState.class, EndpointChannelState::create);

ChannelState get(Endpoint endpoint) {
return cache.get(endpoint);
}

private static EndpointChannelState create() {
return new EndpointChannelState(
Caffeine.newBuilder().weakKeys().maximumSize(10_000).build(_key -> new ChannelState()));
}
}

private static EndpointChannelFactory createEndpointChannelFactory(Channel multiHostQueuedChannel, Config cf) {
Channel queuedChannel = new QueueOverrideChannel(multiHostQueuedChannel);
return endpoint -> {
Expand Down
Loading