Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds support for global circuit budget #6013

Merged
merged 18 commits into from
Jan 2, 2024
+1,017 −3
Merged

Adds support for global circuit budget #6013

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
46629af
Adds support for global circuit budget
davinci26 Dec 12, 2023
589633f
release note
davinci26 Dec 12, 2023
4237680
adds test and a docstring
davinci26 Dec 12, 2023
c42a27a
test metrics
davinci26 Dec 13, 2023
6c7e736
remove reddit parameters
davinci26 Dec 13, 2023
bdf1de3
PR comments
davinci26 Dec 15, 2023
8e354c4
revert change
davinci26 Dec 15, 2023
337431c
regenerate
davinci26 Dec 15, 2023
4151035
fixes pr comments
davinci26 Dec 18, 2023
47df638
Merge remote-tracking branch 'upstream/main' into retryBudget
davinci26 Dec 20, 2023
567d012
update metrics docs
davinci26 Dec 21, 2023
4f4825b
kick ci
davinci26 Dec 21, 2023
509d1ac
revert metric change
davinci26 Dec 21, 2023
f9edfc7
remove metrics part 2
davinci26 Dec 21, 2023
378cee4
remove metrics part 3
davinci26 Dec 24, 2023
5531f35
ws change
davinci26 Dec 24, 2023
5a2ada7
kick ci
davinci26 Dec 25, 2023
e4a5cbb
kick ci 2
davinci26 Dec 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions apis/projectcontour/v1alpha1/contourconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,21 @@ const (
EnvoyServerType XDSServerType = "envoy"
)

type GlobalCircuitBreakerDefaults struct {
// The maximum number of connections that a single Envoy instance allows to the Kubernetes Service; defaults to 1024.
// +optional
davinci26 marked this conversation as resolved.
Show resolved Hide resolved
MaxConnections uint32 `json:"maxConnections,omitempty" yaml:"max-connections,omitempty"`
// The maximum number of pending requests that a single Envoy instance allows to the Kubernetes Service; defaults to 1024.
// +optional
MaxPendingRequests uint32 `json:"maxPendingRequests,omitempty" yaml:"max-pending-requests,omitempty"`
// The maximum parallel requests a single Envoy instance allows to the Kubernetes Service; defaults to 1024
// +optional
MaxRequests uint32 `json:"maxRequests,omitempty" yaml:"max-requests,omitempty"`
// The maximum number of parallel retries a single Envoy instance allows to the Kubernetes Service; defaults to 3.
// +optional
MaxRetries uint32 `json:"maxRetries,omitempty" yaml:"max-retries,omitempty"`
}

// XDSServerConfig holds the config for the Contour xDS server.
type XDSServerConfig struct {
// Defines the XDSServer to use for `contour serve`.
Expand Down Expand Up @@ -689,6 +704,12 @@ type ClusterParameters struct {
// +optional
PerConnectionBufferLimitBytes *uint32 `json:"per-connection-buffer-limit-bytes,omitempty"`

// GlobalCircuitBreakerDefaults specifies default circuit breaker budget across all services.
// If defined, this will be used as the default for all services.
//
// +optional
GlobalCircuitBreakerDefaults *GlobalCircuitBreakerDefaults `json:"circuitBreakers,omitempty"`

// UpstreamTLS contains the TLS policy parameters for upstream connections
//
// +optional
Expand Down
20 changes: 20 additions & 0 deletions apis/projectcontour/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions changelogs/unreleased/6013-davinci26-minor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Support for Global Circuit Breaker Policy

The way [circuit-breaker-annotations](https://projectcontour.io/docs/1.27/config/annotations/) work currently is that when not present they are being defaulted to Envoy defaults. The Envoy defaults can be quite low for larger clusters with more traffic so if a user accidentally deletes them or unset them this cause an issue. With this change we are providing contour administrators the ability to provide global defaults that are good. In that case even if the user forgets to set them or deletes them they can have the safety net of good defaults. They can be configured via [cluster.circuit-breakers](https://projectcontour.io/docs/1.28/configuration/#circuit-breakers) or via `ContourConfiguration`` CRD in [spec.envoy.cluster.circuitBreakers](https://projectcontour.io/docs/1.28/config/api/#projectcontour.io/v1alpha1.GlobalCircuitBreakerDefaults)
5 changes: 5 additions & 0 deletions cmd/contour/serve.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,7 @@
globalRateLimitService: contourConfiguration.RateLimitService,
maxRequestsPerConnection: contourConfiguration.Envoy.Cluster.MaxRequestsPerConnection,
perConnectionBufferLimitBytes: contourConfiguration.Envoy.Cluster.PerConnectionBufferLimitBytes,
globalCircuitBreakerDefaults: contourConfiguration.Envoy.Cluster.GlobalCircuitBreakerDefaults,

Check warning on line 563 in cmd/contour/serve.go

View check run for this annotation

Codecov / codecov/patch

cmd/contour/serve.go#L563 

Added line #L563 was not covered by tests
upstreamTLS: &dag.UpstreamTLS{
MinimumProtocolVersion: annotation.TLSVersion(contourConfiguration.Envoy.Cluster.UpstreamTLS.MinimumProtocolVersion, "1.2"),
MaximumProtocolVersion: annotation.TLSVersion(contourConfiguration.Envoy.Cluster.UpstreamTLS.MaximumProtocolVersion, "1.3"),
Expand Down Expand Up @@ -1122,6 +1123,7 @@
maxRequestsPerConnection *uint32
perConnectionBufferLimitBytes *uint32
globalRateLimitService *contour_api_v1alpha1.RateLimitServiceConfig
globalCircuitBreakerDefaults *contour_api_v1alpha1.GlobalCircuitBreakerDefaults
upstreamTLS *dag.UpstreamTLS
}

Expand Down Expand Up @@ -1192,6 +1194,7 @@
ConnectTimeout: dbc.connectTimeout,
MaxRequestsPerConnection: dbc.maxRequestsPerConnection,
PerConnectionBufferLimitBytes: dbc.perConnectionBufferLimitBytes,
GlobalCircuitBreakerDefaults: dbc.globalCircuitBreakerDefaults,
SetSourceMetadataOnRoutes: true,
UpstreamTLS: dbc.upstreamTLS,
},
Expand All @@ -1217,6 +1220,7 @@
GlobalRateLimitService: dbc.globalRateLimitService,
PerConnectionBufferLimitBytes: dbc.perConnectionBufferLimitBytes,
SetSourceMetadataOnRoutes: true,
GlobalCircuitBreakerDefaults: dbc.globalCircuitBreakerDefaults,
davinci26 marked this conversation as resolved.
Show resolved Hide resolved
davinci26 marked this conversation as resolved.
Show resolved Hide resolved
UpstreamTLS: dbc.upstreamTLS,
},
}
Expand All @@ -1229,6 +1233,7 @@
MaxRequestsPerConnection: dbc.maxRequestsPerConnection,
PerConnectionBufferLimitBytes: dbc.perConnectionBufferLimitBytes,
SetSourceMetadataOnRoutes: true,
GlobalCircuitBreakerDefaults: dbc.globalCircuitBreakerDefaults,
})
}

Expand Down
38 changes: 38 additions & 0 deletions cmd/contour/serve_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,31 @@ func TestGetDAGBuilder(t *testing.T) {
assert.ElementsMatch(t, map[string]string(nil), ingressProcessor.ResponseHeadersPolicy.Remove)
})

t.Run("GlobalCircuitBreakerDefaults specified for all processors", func(t *testing.T) {
g := contour_api_v1alpha1.GlobalCircuitBreakerDefaults{
MaxConnections: 100,
}

serve := &Server{
log: logrus.StandardLogger(),
}
got := serve.getDAGBuilder(dagBuilderConfig{
gatewayControllerName: "projectcontour.io/gateway-controller",
rootNamespaces: []string{},
dnsLookupFamily: contour_api_v1alpha1.AutoClusterDNSFamily,
globalCircuitBreakerDefaults: &g,
})

iProcessor := mustGetIngressProcessor(t, got)
assert.EqualValues(t, iProcessor.GlobalCircuitBreakerDefaults, &g)

hProcessor := mustGetHTTPProxyProcessor(t, got)
assert.EqualValues(t, hProcessor.GlobalCircuitBreakerDefaults, &g)

gProcessor := mustGetGatewayAPIProcessor(t, got)
assert.EqualValues(t, gProcessor.GlobalCircuitBreakerDefaults, &g)
})

t.Run("request and response headers policy specified for ingress", func(t *testing.T) {
policy := &contour_api_v1alpha1.PolicyConfig{
RequestHeadersPolicy: &contour_api_v1alpha1.HeadersPolicy{
Expand Down Expand Up @@ -192,6 +217,19 @@ func TestGetDAGBuilder(t *testing.T) {
// TODO(3453): test additional properties of the DAG builder (processor fields, cache fields, Gateway tests (requires a client fake))
}

func mustGetGatewayAPIProcessor(t *testing.T, builder *dag.Builder) *dag.GatewayAPIProcessor {
t.Helper()
for i := range builder.Processors {
found, ok := builder.Processors[i].(*dag.GatewayAPIProcessor)
if ok {
return found
}
}

require.FailNow(t, "GatewayAPIProcessor not found in list of DAG builder's processors")
return nil
}

func mustGetHTTPProxyProcessor(t *testing.T, builder *dag.Builder) *dag.HTTPProxyProcessor {
t.Helper()
for i := range builder.Processors {
Expand Down
1 change: 1 addition & 0 deletions cmd/contour/servecontext.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,6 +572,7 @@ func (ctx *serveContext) convertToContourConfigurationSpec() contour_api_v1alpha
DNSLookupFamily: dnsLookupFamily,
MaxRequestsPerConnection: ctx.Config.Cluster.MaxRequestsPerConnection,
PerConnectionBufferLimitBytes: ctx.Config.Cluster.PerConnectionBufferLimitBytes,
GlobalCircuitBreakerDefaults: ctx.Config.Cluster.GlobalCircuitBreakerDefaults,
UpstreamTLS: &contour_api_v1alpha1.EnvoyTLS{
MinimumProtocolVersion: ctx.Config.Cluster.UpstreamTLS.MinimumProtocolVersion,
MaximumProtocolVersion: ctx.Config.Cluster.UpstreamTLS.MaximumProtocolVersion,
Expand Down
23 changes: 22 additions & 1 deletion cmd/contour/servecontext_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,8 @@ func TestConvertServeContext(t *testing.T) {
ConnectTimeout: ref.To("2s"),
},
Cluster: &contour_api_v1alpha1.ClusterParameters{
DNSLookupFamily: contour_api_v1alpha1.AutoClusterDNSFamily,
DNSLookupFamily: contour_api_v1alpha1.AutoClusterDNSFamily,
GlobalCircuitBreakerDefaults: nil,
UpstreamTLS: &contour_api_v1alpha1.EnvoyTLS{
MinimumProtocolVersion: "",
MaximumProtocolVersion: "",
Expand Down Expand Up @@ -776,6 +777,26 @@ func TestConvertServeContext(t *testing.T) {
return cfg
},
},
"global circuit breaker defaults": {
getServeContext: func(ctx *serveContext) *serveContext {
ctx.Config.Cluster.GlobalCircuitBreakerDefaults = &contour_api_v1alpha1.GlobalCircuitBreakerDefaults{
MaxConnections: 4,
MaxPendingRequests: 5,
MaxRequests: 6,
MaxRetries: 7,
}
return ctx
},
getContourConfiguration: func(cfg contour_api_v1alpha1.ContourConfigurationSpec) contour_api_v1alpha1.ContourConfigurationSpec {
cfg.Envoy.Cluster.GlobalCircuitBreakerDefaults = &contour_api_v1alpha1.GlobalCircuitBreakerDefaults{
MaxConnections: 4,
MaxPendingRequests: 5,
MaxRequests: 6,
MaxRetries: 7,
}
return cfg
},
},
"global external authorization": {
getServeContext: func(ctx *serveContext) *serveContext {
ctx.Config.GlobalExternalAuthorization = config.GlobalExternalAuthorization{
Expand Down
60 changes: 60 additions & 0 deletions examples/contour/01-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,36 @@ spec:
description: Cluster holds various configurable Envoy cluster
values that can be set in the config file.
properties:
circuitBreakers:
description: GlobalCircuitBreakerDefaults specifies default
circuit breaker budget across all services. If defined,
this will be used as the default for all services.
properties:
maxConnections:
description: The maximum number of connections that a
single Envoy instance allows to the Kubernetes Service;
defaults to 1024.
format: int32
type: integer
maxPendingRequests:
description: The maximum number of pending requests that
a single Envoy instance allows to the Kubernetes Service;
defaults to 1024.
format: int32
type: integer
maxRequests:
description: The maximum parallel requests a single Envoy
instance allows to the Kubernetes Service; defaults
to 1024
format: int32
type: integer
maxRetries:
description: The maximum number of parallel retries a
single Envoy instance allows to the Kubernetes Service;
defaults to 3.
format: int32
type: integer
type: object
dnsLookupFamily:
description: "DNSLookupFamily defines how external names are
looked up When configured as V4, the DNS resolver will only
Expand Down Expand Up @@ -3684,6 +3714,36 @@ spec:
description: Cluster holds various configurable Envoy cluster
values that can be set in the config file.
properties:
circuitBreakers:
description: GlobalCircuitBreakerDefaults specifies default
circuit breaker budget across all services. If defined,
this will be used as the default for all services.
properties:
maxConnections:
description: The maximum number of connections that
a single Envoy instance allows to the Kubernetes
Service; defaults to 1024.
format: int32
type: integer
maxPendingRequests:
description: The maximum number of pending requests
that a single Envoy instance allows to the Kubernetes
Service; defaults to 1024.
format: int32
type: integer
maxRequests:
description: The maximum parallel requests a single
Envoy instance allows to the Kubernetes Service;
defaults to 1024
format: int32
type: integer
maxRetries:
description: The maximum number of parallel retries
a single Envoy instance allows to the Kubernetes
Service; defaults to 3.
format: int32
type: integer
type: object
dnsLookupFamily:
description: "DNSLookupFamily defines how external names
are looked up When configured as V4, the DNS resolver
Expand Down
60 changes: 60 additions & 0 deletions examples/render/contour-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,36 @@ spec:
description: Cluster holds various configurable Envoy cluster
values that can be set in the config file.
properties:
circuitBreakers:
description: GlobalCircuitBreakerDefaults specifies default
circuit breaker budget across all services. If defined,
this will be used as the default for all services.
properties:
maxConnections:
description: The maximum number of connections that a
single Envoy instance allows to the Kubernetes Service;
defaults to 1024.
format: int32
type: integer
maxPendingRequests:
description: The maximum number of pending requests that
a single Envoy instance allows to the Kubernetes Service;
defaults to 1024.
format: int32
type: integer
maxRequests:
description: The maximum parallel requests a single Envoy
instance allows to the Kubernetes Service; defaults
to 1024
format: int32
type: integer
maxRetries:
description: The maximum number of parallel retries a
single Envoy instance allows to the Kubernetes Service;
defaults to 3.
format: int32
type: integer
type: object
dnsLookupFamily:
description: "DNSLookupFamily defines how external names are
looked up When configured as V4, the DNS resolver will only
Expand Down Expand Up @@ -3903,6 +3933,36 @@ spec:
description: Cluster holds various configurable Envoy cluster
values that can be set in the config file.
properties:
circuitBreakers:
description: GlobalCircuitBreakerDefaults specifies default
circuit breaker budget across all services. If defined,
this will be used as the default for all services.
properties:
maxConnections:
description: The maximum number of connections that
a single Envoy instance allows to the Kubernetes
Service; defaults to 1024.
format: int32
type: integer
maxPendingRequests:
description: The maximum number of pending requests
that a single Envoy instance allows to the Kubernetes
Service; defaults to 1024.
format: int32
type: integer
maxRequests:
description: The maximum parallel requests a single
Envoy instance allows to the Kubernetes Service;
defaults to 1024
format: int32
type: integer
maxRetries:
description: The maximum number of parallel retries
a single Envoy instance allows to the Kubernetes
Service; defaults to 3.
format: int32
type: integer
type: object
dnsLookupFamily:
description: "DNSLookupFamily defines how external names
are looked up When configured as V4, the DNS resolver
Expand Down
Loading