Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose Horizontal Pod Autoscaler Behavior and add hpa scaledown test #1077

Merged
merged 17 commits into from
Sep 15, 2022
Merged

Expose Horizontal Pod Autoscaler Behavior and add hpa scaledown test #1077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f4abf28
Add scaledown test for autoscaling
kevinearls Sep 6, 2022
7b51fee
Fix nits
kevinearls Sep 6, 2022
43710b9
Appease the linter
kevinearls Sep 6, 2022
7070f4b
Don't use a default, only set scaleUp/scaleDown if they are in the CR
kevinearls Sep 6, 2022
ae83264
Removed commented out code
kevinearls Sep 6, 2022
19154be
Change defaults for scaleUp/scaleDown
kevinearls Sep 6, 2022
108d0f1
Update autoscaling scaleup/scaledown
kevinearls Sep 8, 2022
037aa23
Run generate
kevinearls Sep 8, 2022
bdc6eb3
Ran make api-docs
kevinearls Sep 8, 2022
60cdfe7
Update HPA implementation to embed HorizontalPodAutoscalerBehavior
kevinearls Sep 15, 2022
54ff0d3
Merge branch 'main' into add-hpa-scaledown-test
kevinearls Sep 15, 2022
bcf71e7
Only set behavior if it exists in the collector CR
kevinearls Sep 15, 2022
0f8e029
Aded some unit tests
kevinearls Sep 15, 2022
a42084c
Merge branch 'main' into add-hpa-scaledown-test
kevinearls Sep 15, 2022
f3a682c
Merge branch 'main' into add-hpa-scaledown-test
kevinearls Sep 15, 2022
c14096d
Add kuttl assertion that hpa scaled down
kevinearls Sep 15, 2022
89a2f48
added whitespace to rerun tests
kevinearls Sep 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions apis/v1alpha1/opentelemetrycollector_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
package v1alpha1

import (
autoscalingv2 "k8s.io/api/autoscaling/v2"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
Expand All @@ -40,6 +41,13 @@ type OpenTelemetryCollectorSpec struct {
// MaxReplicas sets an upper bound to the autoscaling feature. If MaxReplicas is set autoscaling is enabled.
// +optional
MaxReplicas *int32 `json:"maxReplicas,omitempty"`

// Autoscaler specifies the pod autoscaling configuration to use
// for the OpenTelemetryCollector workload.
//
// +optional
Autoscaler *AutoscalerSpec `json:"autoscaler,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: did we consider embedding the HPA spec in here? Or at least embed the autoscaling behavior spec here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just adding enough code to be able to get an e2e test to work within the time allocated, which means we need to scale down much quicker that the default 300 seconds.

@pavolloffay what do you think? Do I need to add the other values of PA scaling rules here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my use case, I know that I would like to be able to specify policies and not just StabilizationWindowSeconds. Embedding only StabilizationWindowSeconds is going to make it so we need to add in each feature on request, making a code change for each one.


// SecurityContext will be set as the container security context.
// +optional
SecurityContext *v1.SecurityContext `json:"securityContext,omitempty"`
Expand Down Expand Up @@ -197,6 +205,12 @@ type OpenTelemetryCollectorList struct {
Items []OpenTelemetryCollector `json:"items"`
}

// AutoscalerSpec defines the OpenTelemetryCollector's pod autoscaling specification.
type AutoscalerSpec struct {
// +optional
Behavior *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty"`
}

func init() {
SchemeBuilder.Register(&OpenTelemetryCollector{}, &OpenTelemetryCollectorList{})
}
9 changes: 9 additions & 0 deletions apis/v1alpha1/opentelemetrycollector_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,15 @@ func (r *OpenTelemetryCollector) validateCRDSpec() error {
return fmt.Errorf("the OpenTelemetry Spec autoscale configuration is incorrect, minReplicas should be one or more")
}

if r.Spec.Autoscaler != nil && r.Spec.Autoscaler.Behavior != nil {
if r.Spec.Autoscaler.Behavior.ScaleDown != nil && *r.Spec.Autoscaler.Behavior.ScaleDown.StabilizationWindowSeconds < int32(1) {
return fmt.Errorf("the OpenTelemetry Spec autoscale configuration is incorrect, scaleDown should be one or more")
}

if r.Spec.Autoscaler.Behavior.ScaleUp != nil && *r.Spec.Autoscaler.Behavior.ScaleUp.StabilizationWindowSeconds < int32(1) {
return fmt.Errorf("the OpenTelemetry Spec autoscale configuration is incorrect, scaleUp should be one or more")
}
}
}

return nil
Expand Down
28 changes: 27 additions & 1 deletion apis/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

124 changes: 124 additions & 0 deletions bundle/manifests/opentelemetry.io_opentelemetrycollectors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,130 @@ spec:
description: Args is the set of arguments to pass to the OpenTelemetry
Collector binary
type: object
autoscaler:
description: Autoscaler specifies the pod autoscaling configuration
to use for the OpenTelemetryCollector workload.
properties:
behavior:
description: HorizontalPodAutoscalerBehavior configures the scaling
behavior of the target in both Up and Down directions (scaleUp
and scaleDown fields respectively).
properties:
scaleDown:
description: scaleDown is scaling policy for scaling Down.
If not set, the default value is to allow to scale down
to minReplicas pods, with a 300 second stabilization window
(i.e., the highest recommendation for the last 300sec is
used).
properties:
policies:
description: policies is a list of potential scaling polices
which can be used during scaling. At least one policy
must be specified, otherwise the HPAScalingRules will
be discarded as invalid
items:
description: HPAScalingPolicy is a single policy which
must hold true for a specified past interval.
properties:
periodSeconds:
description: PeriodSeconds specifies the window
of time for which the policy should hold true.
PeriodSeconds must be greater than zero and less
than or equal to 1800 (30 min).
format: int32
type: integer
type:
description: Type is used to specify the scaling
policy.
type: string
value:
description: Value contains the amount of change
which is permitted by the policy. It must be greater
than zero
format: int32
type: integer
required:
- periodSeconds
- type
- value
type: object
type: array
x-kubernetes-list-type: atomic
selectPolicy:
description: selectPolicy is used to specify which policy
should be used. If not set, the default value Max is
used.
type: string
stabilizationWindowSeconds:
description: 'StabilizationWindowSeconds is the number
of seconds for which past recommendations should be
considered while scaling up or scaling down. StabilizationWindowSeconds
must be greater than or equal to zero and less than
or equal to 3600 (one hour). If not set, use the default
values: - For scale up: 0 (i.e. no stabilization is
done). - For scale down: 300 (i.e. the stabilization
window is 300 seconds long).'
format: int32
type: integer
type: object
scaleUp:
description: 'scaleUp is scaling policy for scaling Up. If
not set, the default value is the higher of: * increase
no more than 4 pods per 60 seconds * double the number of
pods per 60 seconds No stabilization is used.'
properties:
policies:
description: policies is a list of potential scaling polices
which can be used during scaling. At least one policy
must be specified, otherwise the HPAScalingRules will
be discarded as invalid
items:
description: HPAScalingPolicy is a single policy which
must hold true for a specified past interval.
properties:
periodSeconds:
description: PeriodSeconds specifies the window
of time for which the policy should hold true.
PeriodSeconds must be greater than zero and less
than or equal to 1800 (30 min).
format: int32
type: integer
type:
description: Type is used to specify the scaling
policy.
type: string
value:
description: Value contains the amount of change
which is permitted by the policy. It must be greater
than zero
format: int32
type: integer
required:
- periodSeconds
- type
- value
type: object
type: array
x-kubernetes-list-type: atomic
selectPolicy:
description: selectPolicy is used to specify which policy
should be used. If not set, the default value Max is
used.
type: string
stabilizationWindowSeconds:
description: 'StabilizationWindowSeconds is the number
of seconds for which past recommendations should be
considered while scaling up or scaling down. StabilizationWindowSeconds
must be greater than or equal to zero and less than
or equal to 3600 (one hour). If not set, use the default
values: - For scale up: 0 (i.e. no stabilization is
done). - For scale down: 300 (i.e. the stabilization
window is 300 seconds long).'
format: int32
type: integer
type: object
type: object
type: object
config:
description: Config is the raw JSON to be used as the collector's
configuration. Refer to the OpenTelemetry Collector documentation
Expand Down
124 changes: 124 additions & 0 deletions config/crd/bases/opentelemetry.io_opentelemetrycollectors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,130 @@ spec:
description: Args is the set of arguments to pass to the OpenTelemetry
Collector binary
type: object
autoscaler:
description: Autoscaler specifies the pod autoscaling configuration
to use for the OpenTelemetryCollector workload.
properties:
behavior:
description: HorizontalPodAutoscalerBehavior configures the scaling
behavior of the target in both Up and Down directions (scaleUp
and scaleDown fields respectively).
properties:
scaleDown:
description: scaleDown is scaling policy for scaling Down.
If not set, the default value is to allow to scale down
to minReplicas pods, with a 300 second stabilization window
(i.e., the highest recommendation for the last 300sec is
used).
properties:
policies:
description: policies is a list of potential scaling polices
which can be used during scaling. At least one policy
must be specified, otherwise the HPAScalingRules will
be discarded as invalid
items:
description: HPAScalingPolicy is a single policy which
must hold true for a specified past interval.
properties:
periodSeconds:
description: PeriodSeconds specifies the window
of time for which the policy should hold true.
PeriodSeconds must be greater than zero and less
than or equal to 1800 (30 min).
format: int32
type: integer
type:
description: Type is used to specify the scaling
policy.
type: string
value:
description: Value contains the amount of change
which is permitted by the policy. It must be greater
than zero
format: int32
type: integer
required:
- periodSeconds
- type
- value
type: object
type: array
x-kubernetes-list-type: atomic
selectPolicy:
description: selectPolicy is used to specify which policy
should be used. If not set, the default value Max is
used.
type: string
stabilizationWindowSeconds:
description: 'StabilizationWindowSeconds is the number
of seconds for which past recommendations should be
considered while scaling up or scaling down. StabilizationWindowSeconds
must be greater than or equal to zero and less than
or equal to 3600 (one hour). If not set, use the default
values: - For scale up: 0 (i.e. no stabilization is
done). - For scale down: 300 (i.e. the stabilization
window is 300 seconds long).'
format: int32
type: integer
type: object
scaleUp:
description: 'scaleUp is scaling policy for scaling Up. If
not set, the default value is the higher of: * increase
no more than 4 pods per 60 seconds * double the number of
pods per 60 seconds No stabilization is used.'
properties:
policies:
description: policies is a list of potential scaling polices
which can be used during scaling. At least one policy
must be specified, otherwise the HPAScalingRules will
be discarded as invalid
items:
description: HPAScalingPolicy is a single policy which
must hold true for a specified past interval.
properties:
periodSeconds:
description: PeriodSeconds specifies the window
of time for which the policy should hold true.
PeriodSeconds must be greater than zero and less
than or equal to 1800 (30 min).
format: int32
type: integer
type:
description: Type is used to specify the scaling
policy.
type: string
value:
description: Value contains the amount of change
which is permitted by the policy. It must be greater
than zero
format: int32
type: integer
required:
- periodSeconds
- type
- value
type: object
type: array
x-kubernetes-list-type: atomic
selectPolicy:
description: selectPolicy is used to specify which policy
should be used. If not set, the default value Max is
used.
type: string
stabilizationWindowSeconds:
description: 'StabilizationWindowSeconds is the number
of seconds for which past recommendations should be
considered while scaling up or scaling down. StabilizationWindowSeconds
must be greater than or equal to zero and less than
or equal to 3600 (one hour). If not set, use the default
values: - For scale up: 0 (i.e. no stabilization is
done). - For scale down: 300 (i.e. the stabilization
window is 300 seconds long).'
format: int32
type: integer
type: object
type: object
type: object
config:
description: Config is the raw JSON to be used as the collector's
configuration. Refer to the OpenTelemetry Collector documentation
Expand Down
Loading