Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitigate https://github.com/kubernetes/kubernetes/issues/96635 in load test #1822

Merged
merged 1 commit into from
Jun 7, 2021
Merged

Mitigate https://github.com/kubernetes/kubernetes/issues/96635 in load test #1822

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions clusterloader2/testing/load/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,17 @@ tuningSets:
parallelismLimitedLoad:
parallelismLimit: {{$schedulerThroughputNamespaces}}
# TODO(https://github.com/kubernetes/perf-tests/issues/1024): This TuningSet is used only for pod-startup-latency, get rid of it
# Uniform5qps: for each running phase, use 5 qps.
- name: Uniform5qps
qpsLoad:
qps: 5
# Global100qps: use 100 qps globally:
# * split equally qps among all running phases
# * if some phase finishes, other phases will get more qps.
- name: Global100qps
globalQPSLoad:
qps: 100
burst: 1
- name: RandomizedSaturationTimeLimited
RandomizedTimeLimitedLoad:
timeLimit: {{$saturationTime}}s
Expand Down Expand Up @@ -130,6 +138,18 @@ steps:
- basename: daemonset-priorityclass
objectTemplatePath: daemonset-priorityclass.yaml

# Moved from reconcile-objects.yaml to mitigate https://github.com/kubernetes/kubernetes/issues/96635.
# TODO(https://github.com/kubernetes/perf-tests/issues/1823): Merge back to reconcile-objects.yaml once the k/k bug is fixed.
- module:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a TODO (and open an issue) to merge back those sections once the bug in upstream will be fixed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Copy link
Contributor

@jkaniuk jkaniuk Jun 7, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "remove once k/k bug is fixed"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed PTAL

path: /modules/configmaps-secrets.yaml
params:
actionName: create
tuningSet: Global100qps
namespaces: {{$namespaces}}
bigDeploymentsPerNamespace: {{$bigDeploymentsPerNamespace}}
mediumDeploymentsPerNamespace: {{$mediumDeploymentsPerNamespace}}
smallDeploymentsPerNamespace: {{$smallDeploymentsPerNamespace}}

- module:
path: /modules/reconcile-objects.yaml
params:
Expand Down Expand Up @@ -306,6 +326,16 @@ steps:
pvSmallStatefulSetSize: {{$SMALL_STATEFUL_SETS_PER_NAMESPACE}}
pvMediumStatefulSetSize: {{$MEDIUM_STATEFUL_SETS_PER_NAMESPACE}}

- module:
path: /modules/configmaps-secrets.yaml
params:
actionName: delete
tuningSet: Global100qps
namespaces: {{$namespaces}}
bigDeploymentsPerNamespace: {{$bigDeploymentsPerNamespace}}
mediumDeploymentsPerNamespace: {{$mediumDeploymentsPerNamespace}}
smallDeploymentsPerNamespace: {{$smallDeploymentsPerNamespace}}

- name: Deleting PriorityClass for DaemonSets
phases:
- replicasPerNamespace: 0
Expand Down
45 changes: 45 additions & 0 deletions clusterloader2/testing/load/modules/configmaps-secrets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# To mitigate https://github.com/kubernetes/kubernetes/issues/96635 we have to delete
# configmaps and secrets only AFTER the pods using that configmap/secret are deleted.
# TODO(mborsz): Merge this back to reconcile-objects.yaml once the issue above is fixed.

{{$actionName := printf "%s objects" .actionName}}
{{$namespaces := .namespaces}}
{{$tuningSet := .tuningSet}}

{{$bigDeploymentsPerNamespace := .bigDeploymentsPerNamespace}}
{{$mediumDeploymentsPerNamespace := .mediumDeploymentsPerNamespace}}
{{$smallDeploymentsPerNamespace := .smallDeploymentsPerNamespace}}

steps:
- name: {{$actionName}} configmaps and secrets
phases:
- namespaceRange:
min: 1
max: {{$namespaces}}
replicasPerNamespace: {{$bigDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: big-deployment
objectTemplatePath: configmap.yaml
- basename: big-deployment
objectTemplatePath: secret.yaml
- namespaceRange:
min: 1
max: {{$namespaces}}
replicasPerNamespace: {{$mediumDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: medium-deployment
objectTemplatePath: configmap.yaml
- basename: medium-deployment
objectTemplatePath: secret.yaml
- namespaceRange:
min: 1
max: {{$namespaces}}
replicasPerNamespace: {{$smallDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: small-deployment
objectTemplatePath: configmap.yaml
- basename: small-deployment
objectTemplatePath: secret.yaml
12 changes: 0 additions & 12 deletions clusterloader2/testing/load/modules/reconcile-objects.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,6 @@ steps:
replicasPerNamespace: {{$bigDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: big-deployment
objectTemplatePath: configmap.yaml
- basename: big-deployment
objectTemplatePath: secret.yaml
{{if $ENABLE_NETWORKPOLICIES}}
- basename: big-deployment
objectTemplatePath: networkpolicy.yaml
Expand All @@ -119,10 +115,6 @@ steps:
replicasPerNamespace: {{$mediumDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: medium-deployment
objectTemplatePath: configmap.yaml
- basename: medium-deployment
objectTemplatePath: secret.yaml
{{if $ENABLE_NETWORKPOLICIES}}
- basename: medium-deployment
objectTemplatePath: networkpolicy.yaml
Expand All @@ -141,10 +133,6 @@ steps:
replicasPerNamespace: {{$smallDeploymentsPerNamespace}}
tuningSet: {{$tuningSet}}
objectBundle:
- basename: small-deployment
objectTemplatePath: configmap.yaml
- basename: small-deployment
objectTemplatePath: secret.yaml
{{if $ENABLE_NETWORKPOLICIES}}
- basename: small-deployment
objectTemplatePath: networkpolicy.yaml
Expand Down