Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VPA - Vpa full e2e test flakiness #2794

Closed
krzysied opened this issue Feb 4, 2020 · 8 comments
Closed

VPA - Vpa full e2e test flakiness #2794

krzysied opened this issue Feb 4, 2020 · 8 comments

Comments

@krzysied
Copy link
Contributor

krzysied commented Feb 4, 2020

Currently vpa full e2e test is flaky due to timeouts.

Testgrid: https://testgrid.k8s.io/sig-autoscaling-vpa#autoscaling-vpa-full

@krzysied
Copy link
Contributor Author

krzysied commented Feb 4, 2020

I'll be bumping up the timeout for the tests. However it could be worth investigating why there is a ~20 spread on test execution.
/assign

@krzysied
Copy link
Contributor Author

krzysied commented Feb 7, 2020

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-autoscaling-vpa-full/1225139329743982597/build-log.txt

I0207 09:57:45.093] deployment.apps/vpa-admission-controller created
I0207 09:57:45.184] service/vpa-webhook created
I0207 09:57:45.205] /go/src/k8s.io/autoscaler/vertical-pod-autoscaler/e2e /go/src/k8s.io/autoscaler/kubernetes
I0207 10:00:53.893] Feb  7 10:00:53.893: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
I0207 10:00:53.894] === RUN   TestE2E

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-autoscaling-vpa-full/1225260877695946754/build-log.txt

I0206 04:04:08.948] deployment.apps/vpa-admission-controller created
I0206 04:04:09.041] service/vpa-webhook created
I0206 04:04:09.215] /go/src/k8s.io/autoscaler/vertical-pod-autoscaler/e2e /go/src/k8s.io/autoscaler/kubernetes
I0206 04:31:20.105] Feb  6 04:31:20.104: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
I0206 04:31:20.106] === RUN   TestE2E

It seems that test duration is extended due to some unjustified waiting time between vpa setup and test start.

@krzysied
Copy link
Contributor Author

krzysied commented Feb 7, 2020

The line I0206 04:04:09.215] /go/src/k8s.io/autoscaler/vertical-pod-autoscaler/e2e /go/src/k8s.io/autoscaler/kubernetes should be a result of:

The next line is:

go test -mod vendor ./v1beta2/*go -v --test.timeout=60m --args --ginkgo.v=true --ginkgo.focus="\[VPA\] \[${SUITE}\]" --report-dir=/workspace/_artifacts --disable-log-dump

My guess is that the tests are build without sufficient cpu.

@krzysied
Copy link
Contributor Author

krzysied commented Feb 7, 2020

In vpa we include all k8s tests. Lets remove them: #2807

@krzysied
Copy link
Contributor Author

krzysied commented Feb 7, 2020

Looks like #2807 significantly reduced number of unneeded tests
from 2 of 4905 Specs to 2 of 243 Specs.
Lets see if it has impact on test start time.

@krzysied
Copy link
Contributor Author

Removing all k8s e2e tests from vpa e2e: #2812

@krzysied
Copy link
Contributor Author

And now it's reduced to 2 of 37 Specs :)

Hopefully it will reduce the time required by tests to be built.

@krzysied
Copy link
Contributor Author

Currently the test execution times are not varying that much.
I'm closing the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants