Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterize queries per second and queries burstable limit #502

Merged
merged 9 commits into from
Nov 11, 2022
Merged

Parameterize queries per second and queries burstable limit #502

merged 9 commits into from
Nov 11, 2022

Conversation

raghu-nandan-bs
Copy link
Contributor

Changes proposed on the PR:
When we are using ~100 CRs, we can see that number of queries per second made by kubernetes client can go upto 100 and even exceed this limit.

Due to this, calls to k8s API gets throttled and since we use a common k8s client for whole controller, entire process gets throttled for the duration. We should be able to configure this limit to adjust to higher loads.

@raghu-nandan-bs raghu-nandan-bs requested a review from a team as a code owner October 22, 2022 12:37
@raghu-nandan-bs
Copy link
Contributor Author

example error message seen in operator logs:

I1021 15:46:48.067159       1 request.go:601] Waited for 1.007535927s due to client-side throttling, not priority and fairness, request: PUT:https://172.20.0.1:443/apis/policy/v1/namespaces/test-metrics/poddisruptionbudgets/rfr-test-1-ur9p3
I1021 15:46:59.207071       1 request.go:601] Waited for 1.003996959s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/namespaces/test-metrics/statefulsets/rfr-test-1-jdqxx
I1021 15:47:09.237071       1 request.go:601] Waited for 1.004036993s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/namespaces/test-metrics/statefulsets/rfr-test-1-qa72k

@samof76
Copy link
Contributor

samof76 commented Oct 25, 2022

@ese this is significant. please review.

@ese
Copy link
Member

ese commented Nov 11, 2022

Thanks @raghu-nandan-bs , I broke the gofmt in flags.go in the merge with master becouse I resolved the conflicts through the github web interface :/
If you can pass gofmt to the file to solve the issue I will be happy to merge

@raghu-nandan-bs
Copy link
Contributor Author

Thanks @raghu-nandan-bs , I broke the gofmt in flags.go in the merge with master becouse I resolved the conflicts through the github web interface :/ If you can pass gofmt to the file to solve the issue I will be happy to merge

Hi @ese this is done, please review

@ese ese merged commit b9e55b2 into spotahome:master Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants