Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource status aggregating slow down due to informer resync #1260

Closed
Garrybest opened this issue Jan 14, 2022 · 3 comments · Fixed by #1265
Closed

resource status aggregating slow down due to informer resync #1260

Garrybest opened this issue Jan 14, 2022 · 3 comments · Fixed by #1265
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Garrybest
Copy link
Member

What happened:
When the clusters are large-scale, there are many workloads in karmada-control-plane. The controller may have a perf issue.

Here is what I found:

The status aggregation is too slow every 10 hours, because the controller-runtime will resync all bindings by default. See here.

What you expected to happen:
I have 2 methods to fix this problem.

  1. Make resync period as an option, default 0, i.e., we will not resync any more by default.
  2. Make concurrent reconciles as an option, default 1, i.e., we will process serially by default. If not adaptable, set it larger by args.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Karmada version:
  • kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
  • Others:
@Garrybest Garrybest added the kind/bug Categorizes issue or PR as related to a bug. label Jan 14, 2022
@RainbowMango
Copy link
Member

/retitle resource status aggregating slow down due to informer resync

@karmada-bot karmada-bot changed the title Too slow to aggregate status from work to binding resource status aggregating slow down due to informer resync Jan 14, 2022
@RainbowMango
Copy link
Member

Make resync period as an option, default 0, i.e., we will not resync any more by default.

+1

Make concurrent reconciles as an option, default 1, i.e., we will process serially by default. If not adaptable, set it larger by args.

+1

We can fix this bug first(not introduce the options) then port to release-1.0.

@RainbowMango
Copy link
Member

/assign @Garrybest
As you sent the @1265

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants