Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Limit parallel builds on operator #4233

Merged
merged 3 commits into from
Apr 26, 2023

Conversation

christophd
Copy link
Contributor

Usually the builds get strictly queued because of potential image layering in terms of reusing the integration kits. There are situations though where parallel builds on the operator are possible.

  • Avoid many running integration builds
  • Monitor all builds started by the operator instance and limit max number of running builds according to given setting
  • By default use max running builds limit = 3 for build strategy routine
  • By default use max running builds limit = 10 for build strategy pod
  • Add max running builds setting to IntegrationPlatform

Release Note

fix: Limit parallel builds on operator

@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch from 52467de to 077be4d Compare April 6, 2023 07:37
@christophd christophd changed the title fix: Limit running builds on operator fix: Limit parallel builds on operator Apr 6, 2023
pkg/controller/build/build_monitor.go Outdated Show resolved Hide resolved
pkg/platform/defaults.go Show resolved Hide resolved
@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch from 077be4d to 5e66e20 Compare April 6, 2023 14:35
Copy link
Contributor

@squakez squakez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I think it's a good feature to have. I just am not sure to understand why the controller variable slips into the Build resource. Also, I think this requires an e2e in order to validate such feature, ie, a build really queues up when the max limit is reached.

pkg/apis/camel/v1/build_types.go Show resolved Hide resolved
pkg/platform/defaults.go Show resolved Hide resolved
@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch from 5e66e20 to 098577e Compare April 11, 2023 08:19
@christophd
Copy link
Contributor Author

@claudio4j @squakez could you please rerun the failed workflows? I assume that the errors are flaky tests and are not related to the changes made in this PR

@christophd
Copy link
Contributor Author

@squakez YAY, checks are green. I will rebase and resolve the conflicts

@squakez
Copy link
Contributor

squakez commented Apr 12, 2023

@squakez YAY, checks are green. I will rebase and resolve the conflicts

Please, mind my previous comments as well:

I just am not sure to understand why the controller variable slips into the Build resource. Also, I think this requires an e2e in order to validate such feature, ie, a build really queues up when the max limit is reached.

Thanks.

- Avoid many parallel integration builds
- Monitor all builds started by the operator instance and limit max number of running builds according to given setting
- By default use max running builds limit = 3 for build strategy routine
- By default use max running builds limit = 10 for build strategy pod
- Add max running builds setting to IntegrationPlatform
- Add some documentation on build strategy and build queues
@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch from 098577e to 21c2528 Compare April 12, 2023 13:52
@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch 17 times, most recently from a6338b2 to 7d119b5 Compare April 24, 2023 17:27
@christophd christophd force-pushed the issue/CMLK-206/limit-parallel-builds branch from 7d119b5 to 96d28c4 Compare April 24, 2023 18:32
@christophd
Copy link
Contributor Author

@squakez I finally managed to add an E2E test, could you please trigger the workflow once more? thx

@christophd
Copy link
Contributor Author

@squakez no idea what is happening with the jobs here.
all green on my fork christophd#92

@squakez
Copy link
Contributor

squakez commented Apr 25, 2023

@squakez no idea what is happening with the jobs here. all green on my fork christophd#92

let's give another spin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants