-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Added feature to run rendering in synchronous mode. #40
Conversation
…debugging easier and takes load of slow machines.
cc2b81d
to
3cbd585
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about a flag to control the level of parallelism instead of just switching it on and off?
--async=0
to disable--async=1
to enable--async=8
to run maximum of 8 parallel threads
For now, we can skip implementing the last one, but I would like to settle with the flag name.
|
||
// Create the destination file | ||
dstFile, err := os.Create(destination) | ||
if err != nil { | ||
return err | ||
} | ||
defer dstFile.Close() | ||
defer saveClose(dstFile) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Error is logged but not returned.
Named result values are your friend :) https://blog.learngoprogramming.com/5-gotchas-of-defer-in-go-golang-part-iii-36a1ab3d6ef1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, using them now, but got to say, they are NOT my friends looking at the complexity of the resulting code:
saveClose := func(srcFile fs.File) {
closeErr := srcFile.Close()
if closeErr != nil {
if err == nil {
err = closeErr
} else {
err = errors.Join(err, closeErr)
}
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case, what about:
saveClose := func(srcFile fs.File) {
closeErr := srcFile.Close()
err = errors.Join(err, closeErr)
}
errors.Join is ignoring nil errors. If all errors are nil it'll return nil.
Downside you always have to unwrap the error.
This commit adds support for asynchronous processing in the `process` function. It allows tasks to be executed concurrently based on the specified `asyncLevel`. Additionally, test cases have been added to ensure the correct execution of the function.
cmd/root.go
Outdated
@@ -68,6 +69,7 @@ var rootCmd = &cobra.Command{ | |||
func init() { | |||
cobra.OnInitialize(initConfig) | |||
rootCmd.PersistentFlags().StringP("log-level", "l", "info", "Set the logging level") | |||
rootCmd.PersistentFlags().IntVarP(&asyncLevel, "async", "a", 1, "Run asynchronously. Activated by default. Set to 0 to deactivate.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now as async controls the number of parallel routines, I would propose to set some better default.
1 is the same as 0 just using different code.
What about 4 or 8?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about inverting the logic a little bit and doing the following:
- async level is
N
means that maximumN
applications are being processed in parallel, - async level is
0
means the limit is not set, any number of applications can be processed in parallel.
In this case, the default might be 0
. I see it hard to set a viable default value. Moreover, for different operations (sync, render) the "ideal" number would differ.
internal/myks/util.go
Outdated
// run async | ||
if asyncLevel > 0 { | ||
var eg errgroup.Group | ||
semaphore := make(chan struct{}, asyncLevel) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be replaced by eg.SetLimit(asyncLevel)
. No semaphore needed in this case :)
https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit
@fritzduchardt , @Zebradil I added one commit. I think it can be merged now. |
# [1.3.0](v1.2.0...v1.3.0) (2023-09-19) ### Bug Fixes * Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37) * apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f)) * argocd source plugin config type in schema ([520156d](520156d)) * cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a)) * consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516)) * correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7)) * create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e)) * data values of prototype of argocd app ([b5d7ff9](b5d7ff9)) * do not fail on absent rendered directory ([eaf1202](eaf1202)) * do not fail without vendir configs ([2f73cda](2f73cda)) * do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70) * **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7)) * **docker:** specify full image tag ([f3222e5](f3222e5)) * formatting ([fd65f05](fd65f05)) * generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11)) * helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32) * init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd)) * log errors during vendir sync ([5dc1b5e](5dc1b5e)) * make render errors appear in the log with full error message ([c325da2](c325da2)) * process map keys instead of values ([3b86a03](3b86a03)) * reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933)) * search in the default envs directory ([ef4a75e](ef4a75e)) * skip helm rendering ([80a8eb5](80a8eb5)) * **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67)) * update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34)) * use ArgoCD application path relatively to git root ([92f0617](92f0617)) ### Features * add a finalizer to ArgoCD project CR ([acf67fd](acf67fd)) * add argocd-apps prototype ([6772744](6772744)) * add arm binaries ([a74d63e](a74d63e)) * add common overlays example to assets ([39965c5](39965c5)) * add example environment configs ([8edba12](8edba12)) * add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd)) * add git branch detection and refactor data schema ([05e41d4](05e41d4)) * add init command and a data schema file ([c11c27a](c11c27a)) * add prototypes in the init command ([f31471e](f31471e)) * add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6)) * Add vendir authentication via environment ([b0c50c2](b0c50c2)) * Add vendir sync caching ([24ff41c](24ff41c)) * Added docker image ([ae8988d](ae8988d)) * Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b)) * always write data-schema file ([fa83bee](fa83bee)) * ArgoCD support ([#41](#41)) ([e45d585](e45d585)) * configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa)) * create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3)) * detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101)) * do not convert git URL protocol ([2823eb6](2823eb6)) * dump configuration as ytt values ([af65436](af65436)) * fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3) * fine-grained ArgoCD project destination ([04d3b78](04d3b78)) * get git repo URL ([c9b726a](c9b726a)) * **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31) * **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983)) * provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c)) * provide example default values for all environments ([0f9cab6](0f9cab6)) * Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63)) * Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c)) * **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184)) * **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad)) * support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0)) * tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180)) * validate root directory ([f81b719](f81b719)) * vendir sync caching ([7279cc7](7279cc7)) ### Performance Improvements * **docker:** ignore not needed files ([3479804](3479804))
# [2.0.0](v1.2.0...v2.0.0) (2023-09-19) ### Bug Fixes * Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37) * apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f)) * argocd source plugin config type in schema ([520156d](520156d)) * cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a)) * consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516)) * correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7)) * create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e)) * data values of prototype of argocd app ([b5d7ff9](b5d7ff9)) * do not fail on absent rendered directory ([eaf1202](eaf1202)) * do not fail without vendir configs ([2f73cda](2f73cda)) * do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70) * **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7)) * **docker:** specify full image tag ([f3222e5](f3222e5)) * formatting ([fd65f05](fd65f05)) * generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11)) * helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32) * init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd)) * log errors during vendir sync ([5dc1b5e](5dc1b5e)) * make render errors appear in the log with full error message ([c325da2](c325da2)) * process map keys instead of values ([3b86a03](3b86a03)) * reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933)) * search in the default envs directory ([ef4a75e](ef4a75e)) * skip helm rendering ([80a8eb5](80a8eb5)) * **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67)) * update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34)) * use ArgoCD application path relatively to git root ([92f0617](92f0617)) ### Features * add a finalizer to ArgoCD project CR ([acf67fd](acf67fd)) * add argocd-apps prototype ([6772744](6772744)) * add arm binaries ([a74d63e](a74d63e)) * add common overlays example to assets ([39965c5](39965c5)) * add example environment configs ([8edba12](8edba12)) * add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd)) * add git branch detection and refactor data schema ([05e41d4](05e41d4)) * add init command and a data schema file ([c11c27a](c11c27a)) * add prototypes in the init command ([f31471e](f31471e)) * add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6)) * Add vendir authentication via environment ([b0c50c2](b0c50c2)) * Add vendir sync caching ([24ff41c](24ff41c)) * Added docker image ([ae8988d](ae8988d)) * Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b)) * always write data-schema file ([fa83bee](fa83bee)) * ArgoCD support ([#41](#41)) ([e45d585](e45d585)) * configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa)) * create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3)) * detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101)) * do not convert git URL protocol ([2823eb6](2823eb6)) * dump configuration as ytt values ([af65436](af65436)) * fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3) * fine-grained ArgoCD project destination ([04d3b78](04d3b78)) * get git repo URL ([c9b726a](c9b726a)) * **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31) * **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983)) * provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c)) * provide example default values for all environments ([0f9cab6](0f9cab6)) * Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63)) * Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c)) * release 2.0 ([b7b486d](b7b486d)) * **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184)) * **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad)) * support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0)) * tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180)) * validate root directory ([f81b719](f81b719)) * vendir sync caching ([7279cc7](7279cc7)) ### Performance Improvements * **docker:** ignore not needed files ([3479804](3479804)) ### BREAKING CHANGES * release 2.0 This is an empty commit to trigger a major release.
# [2.0.0](v1.2.0...v2.0.0) (2023-09-19) ### Bug Fixes * Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37) * apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f)) * argocd source plugin config type in schema ([520156d](520156d)) * cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a)) * consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516)) * correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7)) * create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e)) * data values of prototype of argocd app ([b5d7ff9](b5d7ff9)) * do not fail on absent rendered directory ([eaf1202](eaf1202)) * do not fail without vendir configs ([2f73cda](2f73cda)) * do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70) * **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7)) * **docker:** specify full image tag ([f3222e5](f3222e5)) * formatting ([fd65f05](fd65f05)) * generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11)) * helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32) * init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd)) * log errors during vendir sync ([5dc1b5e](5dc1b5e)) * make render errors appear in the log with full error message ([c325da2](c325da2)) * process map keys instead of values ([3b86a03](3b86a03)) * reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933)) * search in the default envs directory ([ef4a75e](ef4a75e)) * skip helm rendering ([80a8eb5](80a8eb5)) * **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67)) * update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34)) * use ArgoCD application path relatively to git root ([92f0617](92f0617)) ### Features * add a finalizer to ArgoCD project CR ([acf67fd](acf67fd)) * add argocd-apps prototype ([6772744](6772744)) * add arm binaries ([a74d63e](a74d63e)) * add common overlays example to assets ([39965c5](39965c5)) * add example environment configs ([8edba12](8edba12)) * add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd)) * add git branch detection and refactor data schema ([05e41d4](05e41d4)) * add init command and a data schema file ([c11c27a](c11c27a)) * add prototypes in the init command ([f31471e](f31471e)) * add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6)) * Add vendir authentication via environment ([b0c50c2](b0c50c2)) * Add vendir sync caching ([24ff41c](24ff41c)) * Added docker image ([ae8988d](ae8988d)) * Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b)) * always write data-schema file ([fa83bee](fa83bee)) * ArgoCD support ([#41](#41)) ([e45d585](e45d585)) * configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa)) * create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3)) * detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101)) * do not convert git URL protocol ([2823eb6](2823eb6)) * dump configuration as ytt values ([af65436](af65436)) * fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3) * fine-grained ArgoCD project destination ([04d3b78](04d3b78)) * get git repo URL ([c9b726a](c9b726a)) * **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31) * **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983)) * provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c)) * provide example default values for all environments ([0f9cab6](0f9cab6)) * Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63)) * Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c)) * release 2.0 ([b7b486d](b7b486d)) * **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184)) * **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad)) * support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0)) * tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180)) * validate root directory ([f81b719](f81b719)) * vendir sync caching ([7279cc7](7279cc7)) ### Performance Improvements * **docker:** ignore not needed files ([3479804](3479804)) ### BREAKING CHANGES * release 2.0 This is an empty commit to trigger a major release.
This makes debugging easier and takes load of slow machines. It has proven useful in the past.