-
-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace deprecated gobin
with custom go install
based task runner
#89
Labels
Milestone
Comments
svengreb
added a commit
that referenced
this issue
Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
svengreb
added a commit
that referenced
this issue
Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
svengreb
added a commit
that referenced
this issue
Apr 26, 2021
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
svengreb
added a commit
that referenced
this issue
Apr 26, 2021
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
svengreb
added a commit
that referenced
this issue
Jul 9, 2023
As of Go 1.17 the `go run` command can finally run in module-aware mode [1] while not "polluting" the current module in the working directory, if there is one (`go.mod` file present) 🎉 This finally allows to run commands "on-the-fly" [10] of Go `main` module packages without installing them or without changing dependencies of the current module! To support this feature with wand a new `task.GoModule` [2] has been implemented in a new `golang/run` [3] package. It can be run using a command runner [4] that handles tasks of kind `KindGoModule` [5] so mainly `gotool.Runner` [6]. The new `golang/run.Task` [3] is customizable through the following functions: - `WithArgs(...string) run.Option` - sets additional arguments to pass to the command. - `WithEnv(map[string]string) run.Option` - sets the task specific environment. - `WithModulePath(string) run.Option` - sets the module import path. - `WithModuleVersion(*semver.Version) run.Option` - sets the module version. Next to the new task the `gotool.Runner` [6] has been extended with a new `WithCache(bool)` [9] runner option to toggle the usage of the local cache directory in the root directory of the module. The runner has been made "smart" in the way that it either... - installing the executable through a `golang.Runner` [8], which runs `go install pkg@version` to leverage Go 1.16's feature [12], and execute it afterwards. This is the current default behavior of this runner which will be used when `WithCache(true)` [9] is used. - pass the task to a `golang.Runner` [8], using the new `golang/run` [3] package task, so that it can run `go run pkg@version <args>` instead. This is the new "smart" behavior of the runner which will be used when `WithCache(false)` [9] (default) is used. The new default behavior is to not use a local cache so that caching will be a opt-in! This decision was made because native support for running commands "on-the-fly" should always be preferred to custom logic which is what the local cache directory and [gotool.Runner` [6] purpose is. !!! Note that the minimum Go version for task runners, the new `golang/run` task [3] and the "Elder" wand [7] has been increased to `1.17.0` since this version initially introduced `go run` support in module-awar mode [1]! This is enforced through a build constraint [11] (`go:build go1.17`). !!! The `Elder` [7] reference implementation has also adapted to this new feature by... 1. deprecating the `*elder.Elder.Bootstrap(...string) []error` method! As of wand version `0.9.0` it will be a no-op and will be removed in version `0.10.0`. To install executables anyway the new `*elder.Elder.CacheExecutables error` method should be used instead. To ensure that the wand is properly initialized and operational the `*elder.Elder.Validate(..task.Runner) []error` method is the way to go. A warning message will be printed when the method is called to ensure that users adapt accordionally. 2. providing a new `*elder.Elder.CacheExecutables(...string) error` method which allows to pass paths of Go modules that should be explicitly installed to the local cache directory. This method is a kind of workaround for the, now deprecated, `*elder.Elder.Bootstrap(...string) []error` method to allows users to still cache command executables locally. 3. changing the signature of the `*elder.Elder.Validate() error` method to `*elder.Elder.Validate(...task.Runner) []error` method which allows users to ensure that the _wand_ is properly initialized an operational. Optionally command runner [4] can be passed that will be validated while passing nothing will validate all currently supported runners. [1]: https://go.dev/doc/go1.17#go%20run [2]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#GoModule [3]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run [4]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#Runner [5]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#KindGoModule [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gotool#Runner [7]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder [8]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang#Runner [9]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run#WithCache [10]: https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program [11]: https://pkg.go.dev/cmd/go#hdr-Build_constraints [12]: #89 GH-133
svengreb
added a commit
that referenced
this issue
Jul 9, 2023
As of Go 1.17 the `go run` command can finally run in module-aware mode [1] while not "polluting" the current module in the working directory, if there is one (`go.mod` file present) 🎉 This finally allows to run commands "on-the-fly" [10] of Go `main` module packages without installing them or without changing dependencies of the current module! To support this feature with wand a new `task.GoModule` [2] has been implemented in a new `golang/run` [3] package. It can be run using a command runner [4] that handles tasks of kind `KindGoModule` [5] so mainly `gotool.Runner` [6]. The new `golang/run.Task` [3] is customizable through the following functions: - `WithArgs(...string) run.Option` - sets additional arguments to pass to the command. - `WithEnv(map[string]string) run.Option` - sets the task specific environment. - `WithModulePath(string) run.Option` - sets the module import path. - `WithModuleVersion(*semver.Version) run.Option` - sets the module version. Next to the new task the `gotool.Runner` [6] has been extended with a new `WithCache(bool)` [9] runner option to toggle the usage of the local cache directory in the root directory of the module. The runner has been made "smart" in the way that it either... - installing the executable through a `golang.Runner` [8], which runs `go install pkg@version` to leverage Go 1.16's feature [12], and execute it afterwards. This is the current default behavior of this runner which will be used when `WithCache(true)` [9] is used. - pass the task to a `golang.Runner` [8], using the new `golang/run` [3] package task, so that it can run `go run pkg@version <args>` instead. This is the new "smart" behavior of the runner which will be used when `WithCache(false)` [9] (default) is used. The new default behavior is to not use a local cache so that caching will be a opt-in! This decision was made because native support for running commands "on-the-fly" should always be preferred to custom logic which is what the local cache directory and [gotool.Runner` [6] purpose is. !!! Note that the minimum Go version for task runners, the new `golang/run` task [3] and the "Elder" wand [7] has been increased to `1.17.0` since this version initially introduced `go run` support in module-awar mode [1]! This is enforced through a build constraint [11] (`go:build go1.17`). !!! The `Elder` [7] reference implementation has also adapted to this new feature by... 1. deprecating the `*elder.Elder.Bootstrap(...string) []error` method! As of wand version `0.9.0` it will be a no-op and will be removed in version `0.10.0`. To install executables anyway the new `*elder.Elder.CacheExecutables error` method should be used instead. To ensure that the wand is properly initialized and operational the `*elder.Elder.Validate(..task.Runner) []error` method is the way to go. A warning message will be printed when the method is called to ensure that users adapt accordionally. 2. providing a new `*elder.Elder.CacheExecutables(...string) error` method which allows to pass paths of Go modules that should be explicitly installed to the local cache directory. This method is a kind of workaround for the, now deprecated, `*elder.Elder.Bootstrap(...string) []error` method to allows users to still cache command executables locally. 3. changing the signature of the `*elder.Elder.Validate() error` method to `*elder.Elder.Validate(...task.Runner) []error` method which allows users to ensure that the _wand_ is properly initialized an operational. Optionally command runner [4] can be passed that will be validated while passing nothing will validate all currently supported runners. [1]: https://go.dev/doc/go1.17#go%20run [2]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#GoModule [3]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run [4]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#Runner [5]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#KindGoModule [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gotool#Runner [7]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder [8]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang#Runner [9]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run#WithCache [10]: https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program [11]: https://pkg.go.dev/cmd/go#hdr-Build_constraints [12]: #89 GH-133
svengreb
added a commit
that referenced
this issue
Jul 9, 2023
As of Go 1.17 the `go run` command can finally run in module-aware mode [1] while not "polluting" the current module in the working directory, if there is one (`go.mod` file present) 🎉 This finally allows to run commands "on-the-fly" [10] of Go `main` module packages without installing them or without changing dependencies of the current module! To support this feature with wand a new `task.GoModule` [2] has been implemented in a new `golang/run` [3] package. It can be run using a command runner [4] that handles tasks of kind `KindGoModule` [5] so mainly `gotool.Runner` [6]. The new `golang/run.Task` [3] is customizable through the following functions: - `WithArgs(...string) run.Option` - sets additional arguments to pass to the command. - `WithEnv(map[string]string) run.Option` - sets the task specific environment. - `WithModulePath(string) run.Option` - sets the module import path. - `WithModuleVersion(*semver.Version) run.Option` - sets the module version. Next to the new task the `gotool.Runner` [6] has been extended with a new `WithCache(bool)` [9] runner option to toggle the usage of the local cache directory in the root directory of the module. The runner has been made "smart" in the way that it either... - installing the executable through a `golang.Runner` [8], which runs `go install pkg@version` to leverage Go 1.16's feature [12], and execute it afterwards. This is the current default behavior of this runner which will be used when `WithCache(true)` [9] is used. - pass the task to a `golang.Runner` [8], using the new `golang/run` [3] package task, so that it can run `go run pkg@version <args>` instead. This is the new "smart" behavior of the runner which will be used when `WithCache(false)` [9] (default) is used. The new default behavior is to not use a local cache so that caching will be a opt-in! This decision was made because native support for running commands "on-the-fly" should always be preferred to custom logic which is what the local cache directory and [gotool.Runner` [6] purpose is. !!! Note that the minimum Go version for task runners, the new `golang/run` task [3] and the "Elder" wand [7] has been increased to `1.17.0` since this version initially introduced `go run` support in module-awar mode [1]! This is enforced through a build constraint [11] (`go:build go1.17`). !!! The `Elder` [7] reference implementation has also adapted to this new feature by... 1. deprecating the `*elder.Elder.Bootstrap(...string) []error` method! As of wand version `0.9.0` it will be a no-op and will be removed in version `0.10.0`. To install executables anyway the new `*elder.Elder.CacheExecutables error` method should be used instead. To ensure that the wand is properly initialized and operational the `*elder.Elder.Validate(..task.Runner) []error` method is the way to go. A warning message will be printed when the method is called to ensure that users adapt accordionally. 2. providing a new `*elder.Elder.CacheExecutables(...string) error` method which allows to pass paths of Go modules that should be explicitly installed to the local cache directory. This method is a kind of workaround for the, now deprecated, `*elder.Elder.Bootstrap(...string) []error` method to allows users to still cache command executables locally. 3. changing the signature of the `*elder.Elder.Validate() error` method to `*elder.Elder.Validate(...task.Runner) []error` method which allows users to ensure that the _wand_ is properly initialized an operational. Optionally command runner [4] can be passed that will be validated while passing nothing will validate all currently supported runners. [1]: https://go.dev/doc/go1.17#go%20run [2]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#GoModule [3]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run [4]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#Runner [5]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task#KindGoModule [6]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gotool#Runner [7]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/elder [8]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang#Runner [9]: https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/run#WithCache [10]: https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program [11]: https://pkg.go.dev/cmd/go#hdr-Build_constraints [12]: #89 GH-133
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
This is issue supersedes #78 which documents how the official deprecation of
gobin
in favor of the new Go 1.16go install pkg@version
syntax feature should have been handled for this project. The idea was to replace thegobin
task runner with a one that leverages bingo, a project similar togobin
, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is thatbingo
uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of thego install
command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects.go install
is still not perfectSupport for the new
go install
features, which allow to install commands without affecting themain
module, have already been added in #71 as an alternative togobin
, but one significant problem is still not addressed: install module/package executables globally without overriding already installed executables of different versions.Since
go install
will always place compiled binaries in the path defined bygo env GOBIN
, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions sincego install
still messes up the local user environment.The Workaround: Hybrid
go install
task runnerThe solution is to implement a custom
Runner
that usesgo install
under the hood, but places the compiled executable in a custom cache directory instead ofgo env GOBIN
. The runner will check if the executable already exists, installs it if not so, and executes it afterwards.The concept of storing dependencies locally on a per-project basis is well-known from the
node_modules
directory of the Node package manager npm. Storing executables in a cache directory within the repository (not tracked by Git) allows to usego install
mechanisms while not affect the global user environment and executables stored ingo env GOBIN
. The runner will achieve this by changing theGOBIN
environment variable to the custom cache directory during the execution ofgo install
. This way it bypasses the need for “dirty hacks“ while using a custom output path.The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages.
Note that the runner dynamically runs executables based on the given task so
Validate() error
will be a NOOP.Upcoming Changes
The solution described above will work totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again:
go run
command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed.go install
aware of the-o
flag like thego build
command which is the only reason why the custom runner will be implemented.Further Adjustments
Because the new custom task runner dynamically runs executables based on the given task, the
Bootstrap
method of theWand
reference implementationElder
will additionally allow to pass Go module import paths, optionally including a version suffix (pkg@version
), to install executables from Go module-basedmain
packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task in JetBrains IDEs.The method will also ensure that the local cache directory exists and will create a
.gitignore
file that includes ignore pattern for the cache directory.The text was updated successfully, but these errors were encountered: