Add ko builder #6041

halvards · 2021-06-18T01:00:29Z

Tracking issue for the effort to add the ko builder.

See the design proposal for further details.

Related: #5611, #1958

The text was updated successfully, but these errors were encountered:

halvards · 2021-06-18T01:21:42Z

Related PRs in google/ko:

Small steps, this is just to verify the impact of changes to the dependency graph. Tracking issue: GoogleContainerTools#6041

Small steps, this is just to verify the impact of changes to the dependency graph. Tracking issue: #6041

This adds the initial implementation for the ko builder. Read more about the ko builder in the [design proposal](docs/design_proposals/ko-builder.md). Tracking: GoogleContainerTools#6041

This adds the `Build()` method for building artifacts using ko. It supports both publishing the resulting image to a registry, and sideloading it to the local Docker daemon. The `temporary.go` file contains the structs intended to be added to the schema. This implementation is still missing the following features: - integration test - dependencies (for file watching) - insecure registries - debug mode - support for `go` flags and environment variables (waiting on ko-build/ko#340) - actually plumbing the builder into the Skaffold CLI and API :-) Tracking: GoogleContainerTools#6041

tejal29 · 2021-07-01T21:36:00Z

Thanks for creating the issue!

This adds the `Build()` method for building artifacts using ko. It supports both publishing the resulting image to a registry, and sideloading it to the local Docker daemon. The Skaffold docker client is used in the ko builder. This ensures that any Minikube config set by Skaffold is used. The ko builder uses the docker client when sideloading images to the docker daemon. The `temporary.go` file contains the structs intended to be added to the schema. The additions to the design proposal explain image naming for the ko builder, specifically how container images are named and how Go import paths are resolved when using the proposed ko builder. This commit includes ko builder unit tests for non-current-working-directory workspace dirs. These verify that the ko builder works even if the context specified in `skaffold.yaml` differs from the current working directory. This implementation is still missing the following features: - integration test - dependencies (for file watching) - insecure registries - debug mode - support for `go` flags and environment variables (based on ko-build/ko#340) - actually plumbing the builder into the Skaffold CLI and API :-) Tracking: GoogleContainerTools#6041

* Add core ko builder implementation This adds the `Build()` method for building artifacts using ko. It supports both publishing the resulting image to a registry, and sideloading it to the local Docker daemon. The Skaffold docker client is used in the ko builder. This ensures that any Minikube config set by Skaffold is used. The ko builder uses the docker client when sideloading images to the docker daemon. The `temporary.go` file contains the structs intended to be added to the schema. The additions to the design proposal explain image naming for the ko builder, specifically how container images are named and how Go import paths are resolved when using the proposed ko builder. This commit includes ko builder unit tests for non-current-working-directory workspace dirs. These verify that the ko builder works even if the context specified in `skaffold.yaml` differs from the current working directory. This implementation is still missing the following features: - integration test - dependencies (for file watching) - insecure registries - debug mode - support for `go` flags and environment variables (based on ko-build/ko#340) - actually plumbing the builder into the Skaffold CLI and API :-) Tracking: #6041 * Improve ko Builder unit tests Split out the confusing end-to-end unit test with tests for individual functions. * Reorder test inputs and expected outputs Also fix typo in design proposal. * Add TODO to update import path Reminders to update the import path once the contents of `pkg/skaffold/build/ko/schema` have been added to the real schema in `pkg/skaffold/schema/latest/v1`. * Fix lint errors for TODO comments in import block Adding the TODO in the import block resulted in gci linter errors: https://app.travis-ci.com/github/GoogleContainerTools/skaffold/jobs/530642848#L313

Decide on how ko builder users should specify the location of `package main` for their images. Previous discussion: GoogleContainerTools#6054 (comment) Tracking: GoogleContainerTools#6041 Related: GoogleContainerTools#6054, GoogleContainerTools#6286

* Refine ko builder behavior for main packages Decide on how ko builder users should specify the location of `package main` for their images. Previous discussion: #6054 (comment) Tracking: #6041 Related: #6054, #6286 * Improve explanation of the Target config field Clarify how a blank value works, and also how users can use a pattern with wildcards.

This implements support for the Target config field for the ko builder. The Target field allows users to specify a target for `go build`. This is necessary where the main package is not in the context directory. Tracking: GoogleContainerTools#6041 Related: GoogleContainerTools#6054, GoogleContainerTools#6437

Display the expected tag after successfully building a ko image, for both sideloaded and published images. Also handles the `tagPolicy.sha256` scenarios. Fixes: GoogleContainerTools#6835 Tracking: GoogleContainerTools#6041

Shorten the example values in the config schema. This fixes the issue on the `skaffold.yaml` reference page of the website, where long example values push the description to a very narrow space on the page. Tracking: GoogleContainerTools#6041

Shorten the example values in the config schema. This fixes the issue on the `skaffold.yaml` reference page of the website, where long example values push the description to a very narrow space on the page. Tracking: #6041

* fix(log): Output ggcr logs Previously, Skaffold discarded log messages from [`go-containerregistry`](https://github.com/google/go-containerregistry) (a.k.a `ggcr`). This change enables `ggcr` log messages, with severity tied to Skaffold's log level. Related: #6041 * fix(log): Send Go std `log` to `logrus` Includes `ggcr` logs.

Update explanation of how Go debugging works in Skaffold and add explanations for ko builder users. Also update links to the VS Code Go extension. Related: GoogleContainerTools#6843 Tracking: GoogleContainerTools#6041

Update explanation of how Go debugging works in Skaffold and add explanations for ko builder users. Also update links to the VS Code Go extension. Related: #6843 Tracking: #6041

`ko` adds the `-trimpath` flag to `go build` by default, as this aids in achieving reproducible builds. The `-trimpath` flag removes file system paths from the resulting binary, and this makes interactive debugging more difficult. This change instructs `ko` to _not_ add the `-trimpath` flag when users run `skaffold debug` with the `ko` builder. Bumps the `ko` version to pick up the `Trimpath` build option. Also, simplifies the `ko` example config since it's no longer necessary to add a non-empty `flags` field to `skaffold.yaml` to allow for interactive debugging. Fixes: GoogleContainerTools#6843 Tracking: GoogleContainerTools#6041

`ko` adds the `-trimpath` flag to `go build` by default, as this aids in achieving reproducible builds. The `-trimpath` flag removes file system paths from the resulting binary, and this makes interactive debugging more difficult. This change instructs `ko` to _not_ add the `-trimpath` flag when users run `skaffold debug` with the `ko` builder. Bumps the `ko` version to pick up the `Trimpath` build option. Also, simplifies the `ko` example config since it's no longer necessary to add a non-empty `flags` field to `skaffold.yaml` to allow for interactive debugging. Fixes: #6843 Tracking: #6041

With this change, the `ko` builder no longer prints the image name to `stdout` by default. Originally, this was added to the `ko` builder to mimic the behavior of the `ko` CLI. Other Skaffold image builders do not print the image name in this way. The reason this is useful for existing `ko` CLI users is that some documented workflows rely on capturing the image name from `stdout`, see https://github.com/google/ko/blob/v0.9.3/README.md#build-an-image After some investigation, the better option seems to be to rely on the existing `--quiet` and `--output` Skaffold flags to format the output of the image name(s). This change also updates the `ko` builder documentation to show existing `ko` CLI users how to capture the image name by providing an appropriate Go template to `--output`. Fixes: GoogleContainerTools#6835 Closes: GoogleContainerTools#6836 Tracking: GoogleContainerTools#6041

This change enables `ko` builder users to substitute environment variable values in the `labels` and `env` config fields. These fields are used for image labels and build-time environment variables, respectively. Envvar templating of image labels can be used to add information such as the Git commit SHA to the image, see GoogleContainerTools#6916. Also, environment variable expansion of `flags` and `ldflags` in the `ko` builder configuration now supports Skaffold's templating syntax, for consistency. For backwards compatibility, `ko`'s templating syntax still works with `flags` and `ldflags`: Skaffold: `{{.FOO}}` `ko`: `{{.Env.FOO}}` Tracking: GoogleContainerTools#6041 Fixes: GoogleContainerTools#6916

This change enables `ko` builder users to substitute environment variable values in the `labels` and `env` config fields. These fields are used for image labels and build-time environment variables, respectively. Envvar templating of image labels can be used to add information such as the Git commit SHA to the image, see #6916. Also, environment variable expansion of `flags` and `ldflags` in the `ko` builder configuration now supports Skaffold's templating syntax, for consistency. For backwards compatibility, `ko`'s templating syntax still works with `flags` and `ldflags`: Skaffold: `{{.FOO}}` `ko`: `{{.Env.FOO}}` Tracking: #6041 Fixes: #6916

When using the `ko://` scheme prefix for the Skaffold image name, the rest of the image name, following the prefix, is a Go import path. This import path maps to the Git repository (typically), and won't match an image registry name (e.g., `ko://github.com/org/repo` vs `gcr.io/project_id`). With this change, if _all_ of the following are true, the build results in an error that instructs the user to set the default repo: - the image name in `skaffold.yaml` uses the `ko://` prefix; and - the Skaffold default repo is _not_ set; and - the image is being pushed to a registry (rather than sideloaded to a Docker daemon). Fixes: GoogleContainerTools#6933 Tracking: GoogleContainerTools#6041

Document that `ko` builder users who use the `ko://` prefix must set the default repo if pushing images to a registry. Related: GoogleContainerTools#6933 Tracking: GoogleContainerTools#6041

aaron-prindle · 2022-11-19T18:46:43Z

ko builder is added now 🎊! Marking closed(fixed)

aaron-prindle added kind/todo implementation task/epic for the skaffold team area/build kind/feature-request kind/design discussion labels Jun 18, 2021

halvards added a commit to halvards/skaffold that referenced this issue Jun 18, 2021

Add dependency on the ko module

81865b8

Small steps, this is just to verify the impact of changes to the dependency graph. Tracking issue: GoogleContainerTools#6041

halvards added a commit to halvards/skaffold that referenced this issue Jun 18, 2021

Add dependency on the ko module

3d5ebb8

Small steps, this is just to verify the impact of changes to the dependency graph. Tracking issue: GoogleContainerTools#6041

This was referenced Jun 18, 2021

Add dependency on ko #6042

Merged

Updates to ko builder design proposal to add implementation approach #6046

Merged

tejal29 pushed a commit that referenced this issue Jun 21, 2021

Add dependency on the ko module (#6042)

5c3eb41

Small steps, this is just to verify the impact of changes to the dependency graph. Tracking issue: #6041

halvards mentioned this issue Jun 22, 2021

Add core ko builder implementation #6054

Merged

nkubala added the priority/p2 May take a couple of releases label Jun 23, 2021

halvards mentioned this issue Jul 26, 2021

Restore import path with uppercase characters in custom build ko example #6286

Merged

imjasonh mentioned this issue Aug 9, 2021

RFC: deprecate and remove Watch Mode ko-build/ko#412

Closed

halvards mentioned this issue Aug 16, 2021

Refine ko builder behavior for main packages #6437

Merged

halvards mentioned this issue Nov 11, 2021

docs(ko): Shorter example values in config schema #6837

Merged

halvards mentioned this issue Nov 12, 2021

docs(debug): Improve Go debugging documentation #6852

Merged

halvards mentioned this issue Nov 17, 2021

fix(ko): Do not add -trimpath when debugging #6874

Merged

halvards mentioned this issue Nov 29, 2021

FR: ko builder to support envvars in labels config field #6916

Closed

halvards mentioned this issue Dec 3, 2021

fix(ko): Do not print image name to stdout #6928

Merged

halvards mentioned this issue Dec 7, 2021

feat(ko): Enable templating of labels and env #6944

Merged

halvards mentioned this issue Dec 22, 2021

Introduce log levels ko-build/ko#560

Closed

halvards mentioned this issue Jan 12, 2022

fix(ko): Debug port forwarding for ko:// images #7009

Merged

halvards mentioned this issue Jan 12, 2022

fix(ko): Default repo for pushing ko:// images #7010

Merged

halvards mentioned this issue Feb 18, 2022

Implement ko builder beta features #7131

Open

aaron-prindle closed this as completed Nov 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ko builder #6041

Add ko builder #6041

halvards commented Jun 18, 2021 •

edited

Loading

halvards commented Jun 18, 2021 •

edited

Loading

tejal29 commented Jul 1, 2021

aaron-prindle commented Nov 19, 2022

Add ko builder #6041

Add ko builder #6041

Comments

halvards commented Jun 18, 2021 • edited Loading

halvards commented Jun 18, 2021 • edited Loading

tejal29 commented Jul 1, 2021

aaron-prindle commented Nov 19, 2022

halvards commented Jun 18, 2021 •

edited

Loading

halvards commented Jun 18, 2021 •

edited

Loading