ko hangs when used with ASDF installed Go

[StevenACoffman]

I am able to reproduce either a bug, or a misconfiguration that could perhaps use a better error message of some sort.

The reproduction is here at StevenACoffman/repro.

Repro

If you clone that repo and run:

export GOPRIVATE=github.com/StevenACoffman
cd sub
ko publish -B --bare --local --platform=linux/amd64 --push=false .

The terminal will just hang there until you hit Ctrl-c.

At which point, it will then say:

^CError: error creating builder: 'builds': entry #0 does not contain a valid local import path (./.) for directory (.): err: signal: interrupt: stderr:
2021/12/12 21:00:19 error during command execution:error creating builder: 'builds': entry #0 does not contain a valid local import path (./.) for directory (.): err: signal: interrupt: stderr:

However, if you delete the .ko.yaml file in that directory and run it again, things work as you'd expect.

rm .ko.yaml
ko publish -B --bare --local --platform=linux/amd64 --push=false .

I have tried a number of different possible values for "main" and "dir" and "id" in the .ko.yaml file, including omitting them altogether. For instance, setting the main: to the values github.com/StevenACoffman/repro/sub, ., and main don't seem to affect this hanging behavior.

.ko.yaml contents:

# KO_DOCKER_REPO should be either kind.local
# (for KinD) or ko.local

defaultBaseImage: gcr.io/distroless/static
builds:
  # You can have multiple builds defined as a yaml list
  -
    # ID of the build.
    # Defaults to the project name.
    id: "my-build"

    # Path to project's (sub)directory containing Go code.
    # This is the working directory for the Go build command(s).
    # Default is `.`.
    dir: .

    # Path to main.go file or main package.
    # Notice: when used with `gomod.proxy`, this must be a package.
    #
    # Default is `.`.
    main: github.com/StevenACoffman/repro/sub


    ldflags:
      - -s 
      - -w
      - -X main.AppName=Working

    # Custom environment variables to be set during the builds.
    # Default is empty.
    env:
      - CGO_ENABLED=0

Any help or tips would be greatly appreciated!

[halvards]

Thanks for filing

With ko v0.9.3, I couldn't reproduce the hanging behavior with the repo you provided:

$ ko version
0.9.3

$ git clone https://github.com/StevenACoffman/repro.git

$ export GOPRIVATE=github.com/StevenACoffman

$ cd sub

$ ko publish -B --bare --local --platform=linux/amd64 --push=false .
2021/12/13 14:08:08 Using base gcr.io/distroless/static for github.com/StevenACoffman/repro/sub
2021/12/13 14:08:11 Building github.com/StevenACoffman/repro/sub for linux/amd64
2021/12/13 14:08:12 Loading ko.local/sub:445cfe3dd7dcad8f41b8eb35058687973182ee7d7a3d57270c29dc58143d3293
2021/12/13 14:08:12 Loaded ko.local/sub:445cfe3dd7dcad8f41b8eb35058687973182ee7d7a3d57270c29dc58143d3293
2021/12/13 14:08:12 Adding tag latest
2021/12/13 14:08:12 Added tag latest
ko.local/sub:445cfe3dd7dcad8f41b8eb35058687973182ee7d7a3d57270c29dc58143d3293

However, the build config from .ko.yaml isn't applied, because the dir+main doesn't match the directory containing package main. That's why, when running the image, the output is the following:

$ docker run --detach --rm --publish 8080:8080 ko.local/sub:445cfe3dd7dcad8f41b8eb35058687973182ee7d7a3d57270c29dc58143d3293

$ curl -s localhost:8080
Hello from unknown, !

If I change main to . and rebuild, this is the result:

$ curl -s localhost:8080
Hello from Working, !

A few points to note:

• The -B (--base-import-paths) and --bare flags are not designed to be used together, and in this case, -B takes precedence: https://github.com/google/ko#naming-images

• The main field in .ko.yaml is a directory, not a full Go import path.

• If your .ko.yaml and go.mod are in the same directory as your main package, and if this directory is also your working directory, then you can leave out dir and main.

• ko sets CGO_ENABLED=0 by default.

Based on the above, a simpler version of your .ko.yaml would be this:

builds:
- id: anything
  ldflags:
  - -s
  - -w
  - -X main.AppName=Working

Here's what I got with the simplified .ko.yaml:

$ image=$(ko publish -B --local .)

$ docker run --detach --rm --publish 8080:8080 $image

$ curl -s localhost:8080
Hello from Working, !

If you build ko from the main branch, the output includes a line that indicates which build config was applied (if any):

2021/12/13 14:12:25 Using build config my-build for github.com/StevenACoffman/repro/sub

This line is absent if no build config from .ko.yaml matched the module.

[imjasonh]

A few points to note:

• The -B (--base-import-paths) and --bare flags are not designed to be used together, and in this case, -B takes precedence: https://github.com/google/ko#naming-images
• The main field in .ko.yaml is a directory, not a full Go import path.
• If your .ko.yaml and go.mod are in the same directory as your main package, and if this directory is also your working directory, then you can leave out dir and main.
• ko sets CGO_ENABLED=0 by default.

Each of these sounds like an error or warning we should raise, FWIW, if anybody reading this is interested in making it so.

@StevenACoffman just to check, does a regular go build ./ work for you inside sub? I haven't used GOPRIVATE much myself, so this stands out as something that might cause go build to hang, which would cause ko publish to hang, particularly if the source at github.com/StevenACoffman is differently accessible to you vs me and @halvards (I'm also unable to reproduce, with ko @ HEAD and go 1.17.1)

(For my own reference, docs on GOPRIVATE and private modules is here and here)

edit: And thank you so much for taking the time to share a repro, that's immensely helpful in debugging issues like this. 🙏

edit²: What's your ko version and go version, btw?

[StevenACoffman]

Thanks so much for your helpful responses. It appears that the cause is actually using ASDF for Go installation is incompatible with ko somehow. I'm providing how I came to that conclusion.

I should have initially mentioned more details like I am running on MacOS Monterey 12.0.01.
This is how I installed ko:

curl -L https://github.com/google/ko/releases/download/v0.9.3/ko_0.9.3_Darwin_x86_64.tar.gz |  tar xzf - ko
chmod a+x ko
mv ko ~/go/bin

$ uname -v
Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64
$ ko version
0.9.3
$ go version
go version go1.17.1 darwin/amd64
$ env | grep GO
GOBIN=/Users/steve/go/bin
GOPRIVATE=github.com/StevenACoffman
GO111MODULE=on
$ env | grep KO
KO_DOCKER_REPO=ko.local

I initially encountered the hanging behavior in a private GitHub repository, but am seeing the hanging behavior in the public repro repo also when I unset both GOPRIVATE and GOBIN environment variables. The original context also had KO_DOCKER_REPO set to gcr.io/MYPROJECT/MYAPP.

I have Go installed using ASDF which plays shell games to have multiple GO versions installed. This is my installation method (which should also work on linux btw)

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.8.1
. $HOME/.asdf/asdf.sh # this was also added to my .bashrc
asdf plugin-add golang https://github.com/kennyp/asdf-golang.git
asdf install golang 1.17.5

$ which go
/Users/steve/.asdf/shims/go

When I instead used homebrew to install Go 1.17.2 (yeah, not identical) and set my PATH to prefer it:

brew install go
export PATH=/usr/local/Cellar/go/1.17.2/bin:$PATH

Then ko no longer hangs in the repro sub directory.

$ ko publish -B --local .
2021/12/13 12:43:09 Using base gcr.io/distroless/static:nonroot for github.com/StevenACoffman/repro/sub
2021/12/13 12:43:10 Building github.com/StevenACoffman/repro/sub for linux/amd64
2021/12/13 12:43:11 Loading ko.local/sub:2955d3a49baee0ccfed7f28a4de8506fdfbb5a6f005611b2c77e64f80335db9a
2021/12/13 12:43:11 Loaded ko.local/sub:2955d3a49baee0ccfed7f28a4de8506fdfbb5a6f005611b2c77e64f80335db9a
2021/12/13 12:43:11 Adding tag latest
2021/12/13 12:43:11 Added tag latest
ko.local/sub:2955d3a49baee0ccfed7f28a4de8506fdfbb5a6f005611b2c77e64f80335db9a

If it is helpful, I can build and install ko from source with additional debug messages to see where it is hanging. I have updated the title of this issue based on my updated hypothesis.

[jonjohnsonjr]

What's the output of go env?

I've seen some weirdness around osx install locations:

#305

[StevenACoffman]

Using ASDF for go 1.17.5 (under which ko just hangs):

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/steve/Library/Caches/go-build"
GOENV="/Users/steve/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/steve/.asdf/installs/golang/1.17.5/packages/pkg/mod"
GONOPROXY="github.com/Khan"
GONOSUMDB="github.com/Khan"
GOOS="darwin"
GOPATH="/Users/steve/.asdf/installs/golang/1.17.5/packages"
GOPRIVATE="github.com/Khan"
GOPROXY="direct"
GOROOT="/Users/steve/.asdf/installs/golang/1.17.5/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/Users/steve/.asdf/installs/golang/1.17.5/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s8/3k2blhzd50ldk8c3lld3_p0m0000gn/T/go-build3750308294=/tmp/go-build -gno-record-gcc-switches -fno-common"

ko will operate correctly if I avoid the ASDF Go shim and just set the the ASDF installed go directly in the PATH using:

export PATH="$(go env | grep GOROOT | awk -v val=$num 'BEGIN { FS="\"";}{print $2}')/bin:$PATH"

If I then run go env the only differences are:

GOMODCACHE="/Users/steve/go/pkg/mod"
GOPATH="/Users/steve/go"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s8/3k2blhzd50ldk8c3lld3_p0m0000gn/T/go-build1230195988=/tmp/go-build -gno-record-gcc-switches -fno-common"

However, these are unrelated, as I can export these environment variables with the other values, verify go env is identical, and still run ko successfully in either case.

[jonjohnsonjr]

Can you determine where it's hanging with dtruss?

[imjasonh]

Is it interesting that in the initial report, we don't see a Building github.com/foo/bar for linux/amd64 log line before it hangs? That's logged right before we invoke go build, so it might not be stuck calling go build at all.

AFAIK I don't think we invoke the go CLI any time before that, everything else is Go packages.

[StevenACoffman]

Again, I'm happy to sprinkle log lines in ko source, build ko, and run it if that is helpful.

[jonjohnsonjr]

It's failing in here: https://github.com/google/ko/blob/2ba70fc75bd94147b03e9c24d3e4c5f56d02edfc/pkg/commands/options/build.go#L184-L187

This is kind of a horrifying comment: https://cs.opensource.google/go/x/tools/+/refs/tags/v0.1.8:internal/gocommand/invoke.go;l=208-213;drc=27fc7644e1ac52666992274b9880c9d67e4b5a9c;bpv=1;bpt=1

[StevenACoffman]

A few points to note:

• The -B (--base-import-paths) and --bare flags are not designed to be used together, and in this case, -B takes precedence: https://github.com/google/ko#naming-images
• The main field in .ko.yaml is a directory, not a full Go import path.
• If your .ko.yaml and go.mod are in the same directory as your main package, and if this directory is also your working directory, then you can leave out dir and main.
• ko sets CGO_ENABLED=0 by default.

Each of these sounds like an error or warning we should raise, FWIW, if anybody reading this is interested in making it so.

I would love it if these could be warnings, btw. All of the mistakes I made were from examples I found in the wild searching github for ko uses.

[jonjohnsonjr]

Again, I'm happy to sprinkle log lines in ko source, build ko, and run it if that is helpful.

Can you try setting GOPACKAGESDEBUG=true?

Does go list ./. also hang for you?

[StevenACoffman]

So ko still hangs, but go list ./. does not hang and works normally.

$ export ASDF_GOLANG_VERSION=1.17.5
$ export GOPACKAGESDEBUG=true
$ cd repro/sub
$ go list ./.
github.com/StevenACoffman/repro/sub
$ ko publish -B --local .
^C2021/12/14 11:47:52 21.721165405s for GOROOT= GOPATH= GO111MODULE=off GOPROXY= PWD=. go list -e -f {{context.ReleaseTags}} -- unsafe
Error: error creating builder: 'builds': entry #0 does not contain a valid local import path (./.) for directory (.): err: signal: interrupt: stderr:
2021/12/14 11:47:52 error during command execution:error creating builder: 'builds': entry #0 does not contain a valid local import path (./.) for directory (.): err: signal: interrupt: stderr:

If I just run this:

GOROOT= GOPATH= GO111MODULE=off GOPROXY= PWD=. go list -e -f {{context.ReleaseTags}} -- unsafe

It just hangs as well.

[StevenACoffman]

So I tried various permutations and only the PWD=. appears to cause the hanging behavior. This will hang using ASDF's shimmed go 1.17.5 on darwin:

PWD=. go list -e -f {{context.ReleaseTags}}

I can set or unset all the other environment variables and nothing changes the hanging behavior.

[StevenACoffman]

Also, sorry that I got pulled away yesterday. if more realtime communication would be helpful now or in the future, where does that generally take place for google/ko developers? A channel in Gophers slack or the Kubernetes slack?

[jonjohnsonjr]

A channel in Gophers slack or the Kubernetes slack?

We hang out in k8s slack: https://github.com/google/ko#discuss

[StevenACoffman]

Worth mentioning that users of goenv most likely also experienced the same hanging behavior prior to the fix in #548