dep ensure never comes back

[oomichi]

dep ensure never comes back over 20mins on test-infra repo.
On the other hand, dep ensure -update finishes in 2 mins with huge changes.
After doing dep ensure -update, dep ensure comes back in 9sec.
I tried this on both a virtual/physical machines and faced the same issue on both.

[stevekuznetsov]

Can you strace the process to see what it's doing?

[oomichi]

@stevekuznetsov

That freezes in futex() call like the following:
Cannot you reproduce this problem on your env?

openat(AT_FDCWD, "/home/oomichi/dev/go/src/k8s.io/test-infra/Gopkg.toml", O_RDONLY|O_CLOEXEC) = 3
read(3, "# Gopkg.toml for k8s.io/test-inf"..., 512) = 346
read(3, "", 1190) = 0
openat(AT_FDCWD, "/home/oomichi/dev/go/src/k8s.io/test-infra/Gopkg.lock", O_RDONLY|O_CLOEXEC) = 4
read(4, "# This file is autogenerated, do"..., 512) = 512
read(4, "b.com/bwmarrin/snowflake"\n pack"..., 1024) = 1024
read(4, "e/go-github"\n packages = ["gith"..., 2048) = 2048
read(4, "/common"\n packages = ["expfmt","..., 4096) = 3053
read(4, "", 1043) = 0
mmap(0xc420200000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc420200000
mmap(0xc41ffe8000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc41ffe8000
close(4) = 0
close(3) = 0
stat("/home/oomichi/dev/go/pkg/dep", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/home/oomichi/dev/go/pkg/dep/sources", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/home/oomichi/dev/go/pkg/dep/sm.lock", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
getpid() = 13334
openat(AT_FDCWD, "/home/oomichi/dev/go/pkg/dep/sm.lock.212132414", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
getpid() = 13334
write(3, "13334\n", 6) = 6
linkat(AT_FDCWD, "/home/oomichi/dev/go/pkg/dep/sm.lock.212132414", AT_FDCWD, "/home/oomichi/dev/go/pkg/dep/sm.lock", 0) = 0
lstat("/home/oomichi/dev/go/pkg/dep/sm.lock.212132414", {st_mode=S_IFREG|0600, st_size=6, ...}) = 0
lstat("/home/oomichi/dev/go/pkg/dep/sm.lock", {st_mode=S_IFREG|0600, st_size=6, ...}) = 0
close(3) = 0
unlinkat(AT_FDCWD, "/home/oomichi/dev/go/pkg/dep/sm.lock.212132414", 0) = 0
rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP BUS FPE SEGV STKFLT CHLD PROF RTMIN RT_1], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[INT ILL TRAP BUS FPE SEGV STKFLT CHLD PROF RTMIN RT_1], NULL, 8) = 0
futex(0xc420160910, FUTEX_WAKE, 1) = 1
futex(0xc420160910, FUTEX_WAKE, 1) = 1
futex(0xbeca90, FUTEX_WAIT, 0, NULL

[stevekuznetsov]

I haven't had a moment to reproduce it, hopefully though this helps @cblecker with debugging

[oomichi]

@cblecker

Thank you for your advice, the following is my try based on your comment.

have you tried a go get -u github.com/golang/dep/cmd/dep to update to the latest dep on your local
environment? And then a dep ensure -add ?

I tried the above, but dep ensure -add github.com/go-openapi/spec never comes back.
My try is the following:

$ go get -u github.com/golang/dep/cmd/dep
$ go get k8s.io/test-infra
$ cd $GOPATH/src/k8s.io/test-infra
$ dep ensure -add github.com/go-openapi/spec
Fetching sources...
(never come back..)

The other option is to remove dep from your system, and use the latest
tagged binary (https://github.com/golang/dep/releases/tag/v0.3.2).
I think this is a local env issue.

Does the above mean you never face this issue on your local env?

[cblecker]

So it looks like github.com/go-openapi/spec doesn't have branches/tags to lock specific versions, so what you should be able to do is:

dep ensure
dep prune
hack/update-bazel.sh

We don't even need to mess around with an -add.

No, I'm not able to replicate this in my environment. I'm currently using the latest tagged version of dep (v0.3.2).

[oomichi]

I tried dep v0.3.2 by downloading https://github.com/golang/dep/releases/tag/v0.3.2 , but I still face the same issue::

~/dev/go/src/k8s.io/test-infra$ dep version
dep:
version : v0.3.2
build date : 2017-10-19
git hash : 8ddfc8a
go version : go1.9
go compiler : gc
platform : linux/amd64
~/dev/go/src/k8s.io/test-infra$ dep ensure
(never come back)

I guess my local env is something wrong and I am creating another env on google cloud.
I will report another try on google cloud.

BTW what is a common development environment for k8s?
I am using ubuntu 16.04LTS on virtualbox of windows.
And golang is version 1.8. This might not be recommended, right?

[oomichi]

humm, my try on google cloud also was failed with the same issue.
That was

• Virtual machine instance: vCPU x 1, memory 5.75 GB
• CPU: x86_64
• OS: ubuntu 16.04LTS
• Go: version 1.8
• dep: version 0.3.2

[oomichi]

Now I find what is wrong on my env.
dep ensure requires hg command for getting bitbucket.org/ww/goautoneg , but dep ensure never says such thing and just stopped on env where hg command is not installed.
After installing hg command which is packed in mercurial package, this issue is solved.
Thank you all for your help anyways.

To avoid the same issue, where is good place to write this kind of tip in test-infra?

[cblecker]

https://github.com/kubernetes/test-infra/blob/master/docs/dep.md is the right place for that.

Glad to hear you were able to track it down, and thanks for updating the docs!

[oomichi]

@cblecker Thanks for your help again :-)

#6043 is for that.