"failed to set up "", error Unable to retrieve local repo information" when doing dep ensure -update

[tomwilkie]

Mac OS X Seirra.

$ dep ensure -update
solve error: No valid source could be created:
	failed to set up "", error Unable to retrieve local repo information	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository
ensure Solve(): No valid source could be created:
	failed to set up "", error Unable to retrieve local repo information	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository

I tried deleting some of the repos in the cache:

$ for dir in $(find $GOPATH/pkg/dep/sources -maxdepth 1); do echo $dir; git -C $dir remote -v; done 
/Users/twilkie/Documents/pkg/dep/sources/git---github.com-kr-logfmt
origin  git://github.com/kr/logfmt (fetch)
origin  git://github.com/kr/logfmt (push)
/Users/twilkie/Documents/pkg/dep/sources/git---github.com-sercand-kuberesolver
origin  git://github.com/sercand/kuberesolver (fetch)
origin  git://github.com/sercand/kuberesolver (push)
/Users/twilkie/Documents/pkg/dep/sources/git-gopkg.in-inf.v0
origin  git://github.com/go-inf/inf (fetch)
origin  git://github.com/go-inf/inf (push)
...

But then I got:

$ dep ensure --update
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x133ed4c]

goroutine 275 [running]:
github.com/golang/dep/vendor/github.com/sdboyer/gps.(*SourceMgr).SyncSourceFor(0xc420018c00, 0xc4201b4240, 0x10, 0x0, 0x0, 0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/source_manager.go:461 +0x10c
github.com/golang/dep/vendor/github.com/sdboyer/gps.(*bridge).breakLock.func1(0xc4203a4f00, 0xc4201b4240, 0x10, 0x0, 0x0, 0x1614200, 0xc4203d0640)
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/bridge.go:341 +0x68
created by github.com/golang/dep/vendor/github.com/sdboyer/gps.(*bridge).breakLock
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/bridge.go:345 +0x240

[tomwilkie]

I'm getting the panic no matter what I do now - I've delete everything & re-inited, and I still get:

$ dep ensure -v -update
Root project is "github.com/weaveworks/cortex"
 27 transitively valid internal packages
 70 external packages imported from 29 projects
(0)   ✓ select (root)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x133ed4c]

goroutine 301 [running]:
github.com/golang/dep/vendor/github.com/sdboyer/gps.(*SourceMgr).SyncSourceFor(0xc420018c00, 0xc4201b4260, 0x10, 0x0, 0x0, 0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/source_manager.go:461 +0x10c
github.com/golang/dep/vendor/github.com/sdboyer/gps.(*bridge).breakLock.func1(0xc420392f00, 0xc4201b4260, 0x10, 0x0, 0x0, 0x1614200, 0xc42019f040)
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/bridge.go:341 +0x68
created by github.com/golang/dep/vendor/github.com/sdboyer/gps.(*bridge).breakLock
	/Users/twilkie/Documents/src/github.com/golang/dep/vendor/github.com/sdboyer/gps/bridge.go:345 +0x240

[tomwilkie]

I had a bit of a play and got it to run without panicing: https://github.com/golang/dep/compare/master...tomwilkie:hack-it-up?expand=1

Not found the root problem yet.

[sdboyer]

Ahhh well we totally are leaking file handles there. That's a big derp. Will fix that right now.

I suspect some of this is fixed by sdboyer/gps#187, which we've yet to merge into dep just yet. (same bug as #250).

However, this is concerning:

$ dep ensure -update
solve error: No valid source could be created:
	failed to set up "", error Unable to retrieve local repo information	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository	failed to set up "", error Unable to update repository

I need to work on the error formatting there, for sure, but the big question would be what state the local repos were in that they couldn't be read from. The "Unable to update repository failed to set up" error is emanating from the underlying vcs library that gps uses.

The empty string is obnoxious, and totally just a mistake on my part. sdboyer/gps#192 is to fix up these errors so they're less useless.

[tomwilkie]

I've just opened a PR to improve the error reporting: sdboyer/gps#193 (and one for the FD leak)

[tomwilkie]

I think the root cause was poorly configured git ssh keys on my part. What causes dep to use ssh instead of http?

[tomwilkie]

Okay I see its trying https and ssh... its being really unreliable for me.

[tomwilkie]

Okay the unreliability is too many fds (I added some more debugging to Mastermind/vcs):

solve error: No valid source could be created:
	failed to set up "https://github.com/ugorji/go", error Unable to retrieve local repo information: pipe: too many open files, 	failed to set up "ssh://[email protected]/ugorji/go", error Unable to update repository: pipe: too many open files, 	failed to set up "git://github.com/ugorji/go", error Unable to retrieve local repo information: pipe: too many open files, 	failed to set up "http://github.com/ugorji/go", error Unable to update repository: pipe: too many open files, 
ensure Solve(): No valid source could be created:

[tomwilkie]

Opened Masterminds/vcs#67 for more helpful errors from Mastermind.

[sdboyer]

That change isn't strictly necessary - the data's all made accessible via methods. We could add the Original() bit to gps' unwrapVcsErr(); I think I omitted it initially because it didn't seem to add much, though I don't recall my reasoning now.

[amenzhinsky]

I'm still getting the same error

$ dep init 
Unable to cache golang.org/x/net
Cached gopkg.in/yaml.v2
Cached github.com/go-sql-driver/mysql
Cached github.com/lib/pq
Cached github.com/mattn/go-sqlite3
solve error: Unable to retrieve local repo information: 
Unable to retrieve local repo information: 

[sdboyer]

@amenzhinsky while the error looks similar, I suspect we've got a different underlying cause here. Would you mind opening a new issue to track the problem you're experiencing?

[amenzhinsky]

@sdboyer probably that was just a one-time error
$GOPATH/pkg/dep/sources/https---go.googlesource.com-net/ was an empty directory only with .git in it and removing $GOPATH/pkg/dep/sources/ helped.

I'll file a new issue if the problem occurs again.

[sdboyer]

Ugh, more cache inconsistency problems...

At some point prior to the issue arising, did you interrupt (ctrl-c) a running dep command?

[amenzhinsky]

@sdboyer yes, I definitely did that when it just hung, probably because of networking problems.

[sdboyer]

@amenzhinsky yeah, we likely need to make it kill subprocesses less aggressively (e.g. SIGTERM instead of SIGKILL). More complex for us to manage, but it should help avoid leaving that kind of disk detritus around.

[amenzhinsky]

I think we could clone let's say a git repo to a tmp dir and when it's done we can move it to cache.

[sdboyer]

...that is a great idea :) there's a lot of contention around those segments, so i'd need to make sure it all folds together, but yeah - please make an issue for that!!

(will also have to wait until after we get gps moved in to dep)