dep init does not attempt to use ssh #1041

shoenig · 2017-08-21T03:58:18Z

What version of Go (`go version`) and `dep` (`git describe --tags`) are you using?

$ go version
go version go1.8.3 linux/amd64

$ pwd
~/Documents/go/src/github.com/golang/dep
$ git describe --tags
v0.3.0-144-g1bef80e

What `dep` command did you run?

running the dep executable
dep command is `dep init --skip-tools=true --no-examples=false --gopath=true --v=true`
--- dep command output follows ---
Getting direct dependencies...
Checked 9 directories for packages.
Found 6 direct dependencies.
Searching GOPATH for projects...
remote repository at https://gitlab.internal.com/gophers/rlog.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/rlog
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
remote repository at https://gitlab.internal.com/gophers/3rdparty.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/3rdparty
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
remote repository at https://gitlab.internal.com/gophers/libtest.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/libtest
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
remote repository at https://gitlab.internal.com/gophers/libmysql.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/libmysql
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
remote repository at https://gitlab.internal.com/gophers/elements.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/elements
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
remote repository at https://gitlab.internal.com/gophers/libstatsd.git does not exist, or is inaccessible
Unable to cache gitlab.internal.com/gophers/libstatsd
main.(*gopathScanner).scanGopathForDependencies.func1
	/home/hoenig/Documents/go/src/github.com/golang/dep/cmd/dep/gopath_scanner.go:198
runtime.goexit
	/opt/google/go/src/runtime/asm_amd64.s:2197
Following dependencies were not found in GOPATH. Dep will use the most recent versions of these projects.
  gitlab.internal.com/gophers/rlog
  gitlab.internal.com/gophers/3rdparty
  gitlab.internal.com/gophers/elements
  gitlab.internal.com/gophers/libmysql
  gitlab.internal.com/gophers/libstatsd
  gitlab.internal.com/gophers/libtest
Root project is "gitlab.internal.com/systools/lassie"
 6 transitively valid internal packages
 11 external packages imported from 6 projects
(0)   ✓ select (root)
(1)	  ← no more versions of gitlab.internal.com/gophers/3rdparty to try; begin backtrack
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 2.275350434s
     b-source-exists: 213.605407ms
         select-root:    469.254µs
            new-atom:     25.308µs
  b-deduce-proj-root:      7.152µs
               other:      3.956µs

  TOTAL: 2.489461511s

remote repository at https://gitlab.internal.com/gophers/3rdparty.git does not exist, or is inaccessible

--- end of dep command output ---

What did you expect to see?

Complete success - we are configured to only clone via ssh. This is a hard requirement, and no https based workaround will work. go dep claims to be able to clone via ssh, but evidently did not attempt to do so.

$ git clone [email protected]:gophers/3rdparty.git
Cloning into '3rdparty'...
remote: Counting objects: 10884, done.
remote: Compressing objects: 100% (7221/7221), done.
remote: Total 10884 (delta 3534), reused 9775 (delta 2476)
Receiving objects: 100% (10884/10884), 20.21 MiB | 6.80 MiB/s, done.
Resolving deltas: 100% (3534/3534), done.

Other

Never mind the semantics of the name ("3rdparty") of the repo being cloned, it's just an unfortunate coincidence

The text was updated successfully, but these errors were encountered:

darkowlzz · 2017-08-21T15:45:09Z

@shoenig thanks for trying dep. Sorry that it didn't work in your use case.

I don't think we can solve this issue in dep init itself. Go project repos provide the go-import meta tag, which is read by dep, as well as go get, to determine the repo location and protocol to use. More info.
e.g.:

<meta name="go-import" content="github.com/golang/dep git https://github.com/golang/dep.git">

The issue you are facing is more of a support for private/enterprise patterns, which is being tracked in #286 .

Hope this helps :)

xgfone · 2017-09-05T02:57:57Z

We have a private repository based on gitlab, which only allows to access it by ssh, not http or https directly.

If dep does not support ssh, we have no way to use dep to manage the package dependency.

sdboyer · 2017-09-05T03:34:51Z

@xgfone dep supports ssh, in the same way that go get does - as the non-default option that it falls back to when other things don't work. the necessity of specifying a user tends to make that tricky for pure import path inference.

if you're running a go-get metadata server, though, it should work without problem.

xgfone · 2017-09-05T06:25:46Z

@sdboyer Yes, i found. But i need to add the suffix .git after the repo, for example, i can't import gitlab.example.com/username/project, and import gitlab.example.com/username/project.git is OK.

I feel that gitlab.example.com/username/project.git is UGLY.

In $GOROOT/src/cmd/go/internal/get/vcs.go,

// Github
{
    prefix: "github.com/",
    re:     `^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[\p{L}0-9_.\-]+)*$`,
    vcs:   "git",
    repo:  "https://{root}",
    check: noVCSSuffix,
}

it's very nice to reject the suffix .git.

For git, i can only clone the repo like git clone [email protected]:username/project, not git clone https://gitlab.example.com/username/project.

Now, how can I solve the dependency by the format gitlab.example.com/username/project?

At present, my Gopkg.toml is like

[[constraint]]
    name = "gitlab.example.com/username/project.git"
    version = "v1.0.0"

Then, import gitlab.example.com/username/project.git in my code files.

xgfone · 2017-09-05T06:35:52Z

go get does not support the customized private repository. So in order to support our private repos, i added the code below, which is copied from Github, into vcsPaths:

// gitlab.example.com
{
    prefix: "gitlab.example.com/",
    re:    `^(?P<root>gitlab\.example\.com/(?P<project>[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/[\p{L}0-9_.\-]+)*$`,
    vcs:   "git",
    repo:  "ssh://[email protected]/{project}",
    check: noVCSSuffix,
},

For repo, i only write ssh://[email protected]/{project}, not [email protected]/{project}. Or it will be considered as an invalid URL in the method isSecure.

Finally, it works.

xgfone · 2017-09-05T06:51:19Z

The suffix .git is nothing to tell dep or go get that the repository is based on git.

I think, in this case, dep can be like git, which maybe have a global or local configuration file to recognize it. It's better to indicate it in Gopkg.toml.

xgfone · 2017-09-05T07:40:30Z

I have found a solution from ibrasho.

[[constraint]]
    name = "gitlab.example.com/username/project"
    source = "gitlab.example.com/username/project.git"
    version = "v1.0.0"

It works.

It's not fairly perfect, but can have solved my problem. CHEERS!

sdboyer · 2017-09-05T12:14:46Z

i'm glad you found something that ou feel is acceptable 😄

shoenig · 2017-09-05T14:29:34Z

I do not feel this workaround is acceptable considering projects with vast numbers of transitive 3rd party dependencies. We want go dep to be capable of cloning mirrored dependencies without adding thousands these of hand-written constraints for every project. Please, have some empathy for the enterprise use case.

sdboyer · 2017-09-06T00:29:42Z

@shoenig i quite agree - having to add manually add source information in this way is not an acceptable solution at scale. we never intended it as such.

Please, have some empathy for the enterprise use case.

we do! see e.g. #286 #175. we've got folks working on a registry/intermediary server that should be able to address these sorts of woes, once and for all.

arbitrary custom matching logic, such as that given in #1041 (comment), is not something we can support. if we were to add such support, then we create a situation where projects are resolvable only if everyone knows has the same custom matching logic. and there's no way to make that feature "internal, enterprise-only", which would mean we'd be voluntarily creating a situation where the public ecosystem can fragment itself. that's a non-starter, so we need to wait for registries.

in the meantime, running a go get metadata proxy can do a lot. or using .git infixes in your import paths.

balaji39mit · 2018-03-19T13:25:46Z

Following is the comment from : @xgfone :

go get does not support the customized private repository. So in order to support our private repos, i added the code below, which is copied from Github, into vcsPaths:

// gitlab.example.com
{
prefix: "gitlab.example.com/",
re: ^(?P<root>gitlab\.example\.com/(?P<project>[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/[\p{L}0-9_.\-]+)*$,
vcs: "git",
repo: "ssh://[email protected]/{project}",
check: noVCSSuffix,
},
For repo, i only write ssh://[email protected]/{project}, not [email protected]/{project}. Or it will be considered as an invalid URL in the method isSecure.

Finally, it works.

@xgfone : The above solution is not working for me.! BTW, I am also under the same problem which you faced. We are hosting our code base on gitlab (private repo). I am still seeing that dep is trying to fetch the source using https instead of ssh.

Notes : I can be able to get the source using go get, but not using dep.

FQ400 · 2018-03-21T12:42:13Z

I ran into the issue with a private github repo on a CentOS. I could fix it by adding the following to a .gitconfig.

[url "[email protected]:"]
  insteadOf = https://github.com/

xgfone · 2018-03-26T06:53:37Z

@balaji39mit I think that it's not recommended way to modify the Go tool source. It's only my first test to find one way to support ssh, not only http. @FQ400 is right. Now I'm using this way, too.

gueluelue · 2018-04-17T15:52:35Z

@FQ400 thanks, your approach solved it for me

balaji39mit · 2018-04-19T12:33:47Z

@xgfone : Yeah, it worked. Thanks.

nextgentm1 · 2018-05-04T02:28:54Z

@FQ400 Thanks! Here is a one liner for Git Enterprise users:

git config --global url."git@github.yourEnterprise.com:".insteadOf "https://github.yourEnterprise.com/"

sdboyer added area: init bug labels Aug 27, 2017

sdboyer closed this as completed Sep 5, 2017

tuxlife mentioned this issue Jun 8, 2018

Add the git+ssh rewrite config settings #1889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dep init does not attempt to use ssh #1041

dep init does not attempt to use ssh #1041

shoenig commented Aug 21, 2017

darkowlzz commented Aug 21, 2017 •

edited

Loading

xgfone commented Sep 5, 2017

sdboyer commented Sep 5, 2017

xgfone commented Sep 5, 2017

xgfone commented Sep 5, 2017 •

edited

Loading

xgfone commented Sep 5, 2017 •

edited

Loading

xgfone commented Sep 5, 2017 •

edited

Loading

sdboyer commented Sep 5, 2017

shoenig commented Sep 5, 2017 •

edited

Loading

sdboyer commented Sep 6, 2017

balaji39mit commented Mar 19, 2018 •

edited

Loading

FQ400 commented Mar 21, 2018 •

edited

Loading

xgfone commented Mar 26, 2018

gueluelue commented Apr 17, 2018

balaji39mit commented Apr 19, 2018

nextgentm1 commented May 4, 2018

dep init does not attempt to use ssh #1041

dep init does not attempt to use ssh #1041

Comments

shoenig commented Aug 21, 2017

What version of Go (go version) and dep (git describe --tags) are you using?

What dep command did you run?

What did you expect to see?

Other

darkowlzz commented Aug 21, 2017 • edited Loading

xgfone commented Sep 5, 2017

sdboyer commented Sep 5, 2017

xgfone commented Sep 5, 2017

xgfone commented Sep 5, 2017 • edited Loading

xgfone commented Sep 5, 2017 • edited Loading

xgfone commented Sep 5, 2017 • edited Loading

sdboyer commented Sep 5, 2017

shoenig commented Sep 5, 2017 • edited Loading

sdboyer commented Sep 6, 2017

balaji39mit commented Mar 19, 2018 • edited Loading

FQ400 commented Mar 21, 2018 • edited Loading

xgfone commented Mar 26, 2018

gueluelue commented Apr 17, 2018

balaji39mit commented Apr 19, 2018

nextgentm1 commented May 4, 2018

What version of Go (`go version`) and `dep` (`git describe --tags`) are you using?

What `dep` command did you run?

darkowlzz commented Aug 21, 2017 •

edited

Loading

xgfone commented Sep 5, 2017 •

edited

Loading

xgfone commented Sep 5, 2017 •

edited

Loading

xgfone commented Sep 5, 2017 •

edited

Loading

shoenig commented Sep 5, 2017 •

edited

Loading

balaji39mit commented Mar 19, 2018 •

edited

Loading

FQ400 commented Mar 21, 2018 •

edited

Loading