Support .ssh/config for specifying keys if ssh-agent fails

[alexcrichton]

Update (2018-10-30)

There's a workaround below for those interested:

[net]
git-fetch-with-cli = true

Original description

When cloning an SSH repository the only currently supported method of authenticating is picking up a key through ssh-agent. This can fail, however, for example if it's just not running! Cargo should support parsing .ssh/config and/or otherwise having a reasonable fallback in trying to find public/private keys on the filesystem. Currently libssh2 does not support this, so an external library will be required.

As to the rationale for this issue, apparently when using CircleCI with a deploy key it will add this to ~/.gitconfig:

[url "[email protected]:"]
  insteadOf = https://github.com/

Which means that clones of the index will be rewritten to [email protected] (SSH) instead of HTTPS. The ssh-agent apparently also isn't running, so it relies on ~/.ssh/config to point SSH at the right keys, which Cargo isn't itself looking at.

[travisofthenorth]

Is there a workaround for this for CircleCI? Running into this:

$ cargo build --release --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [23/-1] username does not match previous request

[alexcrichton]

@travisofthenorth that specific bug should be fixed on nightly I believe, although it may still not work on nightly when all put together

[travisofthenorth]

Thanks @alexcrichton. The nightly build does fix that issue. cargo build --verbose still seems to hang. I suppose that's what you were alluding to?

[alexcrichton]

Hm, that's odd that it hangs! It should either always make progress in one way or another or give an error message. Maybe there's just a slow download or slow network performance?

[travisofthenorth]

Network performance seems average on CircleCI. Downloading and installing rust takes ~15s. How long would you expect a build to take, and what output? This is all I saw after a few minutes:

$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`

[alexcrichton]

Weird, is it possible to get a stack trace of what Cargo is doing at that time?

[travisofthenorth]

Hmmm, am I doing something wrong here? I'm so puzzled, I'm getting no additional output:

$ RUST_BACKTRACE=1 cargo build --release --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`

[alexcrichton]

Oh I meant moreso attaching a debugger while it's running (RUST_BACKTRACE is only used on panics).

Another option may be to use RUST_LOG=cargo and see if that prints something useful

[travisofthenorth]

Are there docs on doing this? I'm pretty new to Rust (I'm actually working in a Ruby app and a Rust lib is getting compiled in a gem).

FWIW:

$ RUST_LOG=cargo cargo build --release --verbose
DEBUG:cargo::build: executing; cmd=cargo-build; args=["cargo", "build", "--release", "--verbose"]
DEBUG:cargo::ops::cargo_compile: compile; manifest-path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml
TRACE:cargo::ops::cargo_read_manifest: read_package; path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml; source-id=file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
DEBUG:cargo::ops::cargo_compile: loaded package; package=faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
DEBUG:cargo::core::registry: load/missing  file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::core::source: loading SourceId; file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::ops::cargo_read_manifest: read_package; path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml; source-id=file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::core::resolver: resolve; summary=faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
TRACE:cargo::core::resolver: activating faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
DEBUG:cargo::core::registry: load/missing  registry https://github.com/rust-lang/crates.io-index
TRACE:cargo::core::source: loading SourceId; registry https://github.com/rust-lang/crates.io-index
    Updating registry `https://github.com/rust-lang/crates.io-index`

[alexcrichton]

Probably not, but that looks like it's just a slow clone. So far it doesn't look like it's necessarily a Cargo problem so much as ensuring that a filesystem/network connection is looking good. If you can ssh into the builder (I think circleci allows that?) and attach a debugger to the Cargo process you can find out for sure, but it looks like libgit2's just chugging along cloning the index.

[travisofthenorth]

attach a debugger to the Cargo process

^ that's what I meant when I was asking "are there docs on doing this". These snippets are actually from my ssh session on a Circle box trying to install manually. I let it sit for 30m-1hr and it never finished, so I'm skeptical that it's just git.

[alexcrichton]

Ah yeah, to do that you should just need gdb -p <pid-of-cargo> which you can learn through something like ps aux | grep cargo. You could also just do gdb --args cargo build I believe. Once you're in gdb and it's paused when Cargo is hanging you should just need bt all I think

[travisofthenorth]

@alexcrichton I just ran rm .gitignore and it worked fine. So, I suppose it's a similar issue still present in the nightly?

Edit: I don't know if this is at all helpful, but here's the backtrace before removing the .gitignore file:

#0  0x00007fcef083878d in recv () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fcef1344909 in _libssh2_recv ()
#2  0x00007fcef1341e5f in agent_transact_unix ()
#3  0x00007fcef13424a5 in libssh2_agent_list_identities ()
#4  0x00007fcef131a898 in _git_ssh_authenticate_session ()
#5  0x00007fcef131b2b5 in _git_ssh_setup_conn ()
#6  0x00007fcef131998c in git_smart__connect ()
#7  0x00007fcef12f3095 in git_remote_connect ()
#8  0x00007fcef12f488c in git_remote_fetch ()
#9  0x00007fcef12b4826 in git2::remote::Remote::fetch::h60731caef39218e3 ()
#10 0x00007fcef1125799 in cargo::sources::git::utils::fetch::_$u7b$$u7b$closure$u7d$$u7d$::h2be50155b946ea5e ()
#11 0x00007fcef111e373 in cargo::sources::git::utils::fetch::h9819a463fcc3e7bd ()
#12 0x00007fcef1140397 in cargo::sources::registry::RegistrySource::do_update::he4e9bb63bd164d9b ()
#13 0x00007fcef0fa3d25 in _$LT$sources..registry..RegistrySource$LT$$u27$cfg$GT$$u20$as$u20$core..registry..Registry$GT$::query::h6f58831bb6ae841b ()
#14 0x00007fcef0ff52a4 in _$LT$core..registry..PackageRegistry$LT$$u27$cfg$GT$$u20$as$u20$core..registry..Registry$GT$::query::hd4c3f0cd3e3a40c6 ()
#15 0x00007fcef0fdd5f1 in _$LT$$RF$$u27$a$u20$mut$u20$I$u20$as$u20$std..iter..Iterator$GT$::next::he344e9055ac2d583 ()
#16 0x00007fcef0fd6c86 in cargo::core::resolver::Context::build_deps::h1981aabeeff3bf33 ()
#17 0x00007fcef0fcf495 in cargo::core::resolver::activate::h63b38b2f3bd7dc52 ()
#18 0x00007fcef0fc8e73 in cargo::core::resolver::activate_deps_loop::h373099c8772ce62b ()
#19 0x00007fcef0fc7375 in cargo::core::resolver::resolve::h20a2a85e670c0bae ()
#20 0x00007fcef10095ee in cargo::ops::resolve::resolve_with_previous::hd06ae7cd3726d0b0 ()
#21 0x00007fcef0ffb8c2 in cargo::ops::resolve::resolve_pkg::he4cb4924304ee511 ()
#22 0x00007fcef1005302 in cargo::ops::cargo_compile::resolve_dependencies::hc76d5aefb9b3e742 ()
#23 0x00007fcef1001c36 in cargo::ops::cargo_compile::compile_pkg::h48656ffa9b1541f3 ()
#24 0x00007fcef1000aea in cargo::ops::cargo_compile::compile::h1b43b20047c53d10 ()
#25 0x00007fcef0f29b0b in cargo::call_main_without_stdin::hec0ecdd41bac3cd3 ()
#26 0x00007fcef0f1ad2f in cargo::execute::hab7accd6bf9b5c64 ()
#27 0x00007fcef0f17574 in cargo::call_main_without_stdin::hf099bd36acb849a3 ()
#28 0x00007fcef0f144a2 in cargo::main::h2b219a79f378c1ef ()
#29 0x00007fcef14d9a99 in std::panicking::try::call::hc5e1f5b484ec7f0e ()
#30 0x00007fcef14e4aac in __rust_try ()
#31 0x00007fcef14e4a4f in __rust_maybe_catch_panic ()
#32 0x00007fcef14d94c4 in std::rt::lang_start::h61f4934e780b4dfc ()
#33 0x00007fcef026eec5 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#34 0x00007fcef0f13e29 in _start ()

[alexcrichton]

Oh this may be libssh2 going into the weeds which is being redirected via ~/.gitconfig and replaceWith (or some key named like that...)

[travisofthenorth]

It seems like insteadOf is the only key defined on CircleCI, at least on the box I was using.

[mscharley]

Seems like this is actually an issue with Circle. I worked around it with the following:

checkout:
  post:
    - git config --global --unset url.ssh://[email protected]:.insteadof

But the configuration is actually broken. The value assigned to this variable isn't correct which makes anything that gets transformed by it into a broken url, eg. ssh://[email protected]:/rust-lang/cargo

[wrl]

Is there a reason this is stalled? The current behaviour (ssh-agent being required) is very frustrating.

[yodaldevoid]

@Diggsey Are you on Windows and using the built in OpenSSH ssh-agent? If so, libssh2 doesn't support that ssh-agent, though there is work being done to remedy that.

[Diggsey]

@yodaldevoid yes

[r0fls]

I am experiencing this just trying to go through the getting started guide and run cargo build after adding the ferris-says dep: https://www.rust-lang.org/learn/get-started

Why does it even need an SSH key at all to download this?

[rnestler]

Why does it even need an SSH key at all to download this?

This should only be an issue if specifying dependencies via git with an ssh URL so something seems wrong in your case anyway.

[steveej]

Seems like this is actually an issue with Circle. I worked around it with the following:

checkout:
  post:
    - git config --global --unset url.ssh://[email protected]:.insteadof

But the configuration is actually broken. The value assigned to this variable isn't correct which makes anything that gets transformed by it into a broken url, eg. ssh://[email protected]:/rust-lang/cargo

first, this is very helpful, thank you!
however, as of today this snippet doesn't work on CircleCI. i believe that's for these two reasons:

1. checkout doesn't support post (anymore)
2. there's a typo in the given command.

here's what i find working today:

some-job:
      (...)
      steps:
         - checkout
         - run: git config --global --unset url.ssh://[email protected]

[beeb]

Please also support custom SSH agents. I use my password manager to handle my SSH keys and as such my SSH config looks like this:

Host *
	IdentityAgent "~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_github.pub
    IdentitiesOnly yes

What's more, I use https to ssh rewriting to support private repos in go projects, and my global git config is:

[url "[email protected]:"]
	insteadOf = https://github.com/

This setup absolutely doesn't work with cargo.

[rizalgowandy]

export CARGO_NET_GIT_FETCH_WITH_CLI=true

[BatmanAoD]

Using ssh access without the ssh-agent running seems like a fairly unusual setup to be using intentionally; I ran into this issue because I just forgot to start up the agent on OS X. Perhaps the hint message pointing users to net_git_fetch_with_cli could also suggest starting up ssh-agent as an alternative?