git: SSH config not respected

[uhthomas]

Expected Behaviour

jj git clone should respect the SSH config (~/.ssh/config).

Actual Behaviour

The SSH config is not respected, which may be problematic in some cases. In this case, ProxyCommand is to communicate with a private Git server.

Steps to Reproduce the Problem

1. Use an SSH environment which requires additional configuration via ~/.ssh/config to work.
2. Observe jj git clone hang.

Specifications

• Version: 0.2.0.r917.2916cb2-1
• Platform: Linux x86_64

[martinvonz]

Thanks for your report. I'll be away from my computer for about a week from now. If anyone feels like looking into this while I'm away, I think the right place to start is here.

[valpackett]

libgit2 uses libssh2, not OpenSSH, so of course it doesn't use OpenSSH's configuration. And unfortunately there is no support for proxying at all. See libgit2/libgit2#4872 (one linked PR does something in that direction)

the right place to start is here

Hopefully the hardcoded id_rsa won't remain the only supported key path for non-agent operation there for long… :)

[martinvonz]

By the way, Git has its "credential helpers", which may be what we should use. I don't know enough about it yet.

[bergkvist]

Is it possible to use OpenSSH instead of libssh2?

[martinvonz]

Is it possible to use OpenSSH instead of libssh2?

I was going to say that I think we'd have to ask for such support from libgit2 (or switch to gitoxide or something) but I see you've already asked there (thanks). I have no idea how feasible it is.

[arxanas]

@bergkvist One alternative is what I do: I never use jj git and simply use git for all clone/fetch/push operations, and use jj's colocated repo mode to use jj with the Git repository by running jj init --git-repo .. I had to do this at a previous job where they used a custom version of Git. (I'm assuming that your installation of Git uses OpenSSH instead of libssh2 in whatever way you require.)

[bergkvist]

@arxanas Do you have to rerun jj init --git-repo . every time you do a git pull? Or does it "keep in sync" after you do it once?

[yuja]

jj init --git-repo . is needed only once.

jj git fetch is basically the same as git fetch --prune && jj git import, and for "colocated" repo created by --git-repo ., jj git import runs implicitly every time you do jj <subcommand>.

[bergkvist]

@martinvonz: See libgit2/libgit2#6617

[martinvonz]

@martinvonz: See libgit2/libgit2#6617

Wow, that's great news! Thanks for the update. Will you get a chance to try it?

[bergkvist]

I tested it out now, but I'm getting a segfault when using a git remote like [email protected]:bergkvist/jj. (Building https://github.com/bergkvist/jj will in theory also allow you to reproduce the segfault).

$ git clone [email protected]:bergkvist/jj && cd jj
$ cargo install --path=.
$ ~/.cargo/bin/jj init --git-repo=.
$ ~/.cargo/bin/jj git fetch
Segmentation fault

[bergkvist]

The above segfault has been fixed. Also, I had to enable SSH exec support in git2-rs (bergkvist/git2-rs@9a4d232).

You can test out installing my patched version like this:

git clone [email protected]:bergkvist/jj && cd jj
git checkout 4327e7ed1283329689efc31a5d244ec5cb707e16
cargo install --path=.

For jj, I guess this will be a matter of upgrading git2-rs once the change has propagated there - possibly with a feature flag specifying ssh exec instead of libssh2.

[willhbr]

jj git clone is hanging for me, but I don't have anything weird in my ssh config:

$ jj git clone [email protected]:willhbr/dotfiles.git dotfiles
Fetching into new repo in "/home/will/Projects/dotfiles"
Error: remote rejected authentication: Failed getting response; class=Ssh (23); code=Auth (-16)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?
$ ssh -F /dev/null [email protected]
PTY allocation request failed on channel 0
Hi willhbr! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

I do use an ed25519 key normally, but created ~/.id_rsa with the assumption that libgit2 would look for that. ssh -F /dev/null -i ~/.ssh/id_rsa [email protected] also works.

$ jj --version
jj 0.8.0-8149ec6df6490ac596f92fdf584d2d666e8c0b6b
$ uname -a
Linux brett 6.2.0-26-generic #26-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 10 23:39:54 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[yuja]

jj 0.8.0-8149ec6df6490ac596f92fdf584d2d666e8c0b6b

I think it's #1970, and should be fixed in the next release. For now, you can try installing jj from the git repo, or ssh-add the key to register to ssh agent.

[willhbr]

Confirmed that ssh-add worked:

$ eval `ssh-agent -s`
Agent pid 1721601
$ ssh-add
Identity added: /home/will/.ssh/id_rsa (will@brett)
Identity added: /home/will/.ssh/id_ed25519 (will@brett)
$ jj git clone [email protected]:willhbr/dotfiles.git
Fetching into new repo in "/home/will/Projects/dotfiles"
Working copy now at: b (no description set)
Parent commit      : f Add proper JJ config
Added 34 files, modified 0 files, removed 0 files

[necauqua]

libgit2/libgit2#6617 was just merged 🎉

So now it has to get released, git2-rs updated to enable the feature flag with a cargo feature flag, and then we can update git2-rs and enable the feature flag and it should just work, as @bergkvist showed

[julienvincent]

I am now running into this issue as well. I am using the 1Password IdentityAgent (so my ssh keys are not directly present in ssh-add -l. My config looks as follows:

Host *
	IdentityAgent "~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"

Is this configuration simply not supported or is there some workaround I can do to get this to work?

[bnjmnt4n]

I've made a PR which attempts to upstream libgit2/libgit2#6617 into git2-rs: rust-lang/git2-rs#1031. However, libgit2 still has not made an official release with the OpenSSH changes, so it might take a while for the changes to propagate.

I've tried building a version of jj with a prerelease version of git2-rs, and things are looking somewhat promising: jj git clone and jj git fetch is able to work with a custom ssh-agent (https://github.com/maxgoedjen/secretive), declared only via SSH_AUTH_SOCK, which it previously wasn't detecting. However, jj git push causes a segfault 😢. If I have some time I'll try to debug and see what's going on (might have messed something up in the upstream repo).

(My changes are at https://github.com/bnjmnt4n/jj/tree/ssh-openssh if anyone is interested in trying it out.)

[bnjmnt4n]

I've tested the git2-rs with OpenSSH on a WSL machine and it is able to fetch and push changes fine, so might just be an issue on my M2 Macbook. I guess I need to figure out how to use lldb 😬 Update: fixed!

For now, if anyone's interested in testing:

git clone [email protected]:bnjmnt4n/jj && cd jj
git checkout ssh-openssh
cargo install --path cli --locked --bin jj

[julienvincent]

Nice work @bnjmnt4n! I gave it a try locally and get the same result as you: jj fetch works just fine (and uses my custom IdentityAgent) but jj push segfaults. I am also running on an M2 though :)

[maan2003]

jj git push worked for me. I am on M1 but linux.

[bnjmnt4n]

I realized the issue, there have been changes to the libgit2 functions which have not been reflected in git2-rs FFI 🤦 I'll try to look at what's changed and see if I can update my fork of git2-rs just to get this working, but I'm not very familiar with libgit2 so I'm not sure how far I'll get. No matter what, it will probably be a bit of time before this even lands: we will have to wait for the next version of libgit2 to release, then update git2-rs FFI before this can land.

For now, I think you can treat the above repo as slightly suspect, even if it appears to work.

[maan2003]

finding ffi changes should be easy, just diff include/ from last released version in libgit2

[maan2003]

looks like git_push_options struct changed in libgit2, that explains the segfault

[bnjmnt4n]

Yeah, I've updated the my git2-rs PR and tried building jj locally, it seems to work fine now! Will take a closer look at the other changes.

[bnjmnt4n]

I've updated my fork to include the a now fully functional copy of jj. I will probably maintain it until the upstream PR is merged, so if anyone is interested, please do try and test it out and report bugs!

[bromanko]

I've updated my fork to include the a now fully functional copy of jj. I will probably maintain it until the upstream PR is merged, so if anyone is interested, please do try and test it out and report bugs!

Thanks for the fork! First few experiments work well with my 1Password-based SSH setup.

FWIW It would also be good to update this error message since libssh2 would no longer be in use.

[julienvincent]

Seems to work! Will be using this thanks for doing this @bnjmnt4n

[bnjmnt4n]

FYI: it was pointed out on Discord that the SSH_AUTH_SOCK is supported by libssh2, so if you're just looking to use a different SSH agent and don't have any other SSH configuration, you can just export SSH_AUTH_SOCK and use the latest Jujutsu. Eg. for 1Password.

[MikeJCusack]

@bnjmnt4n That does indeed work. The issue I'm having is I have multiple ssh keys managed by 1Password. It does successfully access the 1Password ssh agent, but it just picks the first key alphabetically. For stock git I have ~/.ssh/config configured for picking the ssh key. Since libssh2 does not use that config file what would be the equivalent? Or do we need to wait until OpenSSH is supported in libssh2?

[bnjmnt4n]

@MikeJCusack if you still require some configuration in ~/.ssh/config, the easiest way is likely to use #3191. OpenSSH is already supported in libgit2, but there are some pending PRs which have to be merged by the Rust binding git2-rs before we can consider merging #3191.

[MikeJCusack]

I was able to specify which key is provided first by creating the agent.toml file for 1Password's agent config, but there is a caveat that if that config is used it will ignore the preference of key in ~/.ssh/config so it will alway provide the first key configured in agent.toml. As I have multiple keys in use this is doable, but inconvenient. I suppose I will just need to wait for the PR to merge.

[MikeJCusack]

Although I suppose it can be improved by configuring the 1Password agent to remember the authorization for longer. I currently have it set to the default of until 1Password locks.

[sethherve]

FYI: in order to get this to work:

For now, if anyone's interested in testing:

git clone [email protected]:bnjmnt4n/jj && cd jj
git checkout ssh-openssh
cargo install --path=.

I had to run cargo install --path cli --locked --bin jj

[jessie828]

Hi all, I depend on this issue to be solved in order for me to be able to introduce jj at my company.

This is why I would like to offer help testing or anything else that is needed.

I am a complete noob when it comes to rust but would not mind a few pointers.

Please let me know what I can do to help this issue along.

[julienvincent]

@jessie828 This feature is dependent on patches landing upstream in git-rs.

See rust-lang/git2-rs#1031

There isn't really much that can happen in jujutsu itself until support has landed in that project.

If you are wanting to help out somehow I would recommend looking there.

[dbaynard]

In my case there's a simple workaround: the only part of the ssh config I needed was the User, which I was able to include in the URL.

jj git remote set-url origin "ssh://${REMOTE_USER?}@${REMOTE_URL?}"

[jessie828]

My problem is I need it to respect my Config Where I have some proxy settings declared.

Am i maybe doing something wrong?

[bnjmnt4n]

As mentioned above, reading from the SSH config file is not supported in Jujutsu currently. We're currently blocked on a PR made to an upstream project which requires further input (rust-lang/git2-rs#1031). However, you're welcome to build a custom jj binary with the following PR #3191 which has been reported to function properly (with regards to reading SSH configuration).

[beeb]

I have another use-case that might not be supported even with the unreleased changes from #3191 . Under WSL (Windows) and in combination with 1Password, I have configured Linux's git to use the ssh.exe from the Windows host, which is configured to use the 1Password agent (via C:\Users\user\.ssh\config):

git config --global core.sshCommand "ssh.exe"

But as I understand, this fix would not honor this configuration since jj uses the OpenSSH library to connect, and not the ssh command.

[bnjmnt4n]

#3191 should be able to handle this case. It refers to libgit2 using the OpenSSH binary (aka the ssh command).