[BUG] GitFS fails to authenticate over SSH on Ubuntu 20.04 (focal) #58898

jvandertil · 2020-11-06T15:44:06Z

Description
Trying to get the GitFS backend to work keeps resulting in this error:

2020-11-06 15:15:37,245 [salt.utils.gitfs :1888][ERROR   ][42996] Error occurred fetching gitfs remote '[email protected]:<USER>/<REPO>.git': Failed to authenticate SSH session: Unable to send userauth-publickey request
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 1864, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/usr/lib/python3/dist-packages/pygit2/remote.py", line 423, in fetch
    check_error(err)
  File "/usr/lib/python3/dist-packages/pygit2/errors.py", line 61, in check_error
    raise GitError(message)
_pygit2.GitError: Failed to authenticate SSH session: Unable to send userauth-publickey request

It takes quite long for the authentication to fail, so I suspect a timeout is reached.

Setup
Configuration

fileserver_backend:
  - roots
  - git

file_roots:
  base:
    - /srv/salt

pillar_roots:
  base:
    - /srv/pillar

gitfs_base: main

gitfs_remotes:
  - [email protected]:<USER>/<REPO>.git:
    - pubkey: /<PATH>/ssh-git-ro.pub
    - privkey: /<PATH>/ssh-git-ro

The SSH key is added as a Deploy Key to the Git repository.
Also tried:

SSH key as key for user
4096 bit SSH key
3072 bit SSH key
chmod 777 on SSH key (+pub)
removing trailing whitespace from private key

All the attempts resulted in the same error. I am able to do a 'git clone' manually of the repository.

Steps to Reproduce the behavior
Installation is done through apt-get using the instructions here.
Installed python3-pygit2 package through apt-get

After setting up the master (and a minion) with the above config.
Create a new GitHub repository with a top.sls file on the main branch.
Generate an SSH key and add the SSH key as a Deploy Key to the GitHub repository.

Run as root:

salt-run cache.clear_git_lock gitfs type=update

salt-run -l debug fileserver.update

Expected behavior
The command succeeds

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3002.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 3.0.7
         Jinja2: 2.10.1
        libgit2: 0.28.3
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.6.1
         pygit2: 1.0.3
         Python: 3.8.5 (default, Jul 28 2020, 12:59:40)
   python-gnupg: 0.4.5
         PyYAML: 5.3.1
          PyZMQ: 18.1.1
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: ubuntu 20.04 focal
         locale: utf-8
        machine: x86_64
        release: 5.4.0-52-generic
         system: Linux
        version: Ubuntu 20.04 focal

Additional context
Also reported as a comment on this closed issue: #52636 (comment)

The text was updated successfully, but these errors were encountered:

welcome · 2020-11-06T15:44:08Z

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

eliasp · 2020-11-10T16:24:33Z

Duplicate of #51495

This is a known issue of libssh2-1.8.0 and should be fixed in libssh2-1.8.2 via libssh2/libssh2@165f05e and libssh2-1.9.0. See also the following items:

According to reports in libssh2/libssh2#336 there are also users still seeing this issue on 1.8.2, but for most it seems to be working.

jvandertil · 2020-11-10T18:34:50Z

Indeed. The issues seems to be as you describe. Unfortunately that means that Ubuntu 20.04 will not work with the libssh2 provided by the Canonical repositories. I have now installed pygit2 and libssh2 (1.9.0) manually, which fixes the problem.

eliasp · 2020-11-10T19:23:57Z

If Ubuntu were to update libssh2-1.8.0 to 1.8.2 this bug would be fixed - it's unlikely they're going to update to 1.9.0.
There are no bugs on Launchpad asking for this fix/update yet.

eliasp · 2020-12-03T23:37:42Z

Bad news for Ubuntu users - libssh2-1.8.0-2.1 (Ubuntu 20.04) already includes the necessary fixes and it still doesn't work...

From the changelog:

Fixed misapplied patch for user auth.

moved MAX size declarations

This is also confirmed by looking at the deb-src pkg of libssh2-1.8.0-2.1:

$ apt source libssh2
[...]
dpkg-source: info: applying Fixed-misapplied-patch-327.patch
dpkg-source: info: applying moved-MAX-size-declarations-330.patch

henrikor · 2020-12-16T15:20:23Z

For Ubuntu 20.04 working solution is to run:

apt remove python3-pygit2; apt install python3-pip; pip3 install pygit2

jvandertil · 2021-03-10T13:46:16Z

chmod 777 on SSH key (+pub)

Big security hole there. Now everyone can read your private key. OpenSSH should refuse to use it.

Yes, that's why I don't do that normally. It was just a debug step taken to exclude any permission issue while tracking down the original issue. I figured I'd note it in the issue to rule it out straight away.

OrangeDog · 2021-03-30T12:29:01Z

apt remove python3-pygit2; apt install python3-pip; pip3 install pygit2

I tried this and it made no difference

Error occurred fetching gitfs remote '[email protected]:example/saltstack.git': Failed to authenticate SSH session: Unable to send userauth-publickey request

What's weird is that it does seem to actually fetch it. Changes are visible, and salt-run fileserver.update works with no issues or error logs.

salt --versions

Salt Version:
          Salt: 3002.6

Dependency Versions:
          cffi: 1.14.5
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 1.1.0
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 1.5.0
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-70-generic
        system: Linux
       version: Ubuntu 20.04 focal

henrikor · 2021-03-30T12:45:23Z

Don't remember all details @OrangeDog. But on my server I have:

libssh2-1/focal,now 1.8.0-2.1build1 amd64 [installed,auto-removable] and
libssh-4/focal-updates,focal-security,now 0.9.3-2ubuntu2.1 amd64 [installed,automatic]

and:

❯ pip3 show pygit2
Name: pygit2
Version: 1.3.0
Summary: Python bindings for libgit2.
Home-page: http://github.com/libgit2/pygit2
Author: None
Author-email: None
License: GPLv2 with linking exception
Location: /usr/local/lib/python3.8/dist-packages
Requires: cffi, cached-property
Required-by:

OrangeDog · 2021-03-30T12:52:32Z

auto-removable

I've already removed that one. I can downgrade to pygit2==1.3.0 and see if that makes any difference.

mattiascockburn · 2021-08-18T07:54:06Z

This is an ongoing problem. I packaged the current version of pygit2 (1.6.0) and libgit2 (1.1.1), but this also leads to the behaviour described above. Installing the same module via pip works, so i'm a bit stumped how to proceed further, as installing random packages via pip is not a way to go for our production environment. Any suggestions on how to debug this further on Ubuntu 20.04?

jvandertil added the Bug broken, incorrect, or confusing behavior label Nov 6, 2020

sagetherage added the needs-triage label Nov 6, 2020

sagetherage assigned Ch3LL Nov 6, 2020

jvandertil closed this as completed Nov 10, 2020

This comment has been minimized.

Sign in to view

cdalvaro mentioned this issue Oct 24, 2021

_pygit2.GitError: Failed to authenticate SSH session: Unable to send userauth-publickey request cdalvaro/docker-salt-master#106

Closed

OrangeDog mentioned this issue Apr 27, 2023

[BUG] onedir blocks installation of shared libraries for pygit2 gitfs in 3006.0 #64121

Open

9 tasks

DaAwesomeP mentioned this issue May 17, 2023

[BUG] Maintain version locking/compatibility/updates with optional modules #64299

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] GitFS fails to authenticate over SSH on Ubuntu 20.04 (focal) #58898

[BUG] GitFS fails to authenticate over SSH on Ubuntu 20.04 (focal) #58898

jvandertil commented Nov 6, 2020 •

edited

Loading

welcome bot commented Nov 6, 2020

eliasp commented Nov 10, 2020 •

edited

Loading

jvandertil commented Nov 10, 2020

eliasp commented Nov 10, 2020

eliasp commented Dec 3, 2020

henrikor commented Dec 16, 2020

This comment has been minimized.

jvandertil commented Mar 10, 2021

OrangeDog commented Mar 30, 2021 •

edited

Loading

henrikor commented Mar 30, 2021

OrangeDog commented Mar 30, 2021

mattiascockburn commented Aug 18, 2021

[BUG] GitFS fails to authenticate over SSH on Ubuntu 20.04 (focal) #58898

[BUG] GitFS fails to authenticate over SSH on Ubuntu 20.04 (focal) #58898

Comments

jvandertil commented Nov 6, 2020 • edited Loading

welcome bot commented Nov 6, 2020

eliasp commented Nov 10, 2020 • edited Loading

jvandertil commented Nov 10, 2020

eliasp commented Nov 10, 2020

eliasp commented Dec 3, 2020

henrikor commented Dec 16, 2020

This comment has been minimized.

jvandertil commented Mar 10, 2021

OrangeDog commented Mar 30, 2021 • edited Loading

henrikor commented Mar 30, 2021

OrangeDog commented Mar 30, 2021

mattiascockburn commented Aug 18, 2021

jvandertil commented Nov 6, 2020 •

edited

Loading

eliasp commented Nov 10, 2020 •

edited

Loading

OrangeDog commented Mar 30, 2021 •

edited

Loading