gitfs broken on Fedora 29

[edgan]

Description of Issue/Question

I am trying to use ssh authentication with gitfs to pull git repositories from Github, but it seems broken on Fedora 29.

I have tried downgrading my libgit2 and pygit2 packages. Which gave a different error. I have double checked my private key for an empty line at the end, like in the release notes. I have successfully downloaded all the git repositories with the key configured. I tried an ECDSA key instead of an RSA key. I tried just doing formulas and not pillars.

Error

Feb 04 02:33:20 storage systemd[1]: Started The Salt Master Server.
Feb 04 02:35:19 storage salt-master[4467]: [ERROR   ] Error occurred fetching gitfs remote '[email protected]:edgan/salt-formulas': Failed to authenticate SSH session: Unable to send userauth-publickey request
Feb 04 02:35:19 storage salt-master[4467]: Traceback (most recent call last):
Feb 04 02:35:19 storage salt-master[4467]:   File "/usr/lib/python2.7/site-packages/salt/utils/gitfs.py", line 1703, in _fetch
Feb 04 02:35:19 storage salt-master[4467]:     fetch_results = origin.fetch(**fetch_kwargs)
Feb 04 02:35:19 storage salt-master[4467]:   File "/usr/lib64/python2.7/site-packages/pygit2/remote.py", line 405, in fetch
Feb 04 02:35:19 storage salt-master[4467]:     check_error(err)
Feb 04 02:35:19 storage salt-master[4467]:   File "/usr/lib64/python2.7/site-packages/pygit2/errors.py", line 64, in check_error
Feb 04 02:35:19 storage salt-master[4467]:     raise GitError(message)
Feb 04 02:35:19 storage salt-master[4467]: GitError: Failed to authenticate SSH session: Unable to send userauth-publickey request
Feb 04 02:35:20 storage salt-master[4467]: [ERROR   ] Error occurred fetching git_pillar remote 'master [email protected]:edgan/salt-pillars': Failed to authenticate SSH session: Unable to send userauth-publickey

Setup

/etc/salt/master:
ext_pillar:
  - git:
    - master [email protected]:edgan/salt-pillars:
      - pubkey: /root/.ssh/id_rsa.pub
      - privkey: /root/.ssh/id_rsa
    - master [email protected]:edgan/salt-grains:
      - pubkey: /root/.ssh/id_rsa.pub
      - privkey: /root/.ssh/id_rsa

extension_modules: /etc/salt/modules
failhard: True
fileserver_backend:
  - git
gitfs_provider: pygit2
gitfs_saltenv_whitelist:
  - base

gitfs_base: master
git_pillar_base: master

gitfs_remotes:
  - [email protected]:edgan/salt-formulas:
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa
  - [email protected]:edgan/salt-grains:
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa

gpg_keydir: /etc/salt/gpgkeys
hash_type: sha256

state_output: mixed
state_top_saltenv: base

reactor:
  - 'minion_start':
    - /etc/salt/reactor/minion_start.sls

Steps to Reproduce Issue

/usr/bin/salt-run git_pillar.update
/usr/bin/salt-run fileserver.update

Versions Report

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.27.4
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.27.2
         Python: 2.7.15 (default, Oct 15 2018, 15:26:09)
   python-gnupg: Not Installed
         PyYAML: 4.2
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 5.0.2
            ZMQ: 4.1.6
 
System Versions:
           dist: fedora 29 Twenty Nine
         locale: UTF-8
        machine: x86_64
        release: 4.20.3-200.fc29.x86_64
         system: Linux
        version: Fedora 29 Twenty Nine

[edgan]

Workaround is to upgrade to libssh2 1.8.1. Which I did by making my own upgraded package.

It makes sense, because this works on Ubuntu which has libssh2 1.8.1. The problem kind of is that 1.8.1 is not an official release, which is why Fedora hasn't updated. I am asking the libssh2 developers about making a 1.8.1 release.

[Ch3LL]

thanks this will be useful to some other users if they run into it as well. for now i'll just tag this as an upstream issue.

[tyhunt99]

I am seeing a similar issue with Ubuntu and followed the same verification steps but even upgrading libssh2 as per suggestion did not resolve it. I have tried both 1.8.1 and 1.9.0 and no luck

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.12.3
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: 1.10.6
          gitdb: 2.0.3
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.2
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 2.7.15+ (default, Nov 27 2018, 23:36:35)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1021-aws
         system: Linux
        version: Ubuntu 18.04 bionic

[Ch3LL]

@tyhunt99 was this working previously? Just want to ensure your authentication is also setup correctly.

[tyhunt99]

Yes I believe all 2018.3.* were working for me. This particular instance I am working on now is a brand new one starting at 2019.2.0 so it would take some work but I can try and downgrade to see if it is still working

[Ch3LL]

what version of pygit2 you running? If 0.27.4 you might need this fix: #51304

[Ch3LL]

Oh i just saw its in your version report. Looks like your on pygit2: 0.28.2. can you try that PR?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.