Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for host key algorithms other than ssh-rsa on local ssh proxy #140

Closed
toadjaune opened this issue Jan 3, 2023 · 6 comments
Closed

Comments

@toadjaune
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

When using packer with the ansible provisionner, packer creates a local ssh server for ansible to connect to, and injects necessary configurations for ansible to connect to it. This design allows packer to use ansible regardless of the type of target and transport (for example, it makes possible to provision a local docker container with ansible, even though there isn't a ssh server in the container).

The only host key signature algorithm currently supported by this local server is ssh-rsa (as evidenced by the detailed logs below)

Openssh 8.8 has ssh-rsa signature algorithm disabled by default (release notes).

This makes a successful ssh handshake between a local ssh client and the packer local ssh server impossible, which in turn makes ansible fail.

Reproduction Steps

  • Have an operating system with openssh client >=8.8
    • Or manually disable ssh-rsa with HostKeyAlgorithms=-ssh-rsa (cf workarounds below on where to put this config)
  • Install packer
  • Install ansible
  • Create the following files in a directory :

debug.pkr.hcl

packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "python" {
  image  = "python:3"
  commit = true
}

build {
  name = "learn-packer"
  sources = [
    "source.docker.python"
  ]

  provisioner "ansible" {
    playbook_file = "./playbook.yml"
    extra_arguments = ["-vvvv"] # Make ansible and ssh client very verbose
    ansible_env_vars = [
      "ANSIBLE_CONFIG=./ansible.cfg", # Just to make the ansible debug output more readable
      "ANSIBLE_PIPELINING=true",
    ]
  }
}

ansible.cfg

[defaults]
stdout_callback=debug
stderr_callback=debug

playbook.yml

- name: Dummy test playbook
  hosts: all
  tasks:
    - debug: "msg=plop"
  • run packer build debug.pkr.hcl
  • Get a Unable to negotiate with 127.0.0.1 port XXXXX: no matching host key type found. Their offer: ssh-rsa error

Here is a detailed (although still abridged) log, the interesting bits are the host key algorithm lines.

Detailed log learn-packer.docker.python: output will be in this color.

==> learn-packer.docker.python: Creating a temporary directory for sharing data...
==> learn-packer.docker.python: Pulling Docker image: python:3
[...]
==> learn-packer.docker.python: Starting docker container...
[...]
==> learn-packer.docker.python: Using docker communicator to connect:
==> learn-packer.docker.python: Provisioning with Ansible...
learn-packer.docker.python: Setting up proxy adapter for Ansible....
==> learn-packer.docker.python: Executing Ansible: ansible-playbook -e packer_build_name="python" -e packer_builder_type=docker -vvvv -e ansible_ssh_private_key_file=/tmp/ansible-key1138914215 -i /tmp/packer-provisioner-ansible3443892612 /path/packer/playbook.yml
[...]
learn-packer.docker.python: PLAYBOOK: playbook.yml *********************************************************
[...]
learn-packer.docker.python: PLAY [Dummy test playbook] *****************************************************
learn-packer.docker.python:
learn-packer.docker.python: TASK [Gathering Facts] *********************************************************
learn-packer.docker.python: task path: /path/packer/playbook.yml:1
==> learn-packer.docker.python: failed to handshake
[...]
learn-packer.docker.python: debug3: ssh_connect_direct: entering
learn-packer.docker.python: debug1: Connecting to 127.0.0.1 [127.0.0.1] port 42711.
learn-packer.docker.python: debug3: set_sock_tos: set socket 3 IP_TOS 0x48
learn-packer.docker.python: debug2: fd 3 setting O_NONBLOCK
learn-packer.docker.python: debug1: fd 3 clearing O_NONBLOCK
learn-packer.docker.python: debug1: Connection established.
learn-packer.docker.python: debug3: timeout: 10000 ms remain after connect
learn-packer.docker.python: debug1: identity file /tmp/ansible-key1138914215 type -1
learn-packer.docker.python: debug1: identity file /tmp/ansible-key1138914215-cert type -1
learn-packer.docker.python: debug1: Local version string SSH-2.0-OpenSSH_8.8
learn-packer.docker.python: debug1: Remote protocol version 2.0, remote software version Go
learn-packer.docker.python: debug1: compat_banner: no match: Go
learn-packer.docker.python: debug2: fd 3 setting O_NONBLOCK
learn-packer.docker.python: debug1: Authenticating to 127.0.0.1:42711 as 'toadjaune'
learn-packer.docker.python: debug3: send packet: type 20
learn-packer.docker.python: debug1: SSH2_MSG_KEXINIT sent
learn-packer.docker.python: debug3: receive packet: type 20
learn-packer.docker.python: debug1: SSH2_MSG_KEXINIT received
learn-packer.docker.python: debug2: local client KEXINIT proposal
learn-packer.docker.python: debug2: KEX algorithms: [email protected],diffie-hellman-group-exchange-sha256,ext-info-c
learn-packer.docker.python: debug2: host key algorithms: [email protected],[email protected],[email protected],ssh-ed25519
learn-packer.docker.python: debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
learn-packer.docker.python: debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
learn-packer.docker.python: debug2: MACs ctos: [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
learn-packer.docker.python: debug2: MACs stoc: [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
learn-packer.docker.python: debug2: compression ctos: [email protected],zlib,none
learn-packer.docker.python: debug2: compression stoc: [email protected],zlib,none
learn-packer.docker.python: debug2: languages ctos:
learn-packer.docker.python: debug2: languages stoc:
learn-packer.docker.python: debug2: first_kex_follows 0
learn-packer.docker.python: debug2: reserved 0
learn-packer.docker.python: debug2: peer server KEXINIT proposal
learn-packer.docker.python: debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
learn-packer.docker.python: debug2: host key algorithms: ssh-rsa
learn-packer.docker.python: debug2: ciphers ctos: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
learn-packer.docker.python: debug2: ciphers stoc: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
learn-packer.docker.python: debug2: MACs ctos: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs stoc: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs ctos: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs stoc: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: compression ctos: none
learn-packer.docker.python: debug2: compression stoc: none
learn-packer.docker.python: debug2: languages ctos:
learn-packer.docker.python: debug2: languages stoc:
learn-packer.docker.python: debug2: first_kex_follows 0
learn-packer.docker.python: debug2: reserved 0
learn-packer.docker.python: debug1: kex: algorithm: [email protected]
learn-packer.docker.python: debug1: kex: host key algorithm: (no match)
learn-packer.docker.python: Unable to negotiate with 127.0.0.1 port 42711: no matching host key type found. Their offer: ssh-rsa
learn-packer.docker.python:
learn-packer.docker.python:
learn-packer.docker.python: PLAY RECAP *********************************************************************
learn-packer.docker.python: default : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
learn-packer.docker.python:
==> learn-packer.docker.python: Provisioning step had errors: Running the cleanup provisioner, if present...
==> learn-packer.docker.python: Killing the container: a79558c88072c48122a6e63257f52bdb9d28ce9815d198e627ed73a283d4a88d
Build 'learn-packer.docker.python' errored after 3 seconds 367 milliseconds: Error executing Ansible: Non-zero exit status: exit status 4

==> Wait completed after 3 seconds 367 milliseconds

==> Some builds didn't complete successfully and had errors:
--> learn-packer.docker.python: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.

### Packer / system versions
  • Packer : Packer v1.8.5
  • Ansible : 2.14.1
  • OS : Fedora 37 Workstation, with Linux 6.0.15-300.fc37.x86_64
  • SSH client : OpenSSH_8.8p1, OpenSSL 3.0.5

Workarounds

Re-enabling ssh-rsa with the HostKeyAlgorithms client configuration is usually enough to work around this problem.

On Fedora (and presumably RedHat), though, there are system-wide policies in /etc/crypto-policies/back-ends, enforcing this deprecations, and making the ssh client discard ssh-rsa when manually added by the user. We'll need the PubkeyAcceptedAlgorithms for that part.

Resuting configurations :

  • In packer configuration
    • extra_arguments = ["-vvvv", "--ssh-extra-args", "-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa"] in the example config file above
    • ansible_ssh_extra_args should do it too, although in my testing, it did not seem to work.
  • In ssh client configuration
    • Add the following at the top of ~/.ssh/config :
    Host 127.0.0.1
      HostKeyAlgorithms +ssh-rsa
      PubkeyAcceptedAlgorithms +ssh-rsa
    

Fixes suggestions

I see a few different ways to fix this on the packer side :

  • Adding HostKeyAlgorithms and PubkeyAcceptedAlgorithms to the default ssh options sent by packer :
    While this option is fast to implement and deploy, if users have defined custom ssh options, this would not apply to them. It is therefore quite brittle
  • Implementing support for newer rsa-based signature algorithms (rsa-sha2-512,rsa-sha2-256) :
    This would allow ssh clients to connect with their default configuration, and I assume does not require much development ?
  • Implementing support for ed25519 host keys :
    This would also work with default client configurations, but I assume represents more work to implement, as we're talking about an entirely distinct key/signature type.
@toadjaune toadjaune added the bug label Jan 3, 2023
@github-actions github-actions bot removed the bug label Jan 3, 2023
@nywilken nywilken transferred this issue from hashicorp/packer Feb 21, 2023
@github-actions
Copy link

This issue has been synced to JIRA for planning.

JIRA ID: HPR-981

@nywilken
Copy link
Contributor

@toadjaune thanks for calling this out. This sounds like a needed enhancement here. I've added this issue to our internal board for prioritization with the team.

@toadjaune
Copy link
Author

Great news, thanks for the update ! :)

@SylvainMartel
Copy link

Any update?

garyservin added a commit to locusrobotics/tailor-image that referenced this issue May 24, 2023
garyservin added a commit to locusrobotics/tailor-image that referenced this issue May 26, 2023
garyservin added a commit to locusrobotics/tailor-image that referenced this issue May 31, 2023
* Move to jammy

* Update docker version

* Manually install catkin

* Update python dependencies

* Increase base image size

* Exit if creating image failed

* Workaround to issue with ansible and amazon-ebs

Ref: hashicorp/packer-plugin-ansible#140
lfit-replication pushed a commit to lfit/releng-common-packer that referenced this issue Jun 29, 2023
As of packer version 1.7.0 HCL2 is the preferred way to write Packer
templates. HCL2 preserves existing workflows while leveraging HCL2’s
advanced features like variable interpolation and configuration
composability.

Migrate packer templates from JSON to HCL2 format. JSON format
templates are deprecated and no longer works with packer
version > 1.9.x.

Packer version 1.9.1 will be minimum required version for packer jobs.
This version requires installing the cloud specific plugins through
packer config and needs to be initalized and downloaded with
`packer init` before running `packer build`.

Add support for host key algorithms to work with local ssh proxy
without which packer builds results in "failed to handshake" error.
Workaround is to pass additional params with "extra_arguments".

Support for '.json' templates will be removed from common-packer in
subsequent releases. All projects specific templates not available in this
repository are required to convert existing '.json' to '.pkr.hcl' format.

This change requires updating CI jobs with the additional steps.

Ref:
https://developer.hashicorp.com/packer/docs/templates/hcl_templates
https://github.com/hashicorp/packer-plugin-openstack/blob/main/README.md
hashicorp/packer-plugin-ansible#140

Issue: RELENG-4764
Change-Id: Ie63d9551bd1bab224dc9335d45a21d5ee3e09550
Signed-off-by: Anil Belur <[email protected]>
@lbajolet-hashicorp
Copy link
Contributor

Hi @toadjaune,

Forgive me if I misunderstand the issue here, but I believe this is essentially similar to #69, isn't it?

If so, this has been fixed in the latest release of the plugin, so now when you are starting the adapter, the Ansible plugin will use ECDSA keys for the SSH communication, and not ssh-rsa by default.

Can you confirm if that's the case for you?

@toadjaune
Copy link
Author

Hi @lbajolet-hashicorp,

I can confim that the issue is no longer present in version 1.1.0 of the plugin. and that the connection successfully establishes with a ecdsa-sha2-nistp384 handshake.

Thanks a lot for this !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants