Use different signature algorithm for SSH host key

[heroin-moose]

Currently this plugin generates RSA keys:

packer-plugin-ansible/provisioner/ansible/provisioner.go

Line 993 in 97b6b77

key, err := rsa.GenerateKey(rand.Reader, 2048)

However, RSA/SHA-1 was deprecated in the latest OpenSSH release (changelog) and my builds fail with the following error:

qemu.example-vm: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Unable to negotiate with 127.0.0.1 port 37115: no matching host key type
found. Their offer: ssh-rsa", "unreachable": true}

Please, consider changing the algorithm to something newer.

[klausenbusk]

I did some research and switching to something else (ex:Ed25519) is more difficult than anticipated, as Go doesn't support the relevant key format (golang/go#37132).

See also this issue: golang/go#37278 for RFC8332 support.

Related issue: hashicorp/packer#8609

[klausenbusk]

I was too fast, this work and is supported by OpenSSH since release 5.7:

diff --git a/provisioner/ansible/provisioner.go b/provisioner/ansible/provisioner.go
index 076f25e..21876aa 100644
--- a/provisioner/ansible/provisioner.go
+++ b/provisioner/ansible/provisioner.go
@@ -7,8 +7,9 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
 	"crypto/rand"
-	"crypto/rsa"
 	"crypto/x509"
 	"encoding/pem"
 	"errors"
@@ -934,7 +935,7 @@ func newUserKey(pubKeyFile string) (*userKey, error) {
 		return userKey, nil
 	}
 
-	key, err := rsa.GenerateKey(rand.Reader, 2048)
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
 		return nil, errors.New("Failed to generate key pair")
 	}
@@ -945,9 +946,12 @@ func newUserKey(pubKeyFile string) (*userKey, error) {
 
 	// To support Ansible calling back to us we need to write
 	// this file down
-	privateKeyDer := x509.MarshalPKCS1PrivateKey(key)
+	privateKeyDer, err := x509.MarshalPKCS8PrivateKey(key)
+	if err != nil {
+		return nil, errors.New("Failed to marshal private key")
+	}
 	privateKeyBlock := pem.Block{
-		Type:    "RSA PRIVATE KEY",
+		Type:    "PRIVATE KEY",
 		Headers: nil,
 		Bytes:   privateKeyDer,
 	}
@@ -990,7 +994,7 @@ func newSigner(privKeyFile string) (*signer, error) {
 		return signer, nil
 	}
 
-	key, err := rsa.GenerateKey(rand.Reader, 2048)
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
 		return nil, errors.New("Failed to generate server key pair")
 	}

[yasn77]

Issue is also reported here #53

I am facing a similar problem with Arch Linux, I am getting around it right now by using docker to run packer, but it's pretty annoying :)

[klausenbusk]

x/crypto supports RSA SHA-2 (golang/crypto@b4de73f) now. So updating golang.org/x/crypto should fix this issue.

[NHAS]

I am also effected by this issue. Has there been any movement on this?

[ispirals]

I reached this as I'm facing the same issue with new versions of OS, is there any plan to fix this?

[ankitwal]

Not a complete solution but if you do not need a proxy adapter, toggling off use_proxy = false, avoids this issue.

[NHAS]

Another way you can get around this, which I think is rather clean is re-enabling the weak rsa-sha1 algorithm within your ssh client.

@ispirals

E.g Add this to your packer json/hcl

 "ansible_ssh_extra_args": [
                "-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"
]

To be fair, this is only for the ansible connection itself, as that will use your local ssh client & configuration

[adds68]

Issue is also reported here #53

I am facing a similar problem with Arch Linux, I am getting around it right now by using docker to run packer, but it's pretty annoying :)

Hey, interested to know how this workaround is used, do you have any examples?

[stefangweichinger]

same here: could use a working example, fails here with virtualbox-iso

==> debian.virtualbox-iso.bullseye: Executing Ansible: ansible-playbook -e packer_build_name="base-debian-amd64" -e packer_builder_type=virtualbox-iso -e packer_http_addr=10.0.2.2:8873 --ssh-extra-args '-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa' -e ansible_ssh_private_key_file=/tmp/ansible-key43104058 -i /tmp/packer-provisioner-ansible2143489858 /home/sgw/projects/packer/my-work/packer_sgw_builds/ansible/*****-debian-11-guest-additions.yml
    debian.virtualbox-iso.bullseye:
    debian.virtualbox-iso.bullseye: PLAY [all] *********************************************************************
    debian.virtualbox-iso.bullseye:
    debian.virtualbox-iso.bullseye: TASK [Gathering Facts] *********************************************************
    debian.virtualbox-iso.bullseye: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /home/sgw/.ansible/tmp/ansible-local-124735o9cwss1j/tmpe8pwoul3 /home/vagrant/.ansible/tmp/ansible-tmp-1651577057.3591588-124739-192345091164490/AnsiballZ_setup.py:\n\n"}
    debian.virtualbox-iso.bullseye:
    debian.virtualbox-iso.bullseye: PLAY RECAP *********************************************************************
    debian.virtualbox-iso.bullseye: default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
    debian.virtualbox-iso.bullseye:
==> debian.virtualbox-iso.bullseye: Provisioning step had errors: Running the cleanup provisioner, if present...
==> debian.virtualbox-iso.bullseye: Cleaning up floppy disk...
==> debian.virtualbox-iso.bullseye: Deregistering and deleting VM...
==> debian.virtualbox-iso.bullseye: Deleting output directory...
Build 'debian.virtualbox-iso.bullseye' errored after 6 minutes 40 seconds: Error executing Ansible: Non-zero exit status: exit status 2

I have this in my "debian-improved.pkr.hcl":

provisioner "ansible" {                                                       
  playbook_file = "ansible/vagrant-debian-11-guest-additions.yml"             
  user          = "${var.ssh_username}"                                       
  ansible_ssh_extra_args = [                                                    
                "-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"
 ]                                                                               
   }        

This is Packer 1.8.0

[adrianlzt]

I'm using this configuration in the .ssh/config file to avoid the error (no need for the ansible_ssh_extra_args config):

Host 127.0.0.1
      HostKeyAlgorithms +ssh-rsa
      PubkeyAcceptedKeyTypes +ssh-rsa

[lsampaioweb]

So the only option to make it work is by using the workaround??? I am having the same problem. I didn't like this option...

[Syntax3rror404]

"ansible_ssh_extra_args": ["-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"]
helped my out with rocky 9

[danwashusen]

Setting use_proxy = false fixed it for me...

https://www.packer.io/plugins/provisioners/ansible/ansible#use_proxy

[Fjan]

For anyone else running into the error pointed out by @stefangweichinger, the solution for that is in hashicorp/packer#11783 so the total snippet that works becomes:

            "ansible_ssh_extra_args": ["-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa -o IdentitiesOnly=yes"],
            "extra_arguments": [ "--scp-extra-args", "'-O'" ],

[manukm9]

"ansible_ssh_extra_args": [
                "-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"
]

after adding the above snippet I am getting the following error. Any help on this

{
    amazon-ebs.ubuntu18:     "changed": false,
    amazon-ebs.ubuntu18:     "msg": "Failed to connect to the host via ssh: command-line line 0: keyword hostkeyalgorithms extra arguments at end of line",
    amazon-ebs.ubuntu18:     "unreachable": true
    amazon-ebs.ubuntu18: }

[lethargosapatheia]

The problem with this solution (+ssh-rsa) is that it ignores the fact that some people actually use the much more secure ed25519 (which should already be standard by now). And that doesn't work in my case in Ubuntu 22.04, but it has been working on Ubuntu 20.04 for some reason. So that's great.

Disabling proxy does work though.

[jdptechnc]

Any plans to fix (with a real fix, and not using rsa) this issue? For ansible control environments on systems using a modern crypto policy (no sha1), just adding +rsa to the ansible config will not work.

[lbajolet-hashicorp]

Hey there,

I've started taking a look at this config, I was reluctant replacing RSA for this code, but from what I understand the keys generated are temporary for the proxy/adapter that we automatically start when provisioning with Ansible (unless use_proxy = false is specified), so since we don't rely on a target machine's support for more recent OpenSSH features, I think generating ED25519 keys would be a good solution indeed.
I'll start working on the code soon, I'll probably have to chat among the team as well to have more views on this, so this may take a bit of time, I hope you all understand.

I'm still a bit confused as to why we always end-up accepting ssh-rsa with sha1 only, since we're using a version of golang.org/x/crypto that accepts, in order, sha256, sha512 and sha1. So that's probably something else to investigate/fix, but using ed25519 won't have the same problem so that in and of itself should suffice.

To help developments on this, can I ask someone to provide a template that I can test the changes upon? Ideally something local like Qemu so I don't need extra infrastructure to test while developing this.

Thanks in advance, let's get this fixed ASAP!

[lbajolet-hashicorp]

Update: PR is open, I didn't go for ed25519 in the end since the serialised key is only accepted by OpenSSH when in its native format, which the Go libraries don't support yet.

I'll probably iterate on the PR before we can merge it, as much as ECDSA should be well-supported, I would like to introduce an option to fallback on RSA-SHA1 for users that desire to do so (if they run Packer/Ansible on an old distribution with OpenSSH that doesn't support ECDSA at all). I'll amend this PR, but in the meantime, I encourage you all to test out this change by locally compiling the plugin and testing it on your configurations.

Hope that fixes this problem! Sorry for letting that linger for this long.

[Kaappihullu]

"ansible_ssh_extra_args": [
                "-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"
]

after adding the above snippet I am getting the following error. Any help on this

{
    amazon-ebs.ubuntu18:     "changed": false,
    amazon-ebs.ubuntu18:     "msg": "Failed to connect to the host via ssh: command-line line 0: keyword hostkeyalgorithms extra arguments at end of line",
    amazon-ebs.ubuntu18:     "unreachable": true
    amazon-ebs.ubuntu18: }

for some reason apparently by no seen reason (none that I could find, one minute it works and then I get this same error) ansible sometimes wraps ansible_ssh_extra_args with '', so the ssh command fails. But if you use

extra_arguments": [ "--ssh-extra-args", "-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa" ]
instead, it will work.

with ansible_ssh_extra_args the parameters appear like this: ssh ... '-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa' and without: ssh ... -o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa

[lazywebm]

"ansible_ssh_extra_args": [
                "-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa"
]

after adding the above snippet I am getting the following error. Any help on this

{
    amazon-ebs.ubuntu18:     "changed": false,
    amazon-ebs.ubuntu18:     "msg": "Failed to connect to the host via ssh: command-line line 0: keyword hostkeyalgorithms extra arguments at end of line",
    amazon-ebs.ubuntu18:     "unreachable": true
    amazon-ebs.ubuntu18: }

for some reason apparently by no seen reason (none that I could find, one minute it works and then I get this same error) ansible sometimes wraps ansible_ssh_extra_args with '', so the ssh command fails. But if you use

extra_arguments": [ "--ssh-extra-args", "-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa" ] instead, it will work.

with ansible_ssh_extra_args the parameters appear like this: ssh ... '-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa' and without: ssh ... -o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa

Addendum for documentatio purposes:
For my pipeline, only "-o HostKeyAlgorithms=+ssh-rsa" was needed to make it work, every other notation failed. I'm using packer to provision VM templates including an running ansible playbook, so the provisioner section now looks like this:

provisioner "ansible" {
  ansible_env_vars = [ "ANSIBLE_BECOME_EXE=/usr/bin/sudo" ]
  extra_arguments = ["--ssh-extra-args", "-o HostKeyAlgorithms=+ssh-rsa", "--extra-vars", "@../template-builder-playbook/inventories/group_vars/all", "-e", "ansible_python_interpreter=auto"]
  playbook_file   = "../template-builder-playbook/builder.yml"
  user            = "${ssh_username}"
}

[eschulma]

The latest version of the ansible plugin fixes this. Add

packer {
  required_plugins {
    ansible = {
      version = ">= 1.1.0"
      source  = "github.com/hashicorp/ansible"
    }
  }
}

to your config file and run packer init .

[msandu62]

Facing the same issue "msg": "Failed to connect to the host via ssh: command-line line 0: keyword hostkeyalgorithms extra arguments at end of line" - any solution?

[lbajolet-hashicorp]

Hi @msandu62,

Regarding the SSH problem, as mentioned above, this problem has been fixed with version 1.1.0 of the plugin, please consider updating to this version or above if you are experiencing the RSA-key issue.

If you do need to specify extra SSH arguments even on the latest version of the plugin, the "extra arguments at end of line" is described by issue #158, a PR is in the pipeline for this, which I'll review today.

[dnv]

Not really sure how to proceed... I have an RHEL 8 JSON template I've previously succesfully used against 8.3, 8.5 and 8.8, now trying to build against 8.9, I am running head first into this very issue. My template is JSON and while my 9 template is HCL, I'd really rather not have to spend the time converting the old 8 JSON template to HCL and tidying it all up just so that I could use "packer init" to enforce requiring a newer Ansible plugin version.

I tried the 'use_proxy = false' workaround suggested here, but somehow, in that mode, packer seemingly isn't even trying to use password authentication and fails all authentication methods.

[lbajolet-hashicorp]

Hi @dnv,

Without updating to HCL2 for required_plugins, you can always install more recent versions of the plugin through packer plugins install, in your case if you want to require v1.1.0 (latest), you can run this:

$ packer plugins install "github.com/hashicorp/ansible" "v1.1.0"

Assuming it's not already installed, this will get v1.1.0 of the plugin, and unless some other version takes precedence over it, Packer will start using it for all your builds that use Ansible.

If you want to make sure you're using it, I encourage you to run packer with the PACKER_LOG=1 environment variable set so you get more verbose logs, which will tell you which plugins are being used/loaded for running your builds.

[dnv]

@lbajolet-hashicorp thank you so much!

This almost got me there, after manually updating the Ansible plugin, I faced another error complaining about a missing /usr/lib/sftp-server, but this was in turn solved by adding the following to ansible provisioner configuration:

"extra_arguments": [ "--scp-extra-args", "'-O'" ]