ssh_key not used with kitchen login

[esmil]

Description

My instance has the ssh_key set in the .kitchen/<instance>.yml,
but kitchen login <instance> doesn't seem to use it.

Kitchen Version

This happened since I updated to v1.11.1 today. I probably skipped a few version so it could be introduced earlier.

Ruby Version

Standard Arch Linux
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

Platform Version

Arch Linux

Replication Case

Create a new instance with the ssh_key set. Then try kitchen login <instance> and see that you'll be prompted for a password.

[cheeseplus]

Could we get an example kitchen.yml and logs?

[esmil]

esmil@stitch~$ cd test
esmil@stitch~/test$ cat .kitchen.yml
---
driver:
  name: qemu

provisioner:
  name: chef_zero

platforms:
  - name: xenial
    driver:
      image: xenial.qcow2

suites:
  - name: default
esmil@stitch~/test$ kitchen converge -l debug
-----> Starting Kitchen (v1.11.1)
-----> Creating <default-xenial>...
       KVM enabled.
       Spawning QEMU..
       Waiting for SSH..
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen"}>
D      [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}> (echo '[SSH] Established')
       [SSH] Established
D      [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}> (sudo sh -s 2>/dev/null <<END
echo '127.0.0.1 default-xenial' >> /etc/hosts
hostnamectl --transient set-hostname default-xenial || hostname default-xenial
END
umask 0022
install -dm700 "$HOME/.ssh"
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIboBIvdtZLruuwDksO8qd90IFqArrUHksiv4292MrehEqEKqpGzto1I86oy7Y1sqi3j8G0PSFaKkXsICwyor7bjVygxTDFFOWq+h78vvFnbgq1qOubWST/E1QY/m9bIryxh9icNS4tLKlFpn9l9a4pmBAsN07DYVaocVUugkOxeCxr/KLUXdvmXj4Xfq837ultL1ggyaN2YqeDwn+GxM8RNm51rIRa4DL6Y1rPgztm3FL+A9MglzEjELbs8s6EDVewAG6do1HwS3LqCGxlfZgRieelzIHkyaqLRU4mzqbkhmfaR/U/fHQppRDYWb3CoCSBilJxxYfjLK1VDyXOWkH kitchen-qemu' > "$HOME/.ssh/authorized_keys"
)
D      [SSH] closing connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}>
       Finished creating <default-xenial> (0m6.93s).
-----> Converging <default-xenial>...
       Preparing files for transfer
D      Creating local sandbox in /tmp/default-xenial-sandbox-20160830-10130-1rwb0t7
       Preparing dna.json
D      Creating dna.json from {:run_list=>[]}
       Berksfile, Cheffile, cookbooks/, or metadata.rb not found so Chef will run with effectively no cookbooks. Is this intended?
       Removing non-cookbook files before transfer
       Preparing validation.pem
D      Using a dummy validation.pem
       Preparing client.rb
D      Creating client.rb from {:node_name=>"default-xenial", :checksum_path=>"/tmp/kitchen/checksums", :file_cache_path=>"/tmp/kitchen/cache", :file_backup_path=>"/tmp/kitchen/backup", :cookbook_path=>["/tmp/kitchen/cookbooks", "/tmp/kitchen/site-cookbooks"], :data_bag_path=>"/tmp/kitchen/data_bags", :environment_path=>"/tmp/kitchen/environments", :node_path=>"/tmp/kitchen/nodes", :role_path=>"/tmp/kitchen/roles", :client_path=>"/tmp/kitchen/clients", :user_path=>"/tmp/kitchen/users", :validation_key=>"/tmp/kitchen/validation.pem", :client_key=>"/tmp/kitchen/client.pem", :chef_server_url=>"http://127.0.0.1:8889", :encrypted_data_bag_secret=>"/tmp/kitchen/encrypted_data_bag_secret", :named_run_list=>{}}
D      [SSH] reusing existing connection [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}>
D      [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}> (sudo -E sh -c '

chef_omnibus_root="/opt/chef"
chef_omnibus_url="https://omnitruck.chef.io/install.sh"
install_flags=""
pretty_version="install only if missing"
sudo_sh="sudo -E sh"
version="true"

tmp_stderr="/tmp/stderr";

# capture_tmp_stderr SOURCE
capture_tmp_stderr() {
  # spool up $tmp_stderr from all the commands we called
  if test -f "$tmp_stderr"; then
    output="`cat $tmp_stderr`";
    stderr_results="${stderr_results}\nSTDERR from $1:\n\n${output}\n";
    rm $tmp_stderr;
  fi
}

# do_curl URL FILENAME
do_curl() {
  echo "Trying curl...";
  curl -sL -D "$tmp_stderr" "$1" > "$2";
  ec=$?;
  # check for 404
  grep "404 Not Found" "$tmp_stderr" 2>&1 >/dev/null;
  if test $? -eq 0; then
    http_404_error "$1";
  fi

  # check for bad return status or empty output
  if test $ec -ne 0 || test ! -s "$2"; then
    capture_tmp_stderr "curl";
    return 1;
  else
    echo "Download complete.";
    return 0;
  fi
}

# do_download URL FILENAME
do_download() {
  echo "Downloading ${1} to file ${2}";

  exists wget;
  if test $? -eq 0; then
    do_wget "$1" "$2" && return 0;
  fi

  exists curl;
  if test $? -eq 0; then
    do_curl "$1" "$2" && return 0;
  fi

  exists fetch;
  if test $? -eq 0; then
    do_fetch "$1" "$2" && return 0;
  fi

  exists python;
  if test $? -eq 0; then
    do_python "$1" "$2" && return 0;
  fi

  exists perl;
  if test $? -eq 0; then
    do_perl "$1" "$2" && return 0;
  fi

  unable_to_download "$1" "$2";
}

# do_fetch URL FILENAME
do_fetch() {
  echo "Trying fetch...";
  fetch -o "$2" "$1" 2>"$tmp_stderr";
  ec=$?;
  # check for 404
  grep "Not Found" "$tmp_stderr" 2>&1 >/dev/null;
  if test $? -eq 0; then
    http_404_error "$1";
  fi

  # check for bad return status or empty output
  if test $ec -ne 0 || test ! -s "$2"; then
    capture_tmp_stderr "fetch";
    return 1;
  else
    echo "Download complete.";
    return 0;
  fi
}

# do_perl URL FILENAME
do_perl() {
  echo "Trying perl...";
  perl -e "use LWP::Simple; getprint(\$ARGV[0]);" "$1" > "$2" 2>"$tmp_stderr";
  ec=$?;
  # check for 404
  grep "404 Not Found" "$tmp_stderr" 2>&1 >/dev/null;
  if test $? -eq 0; then
    http_404_error "$1";
  fi

  # check for bad return status or empty output
  if test $ec -ne 0 || test ! -s "$2"; then
    capture_tmp_stderr "perl";
    return 1;
  else
    echo "Download complete.";
    return 0;
  fi
}

# do_python URL FILENAME
do_python() {
  echo "Trying python...";
  python -c "import sys,urllib2 ; sys.stdout.write(urllib2.urlopen(sys.argv[1]).read())" "$1" > "$2" 2>"$tmp_stderr";
  ec=$?;
  # check for 404
  grep "HTTP Error 404" "$tmp_stderr" 2>&1 >/dev/null;
  if test $? -eq 0; then
    http_404_error "$1";
  fi

  # check for bad return status or empty output
  if test $ec -ne 0 || test ! -s "$2"; then
    capture_tmp_stderr "python";
    return 1;
  else
    echo "Download complete.";
    return 0;
  fi
}

# do_wget URL FILENAME
do_wget() {
  echo "Trying wget...";
  wget -O "$2" "$1" 2>"$tmp_stderr";
  ec=$?;
  # check for 404
  grep "ERROR 404" "$tmp_stderr" 2>&1 >/dev/null;
  if test $? -eq 0; then
    http_404_error "$1";
  fi

  # check for bad return status or empty output
  if test $ec -ne 0 || test ! -s "$2"; then
    capture_tmp_stderr "wget";
    return 1;
  else
    echo "Download complete.";
    return 0;
  fi
}

# exists COMMAND
exists() {
  if command -v "$1" >/dev/null 2>&1; then
    return 0;
  else
    return 1;
  fi
}

# http_404_error URL
http_404_error() {
  echo ">>>>>> Downloading ${1} resulted in an HTTP/404, aborting";
  exit 40;
}

# should_update_chef ROOT VERSION
should_update_chef() {
  if test ! -d "$1"; then
    return 0;
  elif test "$2" = "true"; then
    return 1;
  elif test "$2" = "latest"; then
    return 0;
  fi

  if test -f "${1}/version-manifest.txt"; then
    chef_version="`head -n 1 ${1}/version-manifest.txt | cut -d \" \" -f 2`";
  else
    chef_version="`${1}/bin/chef-solo -v | cut -d \" \" -f 2`";
  fi

  echo "$chef_version" | grep "^${2}" 2>&1 >/dev/null;
  if test $? -eq 0; then
    return 1;
  else
    echo "${2}" | grep "^$chef_version" 2>&1 >/dev/null;
    if test $? -eq 0; then
      return 1;
    else
      return 0;
    fi
  fi
}

# unable_to_download URL FILE
unable_to_download() {
  echo "Unable to download $1 to $2, aborting";

  if test "x${stderr_results}" != "x"; then
    echo "\nDEBUG OUTPUT FOLLOWS:\n${stderr_results}";
  fi

  exit 10;
}

# main
main() {
  should_update_chef "$chef_omnibus_root" "$version"
  if test $? -eq 0; then
    echo "-----> Installing Chef Omnibus (${pretty_version})";

    # solaris 10 lacks recent enough credentials, so http url is used
    platform="`/usr/bin/uname -s 2>/dev/null`";
    platform_version="`/usr/bin/uname -r 2>/dev/null`";
    if test "x${platform}" = "xSunOS" && test "x${platform_version}" = "x5.10"; then
      chef_omnibus_url=`echo "$chef_omnibus_url" | sed -e "s/https/http/"`;
    fi

    do_download "$chef_omnibus_url" /tmp/install.sh;
    $sudo_sh /tmp/install.sh $install_flags;
  else
    echo "-----> Chef Omnibus installation detected (${pretty_version})";
  fi
}

# augment path in an attempt to find a download program
PATH="${PATH}:/opt/local/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/sfw/bin";
export PATH;

main
')
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}>
-----> Installing Chef Omnibus (install only if missing)
       Downloading https://omnitruck.chef.io/install.sh to file /tmp/install.sh
       Trying curl...
       Download complete.
       ubuntu 16.04 x86_64
       Getting information for chef stable  for ubuntu...
       downloading https://omnitruck.chef.io/stable/chef/metadata?v=&p=ubuntu&pv=16.04&m=x86_64
         to file /tmp/install.sh.1787/metadata.txt
       trying curl...
       sha1 72a0a10ba5684d1e51590c1bc022bcb2e99348c6
       sha256   973c2bc9a84822158ba7c0c360d0a25c97420f293ccbe5d8019615a411460785
       url  https://packages.chef.io/stable/ubuntu/14.04/chef_12.13.37-1_amd64.deb
       version  12.13.37
       downloaded metadata file looks valid...
       downloading https://packages.chef.io/stable/ubuntu/14.04/chef_12.13.37-1_amd64.deb
         to file /tmp/install.sh.1787/chef_12.13.37-1_amd64.deb
       trying curl...
       Comparing checksum with sha256sum...

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       You are installing an omnibus package without a version pin.  If you are installing
       on production servers via an automated process this is DANGEROUS and you will
       be upgraded without warning on new releases, even to new major releases.
       Letting the version float is only appropriate in desktop, test, development or
       CI/CD environments.

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       Installing chef 
       installing with dpkg...
       Selecting previously unselected package chef.
       (Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 10702 files and directories currently installed.)
       Preparing to unpack .../chef_12.13.37-1_amd64.deb ...
       Unpacking chef (12.13.37-1) ...
       Setting up chef (12.13.37-1) ...
       Thank you for installing Chef!
D      [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}> (sh -c '
TEST_KITCHEN="1"; export TEST_KITCHEN
sudo_rm="sudo -E rm"
dirs="/tmp/kitchen/clients /tmp/kitchen/cookbooks /tmp/kitchen/data /tmp/kitchen/data_bags /tmp/kitchen/encrypted_data_bag_secret /tmp/kitchen/environments /tmp/kitchen/roles"
root_path="/tmp/kitchen"

$sudo_rm -rf $dirs
mkdir -p $root_path
')
       Transferring files to <default-xenial>
D      TIMING: scp async upload (Kitchen::Transport::Ssh)
D      TIMING: scp async upload (Kitchen::Transport::Ssh) took (0m0.02s)
D      Transfer complete
D      Attempting to execute command - try 1 of 1.
D      [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}> (sh -c '
TEST_KITCHEN="1"; export TEST_KITCHEN
sudo -E /opt/chef/bin/chef-client --local-mode --config /tmp/kitchen/client.rb --log_level auto --force-formatter --no-color --json-attributes /tmp/kitchen/dna.json --chef-zero-port 8889 
')
       Starting Chef Client, version 12.13.37
       Creating a new client identity for default-xenial using the validator key.
       resolving cookbooks for run list: []
       Synchronizing Cookbooks:
       Installing Cookbook Gems:
       Compiling Cookbooks...
       [2016-08-30T12:29:38+00:00] WARN: Node default-xenial has an empty run list.
       Converging 0 resources

       Running handlers:
       Running handlers complete
       Chef Client finished, 0/0 resources updated in 01 seconds
D      Cleaning up local sandbox in /tmp/default-xenial-sandbox-20160830-10130-1rwb0t7
       Finished converging <default-xenial> (0m10.60s).
D      [SSH] shutting previous connection [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}>
D      [SSH] closing connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>16745, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :password=>"kitchen", :user=>"kitchen"}>
-----> Kitchen is finished. (0m17.59s)
esmil@stitch~/test$ cat .kitchen/default-xenial.yml
---
hostname: 127.0.0.1
port: 16745
username: kitchen
password: kitchen
acpi_poweroff: false
ssh_key: "/home/esmil/test/.kitchen/kitchen-qemu.key"
last_action: converge
esmil@stitch~/test$ kitchen login -l debug
D      Login command: ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=VERBOSE -p 16745 [email protected] (Options: {})
Warning: Permanently added '[127.0.0.1]:16745' (ED25519) to the list of known hosts.
[email protected]'s password: 
Authenticated to 127.0.0.1 ([127.0.0.1]:16745).
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-34-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Tue Aug 30 12:29:36 2016 from 192.168.1.2
kitchen@default-xenial:~$ logout
Connection to 127.0.0.1 closed.
Transferred: sent 2232, received 2940 bytes, in 1.1 seconds
Bytes per second: sent 2024.3, received 2666.5
esmil@stitch~/test$ 

[cheeseplus]

Normally one does not make the changes directly to .kitchen/instance.yml, these happen in the root .kitchen.yml. For ssh keys, try setting them under the transport key.

This could also be an issue with the qemu driver and later version of kitchen.

[esmil]

No, I did not make changes to the .kitchen/instance.yml. This is set by kitchen-qemu just like it is set by kitchen-vagrant. This used to work until I updated test-kitchen today.

[cheeseplus]

I see that password is set there as well and this has me suspecting #1095 which will use the password if provided.

[esmil]

Ah, ok. That makes sense. I'll try to make kitchen-qemu not set state[:password]

[esmil]

Ok, this fixes it. Thanks!
esmil/kitchen-qemu@3831032

[cheeseplus]

Glad we could narrow this down so quickly - the root issue was that we couldn't override a password without this fix and it'd always use SSH key.

[esmil]

Yeah, it's just a little funny that kitchen login <instance> doesn't seem to use the password either.

On the other hand I'm glad it didn't otherwise I wouldn't have seen that test-kitchen switched to using passwords underneath ;)