WinRM: dna.json is not updated on target machine during kitchen converge unless deleted

[stuartpreston]

During testing on a bare metal Windows platform, we noticed that a change in the run_list doesn't result in a change to the dna.json on the target machine during a converge.

2nd converge:

D      Creating local sandbox in C:/Users/stp39/AppData/Local/Temp/bss-windows20
08-sandbox-20160301-7596-4atgzn
       Preparing dna.json
D      Creating dna.json from {:run_list=>["recipe[bss::messenger_sysprep_should
_be_broken]"]}

[...]
D      [FileTransporter] File $env:TEMP\kitchen\dna.json is up to date, skipping

Whilst I know the typical use case is with a fresh machine, and probably not changing a run_list per converge this is something that I think should be working, no?

[stuartpreston]

C:\Users\stp39\bss>chef gem list --local winrm

*** LOCAL GEMS ***

winrm (1.7.2, 1.7.0)
winrm-fs (0.3.1)
winrm-s (0.3.7.dev, 0.3.6, 0.3.5, 0.3.4)
winrm-transport (1.0.3)

[mwrock]

I've seen people run into this too but never dug in. Are you seeing this on 1.5 or 1.6?

[stuartpreston]

This is still with 1.5 (for the moment). I'm not sure of the procedure to update a ChefDK binstub to 1.6 after downloading the gem...?

[stuartpreston]

OK, I changed the binstub manually, then got

C:\Users\stp39\bss>kitchen converge
-----> Starting Kitchen (v1.6.0)
-----> Converging <bss-windows2008>...
$$$$$$ Running legacy converge for 'Ssh' Driver
       Preparing files for transfer
       Preparing dna.json
       Preparing cookbooks from project directory
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [Invalid transport 'sspineg
otiate' specified, expected: negotiate, kerberos, plaintext, ssl.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Then changed my kitchen.yml to:

transport:
  name: winrm
  winrm_transport: negotiate

Then got:

C:\Users\stp39\bss>kitchen converge
-----> Starting Kitchen (v1.6.0)
-----> Converging <bss-windows2008>...
$$$$$$ Running legacy converge for 'Ssh' Driver
       Preparing files for transfer
       Preparing dna.json
       Preparing cookbooks from project directory
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [Invalid transport 'negotia
te' specified, expected: negotiate, kerberos, plaintext, ssl.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Investigating further...

[mwrock]

try completely uninstalling winrm-s

[stuartpreston]

Getting the same:

C:\Users\stp39\bss>chef gem uninstall winrm-s

Select gem to uninstall:
 1. winrm-s-0.3.4
 2. winrm-s-0.3.5
 3. winrm-s-0.3.6
 4. winrm-s-0.3.7.dev
 5. All versions
> 5
Successfully uninstalled winrm-s-0.3.4
Successfully uninstalled winrm-s-0.3.5
Successfully uninstalled winrm-s-0.3.6
Successfully uninstalled winrm-s-0.3.7.dev

C:\Users\stp39\bss>kitchen converge -l debug
-----> Starting Kitchen (v1.6.0)
D      winrm requested, loading winrm gem (["~> 1.6"])
D      winrm is loaded.
D      winrm-fs requested, loading winrm-fs gem (["~> 0.3"])
D      winrm-fs is loaded.
-----> Converging <bss-windows2008>...
$$$$$$ Running legacy converge for 'Ssh' Driver
       Preparing files for transfer
D      Creating local sandbox in C:/Users/stp39/AppData/Local/Temp/bss-windows20
08-sandbox-20160301-10028-e116rv
       Preparing dna.json
D      Creating dna.json from {:run_list=>["recipe[bss::messenger_sysprep_should
_be_broken]"]}
       Preparing cookbooks from project directory
D      Using cookbooks from C:/Users/stp39/bss/cookbooks
       Removing non-cookbook files before transfer
       Preparing validation.pem
D      Using a dummy validation.pem
       Preparing client.rb
D      Creating client.rb from {:node_name=>"bss-windows2008", :checksum_path=>"
\#{ENV['TEMP']}\\kitchen\\checksums", :file_cache_path=>"\#{ENV['TEMP']}\\kitche
n\\cache", :file_backup_path=>"\#{ENV['TEMP']}\\kitchen\\backup", :cookbook_path
=>["\#{ENV['TEMP']}\\kitchen\\cookbooks", "\#{ENV['TEMP']}\\kitchen\\site-cookbo
oks"], :data_bag_path=>"\#{ENV['TEMP']}\\kitchen\\data_bags", :environment_path=
>"\#{ENV['TEMP']}\\kitchen\\environments", :node_path=>"\#{ENV['TEMP']}\\kitchen
\\nodes", :role_path=>"\#{ENV['TEMP']}\\kitchen\\roles", :client_path=>"\#{ENV['
TEMP']}\\kitchen\\clients", :user_path=>"\#{ENV['TEMP']}\\kitchen\\users", :vali
dation_key=>"\#{ENV['TEMP']}\\kitchen\\validation.pem", :client_key=>"\#{ENV['TE
MP']}\\kitchen\\client.pem", :chef_server_url=>"http://127.0.0.1:8889", :encrypt
ed_data_bag_secret=>"\#{ENV['TEMP']}\\kitchen\\encrypted_data_bag_secret"}
D      [WinRM] negotiate::http://wt001466:5985/wsman<{:user=>nil, :pass=>nil, :n
o_ssl_peer_verification=>true, :disable_sspi=>false, :basic_auth_only=>false}> (
$chef_omnibus_root = "$env:systemdrive\opscode\chef"
$msi = "$env:TEMP\chef-.msi"
$chef_metadata_url = "https://www.chef.io/chef/metadata?p=windows&m=x86_64&pv=20
08r2&v="
$pretty_version = ""
$version = ""

Function Check-UpdateChef($root, $version) {
  if (-Not (Test-Path $root)) { return $true }
  elseif ("$version" -eq "true") { return $false }
  elseif ("$version" -eq "latest") { return $true }

  Try { $chef_version = (Get-Content $root\version-manifest.txt | select-object
-first 1) }
  Catch {
    Try { $chef_version = (& $root\bin\chef-solo.bat -v) }
    Catch { $chef_version = " " }
  }

  if ($chef_version.split(" ", 2)[1].StartsWith($version)) { return $false }
  else { return $true }
}

Function Get-ChefMetadata($url) {
  Try { $response = ($c = Make-WebClient).DownloadString($url) }
  Finally { if ($c -ne $null) { $c.Dispose() } }

  $md = ConvertFrom-StringData $response.Replace("`t", "=")
  return @($md.url, $md.md5)
}

Function Get-MD5Sum($src) {
  Try {
    $c = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvi
der
    $bytes = $c.ComputeHash(($in = (Get-Item $src).OpenRead()))
    return ([System.BitConverter]::ToString($bytes)).Replace("-", "").ToLower()
  } Finally { if (($c -ne $null) -and ($c.GetType().GetMethod("Dispose") -ne $nu
ll)) { $c.Dispose() }; if ($in -ne $null) { $in.Dispose() } }
}

Function Download-Chef($url, $md5, $dst) {
  Try {
    Log "Downloading package from $url"
    ($c = Make-WebClient).DownloadFile($url, $dst)
    Log "Download complete."
  } Finally { if ($c -ne $null) { $c.Dispose() } }

  if ($md5 -eq $null) { Log "Skipping md5 verification" }
  elseif (($dmd5 = Get-MD5Sum $dst) -eq $md5) { Log "Successfully verified $dst"
 }
  else { throw "MD5 for $dst $dmd5 does not match $md5" }
}

Function Install-Chef($msi) {
  Log "Installing Chef Omnibus package $msi"
  $p = Start-Process -FilePath "msiexec.exe" -ArgumentList "/qn /i $msi" -Passth
ru -Wait

  if ($p.ExitCode -ne 0) { throw "msiexec was not successful. Received exit code
 $($p.ExitCode)" }

  Remove-Item $msi -Force
  Log "Installation complete"
}

Function Log($m) { Write-Host "       $m`n" }

Function Make-WebClient {
  $proxy = New-Object -TypeName System.Net.WebProxy
  $proxy.Address = $env:http_proxy
  $client = New-Object -TypeName System.Net.WebClient
  $client.Proxy = $proxy
  return $client
}

Function Unresolve-Path($p) {
  if ($p -eq $null) { return $null }
  else { return $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFro
mPSPath($p) }
}

$chef_omnibus_root = Unresolve-Path $chef_omnibus_root
$msi = Unresolve-Path $msi

if (Check-UpdateChef $chef_omnibus_root $version) {
  Write-Host "-----> Installing Chef Omnibus ($pretty_version)`n"
  if ($chef_metadata_url -ne $null) {
    $url, $md5 = Get-ChefMetadata "$chef_metadata_url"
  } else {
    $url = $chef_msi_url
    $md5 = $null
  }
  Download-Chef "$url" $md5 $msi
  Install-Chef $msi
} else {
  Write-Host "-----> Chef Omnibus installation detected ($pretty_version)`n"
}
)
D      Cleaning up local sandbox in C:/Users/stp39/AppData/Local/Temp/bss-window
s2008-sandbox-20160301-10028-e116rv
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [Invalid transport 'negotia
te' specified, expected: negotiate, kerberos, plaintext, ssl.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: Failed to complete #converge action: [Invalid transport 'negotia
te' specified, expected: negotiate, kerberos, plaintext, ssl.]
D      ---Nested Exception---
D      Class: RuntimeError
D      Message: Invalid transport 'negotiate' specified, expected: negotiate, ke
rberos, plaintext, ssl.
D      ------Backtrace-------
D      C:/Users/stp39/AppData/Local/chefdk/gem/ruby/2.1.0/gems/winrm-1.7.2/lib/w
inrm/winrm_service.rb:54:in `rescue in initialize'
D      C:/Users/stp39/AppData/Local/chefdk/gem/ruby/2.1.0/gems/winrm-1.7.2/lib/w
inrm/winrm_service.rb:51:in `initialize'

[mwrock]

yuck. I'll be running through this tonight to figure out what's going on.

[mwrock]

Here is what I did starting from a chefdk version of 0.10.0:

1. cup chefdk # updated chefdk to 0.11.2 using chocolatey which is the equivilent of simply running the chefdk installer with all default settings
2. Ran kitchen converge

This resulted in the error: No live threads left. Deadlock? (fatal). I actually tried the above sequence on two separate machines with the same resulting error. After googling, I discovered most got around this by destroying their local gem repository. I deleted c:\users\matt\appdata\local\chefdk and that fixed things. I'm sure this could be tracked down to some single gem but no one seems to know which and there are alot in my case.

After getting past that I started getting the [key not found: "src_md5"] errors that many experience on windows converges with 1.5.0. So I updated to TK 1.6.0 with:

chef gem install appbundle-updater
appbundle-updater chefdk test-kitchen v1.6.0

Thanks @thommay for tipping me off to the appbundle-updater gem. It worked great to update my chefdk Test-Kitchen to 1.6.0.

Now kitchen converge succeeded

I changed the runlist and unfortunately was unable to reproduce @stuartpreston's issue. I reverted to 1.5.0 and was still unable to reproduce. I do know for sure I have seen this in the past. Perhaps it is driver related, I'm using kitchen-vagrant and I believe I saw it previously with kitchen-vsphere.

1.5.0 and 1.6.0 do have some implementation differences with regards to file uplads to the test instances so it is possible this is fixed in 1.6.0 but I'm not certain.

[stuartpreston]

Thanks Matt,

In my case I started with 0.11.2 on a clean machine.

I'll give the appbundle-updater a go tomorrow and see if that changes any other dependencies.

Regarding the driver I am simply using the ssh driver from test-kitchen itself and specifying winrm as the transport. I am also in a domain environment. Will do some more digging later this week when I'm back with the customer.

[stuartpreston]

Back on customer site, I got further with this, using Test Kitchen 0.6.0 (via 0.11.2 ChefDK, appbundle-updater and removing winrm-s gem) and after some debugging, there was a nested exception that was being swallowed:

If you do not specify a value for username and password (as originally worked with the :sspinegotiate transport option) then you end up here:

https://github.com/WinRb/WinRM/blob/master/lib/winrm/http/transport.rb#L146-L151

I didn't see this until I changed the following line:

https://github.com/WinRb/WinRM/blob/master/lib/winrm/winrm_service.rb#L54 to

raise e

.split was being called against a nil user. Net net, right now you must specify a valid username and password to do negotiate auth, I believe?

I guess I'm unblocked for now, but in a domain environment having to put creds in my .kitchen.local.yml file is sub-optimal...

And I still have the issue with the dna.json not getting overwritten when it gets changed...

[mwrock]

Thanks for this info. This uncovers the loss of a feature that we introduced with 1.5.0 and really did not know we had.

SSPINegotiate did not require credentials unlike rubyntlm. SSPINegotiate defaults to the ENV['username'], ENV['userdomain'] and calls native win32 AcquireCredentialsHandle to get a handle. From what I can surmise, when no creds are given, this will retrieve a handle for the currently logged on user. Prior to just about a month ago, the only consumer of wintm-s was knife-windows and it defaulted to Administrator if no user was given and prompted for a password if none was set. So when Test-Kitchen started using it in 1.5.0, we "stumbled" on this feature of using the current logon (only on a windows host) and regrettably now lose that feature.

There are a couple things we can do moving forward:

1. Short-Term: Fix the error handling so that missing credentials does not raise the Invalid Transport error but smething else more actionable.
2. Longer term: Add support for native win32 apis on windows hosts similar to winrm-s and use those instead of rubyntlm on windows.

I really do not think reintroducing winrm-s is a good solution. It simply monkey patches the winrm and httpclient gems and we have had lots of issues where it broke downstream gems. The better answer is to add the features in WinRM. It so happens I am a maintainer but dont have cycles to focus on this in the next few weeks.

[stuartpreston]

Thanks, I agree. I'm unblocked anyway for now. Appreciate the help.

If anyone else finds this and wants to do bare metal Winrm without Vagrant (like I did) then here's my .kitchen.yml which hijacks the built in proxy driver:

---
driver:
  name: proxy
  reset_command: $f="$env:TEMP\kitchen\dna.json";if (Test-Path $f){ Remove-Item $f }
  port: 5985

provisioner:
  name: chef_zero

verifier:
  name: inspec

transport:
  name: winrm

platforms:
  - name: win2008r2

suites:
  - name: default
    run_list:
      - recipe[mycookbook::myrecipe]
    attributes:

.kitchen.local.yml

---
driver:
  host: <FQDN OF MACHINE>
  username: <%= ENV['USERNAME'] %>
  password: <USER PASSWORD>

This is working with ChefDK 0.11.2 plus Test Kitchen 1.6.0 (updated via chef exec appbundle-updater) plus the removal of winrm-s gem.