machine_file resource does not work properly using with_machine_options

[poliva83]

The recipe below will provision the machine resource properly and converge recipe but fail when trying to download files afterwards using machine_file resource. However when you move remove the with_machine_options block and put driver_options & machine_options into a profile it actually works.

Maybe has something to do with how machine resource picks up with_machine_options through calls to new_machine_options or current_machine_options.
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provider/machine.rb#L30
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provider/machine.rb#L38
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provider/machine.rb#L130

machine_file resource doesn't appear to have such methods and ends up with Chef::Config object without the correct machine options instead of Cheffish::MergedConfig object.
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provider/machine_file.rb#L24
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provider/machine_file.rb#L19
https://github.com/chef/chef-provisioning/blob/master/lib/chef/provisioning/chef_run_data.rb#L93

Except in case you put machine options inside your profile and then machine_file will end up getting a Cheffish::MergedConfig object with the options defined.

chef-client -c ~/.chef/knife.rb ./provision.rb

• provision.rb

directory '/tmp/stash' do
  recursive true
end

one_endpoint     = "http://10.10.10.10/fake"
one_credentials  = "Account:fake"

with_driver "opennebula:#{one_endpoint}",
  :credentials => "#{one_credentials}"

base_image_name  = "ubuntu-14.04.1-pre-prod-20150305-40G"
base_image_uname = "chef"

with_machine_options :bootstrap_options => {
  :template => {
    "HTTPBASE" => "http://fake-cli.fake.net/~chef/context",
    "HTTPBASE2" => "http://fake-cli.fake.net/~Mandolin/chef-client-package",
    "MEMORY" => "2048",
    "CPU" => "2",
    "VCPU" => "2",
    "OS" => {
      "ARCH" => "x86_64"
    },
    "DISK" => {
      "IMAGE" => base_image_name,
      "IMAGE_UNAME" => base_image_uname,
      "DRIVER" => "qcow2"
    },
    "NIC" => {
      "NETWORK" => "chef12backend",
      "NETWORK_UNAME" => "Mandolin"
    },
    "GRAPHICS" => {
      "LISTEN" => "0.0.0.0",
      "TYPE" => "vnc"
    },
    "CONTEXT" => {
      "DOMAIN" => "chef.fake.net",
      "NETWORK" => "YES",
      "TEST_KITCHEN" => "YES",
      "HOSTNAME" => "$NAME",
      "INSTALL_CHEF_CLIENT_COMMAND" => "dpkg -E -i /mnt/chef_12.4.1-1_amd64.deb",
      "SSH_USER" => 'local',
      "SSH_PUBLIC_KEY" => ::File.read("#{ENV['HOME']}/.ssh/id_rsa.pub"),
      "FILES" => "$HTTPBASE/01_chef $HTTPBASE2/chef_12.4.1-1_amd64.deb"
    }
  }
},
:ssh_user => 'local',
:ssh_options => {
  :keys_only => false,
  :forward_agent => true,
  :use_agent => true,
  :user_known_hosts_file => '/dev/null'
},
:ssh_execute_options => {
  :prefix => 'sudo '
},
:cached_installer => true

machine 'bootstrap-backend' do
  tag ''
  recipe 'bb-chef-server-cluster'
  recipe 'chef-server-cluster::bootstrap'
  action :converge

  files ({
    '/etc/environment' => { :content => <<END_CLIENT_RB
END_CLIENT_RB
    }
  })
end

%w{ actions-source.json webui_priv.pem }.each do |analytics_file|
  machine_file "/etc/opscode-analytics/#{analytics_file}" do
    local_path "/tmp/stash/#{analytics_file}"
    machine 'bootstrap-backend'
    action :download
  end
end

%w{ pivotal.pem webui_pub.pem }.each do |opscode_file|
  machine_file "/etc/opscode/#{opscode_file}" do
    local_path "/tmp/stash/#{opscode_file}"
    machine 'bootstrap-backend'
    action :download
  end
end

CHEF_PROFILE=default chef-client -c ~/.chef/knife.rb ./provision.rb

• ~/.chef/knife.rb

profiles({
  'default' => {
    :driver => "opennebula:http://10.10.10.10/fake",
    :driver_options => {:credentials => "Account:fake"},
    :machine_options => { :bootstrap_options => {
        :template => {
            "HTTPBASE" => "http://fake-cli.fake.net/~chef/context",
            "HTTPBASE2" => "http://fake-cli.fake.net/~Mandolin/chef-client-package",
            "MEMORY" => "2048",
            "CPU" => "2",
            "VCPU" => "2",
            "OS" => {
              "ARCH" => "x86_64"
            },
            "DISK" => {
              "IMAGE" => base_image_name,
              "IMAGE_UNAME" => base_image_uname,
              "DRIVER" => "qcow2"
            },
            "NIC" => {
              "NETWORK" => "chef12backend",
              "NETWORK_UNAME" => "Mandolin"
            },
            "GRAPHICS" => {
              "LISTEN" => "0.0.0.0",
              "TYPE" => "vnc"
            },
            "CONTEXT" => {
              "DOMAIN" => "chef.fake.net",
              "NETWORK" => "YES",
              "TEST_KITCHEN" => "YES",
              "HOSTNAME" => "$NAME",
              "INSTALL_CHEF_CLIENT_COMMAND" => "dpkg -E -i /mnt/chef_12.4.1-1_amd64.deb",
              "SSH_USER" => 'local',
              "SSH_PUBLIC_KEY" => ::File.read("#{ENV['HOME']}/.ssh/id_rsa.pub"),
              "FILES" => "$HTTPBASE/01_chef $HTTPBASE2/chef_12.4.1-1_amd64.deb"
            }
          },
      :ssh_user => 'local',
      :ssh_options => {
        :keys_only => false,
        :forward_agent => true,
        :use_agent => true,
        :user_known_hosts_file => '/dev/null'
      },
      :ssh_execute_options => {
        :prefix => 'sudo '
      },
      :cached_installer => true
    }}
  }
})

[tyler-ball]

Hey @poliva83 - I am trying to confirm this. What version of chef-provisioning are you using? I don't have your opennebula driver so I cannot test with that, but I am unable to reproduce this behavior using chef-provisioning-aws.

I agree with you that machine_file resource calls the connect_to_machine method with different options than machine uses. But in chef-provisioning-aws this isn't an issue because the ssh_username is saved to the machine_spec which is the record of the state of the machine. Then when chef-provisioning calls connect_to_machine it calls into the driver's connect_to_machine method. Then the following call chain happens in c-p-a 1 2 3 4

The last call is where it loads the username from the spec or the machine_options.

I tried updating chef_run_data.rb to the following:

...
merged_config = Cheffish::MergedConfig.new({ :machine_options => current_machine_options }, config)
Chef::Provisioning.connect_to_machine(machine_spec, merged_config)

This ends up not mattering in the chef-provisioning-aws driver because the connect_to_machine method doesn't even look at the machine_options - only at the machine_spec.reference

It took me wayyyyyyy to long to figure this stuff out, which shows how confusing it is. But I think the idea of the machine_spec is to store information like the ssh_username and other related information needed for later interaction with the machine. I'll have to check with @jkeiser to confirm this, but chef-provisioning may be working correctly as-is.

Can you try updating your driver to have the connect_to_machine method use a similar logic to connect using the machine_spec?

Something we probably want to do is add this logic into chef-provisioning core (so driver authors are less confused). There is a lot of logic in chef-provisioning-aws that should move into chef-provisioning core, but until we have time to do that we may have to duplicate logic across drivers.

[tyler-ball]

Yeah, I talked to @jkeiser. Some background - we don't support resource cloning in chef-provisioning but we still want the following to work:

machine "my_machine" do
  machine_options :ssh_username => "centos"
  action :setup
end
# This will allocate the machine and add the correct files so that it is ready for convergence

machine "my_machine" do
  run_list "recipe[foo::default]"
end

The purpose of the machine_spec is to store any information (and only the minimum amount of information) necessary to connect to the machine later. The reason for this is that later references to the same machine don't need to supply connectivity information. It is loaded by the driver from the machine_spec.

The same would happen if we had resource cloning, but chef-provisioning doesn't have resource cloning.

[jkeiser]

Additionally, we want it to work even if you don't run the recipe with the machine_options in it, at all. People should be able to run that second bit as its own recipe, and have it work--specifically, it's really surprising if you take a resource out of a recipe, run it on its own, and it does something different than before.

[poliva83]

@tyler-ball I was using chefdk 0.6.0 which has chef-provisioning-1.1.1 gem installed.

Seems your right about how we implemented transport_for method in our driver. We expect that certain values like :ssh_user be set in machine_options. I will make changes to our driver based on your above suggestion to use machine_spec to store any information (and only the minimum amount of information) necessary to connect to the machine later.

opennebula_driver/driver.rb

  def machine_for(machine_spec, machine_options)
          if @one.get_resource('vm', { :id => machine_spec.location['server_id'].to_i }).nil?
            raise "#{machine_spec.name} (#{machine_spec.location['server_id']}) does not exist!"
          end
          # TODO: Support Windoze VMs (see chef-provisioning-vagrant)
          Chef::Provisioning::Machine::UnixMachine.new(machine_spec, transport_for(machine_spec, machine_options), convergence_strategy_for(machine_spec, machine_options))
        end

  def transport_for(machine_spec, machine_options)
          # TODO: Support Windoze VMs (see chef-provisioning-vagrant)
          transport = Chef::Provisioning::Transport::SSH.new(
            machine_spec.location['ip'],
            machine_options[:ssh_user],
            machine_options[:ssh_options] || {},
            machine_options[:ssh_execute_options] || { :prefix => 'sudo ' },
            machine_options[:ssh_config] || config)

          # wait up to 5 min to establish SSH connection
          100.times {
            break if transport.available?
            sleep 3
            Chef::Log.debug("Waiting for SSH server ...")
          }
          raise "Failed to establish SSH connection to #{machine_spec.name}" if !transport.available?
          transport
        end

@jkeiser My current workaround of putting these machine options in a profile allows machine_file to pick up the values. Is that because all machine options you put in profile becomes globally available to all the resources?

[tyler-ball]

@poliva83 It is because the chef_config are defaulted to the profile here. The chef_config is passed to all the connect methods, so even though machine_options does not have the ssh_user it is pulled from the chef_config which is populated from the profile. This is why it works when you are using profiles. I can figure out the whole call stack, but its pretty detailed so I'll leave it out unless you just want to know.

To be consistent with the AWS driver (most developed driver) you should be using ssh_username instead of ssh_user and machine_spec.reference instead of machine_spec.location.

Did you copy those from an existing driver, or is the driver author 'how to' information incorrect?

[poliva83]

@tyler-ball We started development of our driver sometime in January and I think the documentation changed in February about using machine_spec.reference instead of machine_spec.location.

see commit: 95254ff

Not sure about ssh_username vs ssh_user but ill make a note to make our driver follow current convention.

[poliva83]

As for the figuring out the whole call stack I don't think that is required. I already looked at that before in detail and needed to pop a couple aspirin afterwards because my brain hurt :)

So having values in machine_spec (ex. ssh_username) being first priority would mean once their set initially they can never be overridden by machine options later?

[tyler-ball]

So having values in machine_spec (ex. ssh_username) being first priority would mean once their set initially they can never be overridden by machine options later?

Correct. But that was the behavior we wanted in AWS. Once the SSH key is tied to an instance you cannot change that. Is Opennebula different?

[poliva83]

This is not different in Opennebula. Once I specified ssh_username variable in template and instantiate Opennebula vm I cannot change it.

[pburkholder]

@poliva83 @tyler-ball So, what needs to happen here for this issue to be fixed, and/or for the OpenNebula folks to be able to move forward?

[tyler-ball]

Can you try updating your driver to have the connect_to_machine method use a similar logic to connect using the machine_spec?

Basically that (quoted from #390 (comment))

[poliva83]

@tyler-ball @pburkholder I've logged issue internally and will update our driver soon to follow the newer conventions. We are planning on opensourcing driver but there are still some features/issues we need to address (this being one of them) before we feel comfortable to release.

[tyler-ball]

@poliva83 I'm going to close this since we have a plan of attack. Thanks!

[poliva83]

@tyler-ball @pburkholder Just a small update for final closer of this issue. We updated our driver based on above recommendations and I am no longer experiencing original issue. Thanks again.