berks upload fails with gem error on 0.10.0

[marcusn]

After upgrade to 0.10.0 I get the following error with berks upload:

ERROR: Gem::ConflictError: Unable to activate berkshelf-4.0.1, because httpclient-2.7.0 conflicts with httpclient (~> 2.6.0)

[tyler-ball]

Hey @marcusn - do you have a ~/.chefdk folder? If so, can you try moving that to ~/.old.chefdk and see if it fixes your issue?

Any time you do chef gem install it adds the gems to that folder in your home directory and chef loads gems from there. Thats typically how dependency issues arise when updating the ChefDK.

[julian1]

I have the same,

rm  -rf ~/.chefdk 
chef gem install berkshelf
chef gem install knife-solo


knife solo cook ...
WARNING: #<Gem::ConflictError: Unable to activate berkshelf-4.0.1, because httpclient-2.7.0 conflicts with httpclient (~> 2.6.0)>
WARNING: Berkshelf could not be loaded
WARNING: Please add the berkshelf gem to your Gemfile or install it manually with `gem install berkshelf`

[marcusn]

Hi,

There is no .chefdk,this is an install with chef-dk cookbook on a vm with no gems installed.

[tyler-ball]

@julian1 If you have installed the ChefDK, you shouldn't need to chef gem install berkshelf - it is already included. If you don't do that do you still get the error?

@marcusn Is this the cookbook you use? And to make sure I understand - you have Chef installed on a VM and then run that cookbook to install the ChefDK, correct?

We test for these load errors so something is different between our test environment and your VM. We typically install by running the install.sh script from https://omnitruck.chef.io/install.sh - my guess is that the cookbook is doing something different. I took a quick look at the cookbook and I don't see any obvious issues. Can you run your berks upload command with the -d debug flag so we can try and get a full stacktrace from the error?

[alexanderadam]

My local ChefDK installation contains a chef binstub created by Appbundler (/opt/chefdk/bin/chef) containing in line 57:

gem "httpclient", "= 2.7.0"

The exact same installation also installed berkshelf 4.0.1.
It's dependencies look like (chef gem dependency berkshelf):

Gem berkshelf-4.0.1
[…]
  httpclient (~> 2.6.0)
[…]

Anyway issue 599 definitely looks like a duplicate.

I just skipped the ChefDK environment to use the "traditional" way (with dependency resolving in Bundler / Gemfile) and everything runs fine here (httpclient 2.6.0.1).

[tyler-ball]

@alexanderadam Thanks for pulling that info out - I agree with you, the chef executable pins to httpclient 2.7.0. In theory this should be fine because the appbundled berks executable (/opt/chefdk/bin/berks) pins to a different version (gem "httpclient", "= 2.6.0.1"). We can ship multiple versions of a gem in the ChefDK omnibus and activate different ones for different executables.

I do agree with you that if I use the chef executable to activate the berkshelf gem, then I should see the error that @marcusn reported. I ran chef exec berks search sudo on a cookbook and it didn't blow up, which surprised me.

@marcusn If you run chef exec berks search sudo on the affected machine do you see an error?

I don't think its an issue specific to knife-solo - its dependencies look fine.

[test-tk]$ chef gem dependency knife-solo
Gem knife-solo-0.5.1
  berkshelf (>= 3.0.0.beta.2, development)
  bundler (>= 0, development)
  chef (>= 10.20)
  coveralls (>= 0, development)
  erubis (~> 2.7.0)
  ffi (< 1.9.1, development)
  fog (>= 0, development)
  librarian-chef (>= 0, development)
  minitest (~> 4.7, development)
  mocha (>= 0, development)
  net-ssh (< 3.0, ~> 2.7)
  parallel (>= 0, development)
  rake (>= 0, development)
  rdoc (>= 0, development)

[janskarvall]

I have a similar problem, likely the same. Don't know how to work around it.

$ knife solo cook <my target host>
Running Chef on <my target host>...
Checking Chef version...
<my login name>@<my target host>'s password: 
<my login name>@<my target host>'s password: 
Enter the password for <my login name>@<my target host>: 
WARNING: #<Gem::ConflictError: Unable to activate berkshelf-4.0.1, because httpclient-2.7.0 conflicts with httpclient (~> 2.6.0)>
WARNING: Berkshelf could not be loaded
WARNING: Please add the berkshelf gem to your Gemfile or install it manually with `gem install berkshelf`
Uploading the kitchen...
<my login name>@<my target host>'s password: ^C
$
$ chef gem dependency knife-solo
WARN: Unresolved specs during Gem::Specification.reset:
      rake (~> 10.1)
      hashie (< 4.0, >= 2.0)
      rack (>= 0)
      rspec (~> 3.0)
      httpclient (>= 2.2.0.2, ~> 2.2)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Gem knife-solo-0.5.1
  berkshelf (>= 3.0.0.beta.2, development)
  bundler (>= 0, development)
  chef (>= 10.20)
  coveralls (>= 0, development)
  erubis (~> 2.7.0)
  ffi (< 1.9.1, development)
  fog (>= 0, development)
  librarian-chef (>= 0, development)
  minitest (~> 4.7, development)
  mocha (>= 0, development)
  net-ssh (< 3.0, ~> 2.7)
  parallel (>= 0, development)
  rake (>= 0, development)
  rdoc (>= 0, development)

$

I'm stuck and would be happy to get a workaround, or a hope for a solution. Would reverting to chefdk 0.9.0 from 0.10.0 help?

I've also tried $ gem clean knife-solo but surprisingly knife-solo still works. How can that be?

$ gem clean knife-solo
Cleaning up installed gems...
Clean Up Complete
$ knife solo cook <my target host>
Running Chef on <my target host>...
Checking Chef version...
<my login name>@<my target host>'s password: 
<my login name>@<my target host>'s password: 
Enter the password for<my login name>@<my target host>: 
WARNING: #<Gem::ConflictError: Unable to activate berkshelf-4.0.1, because httpclient-2.7.0 conflicts with httpclient (~> 2.6.0)>
WARNING: Berkshelf could not be loaded
WARNING: Please add the berkshelf gem to your Gemfile or install it manually with `gem install berkshelf`
Uploading the kitchen...
<my login name>@<my target host>'s password: ^C

[marcusn]

@tyler-ball No errors on that:

vagrant@default-ubuntu-1404:~$ chef exec berks search sudo
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
sudo (2.7.2)
sudo_rules (0.1.4)

[marcusn]

Oddly enough, I cannot reproduce this in a vagrant vm now (I had to rollback the production deployment). Maybe this was the result of doing an upgrade?

[artursmolarek]

@marcusn: I can confirm the same problem on 0.10.0 on the similar env.

➜ ~ chef exec berks search sudo /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here sudo (2.7.2) sudo_rules (0.1.4)

[marcusn]

I just realized that berks was not the problem. The error came from a build log, and I mistook the error as coming from berks, but it actually comes from knife:

10:51 $ knife environment from file environments/*.rb -z
ERROR: Gem::ConflictError: Unable to activate berkshelf-4.0.1, because httpclient-2.7.0 conflicts with httpclient (~> 2.6.0)

We use berkshelf as a library in our environment files (ruby dsl), using require 'berkshelf'. Any ideas?

[tyler-ball]

@marcusn we've got a release coming up soon - I added this to the fix list. Hopefully we can just fix the dependency conflict as part of that

[xi-ao]

Hi,
I was wondering when the fix would be released, thanks in advance.

[danielsdeleo]

FYI, we can't guarantee that we will ship knife and ChefDK with compatible dependency sets, especially given the various dependencies used by different plugins. You will be better off if you can shell out to berks or write a script that uses berks and returns the data you need in a serialized form (i.e., JSON/YAML/etc.). We are in the process of updating httpclient everywhere but since we cannot guarantee that this use case won't be broken again in the future, I'm going to close this.

[alexanderadam]

What a pitty. I really thought ChefDK with Appbundler would bring consistency and a functioning "working together" instead of bringing new (or old) issues.

But thank you for looking into this.

[matschaffer]

@danielsdeleo this is a bummer for folks like myself writing knife plugins.

With no guarantee of compatibility between the different top-level tools in chefdk, all plugin authors will need to shell out to other plugins.

At that point you've got multiple interpreters running and you're more prone to errors given everyone now has to re-parse tool output to make decisions rather than integrating at the ruby level.

[el-dude]

I am getting this error when trying to use knife:
[vagrant@localhost ~]$ cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[vagrant@localhost ~]$ rpm -qa | grep [c]hef
chefdk-2.0.26-1.el7.x86_64
[vagrant@localhost ~]$ sudo knife solo init chef_dir/
/opt/chfdk/embedded/lib/ruby/2.4.0/rubygems/specification.rb:2291:in 'raise_if_conflicts': Unable to activate knife-solo-0.6.0, because net-ssh-4.1.0 conflicts with net-ssh (< 4.0, >= 2.7) (Gem::ConflictError) from /opt/chefdk/embedded/lib/ruby/2.4.0/rubygems/specification.rb:1411:in 'activate' from /opt/chefdk/embedded/lib/ruby/2.4.0/rubygems.rb:220:in 'rescue in try_activate' from /opt/chefdk/embedded/lib/ruby/2.4.0/rubygems.rb:213:in 'try_activate' from /opt/chefdk/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:126:in 'rescue in require' from /opt/chefdk/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in 'require' from /root/.chefdk/gem/ruby/2.4.0/gems/knife-solo-0.6.0/lib/chef/knife/cook.rb:1:in '<top (required)>' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:85:in 'load' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:85:in 'block in load_commands' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:85:in 'each' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:85:in 'load_commands' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:95:in 'load_command' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife/core/subcommand_loader.rb:109:in 'command_class_from' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife.rb:153:in 'subcommand_class_from' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/knife.rb:214:in 'run' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/lib/chef/application/knife.rb:156:in 'run' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.2.20/bin/knife:25:in '<top (required)>' from /bin/knife:264:in 'load' from /bin/knife:264:in '<main>' [vagrant@localhost ~]$

[matschaffer]

@el-dude that appears to be the same error as matschaffer/knife-solo#517 - the 0.7.0 pre-release of knife-solo should fix it.