chef-client run never ends

[jtzl]

My goal is to build a workstation that will run test-kitchen and be able to launch LXCs inside it. I'm doing this with vagrant, and the vagrant image is getting provisioned properly with the chef_client provisioner.

From there, the "lxc" and "lxc::containers" recipes are added (in the Vagrantfile), along with a custom wrapper cookbook in which I'm using an lxc_container lwrp, as described in the lxc cookbook readme.

My vagrant image launches, downloads chef, and starts converging. It appears to hang after:

[2013-09-23T20:01:54+00:00] INFO: template[/var/lib/lxc/ubuntu_1204/rootfs/root/.ssh/authorized_keys] created file /var/lib/lxc/ubuntu_1204/rootfs/root/.ssh/authorized_keys
[2013-09-23T20:01:54+00:00] INFO: template[/var/lib/lxc/ubuntu_1204/rootfs/root/.ssh/authorized_keys] updated file contents /var/lib/lxc/ubuntu_1204/rootfs/root/.ssh/authorized_keys
[2013-09-23T20:01:54+00:00] INFO: template[/var/lib/lxc/ubuntu_1204/rootfs/root/.ssh/authorized_keys] mode changed to 600
[2013-09-23T20:01:54+00:00] INFO: ruby_block[lxc lock_default_users] called

And that's it. Here is the full chef run output: https://gist.github.com/jtzl/6678363

Is the apparent hanging related to my usage of the cookbook? Has anyone experienced and/or resolved this issue in their own environment?

Thanks!

Edit:

with VAGRANT_LOG=debug, it finishes lxc-create successfully and then goes to:

DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...

Over and over. It also did this while waiting on lxc-create, but it will go for over an hour without intervention.

[chrisroberts]

@jtzl Thanks for the report. I'll get a fresh build going and see if I can replicate. Is the host system precise?

[jtzl]

@chrisroberts Thanks for following up - I'm now having the same issue on 12.10, though I had the same problem on 12.04 before do-release-upgrade.

[chrisroberts]

@jtzl I haven't been able to get this hang to replicate. Would it be possible to post a gist of the provision with logging turned up to debug?

[mal]

Here's the end of the debug log from chef:

vagrant@mal-berkshelf:~$ tail -n100  /tmp/chef.log
[2013-12-07T14:55:08+00:00] INFO: Processing ruby_block[lxc lock_default_users] action run (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/lxc/providers/container.rb line 176)
[2013-12-07T14:55:08+00:00] DEBUG: Platform ubuntu version 12.04 found
[2013-12-07T14:55:08+00:00] INFO: ruby_block[lxc lock_default_users] called
[2013-12-07T14:55:08+00:00] INFO: Processing ruby_block[lxc default_password_scrub] action run (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/lxc/providers/container.rb line 197)
[2013-12-07T14:55:08+00:00] DEBUG: Skipping ruby_block[lxc default_password_scrub] due to not_if command `grep 'root:*' /var/lib/lxc/my_container/rootfs/etc/shadow`
[2013-12-07T14:55:08+00:00] INFO: Processing ruby_block[lxc start[my_container]] action run (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/lxc/providers/container.rb line 212)
[2013-12-07T14:55:08+00:00] DEBUG: Platform ubuntu version 12.04 found
[2013-12-07T14:55:08+00:00] DEBUG: sh(lxc-start -n my_container -d)
[2013-12-07T14:55:08+00:00] DEBUG: sh(lxc-info -n my_container)
[2013-12-07T14:55:09+00:00] DEBUG: sh(lxc-info -n my_container)
[2013-12-07T14:55:10+00:00] DEBUG: sh(lxc-info -n my_container)
...
[2013-12-07T14:58:35+00:00] DEBUG: sh(lxc-info -n my_container)
[2013-12-07T14:58:37+00:00] DEBUG: sh(lxc-info -n my_container)

This appears to happen indefinitely ;(

[mal]

And if I try to run that manually, it says that it isn't running, so I tried to start it:

vagrant@mal-berkshelf:~$ sudo lxc-start -n my_container
lxc-start: No such file or directory - failed to mount '/opt/file_store' on '/usr/lib/lxc/root//opt/file_store'
lxc-start: failed to setup the mounts for 'my_container'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'my_container'

[mal]

Creating the /opt/file_store directory prior to the lxc-start -n my_container -d fixes this for me and allows the chef run to complete, albeit unsucessfully; stack trace

[chrisroberts]

interesting. i'll have a look at the file store bit. the error you were running into in the stack trace is resolved in the latest release.

[mal]

That was my mistake, was trying to mount /opt/file_store which I copied from the example not appreciating that it was just a mount, as a result I was trying to mount a non-existent directory. Ooops!

Not had any problems now I'm using the new version, and using it correctly 😀