Cache vagrant-omnibus packages

[tknerr]

Opscode recently switched to provisionerless baseboxes and now installs Omnibus Chef via a Vagrant plugin:
https://github.com/schisamo/vagrant-omnibus

I just waited for >30 mins for the omnibus package to download via 3G (2.5G rather), but unfortunately it did not get cached as the vagrant-omnibus plugin kicks in before vagrant-cachier.

Given that this is a common big upfront download all the time, would it be possible to cache that as well?

[tknerr]

Install vagrant-omnibus:

vagrant plugin install vagrant-omnibus

Sample Vagrantfile:

require 'vagrant-omnibus'

Vagrant.configure("2") do |config|

  # use bare os baseboxes
  config.vm.box = "opscode_ubuntu-13.04_provisionerless"
  config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_ubuntu-13.04_provisionerless.box"

  config.omnibus.chef_version = "11.4.4"

  config.vm.define :foo do |foo_config|

    # with cache
    foo_config.cache.auto_detect = true

    foo_config.vm.provision :chef_solo do |chef|
      chef.cookbooks_path = [ './cookbooks' ]
      chef.add_recipe "apt"
      chef.add_recipe "apache2"
    end
  end
end

After vagrant up I can find the apache2 apt packages in the cache but not the omnibus apt package

[patcon]

Hm. I couldn't sworn I'd submitted an issue for this either here or in the vagrant-omnibus queues, but I guess it got lost in one of my many tabs before submitting. Thanks!

EDIT: Oh hey, it was vagrant-vbguest that I had the concern for: dotless-de/vagrant-vbguest#67. Seems like it's a common solution.

cc: @fnordfish @dotless-de

[patcon]

cc: @schisamo

[tknerr]

@patcon common issue or common solution? :-)

[fgrehm]

@tknerr I haven't got the time to test this one right now but would you be able to gist us VAGRANT_LOG=debug vagrant up? I might be able to find out what's going by reading it ;)

[tknerr]

@fgrehm tons of log for you here: https://gist.github.com/tknerr/5757832 :-)

[fgrehm]

@tknerr tks, I'll look into it ASAP, but I can see you are using Windows, I've never used Vagrant on Windows and I hope this has nothing to do with it :-)

[patcon]

@tknerr Actually, I was thinking that vagrant-omnibus used a package manager, but it just downloads the deb/rpm directly I believe. So it's more an example of how extending vagrant-cachier via other plugins (#5 (comment)) might be helpful. ie. vagrant-omnibus is simply running an install script, and saving it's files to /tmp:
https://gist.github.com/tknerr/5757832#file-log-L2178-L2186

I think accomodating vagrant-omnibus would require that opscode's install.sh script accepted a flag to determine download path. This is an out-of-date version of the install script, but it looks like @btm might be the guy to bring it up with:
https://github.com/opscode/omnibus/commits/master/source/install.sh

(vagrant-vbguest would seem to be the plugin that's easiest to accomodate, since it does apt-get installs. But I'll leave that for another issue :) I'd suggest renaming this issue, since it doesn't really have to do with when vagrant-cachier does it's thing. )

[tknerr]

@patcon I see, thanks for investigating. It would be nice though if the omnibus install.sh would use the platform-specific default for @file_cache_path rather than just /tmp (@btm thoughts?). It would be automatically solved via #14 then?

Suggestions for renaming the issue?

[patcon]

True that. If install.sh had an argument to select the download path, vagrant-omnibus could pass in the file_cache_path value.

To the opscode guys, do you keep the install script in a repo somewhere for pull requests?

cc: @schisamo

[tknerr]

We should be careful with cache_file_path now comes to me as a second thought, e.g.:

• the cache should be vm scoped rather than box scoped by default
• we might want to filter specific file patterns, e.g. *.tar.gz files only

The dangerous bit about file_cache_path is that files that are downloaded there are not necessarily properly versioned, e.g. http://some.host.org/foo-1.2.3.zip might be saved locally as foo.zip...

[patcon]

Hm. Really good point. Yeah, that could be a really hair-pulling problem to troubleshoot if you're new to chef and not realizing what vagrant-cachier is doing... I was going to ask if we could force machine scope when using this bucket, but thinking maybe the better default overall is machine-scope... most beginners leave the machine name as default, so they wouldn't notice a difference. I'd say that if you're using multi-vm or specifically using config.vm.define "my-machine", then you're probably an advanced user who can delve into scoping in vagrant-cache. (I'm not a fan of filtering specific patterns, btw.)

Thoughts?

Speaking of which, I'm thinking it would be great to have a command to clear all or individual caches. It might make trouble-shooting quirky states simpler if we could just start with saying "run vagrant cache clear [all]. I'll open a new issue.

[tknerr]

👍 for vagrant cache clear

Machine scope by default is definitely safer, but I'm fine with either...

[fgrehm]

@tknerr @patcon I'm pretty sure you guys could change the default scope from ~/.vagrant.d/Vagrantfile, if it doesn't work then we have a bug on config merging ;)

[patcon]

@fgrehm Totally, but most users are just going to add the plugin, enable it, and start working away. Trying to think through the safest option for them :)

I know it wouldn't affect puppet users, but any chef user will hit this once we add file_cache_path support. It makes vagrant break it's promise of vagrant destroy && vagrant up being a fresh start. Perhaps obvious since we're talking about caching, but chef is particularly brittle here since remote_file resources use the :create_if_missing action all over the place

[fgrehm]

@patcon well, I think I have a better solution to this. what about requiring users to set the scope in order for the plugin to work and add a note to the readme about setting defaults from ~/.vagrant.d/Vagrantfile? this will make them think about their caching scheme before going crazy and caching all the things :)
either way, maybe we could "fork" this discussion to another issue?

[patcon]

👍 on all. Honestly, we can't even do chef caching until core vagrant gets patched, so the conversation is probably premature optimization :) Let's create an issue for a solution after progress on #14 is even possible.

I created a red "blocked" tag for issues like this one (blocked by install.sh script), and #14 (blocked by vagrant :chef_solo provisioner code). Feel free to remove if you think it too much clutter.

[tknerr]

Darn, I thought there is a workaround by using a modified install.sh which downloads to /var/chef/cache/, but it doesn't work due to the fact that vagrant-cachier kicks in after vagrant-omnibus :-(

More details in the comment here: chef-boneyard/vagrant-omnibus#22 (comment)

One solution would be to hook in before the VagrantPlugins::Omnibus::Action::InstallChef action if vagrant-omnibus is installed. See tmatilai/vagrant-proxyconf#13 for an example.

[fgrehm]

@tknerr starting with 0.3.0 we are no longer using hooks to configure buckets, we changed it to a monkey patch so that we can reconfigure buckets between provisioners runs and we are only relying on hooks to ensure a single cache root dir exists in order to deal with 0.2 -> 0.3 upgrades.
if the hook magic is the only way we can make this happen, it'll be a completely new action / middleware which makes our lives easier ;)

[tknerr]

@fgrehm I think I didn't fully get it yet. So you suggest to switch back to using hooks again instead of the monkey patch?

[patcon]

Not sure if others found this, but we can now use a custom install.sh url:
https://github.com/schisamo/vagrant-omnibus/blob/master/lib/vagrant-omnibus/action/install_chef.rb#L27

We can use something like this script did: https://gist.github.com/hectcastro/6443633

The downside is that it requires setting an ENV variable (might be possible to set at top of Vagrantfile). Would be great to have that be an official config setting. Seems like the cleanest route until opscode makes it possible to submit PRs for install.sh

[fgrehm]

@tknerr sorry, I meant we could create a new hook that does what we want, but I think I like @patcon's idea to use a custom script :)

@patcon just to confirm, are you suggesting that we bundle that script with vagrant-cachier and provide a config that automatically set things up or do you think it should be enough just to have documentation around on setting that script available on the readme?

[patcon]

I don't normally like to move critical code outside the repo, but thinking it might be more beneficial to host in a gist?

If I'm not mistaken, we can set ENV in plugin.rb, and cachier comes before omnibus. Longer term, we can submit a PR to allow the install url to be set in either code or env var?

[tknerr]

@patcon @fgrehm I have tried the same approach by setting the ENV in the Vagrantfile:

Vagrant::configure("2") do |config|
  ENV['OMNIBUS_INSTALL_URL']="https://gist.github.com/hectcastro/6443633/raw/install.sh"
  ...

It picks up the alternative install.sh and it just works with the install.sh from @hectcastro - yes!! :-)

I was previously under the impression that vagrant-cachier comes after omnibus because I was looking for the "Configuring cache buckets..." part in the output:

[sample-app] Mounting shared folders...
[sample-app] -- /vagrant
[sample-app] -- /tmp/vagrant-cache
[sample-app] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks
[sample-app] Installing Chef 11.6.0 Omnibus package...
[sample-app] Configuring cache buckets...
...

My initial approach failed because I cached the omnibus package in /var/chef/cache which is available / symlinked only after the cache buckets are configured.

@hectcastro's approach is to download the omnibus package to /tmp/vagrant-cache/chef which is mounted as a shared folder at this time already, thus it works. Thanks @hectcastro! 👍

It's only a workaround, but it finally works! :-)

[tknerr]

Finally added this to my $HOME/.vagrant.d/Vagrantfile, taking advantage of the load order thus not having to repeat it everywhere (plays well with bindler too):

if (defined? VagrantPlugins::Cachier && defined? VagrantPlugins::Omnibus)
  ENV['OMNIBUS_INSTALL_URL']="https://gist.github.com/hectcastro/6443633/raw/install.sh"
  puts "setting custom OMNIBUS_INSTALL_URL for vagrant-cachier: #{ENV['OMNIBUS_INSTALL_URL']}"
end

[fgrehm]

@tknerr that's AWESOME! :)

@patcon I think that we could start off by documenting what @tknerr have just done on our README and spread the word about it :) after that is proven to work we can then think about further improvements.

/cc @tmatilai @cromulus

[lamont-granquist]

just got back from vacation today. the changes are all in preprod now and i've run it through jenkins and done some manual testing. i need our Q/A to sign off on it and figure out what the new release process in 2014 looks like and then it should all get shipped before end of week (hopefully by wed or so...)

preprod can be hit here if you want to test it:

curl -L https://www-rs-preprod.opscode.com/chef/install.sh

i'm soft-launching a flag to install open source server with install.sh as well:

curl -L https://www-rs-preprod.opscode.com/chef/install.sh | bash -s -- -P server

[bflad]

Any update @lamont-granquist? Feels like we're really close.

[lamont-granquist]

yeah, still stuck on Q/A, i think they've got a bunch to do with EC releases and ohai 7, but i'll try to unblock it next week..

[lamont-granquist]

okay its live, give it a shot @fgrehm @tmatilai @tknerr etc...

[fgrehm]

AWESOME! I've just removed the blocked tag :-)

@lamont-granquist @tmatilai and others, thanks for making it happen 🍻

[tknerr]

@tmatilai @lamont-granquist either it's way too late and I don't see the obvious or the updated install.sh still does not work as expected :-(

Roadblocks I stumbled upon:

• you can pass -d /path/to/some/dir but it does not get created if it does not exist (just a missing mkdir -p)
• if the file already exists in the specified -d directory it will be downloaded again and overwritten rather than checksumming / validating and using the existing file

Am I just using it wrong?

[lamont-granquist]

well i consciously assumed that the first thing was the responsibility of the 'caller' to create that directory.

for the second thing i could see how that's a real problem since you don't know the filename to download. problem is that install.sh doesn't know that either unless it hits the web service. i could fix that with a check for existence of the file, but that'll require that the client is online and can hit the metadata service....

hmmm re-reading it bug it seems like checking for existence would fix it and we're not trying to address offline use as well?

[tknerr]

@lamont-granquist well I was under the impression that the new install.sh would use an offline file if filename and checksum matches. Creating the directory if not exists should be trivial, we doing it for $tmp_dir as well here.

I will modify the install.sh as I think would be required and play around with it using the OMNIBUS_INSTALL_URL env var. Once it works I will prep some pull requests so you can review.

The main goal is still to use vagrant-cachier for caching the omnibus packages downloaded via vagrant-omnibus, and this requires not downloading the file again if it already exists and is valid.

[lamont-granquist]

the problem is that we need to check the web service to get the filename and the md5 to check integrity, so if all you pass is a directory then if we're offline we have no idea of which file we need to check or install.

there is an -f option which passes the whole path to the file in which case it becomes pretty easy (as long as you don't care as much about integrity, and just check for existence).

[tknerr]

yup, fully agree. Actually it should not be totally offline, just the "big" omnibus package should not have to be downloaded again if it is already present AND the checksum matches.

The idea is as follows:

• the metadata file is always downloaded to $tmp_dir (which will be removed afterwards)
• if neither -d or -f are specified the omnibus package is downloaded to $tmp_dir as well (i.e. it will also be cleaned up afterwards)
• if -d (directory) is specified the omnibus package will be downloaded to that directory with its original filename (as specified in the metadata file). If that file already exists in the given location AND the checksum matches it will just use the already present file instead of downloading again.
• if -f (path to filename) is specified the omnibus package will be downloaded to that file. If a file with that name already exists AND the checksum matches it will just use the already present file instead of downloading again.

In essence this means that the we will always check the web service for the latest filename / checksums, but allow for caching if either -d or -f are used.

@lamont-granquist @tmatilai what do you think?

[lamont-granquist]

works for me

[tmatilai]

@tknerr that sounds good and correct. I just can't understand how I missed the whole thing... =)

[tknerr]

@lamont-granquist @tmatilai install.sh PR is ready for you to review (see above)

I have put together a working scenario with vagrant-omnibus and vagrant-cachier here
https://github.com/tknerr/vagrant-managed-servers/tree/cache-omnibus-download

To test it just run bundle install && rake acceptance.

It requires vagrant-omnibus to use the -d option and uses this gisted install.sh in the meantime

[tknerr]

To bring this lengthy issue to an end, everything seems to be in place now:

Prerequisites:

1. Cache omnibus download chef/omnitruck#33: you can now pass the download directory via -d to install.sh (to be released; in the meantime you can use ENV['OMNIBUS_INSTALL_URL'] with this gist)
2. Allow for caching the omnibus download by passing -d option to install.sh chef-boneyard/vagrant-omnibus#68: vagrant-omnibus considers passes the download directory via -d iff ENV['OMNIBUS_DOWNLOAD_DIR'] is specified (to be merged and released)

Options for integrating with vagrant-cachier:
First I tried to do it properly by introducing a new vagrant-omnibus bucket and capability. Unfortunately this does not work because the bucket is installed too late (i.e. after vagrant-omnibus runs the install.sh)

Since this approach failed I implemented a workaround which creates a vagrant omnibus "pseudo bucket" directly when the root bucket is configured. This is early enough to set ENV['OMNIBUS_DOWNLOAD_DIR'], i.e. before vagrant-omnibus runs the install.sh.

The last option would be to not integrate it with vagrant-cachier and handle it manually in your Vagrantfile:

  ...
  ENV['OMNIBUS_DOWNLOAD_DIR'] = "/tmp/vagrant-cache/vagrant_omnibus"
  config.omnibus.chef_version = "11.10.4"
  config.cache.scope = :box
  ...

@fgrehm what's your preference?

[tknerr]

Created PR #92 on contingency. If you have better ideas let me know...

[tknerr]

@schisamo proposed to let vagrant-omnibus detect whether vagrant-cachier is enabled and set the OMNIBUS_DOWNLOAD_DIR env var (or even better a config option) from inside vagrant-omnibus:
chef-boneyard/vagrant-omnibus#68 (comment)

This way we would not need to change anything in vagrant-cachier, and thus would not need to introduce a "pseudo bucket" with #92

[rossmeissl]

+1 for resolving this issue

[lamont-granquist]

the omnitruck deploy to prod is imminent

[lamont-granquist]

@tknerr your omnitruck change just went live in prod

[tknerr]

@lamont-granquist thanks! 👍
Am 18.03.2014 23:03 schrieb "lamont-granquist" [email protected]:

@tknerr https://github.com/tknerr your omnitruck change just went live
in prod

Reply to this email directly or view it on GitHubhttps://github.com//issues/13#issuecomment-37994606
.

[hectcastro]

I'd buy a beverage to anyone who can summarize how to cache the Omnibus package and all of the changes to the associated plugins. :)

[tknerr]

@hectcastro yes what a mess ;-)

1. the new install.sh is already live
2. Cache omnibus download, expose config options chef-boneyard/vagrant-omnibus#73 is ready but not merged

Once 2. is merged and released you just need to grab that new release of vagrant-omnibus. No changes in vagrant-cachier or any other plugins required. 🍻

Jump in for testing chef-boneyard/vagrant-omnibus#73 to get your beverage back ;-)

[tknerr]

Closing in favor of chef-boneyard/vagrant-omnibus#73