macOS EOL'd system ruby makes running sprout difficult

[aramprice]

macOS (v10.12) ships with an EOL'd version of ruby (v2.0.0p648),

This is problematic for running sprout because many ruby gems are dropping support for this version of ruby. The following version locks are currently required to run sprout: https://github.com/pivotal-sprout/sprout-chruby/blob/master/Gemfile#L12-L20

There are a number of possible options for addressing the lagging ruby support in macOS.

One Idea is to replace the sprout ecosystem with something based on Brewfile plus some sort of dotfiles tooling. This would be losing the investment in cookbooks that have been created.

Another idea would be for sprout to bring its own ruby. This would increase boot-strap time but give a consistent ruby env which sprout and its associated gems could depend on.

I'm opening this issue as a place to further this discussion. Paging @wendorf, @cunnie, @hiremaga, ... Please tag anyone who might be interested.

[infews]

It seems more problematic than just installing Ruby early. Because you need Homebrew to install Ruby (at least with ram), but need a Ruby > 2.0.0 to install Homebrew.

[hiremaga]

The idea of an omnibus for Sprout has come up before. Perhaps this just tipped the need for this. Aram, is this what you mean about Sprout bringing its own Ruby?

…

On Mon, 6 Feb 2017 at 3:57 pm, Davis W. Frank ***@***.***> wrote: It seems more problematic than just installing Ruby early. Because you need Homebrew to install Ruby (at least with ram), but need a Ruby > 2.0.0 to install Homebrew. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#99 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAfIrrHJMKdE1nsSVawsOb8mDanYx_-ks5rZqhNgaJpZM4LmHyB> .

[infews]

Here's what I did yesterday that worked, mostly:

• New Sierra install
• open Terminal
• Install Homebrew per this page
  • Used this command: ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • It installs the guts of XCode and the command line tools
  • Then it installs Homebrew
• Install RVM and latest stable Ruby to get Ruby 2.3 (yes, I am an RVM fan still; happy to have alternatives)
• New Terminal window, ensure RVM is on the path
• Clone sprout-wrap
• Edit soloistrc, ensure that the homebrew recipes are present
• Run caffeinate ./sprout

This completed the sprout run without error. Most apps seem installed at least. And bash-it, etc., are present.

However, Homebrew complains that the Casks are in /opt/. Not sure if/how to fix this.

I'm fine with this approach. I feel leaving Sprout is going to be a move backwards.

[aramprice]

Regarding #99 (comment) homebrew still/currently uses system ruby on macOS. Unfortunately this is ruby 2.0.0:

→ /usr/bin/ruby -v
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]

If one is willing to double-bootstrap (first brew.sh, then sprout) then the recipe above works fine. This may be the way to go ultimately. If so then sprout should stop trying to install brew see: chefcookbooks/homebrew. Currently sprout assumes that it is responsible for installing brew (and other formulae, casks, etc) so pre-installing brew might cause problems.

Regarding Casks in /opt it is likely that you have an outdated soloistrc or outdated cookbooks. This isn't the behavior I'm seeing on an up-to-date sprout-wrap.

[trinitronx]

Abandoning sprout after so much investment feels backwards, especially since it still works fine on new Ruby + Chef versions. Additionally, Brewfile doesn't take into account things that are not in Homebrew, or otherwise hard to create Homebrew formulas for, making sprout-wrap a much better tool for solving the problem that sprout set out to solve, as it gains the Homebrew library through the homebrew cookbook while still harnessing the power of Chef for system configuration & provisioning.

After using many configuration & provisioning tools, I've seen that bootstrapping is one of the major problematic and difficult issues on any OS. This is especially true for Mac OSX due to the hard to automate dependencies such as XCode + xcode-command-line-tools, and the out of date system ruby.

So really, our problem is bootstrapping. This is a solvable problem, as myself, @infews, and others have managed to get from a fresh OSX install to a working one with RVM + Chef 12.x + soloist. So that means a path from fresh -> provisioned with sprout-wrap exists. The hard part is going to be tracking down the right gem dependencies for a Gemfile.lock, and managing any automation hiccups from fresh slate that come up during soloist provisioning. I've had to do a bit of that work in my fork, so it's available for reference here.

Just as with any configuration management & provisioning software, starting from scratch becomes important for testing to ensure missing dependencies are handled. Knowing this, I had originally intended to set up a test-kitchen suite for running sprout cookbooks against a fresh VM for testing. However, with OSX this also became problematic due to the difficulty in finding Vagrant or VirtualBox VMs for Mac OSX (mostly it was licensing concerns that prevented people from publishing images, so you have make one yourself). I had some problems with VirtualBox + veewee, had minor success using VMWare Fusion, but then ran out of space & time to complete it. (disk space that is ;-)

So I still have yet to get a good test-kitchen provisioning toolchain for Mac working properly. If we had that, finding the right "golden image" set of gems for bootstrapping on top of system ruby would be much easier. I haven't had time myself to learn & set up sprout-orchard, but I'd imagine maybe someone at Pivotal knows. Maybe just starting with RVM + ruby-2.0.0p648, creating a gemset & starting to pare down on the gem dependencies will get us 80-90% there.

Background: I've used sprout-wrap myself at prior jobs & maintain a personal fork also for using it on a semi-frequent basis to manage & maintain my various Mac laptops. I also have a bootstrap script for XCode + CLT here. My attempts at getting an OSX VirtualBox VM created have been shared in the veewee project here and here.

[trinitronx]

Update: So it looks like the chef/bento project has templates now for macosx-[7-12]. These are probably worth a shot at using with test-kitchen as they'll be much more likely to work out of the ".box" (pun intended).

[ClayShentrup]

@trinitronx Thanks for that. Have there been any conversations since then? Is Pivotal actively using this tool? I just unboxed our first proper pairing station here at Periscope Data and I'd love to get us on Sprout-Wrap.

[aramprice]

@ClayShentrup I made some changes to the Gemfile and .travis.yml which should ensure that sprout-wrap runs on the latest macOS.

Before adopting the soloistrc contents of this repo though I'd suggest auditing the cookbook list. For example sprout-mysql and sprout-postgresql should probably be dumped in favor of brew installing these (via homebrew: {formulas: ...} in soloistrc)

[ClayShentrup]

Thanks @aramprice! I'll give it a spin soon.

[aramprice]

See #102 for changes

[aramprice]

Related Homebrew/homebrew-core#14418

[wendorf]

Good news: macOS 10.13 beta 3 has a system Ruby of 2.3.3p222

[trinitronx]

As @wendorf points out, now that High Sierra's out we have:

$ ruby --version
ruby 2.3.3p222 (2016-11-21 revision 56859) [universal.x86_64-darwin17]

[aramprice]

@trinitronx @wendorf good news is that un-pinning gems allows sprout to work on High Sierra (macOS 10.13) but unfortunately Travis CI doesn't support 10.13 as noted here.

[trinitronx]

@aramprice : Of course... d'oh! It looks like they have planned support for 10.13 soon:

This image still run on macOS 10.12 Sierra because we wanted to make Xcode 9.2 available as soon as possible. We will be releasing a new image with macOS 10.13 and Xcode 9.2 in the near future. Thank you for your patience!

Technically if sprout were using rvm, it would be possible to use newer ruby version:

Ruby versions installed with rvm
ruby-2.3.5
ruby-2.4.2 (default)

But of course, that defeats the purpose of testing the bootstrapping process using pre-existing system ruby.

[trinitronx]

Not sure if you guys have tested it yet, but it looks like Travis CI now supports macOS 10.13!

Blog Post: Say Bonjour to Xcode 9.3 and macOS High Sierra!

[aramprice]

I've been converting some of the cookbooks to use this, notably https://github.com/pivotal-sprout/sprout-base, https://github.com/pivotal-sprout/sprout-chruby, and https://github.com/pivotal-sprout/sprout-git

See https://github.com/search?q=org%3Apivotal-sprout+xcode9.3&type=Code

Ex: https://github.com/pivotal-sprout/sprout-base/blob/master/.travis.yml#L3

[aramprice]

Many of the sprout- repos should probably not be converted since their functionality can be accomplished by using brew.

I mention this hear since sprout-wrap references a number of them here and until they are either removed, or converted tests for this repo are unlikely to pass.

[trinitronx]

@aramprice: Thanks for pointing out the ones using the new Travis CI image! I'll have to look into adding some testing to my sprout-wrap fork, mainly surrounding the ruby version & bootstrap parts.
I also need to get my fork up to date and test out the new sprout script and it's own environment instead of system ruby or RVM (although both worked at various times). One of the more difficult things had been migrating ruby versions & getting the gem version dependencies right. This week I had the need to update dependencies and go through bootstrap again, so I had to make the time to do it.

Sounds good about migrating to brew for most things, although it opens the door to less stability when important formulae or casks change. I've noticed over the years that some will change names unexpectedly, others get completely removed, and still others have to either be migrated to custom taps or build your own tap. I had to create a couple custom taps and chef recipes for a couple things that became deprecated by Homebrew. However, the vast majority of them keep the same name and stick around, especially if it's a more popular tool or app. Just be aware if that's the route you guys decide to go 😉

[aramprice]

@trinitronx I may have spoken a tad too soon about the viability of the recipes noted above. I ported the changes I had from other repos to #103 and the travis run appears to have succeeded:

• https://travis-ci.org/pivotal-sprout/sprout-wrap/builds/374699424

I still think that nearly all of the recipes should be dropped in favor of a Brewfile, or at the very least replaced with the homebrew LWRP, but that's a discussion for another day.

FWIW we seem to be moving away from sprout mostly because folks have limited familiarity with chef and building enough context to understand it drives most folks to use bash + brew. The indirection of sprout => soloist => chef => cookbook => brew tends to try folks patience.

Personally I'm experimenting with dotfiles and a Brewfile

[ClayShentrup]

@aramprice It's interesting to hear about that experience with "indirection". Do you feel similarly about this project from Labs Seattle? (Based on Ansible apparently.)

https://github.com/seattle-beach/alfalfa

[aramprice]

@ClayShentrup I haven't looked that closely at seattle-beach/alfalfa but I think there is a similar (slightly lower?) level of indirection.

For sprout at least we decided at one point that it would be macOS only. At that point there was no need for the multi-OS capabilities of chef. It's not clear that brew was feature-full enough at that point (~2yrs ago?) to replace sprout-wrap/soloist/sprout-cookbooks/chef but I think that today brew + Brewfile covers close to 80%-90% of what sprout-wrap/soloist/sprout-cookbooks/chef does with a fraction of the complexity.

Another lens to see this through: why should a developer invest in understanding a technology they wouldn't otherwise use (in this case chef) in order to automate something (brew + Brewfile) that doesn't really (at this point) need it? This argument breaks down if developers live and breath chef (or some other tool). Most developers don't provision fleets of servers (chef's sweet-spot) so it has become an esoteric toolset.

[cunnie]

@aramprice I agree. In fact, the last two times I rebuilt a macOS machine from scratch I bypassed Sprout-wrap entirely because I was having too much trouble wrestling with the recipes.

[trinitronx]

Hmm, interesting... hadn't realized Brewfile was a thing now... definitely worth investigation. Although, sprout-wrap + soloist still are very useful for those of us who have developed custom Chef cookbooks for OSX apps & software (like me). In fact, this project has safely migrated my macOS system setup across 3 separate work laptops, through 2 or 3 (lost count) OS upgrades, and a company acquisition! Definitely would have lost much more time if I didn't have sprout-wrap! Great work to all involved in making this project a reality 🥇 👍 😄

Also, it's proven very useful to install DAW software + plugins and other things on my home laptop.

I wonder if Brewfile and a bit of shell script glue could make the Chef bootstrap process easier by using the chefdk Cask. Then, sprout-wrap could use ChefDK + embedded Ruby to avoid inherent problems with system Ruby? So process would look like:

1. Bash script to install Homebrew + brew cask
2. Brewfile or other one-off command to brew cask install chef/chef/chefdk

• ChefDK + embedded Ruby now available under /opt/chefdk
• All dependencies can be better trusted to be compatible according to ChefDK release process + CI testing at Chef

3. sprout-wrap + soloist can now continue for custom macOS Chef cookbooks!

[aramprice]

Thanks for the success story (stories), it is great to hear about sprout automation paying off folks!

The chefdk thing seems interesting. It sounds like a good path forward for folks who have an existing investment in sprout/chef cookbooks.

Based on the puzzled looks I get from folks when explaining chef cookbooks to folks though I suspect that it won't be enough to revitalize sprout-wrap for the general population though :-/

[trinitronx]

I just had to provision a new MacBook Pro (15-inch, 2019, macOS 10.14.6 Mojave) for work... it turns out that after a couple hurdles with the new XCode + CLI tools, and the missing ruby.h header files, I was able to successfully run soloist on my sprout-wrap fork!

This was starting out with system ruby too. I avoided installing rvm until after getting a fully successful Chef run (also, sous-chefs/rvm cookbook is presently unstable). So as long as Apple still provides the macOS_SDK_headers_for_macOS_10.14.pkg package, everything still works, and it's possible to use sprout-wrap on the latest macOS & macbook models.

I've made some fixes to my bootstrap script, and finally got a successful Chef run after removing or migrating deprecated homebrew casks, recipes, etc... The related changes are in work-laptop-dependency-updates branch if you're curious.

[trinitronx]

I've gotten sprout + soloist working on macOS Big Sur, Monterey, and even patched the homebrew cookbook to get Homebrew working again on the M1 / arm64 chip! (Note: as of this writing, even Chef's built-in homebrew provider does not yet support this)

In case anyone's still interested in using sprout / sprout-wrap... I've managed to refactor my bootstrap script to install via rvm ruby + gemset with Gemfile.lock. This seems to be the way most projects are going these days, and there are some bloggers that warn against using system ruby. It seems to me that they make good points about versions changing across macOS releases, and the ensuing gem installation hassles that I'm sure we've all experienced (hence this unresolved issue).

I've moved my sprout-wrap repository under my 'sole-proprietor' org here: LyraPhase/sprout-wrap. I've been wondering if I should create a proper hard fork & rename the project to avoid the occasional upstream accidental pull request (GitHub defaults to merging back to this repo), or if this project is going to be revived?

[cunnie]

if I should create a proper hard fork & rename the project to avoid the occasional upstream accidental pull request (GitHub defaults to merging back to this repo), or if this project is going to be revived?

I can't speak for anyone else, but in my eyes you are the steward of sprout-wrap, and I like the idea of a proper hard fork & rename to avoid the accidental PR.