x/build: tracking bug for macOS virtualization #9495

bradfitz · 2015-01-03T07:03:22Z

I'd like to get OS X builders running in VMs too.

(Related to #9492 and https://golang.org/s/builderplan)

Looks like virtualizing OS X 10.7+ is legal (but not 10.6) as long as it's only 1 copy, and on official Apple hardware. Considering that we already run Go builders on official Mac hardware in the office, we can continue to do so, but with a VM solution.

And looks like VMWare Fusion has the "vmrun" command, documented at http://www.vmware.com/pdf/vix162_vmrun_command.pdf , so we can write a little API server that runs on the OS X host and calls vmrun.

/cc @adg

c4milo · 2015-01-03T14:13:51Z

You can also use the VIX bindings I wrote just in case vmrun falls short for any reason: https://github.com/hooklift/govix

bradfitz · 2015-01-05T04:25:43Z

@c4milo, any chance you want to own this whole bug? :)

c4milo · 2015-01-05T17:15:10Z

hahaha, @bradfitz let me look into it in detail before making any commitments.

c4milo · 2015-01-05T17:33:22Z

@bradfitz alright, how were you envisioning the little API server you mentioned above?

bradfitz · 2015-01-05T17:54:16Z

I'd need API calls to:

start a pre-configured OS X VM, and return me an ID for it (so I can delete it later), and IP:port I can use to hit its port 80.
an API call to destroy that VM

That's it. The VM would need to run a couple line shell script on boot.

c4milo · 2015-01-05T18:00:10Z

Working on it.

c4milo · 2015-01-05T18:05:17Z

That's it. The VM would need to run a couple line shell script on boot.

As root?

bradfitz · 2015-01-05T18:18:48Z

@c4milo, yeah, root works. I can drop down to a less-privileged user if I need to. But some of our net tests require root, or at least more capabilities.

The script you'll want to run as root is just:

#!/bin/bash
set -e
curl --fail -o /usr/local/bin/buildlet https://storage.googleapis.com/go-builder-data/buildlet.darwin-amd64
chmod +x /usr/local/bin/buildlet
exec /usr/local/bin/buildlet

That should work already. Be careful with that server: it is a remote code execution daemon. Don't run it on your real machine if it's exposed to the internet.

But you should be able to hit it in a VM and see it reply "I am a buildlet" or whatever at its root path.

bradfitz · 2015-01-05T18:19:18Z

(and don't bake in the buildlet to the VM image... only that script. That lets us update the buildlet over time easier than updating the whole VM image)

c4milo · 2015-01-05T18:22:01Z

Got it.

bradfitz · 2015-01-05T18:31:50Z

Also, tell me what to buy. :)

VMWare Fusion 7 Pro? http://store.vmware.com/store/vmware/en_US/DisplayProductDetailsPage/ThemeID.2485600/productID.304322800

I have a iMac 5K running Yosemite and a Macbook Pro running Mavericks.

Ideally we'll want to test the Go builders with OS X 10.8, 10.9, and 10.10. Maybe 10.7 too. So can I just buy those OS X discs from Apple somehow if I can no longer find my original CDs? Or do I need "Server CDs"? I know you're not a lawyer, and I'm not asking for legal advice. We're definitely not looking to avoid paying for things. I'm happy giving Apple plenty of (more) money here... I just want to know technically what's required, once we pay for all the parts the Apple-sanctioned ways.

c4milo · 2015-01-05T18:46:14Z

I'm using VMWare Fusion 7 Pro. I was also thinking in using OSX Server as the guest OS, since it seems to boot faster, but I might need to double check that once I start testing the API.

adg · 2015-01-05T19:55:59Z

On 6 January 2015 at 05:19, Brad Fitzpatrick [email protected]
wrote:

@c4milo https://github.com/c4milo, yeah, root works. I can drop down to
a less-privileged user if I need to. But some of our net tests require
root, or at least more capabilities.

Really? None of the builders I have configured have executed the tests as
root, and I don't think we should be doing that.

bradfitz · 2015-01-05T19:59:37Z

They skip the tests when they're not root.

I don't think it's a big deal, and let's not get sidetracked on this bug.

minux · 2015-01-05T20:16:30Z

I think at least for 10.6, you are not allowed to run regular OS X inside
VMWare Fusion.
You must use OS X Server.

It's probably also true for 10.7.

PS: we do need a builder for 10.6, because 10.6 is the earliest version
that we still
support.

bradfitz · 2015-01-05T20:17:24Z

We can do an old-style builder for 10.6, for as long as we still support it, which probably won't be much longer. It's possible we even drop 10.6 support in Go 1.5.

minux · 2015-01-05T21:36:06Z

Removing OS X 10.6 support in 1.5 SGTM.
It's causing too much trouble in support, and its support is crippled by the
fact that external linking is not working anyway.

Update golang/go#9495 Change-Id: I732cfdee952ad3bf0b3411d0ce45723900acedb4 Reviewed-on: https://go-review.googlesource.com/2472 Reviewed-by: Andrew Gerrand <[email protected]>

bradfitz · 2015-01-13T06:53:47Z

Moving from email thread, I confirmed I can boot into Yosemite into a desktop running Terminal and the buildlet in under 1 minute. It's about 57 seconds on my iMac 5K i7 and VMWare Fusion Pro.

The buildlet is accessible on port 8080 from the host, if you know its IP:

And you can create a start-up item of a +x file named start-buildlet.command (the *.command extension is key) which launches a shell script in Terminal:

We can then snapshot that and create VMs which boot like that, in ~57 seconds. The host API server on the outside should probably cache the ~6MB buildlet too, to reduce network I/O. We can do that later, though.

c4milo · 2015-01-13T17:15:16Z

Here is the first draft of the API: https://github.com/c4milo/osx-builder.

@bradfitz, keep in mind that Login Items will only run upon the user logging in. It might be more convenient for the API if they run on boot as there won't be need to log into the guest OS in order to make the script run. As I mentioned in our email thread, this can be done through Launchd. There is more information about Launchd here: https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/BPSystemStartup.pdf

bradfitz · 2015-01-13T17:16:48Z

I configured the user to automatically log in, so the Login Items works great.

c4milo · 2015-01-13T17:17:35Z

oh, yeah that works too. nice!

Update golang/go#9495 Change-Id: I732cfdee952ad3bf0b3411d0ce45723900acedb4 Reviewed-on: https://go-review.googlesource.com/2472 Reviewed-by: Andrew Gerrand <[email protected]>

c4milo · 2015-02-03T19:19:22Z

@bradfitz what is left here?

adg · 2015-02-05T13:45:43Z

@c4milo we're going to set up a small cluster of OS X machines to use as builders (@crawshaw is doing that). Once they're set up, we'll know what else needs to be done, if anything.

bradfitz · 2015-02-05T16:10:16Z

@c4milo in addition to what Andrew said, I also haven't had found time yet to use what you've made. I created a VM in VMWare that boots right into a buildlet and I could hit it from the outside, which was enough to convince me that the concept would work, but I haven't tested it end-to-end.

We'll also need to add support for the daemon that runs on the host to (your code) to connect to the build coordinator and setup a reverse tunnel back to it, for our farm of Macs running in an isolated network.

This starts the process of making the coordinator about to use the buildlet on things other than GCE. Update golang/go#8647 (ARM, reverse proxy or online.net) Update golang/go#10267 (windows 2003 on AWS) Update golang/go#9495 (OS X VMWare VMs on racked Mac Minis) Change-Id: I5df79ea67e0ececba8b880e81bd93d4c80897455 Reviewed-on: https://go-review.googlesource.com/8198 Reviewed-by: David Crawshaw <[email protected]>

Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2016-08-31T05:00:35Z

CL https://golang.org/cl/28218 mentions this issue.

Updates golang/go#9495 Change-Id: Id01533737c14453aaa1bdccaa3124dd7e30a2465 Reviewed-on: https://go-review.googlesource.com/28218 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2016-09-06T19:00:42Z

CL https://golang.org/cl/28572 mentions this issue.

gopherbot · 2016-09-06T19:00:43Z

CL https://golang.org/cl/28570 mentions this issue.

Updates golang/go#9495 Change-Id: Id87d401d3f7e6fd464039cff4e58a5eda14424cf Reviewed-on: https://go-review.googlesource.com/28570 Reviewed-by: Chris Broadfoot <[email protected]>

We'll revert them to snapshot later. Updates golang/go#9495 Change-Id: I93e67598977484f0f3c7eecab07d76370e85fa1d Reviewed-on: https://go-review.googlesource.com/28580 Reviewed-by: Brad Fitzpatrick <[email protected]>

…er VMs The buildlet now checks the GO_BUILDER_ENV environment variable on start-up and configures itself automatically if set to a known value. The only currently-known value is "macstadium_vm", which means that the buildlet will get its role & build key from the OS X version (using "sw_vers") and the VMWare VM metadata, respectively. Updates golang/go#9495 Change-Id: I9651b2d589fed46ff3d087a97acc0ea7171e2b24 Reviewed-on: https://go-review.googlesource.com/28572 Reviewed-by: Brad Fitzpatrick <[email protected]>

Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2016-09-07T06:00:42Z

CL https://golang.org/cl/28584 mentions this issue.

Currently it's just a package main. It might split in two later. It might stop shelling out to govc and use the underlying API directly too, but it's hairy. Easier to work like this for now. Updates golang/go#9495 Change-Id: I0d2e19abcb5114ab7fe2e2c543d14e50897d4cbb Reviewed-on: https://go-review.googlesource.com/28584 Reviewed-by: Quentin Smith <[email protected]>

quentinmit · 2016-09-29T23:10:30Z

@bradfitz Can we closed this bug now? farmer thinks there are a bunch of running darwin buildlets, which I think are in virtualization.

bradfitz · 2016-09-30T04:11:08Z

The coordinator doesn't yet directly manage the creation of VMs.

The current setup is a transitional hack to get us going where they're VMs, but they're created by a tool which creates them statically once, by hand, and then they just act like normal static machines in reverse mode. But if they die, they never come back to life, and we can't react to load or demand. Right now we only create one OS X 10.8 VM, 3 10.8 VMs, and the rest 10.11. That's pretty arbitrary. If four people wanted OS X 10.8 VMs via gomote, we can't create that right now.

So I want to keep this open until things are better integrated.

gopherbot · 2017-02-25T00:55:50Z

CL https://golang.org/cl/37457 mentions this issue.

Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2017-12-06T23:07:12Z

Change https://golang.org/cl/82355 mentions this issue: cmd/buildlet: always halt Mac VMs after builds, regardless of macOS version

…ersion We used to do this only by necessity on Sierra, due to Sierra issues, but now that x/build/cmd/makemac is better, we can just do this all the time. Doing it always is better anyway, to guarantee fresh environments per build. Also add the forgotten sudo in the Mac halt. The env/darwin/macstadium/image-setup-notes.txt even calls out how to setup password-free sudo to the images, but then I forgot to use it. It only worked before (if it did?) because the process ended and failed its heartbeat, at least some of the time. It's also possible it was never working. The old reason that Sierra machines were special-cased to reboot was reportedly fixed anyway, in golang/go#18751 (comment). Updates golang/go#9495 Change-Id: Iea21d7bc07467429cde79f4212c2b91458f8d8d8 Reviewed-on: https://go-review.googlesource.com/82355 Reviewed-by: Brad Fitzpatrick <[email protected]>

bradfitz added the builder label Jan 3, 2015

bradfitz changed the title ~~builders: track bug for OS X virtualization~~ builders: tracking bug for OS X virtualization Jan 3, 2015

bradfitz assigned c4milo Jan 5, 2015

adg mentioned this issue Jan 5, 2015

build: drop support for OS X 10.6 #9511

Closed

bradfitz mentioned this issue Jan 6, 2015

x/build/builders: tracking bug for new-style SmartOS Solaris builder #9515

Closed

rsc added this to the Unplanned milestone Apr 10, 2015

rsc changed the title ~~builders: tracking bug for OS X virtualization~~ x/build/builders: tracking bug for OS X virtualization Apr 14, 2015

rsc modified the milestones: Unreleased, Unplanned Apr 14, 2015

rsc removed the builder label Apr 14, 2015

rsc added the Builders x/build issues (builders, bots, dashboards) label Jun 11, 2015

bradfitz mentioned this issue Aug 19, 2015

x/build: mac builders not logged in #12185

Closed

bradfitz changed the title ~~x/build/builders: tracking bug for OS X virtualization~~ x/build/builders: tracking bug for macOS virtualization Nov 30, 2018

bradfitz added the OS-Darwin label Nov 30, 2018

dmitshur changed the title ~~x/build/builders: tracking bug for macOS virtualization~~ x/build: tracking bug for macOS virtualization Jun 17, 2022

dmitshur unassigned c4milo Sep 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: tracking bug for macOS virtualization #9495

x/build: tracking bug for macOS virtualization #9495

bradfitz commented Jan 3, 2015

c4milo commented Jan 3, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

adg commented Jan 5, 2015

bradfitz commented Jan 5, 2015

minux commented Jan 5, 2015

bradfitz commented Jan 5, 2015

minux commented Jan 5, 2015

bradfitz commented Jan 13, 2015

c4milo commented Jan 13, 2015

bradfitz commented Jan 13, 2015

c4milo commented Jan 13, 2015

c4milo commented Feb 3, 2015

adg commented Feb 5, 2015

bradfitz commented Feb 5, 2015

gopherbot commented Aug 31, 2016

gopherbot commented Sep 6, 2016

gopherbot commented Sep 6, 2016

gopherbot commented Sep 7, 2016

quentinmit commented Sep 29, 2016

bradfitz commented Sep 30, 2016

gopherbot commented Feb 25, 2017

gopherbot commented Dec 6, 2017

x/build: tracking bug for macOS virtualization #9495

x/build: tracking bug for macOS virtualization #9495

Comments

bradfitz commented Jan 3, 2015

c4milo commented Jan 3, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

bradfitz commented Jan 5, 2015

c4milo commented Jan 5, 2015

adg commented Jan 5, 2015

bradfitz commented Jan 5, 2015

minux commented Jan 5, 2015

bradfitz commented Jan 5, 2015

minux commented Jan 5, 2015

bradfitz commented Jan 13, 2015

c4milo commented Jan 13, 2015

bradfitz commented Jan 13, 2015

c4milo commented Jan 13, 2015

c4milo commented Feb 3, 2015

adg commented Feb 5, 2015

bradfitz commented Feb 5, 2015

gopherbot commented Aug 31, 2016

gopherbot commented Sep 6, 2016

gopherbot commented Sep 6, 2016

gopherbot commented Sep 7, 2016

quentinmit commented Sep 29, 2016

bradfitz commented Sep 30, 2016

gopherbot commented Feb 25, 2017

gopherbot commented Dec 6, 2017