builders: move OpenBSD builder to GCE

[bradfitz]

This bug is about moving the OpenBSD builders to GCE per the document
http://golang.org/s/builderplan

The OpenBSD buildbot will be similar to Windows (issue #8640) and FreeBBSD (issue #8639)

In summary, we want a script (that we can run on Linux, in Docker) that can generate a
GCE-bootable OpenBSD image.

That OpenBSD image must then, upon boot, fetch its GCE instance-level metadata value for
"buildmaster-binary-url" and exec it.

Basically, on boot (/etc/rc.local? is that a thing?), I want it to run:

curl -o /buildmaster $(curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/attributes/buildmaster-binary-url)
chmod +x /buildmaster
exec /buildmaster
halt (if the exec failed)

And then we will occasionally update the buildmaster-binary-url property to a new
cross-compiled-to-OpenBSD (/Windows/FreeBSD) Go binary that implements the builder RPC
server, as described in issue #8640.

The OpenBSD kernel should be able to run both 32-bit or 64-bit binaries. Alternatively,
if easier or desired, prepare me two images.

Can some OpenBSD person own making the tool to prepare this image?

[mdempsky]

Comment 1:

Is qemu usable to generate the image?
At http://marc.info/?l=openbsd-misc&m=138757967321855&w=2 I posted my notes on how I was
building VM images on Ubuntu when I was initially trying to get OpenBSD running on GCE. 
It can probably be cleaned up further / better streamlined, and also extended to install
any extra third-party packages needed.
I'll try to take a stab when I've got a chance, or I can help with any OpenBSD/GCE
issues others run into if they beat me to it.

[bradfitz]

Comment 2:

Can it be done without any human interaction?
I want to push a button and get a bootable image, ideally.
If we _have_ to cheat and manually prep the image (like we're planning on doing for
Windows in issue #8640) that's acceptable, as long as every step & complicated key press
is documented in a text file in the go.tools/dashboard/env/openbsd/ directory.

[bradfitz]

Comment 3:

But to answer the question: using qemu is fine, since qemu can run anywhere, including
under Docker.

[mdempsky]

Comment 4:

Summarizing Brad and my discussion about this just now:
It should be possible to fully automate.  I was running qemu in -nographic mode using
stdin/stdout as a serial port, so worst case it should be possible to script the
installer using something like "expect".  (For simplicity when I was initially doing it,
I was using a dumb script that blindly fed installer answers at certain time intervals.)
OpenBSD's installer also supports an "autoinstall" option that uses response files
(documented here:
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/autoinstall.8?query=autoinstall&sec=8),
but I've never used those so not sure how suitable they are for this use case.  One
concern is they seem to want to be fetched from a DHCP server, whereas I suspect it's
easier if we can configure qemu to run without a network connection.
As a first step, I'll work on an Ubuntu-compatible script that fetches an OpenBSD
install ISO and then creates a GCE-compatible image, and Brad says he can take it from
there.

[bradfitz]

Comment 5:

If you want to do it first by hand and want to parallelize, I could get started with the
manually-prepared bootable image while you work on expect-ifying it.
Setting up a DHCP server inside the Docker container should be fine: each Docker
container runs by default with its own isolated network. But avoiding it would be
easier, unless it's too painful.

[mdempsky]

Comment 6:

Sure, I should be able to get you a manually usable script in a day or so.

[4a6f656c]

Comment 7:

OpenBSD/amd64 does not run 32-bit binaries, hence they will need to be two separate
builders (as they are today).

[bradfitz]

Comment 8:

Sounds good. Two images is fine.
Another bit from talking to mdempsky earlier: we could even one day automate running the
image generator to have an "OpenBSD nightly" or "Linux nightly" kernel.
Btw, we've decided on a better name for what I called "buildmaster" above: "buildlet".
The buildlet is the daemon that runs on each VM (be it Windows, *BSD, or Linux ARM in
qemu (issue #8647)) and to which the Coordinator does RPCs (over HTTP). The buildlet
doesn't yet exist.

[4a6f656c]

Comment 9:

I would recommend using autoinstall (unless it is absolutely impossible) - it will make
repeatable/reproducible builds much easier (also keeping in mind that they will need to
be updated every ~6 months to track the stable OpenBSD releases).

[mdempsky]

Comment 10:

jsing: My approach was going to be to write a script to:
  - Download installXX.iso.
  - Create a siteXX.tgz with whatever custom scripts/tools the buildlet needs.
  - Add the siteXX.tgz to installXX.iso and also add "set tty com0" in boot.conf
  - Boot installXX.iso via qemu -nographic, and use a tool like "expect" to automate answering the questions.
  - Create the GCE-compatible .tgz file from the qemu disk image.
Using an autoinstall(8) response file instead of expect is fine with me too, but it just
seems like extra work because it requires configuring DHCP and TFTP servers to provide
the install.conf file to the VM.

[bradfitz]

Comment 11:

Labels changed: added builder.

[mdempsky]

Comment 12:

Attached is an automated script that I believe should satisfy the initial request at
least for constructing an OpenBSD/amd64 VM image from the latest snapshot install CD. 
Assuming that works, it should be easy to generalize to create i386 images and also
using official release CDs instead.
Just run "./make.bash" and about 6 minutes later you'll have openbsd-amd64-gce.tar.gz
that *should* work on GCE (though I wasn't able to test that far yet).

Attachments:

1. make.bash (3978 bytes)

[bradfitz]

Comment 13:

Thanks, I'm playing with this now. Can we prevent some of this unnecessary (X?) stuff
from being installed?
Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-' to the set name, file name pattern or 'all'. Selected
sets are labelled '[X]'.
    [X] bsd           [X] base56.tgz    [X] game56.tgz    [X] xfont56.tgz
    [X] bsd.rd        [X] comp56.tgz    [X] xbase56.tgz   [X] xserv56.tgz
    [ ] bsd.mp        [X] man56.tgz     [X] xshare56.tgz  [ ] site56.tgz
Set name(s)? (or 'abort' or 'done') [done] +*
    [X] bsd           [X] base56.tgz    [X] game56.tgz    [X] xfont56.tgz
    [X] bsd.rd        [X] comp56.tgz    [X] xbase56.tgz   [X] xserv56.tgz
    [X] bsd.mp        [X] man56.tgz     [X] xshare56.tgz  [X] site56.tgz
Set name(s)? (or 'abort' or 'done') [done] done
Directory does not contain SHA256.sig. Continue without verification? [no] yes
Installing bsd          100% |**************************| 11559 KB    00:01
Installing bsd.rd       100% |**************************|  8878 KB    00:01
Installing bsd.mp       100% |**************************| 11603 KB    00:01
Installing base56.tgz   100% |**************************| 58189 KB    00:37
Extracting etc.tgz      100% |**************************|   164 KB    00:00
Installing comp56.tgz   100% |**************************| 46848 KB    00:31
Installing man56.tgz    100% |**************************|  8894 KB    00:07
Installing game56.tgz   100% |**************************|  2721 KB    00:01
Installing xbase56.tgz  100% |**************************| 16595 KB    00:09
Extracting xetc.tgz     100% |**************************|  9165       00:00
Installing xshare56.tgz 100% |**************************|  4449 KB    00:06
Installing xfont56.tgz  100% |**************************| 38994 KB    00:19
Installing xserv56.tgz   31% |********                  |  6016 KB    00:06 ETA

[bradfitz]

Comment 14:

Also, the make.bash script doesn't seem to ever get to the final "tar" step.
The "halt" command seems to just reboot it, and qemu-system-x86_64 seems to not ever
exit.
I see:
Making all device nodes...done.
quirks-2.10 signed on 2014-09-05T15:03:42Z
quirks-2.10: ok
bash-4.3.24:libiconv-1.14p1 (installing)ldconfig: /var/run/ld.so.hints: No such file or
directory
bash-4.3.24:libiconv-1.14p1: ok
bash-4.3.24:gettext-0.19.1p0: ok
Shell /usr/local/bin/bash appended to /etc/shells
bash-4.3.24: ok
curl-7.37.1:libidn-1.29: ok
curl-7.37.1: ok
Extracted 29633422 from 29636303
CONGRATULATIONS! Your OpenBSD install has been successfully completed!
To boot the new system, enter 'reboot' at the command prompt.
When you login to your new system the first time, please read your mail
using the 'mail' command.
# halt
syncing disks... done
The operating system has halted.
Please press any key to reboot.
rebooting...
>> OpenBSD/amd64 BOOT 3.28
boot>
booting hd0a:/bsd: 7698960+2124204+1092216+0+581376 [100+552720+367859]=0xfd7eb0
entry point at 0x10001e0 [7205c766, 34000004, 24448b12, 1c40a304]
[ using 921520 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2014 OpenBSD. All rights reserved.  http://www.OpenBSD.org
OpenBSD 5.6-current (GENERIC) #345: Tue Sep  9 00:22:24 MDT 2014
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 385863680 (367MB)
avail mem = 367001600 (350MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfd9c0 (10 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2007
bios0: Bochs Bochs
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 1.0, 2809.59 MHz
cpu0:
FPU,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,NXE,LONG,LAHF,SVM,ABM,SSE4A
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2
cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 1
acpihpet0 at acpi0: counter not incrementing
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to
compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 1.0> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 1 int 10
iic0 at piixpm0
iic0: addr 0x18 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x1a 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x29 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x2b 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000
01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00: Virtio Network
Device
vio0 at virtio0: address 52:54:00:12:34:56
virtio0: apic 1 int 11
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00: Virtio Block Device
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
virtio1: apic 1 int 11
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown
nvram: invalid checksum
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (280aa5820cbd97c0.a) swap on sd0b dump on sd0b
clock: unknown CMOS layout
Automatic boot in progress: starting file system checks.
/dev/sd0a (280aa5820cbd97c0.a): file system is clean; not checking
/dev/sd0k (280aa5820cbd97c0.k): file system is clean; not checking
/dev/sd0d (280aa5820cbd97c0.d): file system is clean; not checking
/dev/sd0f (280aa5820cbd97c0.f): file system is clean; not checking
/dev/sd0g (280aa5820cbd97c0.g): file system is clean; not checking
/dev/sd0h (280aa5820cbd97c0.h): file system is clean; not checking
/dev/sd0j (280aa5820cbd97c0.j): file system is clean; not checking
/dev/sd0i (280aa5820cbd97c0.i): file system is clean; not checking
/dev/sd0e (280aa5820cbd97c0.e): file system is clean; not checking
setting tty flags
pf enabled
starting network
DHCPDISCOVER on vio0 - interval 3
DHCPOFFER from 10.0.2.2 (52:55:0a:00:02:02)
DHCPREQUEST on vio0 to 255.255.255.255
DHCPACK from 10.0.2.2 (52:55:0a:00:02:02)
bound to 10.0.2.15 -- renewal in 43200 seconds.
openssl: generating isakmpd/iked RSA key... done.
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA ED25519
starting early daemons: syslogd pflogd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
fw_update: No devices found which need firmware files to be downloaded.
starting local daemons: cron.
Tue Sep  9 08:27:45 PDT 2014
OpenBSD/amd64 (buildlet.my.domain) (tty00)
login: syncing disks... done

[bradfitz]

Comment 15:

Also: can we install a fixed (pre-generated, well-known) ssh server key, so it doesn't
need to be slowly created on first boot?
These VMs won't be internet-facing anyway, so I'm not concerned about its lack of
security by having a well-known private key.

[mdempsky]

Comment 16:

[re #13] We can omit the X sets, but unless it's really a space problem, I wouldn't
recommend it.  It's generally recommended to still install them even if you don't plan
on running X, and some packages will include X dependencies for fonts, rendering
libraries, etc.
[re #14] The reboot is intentional so that the "first time" bootup steps can be done at
image creation time rather than each VM bootup time.  After the second "syncing disks...
done", qemu *should* exit, and then you should just be waiting for the tar command on
the host platform to run.  (The script should probably add a message between qemu and
tar.)
If you're sure that qemu isn't exiting even a few minutes after "syncing disks... done",
that seems like a bug and I'll try to reproduce it.
[re #15] Yeah, that should be possible.  Just let me know how you want the key generated
or where it should come from, and I'll look into updating the script accordingly.

[bradfitz]

Comment 17:

I think I was impatient and it did finish. I haven't tried to boot it yet, though.
I'd rather remove the X sets. I can't imagine what would depend on them and I'd rather
start as minimal as possible and find any problems. If something accidentally depends on
something, that's a bug. All the Linux builders don't have X or fonts, for instance.

[mdempsky]

Comment 18:

Sure, your call. Removing the X sets is just a matter of answering "-x*" between "+*"
and "done" when the installer is asking about sets to install.

[mdempsky]

Comment 19:

To allocate 1G of disk space to swap and then the remaining space for a single root
partition, you can apply the changes below.  If you want a smaller swap partition, just
change "1G\n" to "256M\n" or something.
-send "auto\n"
+send "custom\n"
+
+expect "> "
+send "z\n"
+
+expect "> "
+send "a b\n"
+expect "offset: "
+send "\n"
+expect "size: "
+send "1G\n"
+expect "FS type: "
+send "swap\n"
+
+expect "> "
+send "a a\n"
+expect "offset: "
+send "\n"
+expect "size: "
+send "\n"
+expect "FS type: "
+send "4.2BSD\n"
+expect "mount point: "
+send "/\n"
+
+expect "> "
+send "w\n"
+expect "> "
+send "q\n"

[mdempsky]

Comment 20:

Attached "make.bash" script is an update of the one from #12 with the following changes:
  - includes a workaround for a recent change in GCE's DHCP server behavior.  (See https://code.google.com/p/google-compute-engine/issues/detail?id=77 for details.)
  - uses a custom single disk partition.
  - only installs the "base" and "comp"[iler] sets.
  - enables the SMP kernel (which is tested to at least also work on n1-standard-4)
  - tested that with "--metadata=buildmaster-binary-url:http://storage.googleapis.com/go-builder-data/buildlet.openbsd-amd64" it downloads and runs the buildlet.openbsd-amd64 script

Attachments:

1. make.bash (4678 bytes)

[bradfitz]

I now see:

+ cat /etc/resolv.conf
# Generated by vio0 dhclient
nameserver 10.0.2.3
lookup file bind
+ dig metadata.google.internal

; <<>> DiG 9.4.2-P2 <<>> metadata.google.internal
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 37752
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;metadata.google.internal.      IN      A

;; AUTHORITY SECTION:
.                       1408    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2014122901 1800 900 604800 86400

;; Query time: 33 msec
;; SERVER: 10.0.2.3#53(10.0.2.3)
;; WHEN: Mon Dec 29 13:47:52 2014
;; MSG SIZE  rcvd: 117

+ set -e
+ export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
+ curl -H Metadata-Flavor: Google http://metadata.google.internal/computeMetadata/v1/instance/attributes/buildmaster-binary-
url
/etc/rc.local[17]: curl: not found
+ curl -o /buildlet
/etc/rc.local[17]: curl: not found
+ echo giving up
giving up
+ sleep 10
+ halt -p
Dec 29 13:48:07 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
Dec 29 13:48:07 init: kernel security level changed from 1 to 0
Dec 29 13:48:09 init: single user shell terminated.
syncing disks... done
Zipping disk.raw... (this may take a while)
Done. GCE image is openbsd-amd64-gce.tar.gz.

Note the curl failures.

/cc @mdempsky

[bradfitz]

Running it again, I see the error earlier:

Making all device nodes...done.
quirks-2.46 signed on 2014-12-26T12:42:02Z
quirks-2.46: ok
Can't install libiconv-1.14p1 because of libraries
|library c.78.0 not found
| /usr/lib/libc.so.77.2 (system): bad major
Can't install libiconv-1.14p1 because of libraries
Can't install gettext-0.19.3: can't resolve libiconv-1.14p1
Can't install bash-4.3.30: can't resolve gettext-0.19.3,libiconv-1.14p1
Can't install libidn-1.29p0: can't resolve libiconv-1.14p1,gettext-0.19.3
Can't install curl-7.39.0: can't resolve gettext-0.19.3,libidn-1.29p0,libiconv-1.14p1
Extracted 20650 from 20998
Multiprocessor machine; using bsd.mp instead of bsd.

[bradfitz]

This is now booting on GCE and working.

[bradfitz]

Commit, since I messed up the issue syntax: golang/tools@b2523aa

[bradfitz]

This is now running and working at https://build.golang.org/ ... note the new openbsd-amd64-gce56 builder. I've updated the wiki DashboardBuilders page.

[bradfitz]

We still need 32-bit.

[mdempsky]

32-bit should be roughly just s/(amd64|x86_64)/i386/g, but maybe some of the installer questions differ slightly too.

One other comment: it's probably worth double-checking that non-Gophers can't SSH to the host (e.g., by using GCE firewall rules?), since the make.bash script includes username/password credentials for root and the gopher user.

[bradfitz]

@mdempsky, yes, these VMs are firewalled, including ssh.